CN110687999B - A method and device for semantic processing of electroencephalogram signals - Google Patents

Abstract

The invention discloses a semantic processing method and a corresponding device for electroencephalogram signals, and relates to a data mining technology for electroencephalogram signals. Current BCI studies on brain electrical signals rely on a few classical brain electrical paradigms that make use of only a few of the brain electrical signals, with little improvement to BCI methodologies, most of which do not introduce semantic concepts. A number of potential paradigms in the actual measurement have not been discovered yet and have not been summarized as valid semantics and new paradigms. The invention realizes a new semantic paradigm framework: the electroencephalogram data is taken as a research object, related grammar, semantic processing rules and algorithms are designed, grammar and semantic connotation are given to complex signals, and semantic interpretation is constructed for the deep neural network. The electroencephalogram signals are analyzed through the paradigm framework, researchers are helped to find out information blocks with specific semantics and semantic combinations among the information blocks, and efficient filters are automatically learned.

Description

A method and device for semantic processing of electroencephalogram signals

Technical Field

The present invention relates to a data mining technology for brain electrophysiological signals, and in particular to a processing method and device for mining and analyzing semantic information contained in brain electrophysiological signals (EEG/ECoG).

Background Art

The original novel of The Diving Bell and the Butterfly, which won the 65th Golden Globe Award for Best Foreign Language Film in 2008, was written by Jean-Dominique Bobby, the editor-in-chief of French ELLE magazine. He fell into a deep coma for more than 20 days after suffering an acute stroke. After waking up from the coma, he could not move any part of his body except his left eyelid, could not talk to people, had no expression, and no one could tell whether he was still conscious. But he did not give up. With the help of his speech therapist, Bobby learned to communicate using the alphabet and "wrote" the novel of the same name with his left eyelid. In Bobby's view, his body was like a heavy diving bell, but his heart was like a butterfly flying freely. Bobby's story is actually a typical case of "locked-in syndrome". In addition, according to statistics, there are 85.02 million disabled people in my country[1], many of whom are in a state of difficulty in taking care of themselves. This situation is widespread around the world and is by no means a rare occurrence. Stephen William Hawking, a famous theoretical physicist who passed away not long ago, and Christopher Reeve, who fell off a horse in 1995, both suffered from Lou Gehrig's disease. Christopher could not even communicate with others by blinking. All of this reminds doctors and researchers in related fields to try their best to help these severely disabled patients, find and promote better treatment or rehabilitation technologies, and provide help to patients whose disabilities are not as severe as theirs to improve their quality of life [2]. An important piece of background information is that Christopher Reeve lived in this state until his death in 2004. He was actually quite wealthy. He enjoyed the best medical insurance in the world, received research from the best scientific research and medical teams in the world, and a foundation that vigorously promoted research on this problem continued to invest a lot of money. Despite this, he still suffered unspeakable pain and was not cured until his death. The reason why it is so difficult to solve such problems is that from the perspective of rehabilitation, the regeneration of nerve cells has always been a world-class problem. It was not until the 2012 Nobel Prize in Physiology or Medicine was awarded to British scientist John Gurdon and Japanese medical professor Shinya Yamanaka, who made revolutionary contributions in the field of "somatic cell reprogramming technology", that some hope appeared. However, the implantation of iPS cells (Induced Pluripotent Stem Cells) into the human body may cause gene mutations and cancer, so there is still a long way to go; from the perspective of embodying ultimate care and improving the quality of life, the brain-computer interface technology used to help him also faced many challenges at the time, whether it was the reliability, stability, long-term nature of data acquisition, the location of the signal source, or the perfection, real-time nature, and accuracy of the analysis of complex EEG signals. There are many technical problems that need to be solved. In recent years, with the development of science and technology, humans have gradually acquired the ability to accept the above challenges in the field of brain-computer interfaces. Once brain-computer interface technology is maturely applied in the future, it will greatly benefit mankind. It will not only allow users to live with dignity in the later stages of their lives, but from a scientific point of view, it will also further deepen our understanding of the physiological functions of the brain and the principles of its interaction with the mind, and promote new scientific and technological advances.

Current research on BCI often relies on specific EEG patterns as the basis for recognition, and then uses the recognized specific patterns as input to trigger a special BCI system. For example, in the BCI system based on P300 character spelling input, an improved T9 method with word association function is used to reduce input time [15]; there are also schemes that use steady-state visual evoked potential (SSVEP, evoked potential) [88]. These methods only use a small part of the EEG signal, and are all translated through existing EEG paradigms. There is little improvement in BCI methodology. The human brain uses the sensory motor and language functions of the cerebral cortex to exchange information with the outside world [88], especially the language function with semantic information plays a key role. However, to date, there are still very few studies that consider directly analyzing signals containing semantic information. For example, in the exploration of applying semantics to BCI technology, a study conducted an experiment with two healthy subjects [93], using true and false statements as conditioned stimuli to examine distinguishable conditioned responses. After obtaining the conditioned responses, these conditioned responses correspond to true and false semantic information. This study used support vector machines (a radial basis function classifier) for offline classification. In the idea of distinguishing between the conditions "yes" and "no", the average classification accuracy of a single trial reached 50.5%, indicating that its classification accuracy is very low and the method still belongs to the category of traditional methods. Other studies have shown that the semantics of a single concept can be decoded from a single EEG data trial. The CSP method is used to extract the whole scalp synchronous activity component, and the general SVM machine learning algorithm is used, combined with the measurement results of spectral power to predict the semantic category of each trial. In some other studies [10], the semantically induced gamma EEG activity during sentence reading was studied. The changes in the EEG power spectrum caused by semantics were examined to examine the important influence of irregularities in sentences on sentence correctness, and the FFT method was used for processing. The survey shows that the current research on semantics in EEG is still in the exploratory stage, using traditional machine learning methods, which is completely different from the semantic paradigm model in this study.

Considering that many current models and methods for EEG signals in brain-computer interfaces (BCI) are mainly used to deal with several conventional target tasks, such as translating a known paradigm into a "yes or no" choice problem, most of these paradigms do not introduce the concept of semantics; a large number of potential paradigms have not been discovered in actual measurements, and have not yet been sorted out and summarized into effective semantics and new paradigms. In addition, most of the processing methods for brain-computer interfaces of movement-related potentials mainly rely on experience to find feature information, and rarely use tools and algorithms to automatically search for effective features. For a long time in the past, the reasons for this situation may be that the relevant data volume is not large enough, the data processing process is complicated, and the method is not efficient, which restricts the achievement of the above goals. In recent years, the development of deep learning and the advancement of hardware technology, combined with the semantic paradigm model proposed in this article, have the potential to apply these new models to signal processing, automatic feature search and data mining of brain-computer interfaces. This study has made attempts and explorations on the above goals, and used several data sets for testing, and output presentation results. The openness, extensibility and standardization of the semantic paradigm model provide an effective solution for building a reusable brain-computer interface marker feature database based on this semantic paradigm model in the future.

At present, the paradigm models in the field of BCI are limited, and the types of interface commands are few. Only a small number of studies have analyzed the types and parameters of actual movements/motor imagination, and there are no satisfactory results. In order to extract more information with clear semantics from EEG signals (including scalp EEG signals, cortical EEG signals, ECoG signals, magnetoencephalogram signals, MEG signals and other electrophysiological signals), the present invention connects three fields with a large span (fuzzy electrophysiological signals, deep learning compatible with multiple signals and clear semantic models) to achieve a new semantic paradigm framework: taking EEG signal data as the research object, designing relevant grammar, semantic processing rules and algorithms, giving complex signals grammar, syntax and semantic connotations, and constructing semantic interpretations for deep neural networks. Through this paradigm framework, EEG signals are analyzed to help researchers find information blocks with specific semantics and semantic combinations between these information blocks, and automatically learn efficient filters. If a newly collected, larger data set of EEG signals is used for training and testing, more meaningful information can be extracted from the EEG signals, and the self-feedback mechanism can be combined to adjust the model parameters and update the learning model, achieving high accuracy, large transmission flux, excellent fitting, and strong universality for EEG data sets of different sizes.

Summary of the invention

In order to solve the above problems and achieve the above objectives, the present invention provides a method and device for processing grammar, syntax and semantic paradigms of EEG signals, including:

Process the EEG signal data to obtain feature information;

The feature information is further divided into groups with different semantics, and each group is converted into a feature string;

Process the EEG signal information to obtain a syntax tree;

The generated syntax trees (the number of which can be 1 or greater) form a grammar G[s].

The electroencephalogram (EEG) signals include scalp EEG signals and cortical EEG signals. The method can be used for processing other electrophysiological signals.

Convert EEG signal information into corresponding decision trees through machine learning models;

Convert the decision tree into a syntax tree.

The processing of EEG signals includes decoding the data, that is, the raw EEG signal data collected is input, and the decoding module processes the pre-processed EEG data to obtain a discrete "label sequence" matrix, and each label has a corresponding semantic block, as shown in Formula 4.2. When processing data, the machine learning module will extract, reduce the dimension and filter the local features.

The specific implementation of the decoding process is divided into various machine learning modes such as CNN network mode, CTC+LSTM network mode, other RNN mode, wavelet transform mode, etc.; for the CNN network model, the matrix is equivalent to α2 in Figure 7, and the matrix elements are equivalent to the gray blocks in α2; thus, the label sequence, the feature data corresponding to the label sequence, and the relationship between the two are obtained. If it is based on the CTC+LSTM network model, the matrix corresponds to the label sequence matrix; the label sequence, the feature data corresponding to the label sequence, the relationship between the two, and the sequence length data are obtained. The sequence length is one-dimensional data, and its expression can be: [max_time_step,…,max_time_step], its length is batch_size, and the specific value is max_time_step. It can also be a feature matrix formed by the output formed by wavelet transform.

The source of each matrix element value in the mother matrix (Formula 4.2) can come directly from the feature information after feature extraction, dimensionality reduction and filtering of local features. In addition, the element information in the mother matrix can also come from the weight matrix in the machine learning network or deep learning neural network model.

Optionally, the EEG signal may be preprocessed before data decoding;

The machine learning models include convolutional neural network (CNN) models, long short-term memory network (LSTM) models, other RNN models, wavelet transform models, and other machine learning models;

The decoding of EEG signal data is to prepare for the subsequent semantic analysis and processing, the purpose is to process the data into an ordered form with semantic features and labels at different levels. At the same time, the corresponding decision tree and syntax tree are obtained. The semantic analysis and processing requires mapping the obtained labeled features and trees (decision trees and syntax trees).

Generate sentence patterns and phrases according to the syntax tree for high-level grammar, syntax and semantic analysis.

There is another way to generate grammar: write each characteristic string obtained into production rules in the form of grammar to form grammar G[s];

Generate feature syntax tree according to grammar;

A method for constructing a syntax tree, characterized in that the process of constructing the syntax tree is to form a feature alphabet according to the semantic relationship results of the current experimental analysis of the input information, and the relationship between the key feature information blocks after dimensionality reduction, and then form a semantic alphabet through the feature alphabet to form a feature grammar, form a feature generation formula set (a non-empty finite set composed of generation formulas or rules), and then form a feature syntax tree. Construct a sentence pattern according to the feature syntax tree so that the algorithm can be used in subsequent processing. With the newly formed grammar and feature syntax tree, the sentence pattern of the newly collected EEG signal is formed, its phrases, handles and grammar are determined, and the semantics therein is understood.

Not only can the grammar tree be obtained through the machine learning model, but the grammar tree can also be constructed and modified through experiments. The process is: the input information is analyzed according to the semantic relationship results of the current experiment, and the relationship between the key feature information blocks after dimensionality reduction is formed into the production of grammar, forming a feature signal grammar, and then forming a feature grammar tree. Construct a sentence pattern according to the feature grammar tree so that the algorithm can be used in subsequent processing. With the newly formed grammar and feature grammar tree, the sentence pattern of the newly collected EEG signal is formed, its phrases, handles and grammar are determined, and the semantics therein is understood.

The syntax trees obtained by the above two methods can generate sentence patterns and phrases for high-level grammar, syntax and semantic analysis.

The two different paths for forming grammar and syntax tree described above, and the characteristic signal grammar formed by them, all rely on the following 12 concept definitions and rule definitions:

Definition 1: The feature is the feature matrix alphabet, which is defined as: . Feature matrix alphabet: All input information is processed by function f into a feature matrix _M1 . Each element of _M1 is also a matrix _M2 . The elements of _M2 are represented by numerical values in binary or hexadecimal form. All elements of _M1 belong to the same feature matrix alphabet, which is composed of non-repeating elements and is represented by A _M.

Definition 2. Semantic alphabet: All elements of the input information _M2 can be semantically mapped into specific symbols by the function _fM->a . These symbols can be elements of the character set recognized by the computer system. These symbols, operators and separators constitute a finite non-empty set A, in which each element is a character that can be recognized (such as a computer system) or multiple characters. Therefore, within the scope of this article, it can also be represented by V. That is to say, the semantic alphabet and the vocabulary are regarded as the same, and are referred to as the alphabet for short.

Definition 3. Characteristic symbol string: Any finite sequence of specific symbols obtained by Definition 1 is called a symbol string ∑, also known as a dictionary.

Definition 4. Characteristic symbol matrix: A two-dimensional symbol matrix composed of multiple symbol strings arranged in rows. Similarly, it can be expanded into a three-dimensional symbol matrix.

Definition 5. Prefix of a characteristic symbol string: Assume that p is a characteristic symbol string. The remaining part after deleting n (n is a natural number less than the length of p) symbols from the end of p is the prefix of the characteristic symbol string p.

Definition 6. Characteristic symbol string suffix: Assume p is a characteristic symbol string, and the remaining part after deleting n (n is a natural number less than the length of p) symbols from the beginning of p is the suffix of the characteristic symbol string p.

Definition 7. Substring of a characteristic symbol string: refers to the part of the characteristic symbol string that remains after deleting a characteristic symbol string prefix and a characteristic symbol string suffix from the characteristic symbol string p.

Definition 8. Connection of feature symbol strings: Assuming that p and q are two feature symbol strings respectively, then the combined pq is the feature symbol string q connected to the feature symbol string p.

Definition 9. Definition of feature grammar G: G = (V _n , V _t , P, S)

V _n (non-terminal symbol set) is a non-empty finite symbol set consisting of non-terminal symbols (such as uppercase letters or <Chinese characters>, or other specific symbols). The non-terminal symbols refer to symbols that include a finite number of mathematical operation rules that need to be defined, which is the grammatical category at the semantic level.

V _t (terminal symbol set) is a non-empty finite symbol set consisting of terminal symbols (such as lowercase letters, numbers, punctuation marks, or other specific symbols). The terminal symbols are basic symbols that do not need to be further defined in the grammar and have clear operator meanings at the mathematical or semantic level. V = V _t ∪ V _n = A, where V is the alphabet A or vocabulary of the feature grammar G.

Definition 10. P (feature generation set) is a non-empty finite set of generation rules or rules.

The form of the feature generator is: or

The set to which the elements on the left of the feature generator belong is expressed as α∈V+, so α cannot be empty. The set to which the elements on the right of the generator belong is expressed as β∈V ^* . or is consistent with the connotation of, which means "composed of..." or "defined as".

S is the starting symbol of the feature grammar, S∈V _n , and the necessary condition is that S must appear at least once on the left side of a production.

Definition 11. Feature grammar parse tree: Due to the mapping logic of Definition 1 and Definition 2 above, the leaf nodes of the grammar parse tree can be either elements of A or elements of _AM . Suppose there is a feature grammar G = ( _Vn , _Vt , P, S). For any sentence type of the feature grammar G, a corresponding feature grammar tree can be found, which supports the following four premises:

Each node in the syntax tree has an identifier, which is a characteristic symbol in the set V.

The root node of the syntax tree is the start symbol S of the feature grammar.

If a node on the syntax tree has at least one direct successor node, then the node must be a non-terminal feature symbol. If a node N on the syntax tree has several direct successor nodes, they are sorted from left to right/right to left.

Definition 12. If the regular expressions in the feature grammar have the following expressions:

Let Cn be a non-terminal feature symbol, and its set is U∈V _n , u∈V ^* , that is, the left side of the rule in the feature grammar must be a Cn symbol, and the right side u of the rule expression is a feature symbol sequence expression on V.

Compared with the prior art, the technical solution of the present application includes: after converting EEG information into information with clear semantics, clear semantics are added to the grammar, and the various semantic feature blocks obtained are combined into a semantic paradigm.

The present invention implements a new semantic paradigm framework: taking EEG data as the research object, relevant grammar, semantic processing rules and algorithms are designed, complex signals are given grammar, syntax and semantic connotations, and semantic interpretation is constructed for deep neural networks. Through the paradigm framework, EEG signals are analyzed to help researchers find information blocks with specific semantics and semantic combinations between these information blocks, and automatically learn efficient filters. If more and larger newly collected brain-computer interface data sets are used for training and testing, more information with clear meaning can be extracted from EEG signals, and the self-feedback mechanism can be combined to adjust the model parameters and update the learning model, so as to achieve high accuracy, large transmission flux, good fit and strong universality for EEG data sets of different sizes.

The electroencephalogram signal includes scalp electroencephalogram signal and cerebral cortex electroencephalogram signal. The method of the present invention can be used for other electrophysiological signal processing.

Accordingly, the present invention provides a device for processing electroencephalogram signals, cerebral cortex electrical signals and other electrophysiological signals, characterized in that the device comprises (as shown in FIG. 17 ):

Signal acquisition unit;

A feature signal processing unit that processes the EEG signal data to obtain feature information;

The feature information is divided into groups with different semantics, and each group is converted into a feature string forming unit of the feature string;

forming alphabet units of the alphabet from the characteristic string;

Processing the EEG signal information to obtain a syntax tree unit for forming a syntax tree;

The generated syntax trees (the number of which can be 1 or greater) form grammar units of the grammar G[s];

As shown in Figure 3, the part enclosed by the dotted box is the decoding process, and the part enclosed by the solid box is the grammar, syntax and semantic paradigm analysis process; after the decoding process, the original data is simplified, the data dimension is reduced, the parameters are reduced, and the feature information with key information highlighted is obtained; the decoding process in the figure includes various special pre-processing of the EEG signal listed by the method of the present invention before inputting it into the machine learning model.

Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

Figure 1 shows the existing traditional processing flow

FIG. 2 is a processing flow of the present invention including semantic analysis

Figure 3 shows the overall workflow

Figure 4 is a three-dimensional visualization of the index of category 1 after dimensionality reduction

Figure 5 is the feature semantic syntax tree of the sentence pattern ddefdd

Figure 6 is a phrase for finding a sentence pattern through a syntax tree

Figure 7 shows the convolutional layer filtering

Figure 8 shows the potential sequence forming the matrix alphabet and labeling

Figure 9 shows the prefix search decoding on the alphabet

Figure 10 Decision tree formation

Figure 11 shows the weights to be updated in the LSTM Block

Figure 12 shows the semantic analysis and interpretation process

Figure 13 is the feature information matrix

Figure 14 is the feature matrix

Figure 15 is a feature matrix with the terminal symbol r marked

Figure 16 shows how to obtain the original features from the visualization graph.

FIG. 17 is a diagram showing the structure of the device of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the present invention more clear, the embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

The current BCI model processing only solves the yes or no problem of several simple target tasks. These tasks correspond to some clear paradigms, and the accuracy can only be indicated by probability. There is no semantic conceptual model. A large number of potential paradigms in actual tests cannot summarize effective semantics and new paradigms. In addition, the search for feature information models is basically based on existing experience, and it is difficult to automate the search with the help of tools and algorithms. One of the reasons for this problem is the complexity of data processing, which leads to low efficiency and restricts the achievement of the above task goals. At the same time, the development of deep learning and hardware progress have been rapid in recent years, but these technologies have not yet had clear and interpretable models for use in the BCI field. This study considers using a clear grammar model to connect the latest research results in the two fields to improve the BCI data model and semantic model, improve the efficiency of data utilization, and provide ideas and tools for creating new models and paradigms.

Although deep neural networks have achieved good results in many visual tasks at present, they have not been widely used in data mining of EEG signals based on brain-computer interfaces, and there are still some problems to be solved. The goal of the present invention is to apply several methods such as improved methods based on traditional machine learning, deep learning methods and semantic paradigm models to data mining of EEG signals to meet the above challenges and problems. At present, the experimental paradigm models in the field of BCI are not only limited, but also have few types of interface commands, which makes BCI technology always rely on typical paradigms for translation. How to directly understand human thinking and construct simulation models has always been a difficult problem. Due to the difficulty of interpreting the original meaning and the low level of analysis methods, this problem has not made any progress. In order to explore the solution to this problem, the researchers connected three fields with a large span (fuzzy electrophysiological signals, deep learning compatible with various heterogeneous signals, and clear semantic models), applied the latest deep learning technology to the field of BCI data mining, decoded EEG signals, processed EEG data into discrete label sequences that can address corresponding semantic blocks, and converted deep neural networks into corresponding decision trees and syntax trees through semantic analysis of deep neural networks. As a result, we realized a new semantic paradigm framework with EEG signal data as the research object, and made preliminary but very meaningful explorations for the semantic analysis of EEG information: relevant grammar, semantic processing rules and algorithms were designed, complex signals were given grammar, syntax and semantic connotations, and semantic interpretations were constructed for deep neural networks. The research on EEG information was elevated from a simple command paradigm to the height of semantics and syntax, and the efficiency of complex pattern research on EEG signals was improved. Through this paradigm framework, semantic paradigm analysis of information in EEG signals can be performed to help researchers find information blocks with specific semantics contained therein and the semantic combinations between these information blocks, and automatically learn efficient filters to achieve high accuracy, large transmission flux and strong universality. This framework can be used to reduce the dimension and obtain key feature information, efficiently summarize new paradigm features, and optimize the learning model. At the same time, a series of visualization functions can be used to observe the training status of the data, and the corresponding original features can be obtained through the results of data dimensionality reduction, thereby improving the efficiency of analyzing EEG signal features.

As shown in Figures 1 and 2, in addition to the traditional processing flow, a semantic processing module can be added. The semantic analysis results can be fed back to the semantic module and the machine learning module to help improve the semantic model and the internal model. The improvement of the semantic model can add a mapping relationship between feature information and semantics, and the improvement of the internal model can highlight key features and discover new features and paradigms.

After obtaining the new semantic model (including feature position information), the new data can be used directly: the feature position of the new data is no longer repeatedly learned, but a new spatiotemporal filter model is discovered, or the potential logical association between different features is discovered. The existing training results are used to predict the label type of the new data. If the calculation finds that the accuracy probability reaches the recognition standard (the standard can be defined by oneself), there is no need to modify the model or add a new semantic derivation formula; if the accuracy is not enough, the new data is used as training data, and the model can be retrained in the future. This algorithm will correct the overfitting and underfitting problems. More importantly, it makes the results obtained from each calculation become results with specific semantics and recorded. Researchers can use this record to deduce and formulate the next filter design plan. In addition, the semantic relationship tree formed can easily use dependency feature processing for better preprocessing. This solution makes the past calculation results have specific meanings. It will not re-train a lot with the increase of the amount of new data, and the learning results will be difficult to converge. On the contrary, it will greatly speed up the research progress, so that the analysis of specific problems can converge in a faster way, and the amount of calculation will gradually decrease. The later the hardware requirements will be reduced, and the speed requirements for online analysis will be guaranteed.

Although neural networks have achieved good results in many visual tasks, the interpretability of the model itself still faces many challenges. The interpretability of a model determines whether it can be regarded as a black box or a white box. Obviously, the more obvious the tendency to be a white box is, the clearer the occasions where the model can be applied and the higher the degree of optimization. However, in reality, many developers dare not use it boldly because the interpretability of the model is still unclear, let alone how to optimize and expand the model. Here, we abstract the processing of EEG signals by neural networks into the formation process of human language semantics, and the formal method of mathematical modeling of human language grammar was created by Chomsky. We will propose a specific formal description that can adapt to EEG signals and neural network processing and conform to grammar rules. This method is suitable for processing scalp EEG signals, cortical EEG signals, and magnetoencephalogram signals (which can be extended to various other electrophysiological signals). It can interpret more semantic information from these signals and form grammars that can be used for higher-level applications. Moreover, the method of the present invention can be used for other electrophysiological signal processing.

Judging from the historical process of BCI research, EEG signals are submerged in various noises, and human research on the electrophysiological mechanisms of the brain is slow, so research on BCI is still in its early stages. New tools and ideas are needed to help improve research efficiency and discover new semantic models or paradigms.

It has been verified that the data obtained by the same neural network model after dimensionality reduction of the same data set is stable and unchanged, which allows us to calculate the corresponding grammatical and semantic valid sets based on the existing data set; using this existing valid set as a template, we can carry out targeted dimensionality reduction and denoising on the new test data set to achieve the purpose of substantial dimensionality reduction without affecting the judgment results. Taking into account the signal fluctuations of the subjects between different test batches, the newly introduced signals are not fully matched, but a certain proportion of errors are allowed. This operation logic is the same as the calculation of the full data set. Since the data dimension has been greatly reduced, the matching operation speed at this time will be very fast.

The machine learning models include various machine learning models such as convolutional neural network (CNN) models, long short-term memory network (LSTM) models, other RNN models, wavelet transform models, etc.

The overall workflow is shown in Figure 3. The left side of the figure is a flowchart, and the right side is the data representation, internal form and data structure corresponding to the current left step. The right part of Figure 3 is enlarged separately to become Figure 8.

The part enclosed by the dotted line in the figure is the decoding process, and the part enclosed by the solid line is the grammar, syntax and semantic paradigm processing. The decoding process is to prepare for the subsequent semantic analysis process, and the purpose is to process the data into an ordered form with semantic features and labels according to different levels. The semantic analysis process is to construct a decision tree and a syntax tree with the obtained labeled features, and map the two. The key modules in the figure and the concepts used are explained in detail below.

The present invention involves the following concepts and rule definitions.

Since EEG signals themselves cannot have clear values like computer signals, it is impossible to express the semantics of EEG signals with certain randomness and probability in the current computer system. Therefore, before we can obtain the semantic features of EEG information through some conversion method, we must have a corresponding model to decode EEG signals and convert this vague information into clear information. Therefore, we must define a unique concept system for grammatical semantics and semantic paradigm models in order to distinguish them from the functions of clear computer systems. Therefore, there are the following series of definitions.

Definition 1. Feature matrix alphabet: All input information is processed by function f into a feature matrix _M1 . Each element of _M1 is also a matrix _M2 . The elements of _M2 are represented by binary or hexadecimal numbers. All elements of _M1 belong to the same feature matrix alphabet, which consists of non-repeating elements and is represented by A _M.

Definition 2. Semantic alphabet: All elements of the input information _M2 can be semantically mapped into specific symbols by the function _fM->a . These symbols can be elements of the character set recognized by the computer system. These symbols, operators and separators constitute a finite non-empty set A, in which each element is a character that can be recognized (such as a computer system) or a string of multiple characters. Therefore, within the scope of this article, it can also be represented by V. That is to say, the semantic alphabet and the semantic vocabulary are regarded as the same, and are referred to as the alphabet for short.

The elements of the alphabet are represented by characters or strings, all elements belong to the same alphabet, and the alphabet is composed of non-repeating elements. The elements of the alphabet are defined by a character set, and the character set can be an ASCII character set, a double-byte character set DBCS, or a Unicode character set.

For example, the members of V are 0xA, 0xB, 0x1A2B, 0x1A2B3C, etc. These binary strings are binary expressions of effective features after dimensionality reduction by the neural network.

The operators have specific mathematical definitions, contain symbols with a finite number of mathematical operation rules that must be defined, and are grammatical categories at the semantic level.

The separator is used to separate different semantic units, and can be represented by a space or other unambiguous definitions. For example, in an EEG signal, the signal other than the EEG characteristic signal with clear semantics that is currently being focused on can be translated as a separator.

Definition 3. Characteristic symbol string: Any finite sequence of specific symbols obtained by Definition 1 is called a symbol string ∑, also known as a dictionary.

For example, assuming V = {1, a, b}, then 1, a, b, 11, 1a, 1b, a1, ab, aa, b1, ba, 111, aaa, bbb are all symbol strings on V. The length of the symbol string 1ab is 3, denoted by |1ab| = 3.

Definition 4. Characteristic symbol matrix: A two-dimensional symbol matrix composed of multiple symbol strings arranged in rows. Similarly, it can be expanded into a three-dimensional symbol matrix.

Taking the information input into the neural network as an example, suppose that the information of a certain channel is processed by the CNN/LSTM network to obtain a binary string containing key feature information. If the string is converted into hexadecimal expression, a hexadecimal string can be obtained. The string can be composed of several elements in the alphabet, and the hexadecimal string with key feature information is the symbol string of Definition 3 or the symbol matrix of Definition 4. Taking the binary classification problem of left and right hand movements as an example, after the neural network processing, the relevant visualization results of the index of category 1 after dimensionality reduction are shown in Figure 4:

The original signals corresponding to the above indexes constitute a symbol string of this category. Assuming it is 0x1a2b34567890, it can be expressed as: L->1a2b34567890; we can also know the original signals nearby corresponding to a certain index information, and correspond the symbol string composed of these original signals to the index: 22->0d265b8792c; the advantage of this is that the local features of a specific signal can be defined. Taking the P300 signal as an example, a P300 signal can be composed of several indexes, such as 52, 57, and 7 (assuming that the three adjacent indexes in sub-graph b of Figure 4 constitute the typical feature values of the P300 signal after dimensionality reduction). After being processed by the neural network model, the EEG information of the P300 signals of different subjects can be converted into a variety of possible character strings. These character strings have certain similarities, but they can all express the key features of P300. For easier understanding, L can be regarded as a sentence, 22 as a word, and the characters on the right side of -> are all elements in the alphabet. When expressing grammatical semantics, the symbol matrix can be converted into a symbol string for unified processing, while it is still stored and calculated as a tensor when performing other processing.

From the above examples, we can see that in order to facilitate operations on strings and their subsets, we also need to define some new concepts and rules for operations between these concepts.

Definition 5. Prefix of a characteristic symbol string: Assume that p is a characteristic symbol string. The remaining part after deleting n (n is a natural number less than the length of p) symbols from the end of p is the prefix of the characteristic symbol string p.

For example, 1, 1a, 1a2b34567890 are all prefixes of 1a2b34567890. In the semantics of this article, the characteristic symbol string can be referred to as the symbol string.

Definition 6. Characteristic symbol string suffix: Assume that p is a characteristic symbol string. The remaining part after deleting n (n is a natural number less than the length of p) symbols from the beginning of p is the suffix of the characteristic symbol string p.

Definition 7. Substring of a characteristic symbol string: refers to the part of the characteristic symbol string that remains after deleting a characteristic symbol string prefix and a characteristic symbol string suffix from the characteristic symbol string p.

Definition 8. Connection of feature symbol strings: Assuming that p and q are two feature symbol strings respectively, then the combined pq is the feature symbol string q connected to the feature symbol string p.

Definition 9. Definition of feature grammar G: G = (V _n , V _t , P, S)

V _n (non-terminal symbol set) is a non-empty finite symbol set consisting of non-terminal symbols (such as uppercase letters or <Chinese characters>, or other specific symbols). The non-terminal symbols refer to symbols that include a finite number of mathematical operation rules that need to be defined, which is the grammatical category at the semantic level.

V _t (terminal symbol set) is a non-empty finite symbol set consisting of terminal symbols (such as lowercase letters, numbers, punctuation marks, or other specific symbols). The terminal symbols are basic symbols that do not need to be further defined in the grammar and have clear operator meanings at the mathematical or semantic level. V = V _t ∪ V _n = A, where V is the alphabet A or vocabulary of the feature grammar G.

Definition 10. P (feature generation set) is a non-empty finite set of generation rules or rules.

The form of the feature generator is: or α→β

The set to which the elements on the left of the feature generator belong is expressed as α∈V+, so α cannot be empty. The set to which the elements on the right of the generator belong is expressed as β∈V ^* . or is consistent with the connotation of, which means "composed of..." or "defined as".

S is the starting symbol of the feature grammar, S∈V _n , and the necessary condition is that S must appear at least once on the left side of a production.

For example, after semantic analysis of EEG signals, a special semantic operator can be defined as the terminal symbol according to the relationship between semantic blocks.

Definition 11. Feature grammar parse tree: Due to the mapping logic of Definition 1 and Definition 2 above, the leaf nodes of the grammar parse tree can be either elements of A or elements of _AM . Suppose there is a feature grammar G = ( _Vn , _Vt , P, S). For any sentence type of the feature grammar G, a corresponding feature grammar tree can be found, which supports the following four premises:

Each node in the feature syntax tree has an identifier, which is a feature symbol in the set V.

The root node of the feature syntax tree is the start symbol S of the feature grammar.

If a node in the syntax tree has at least one direct successor node, then the node must be a non-terminal feature symbol.

If a node N on the feature syntax tree has several direct successor nodes, they are sorted from left to right/right to left.

For example, based on the results of EEG signal analysis, the key binary features are mapped to custom symbols to generate the following feature grammar:

G[S]:

S→dDS

D→SD

S→d

D→efd

According to the existing feature syntax tree, there are two derivation methods: top-down and bottom-up; you can choose any one of these methods to perform derivation. For example, when the feature syntax tree is derivation is established from top to bottom, the derivation constructed is the rightmost feature derivation, as shown in Figure 5, the feature sentence pattern is derived using the production rules of the grammar G[S].

The method to find a phrase of a characteristic sentence pattern is to use a syntax tree, starting from the start symbol of the grammar, and construct a characteristic syntax tree corresponding to the sentence pattern through production rules. As shown in Figure 5, d, efd, defd, and ddefdd are all phrases of the sentence pattern ddefdd. As long as these symbols are given specific mathematical meanings, specific semantics can be formed between binary features, such as Figure 6. Referring to Figure 6, it can be deduced that the characteristic sentence pattern The phrase is: (A×S), A×S, k. The phrases and non-leaf nodes in Figures 5 and 6 can be used as key feature information after a large-scale dimensionality reduction, and they all correspond to the probability of a certain event or situation in the EEG event. Therefore, these expressions can find corresponding concepts and forms in the decision tree theory, thus associating the seemingly irregular EEG signals with grammar semantics and decision trees.

Summarize:

Derivation is like a convolution operation, each subtree is equivalent to each convolution function, and the handle is equivalent to an ordered subgraph. Due to the grammar derivation tree, it is very easy to require a phrase. This is like finding the phrase P300 based on a known image. Each "→" expression in the grammar is equivalent to multiple possible waveforms, waveforms of different subjects. What we need to do is to find various possibilities, which are equivalent to various results of neural network training.

Definition 12. If the regular expressions in the feature grammar have the following expressions:

Let Cn be a non-terminal feature symbol, and its set is U∈V _n , u∈V ^* , that is, the left side of the rule in the feature grammar must be a Cn symbol, and the right side u of the rule expression is a feature symbol sequence expression on V.

The grammar can describe the variable names that appear in the expression and the possible positions. It corresponds to the memory output of LSTM.

Semantic interpretation of LSTM

Different from the tree-shaped LSTM, this study uses the traditional LSTM model as the research object. The tree-shaped LSTM has its own decision tree features, and its semantic model is essentially the same as the traditional model. When processing serialized information, the LSTM model can not only predict the information of future time steps, but also classify and identify a complete time series. These two methods have different granularity of input information recognition and different output forms, but the internal logical process is the same.

Let x _i ∈ ^RH×W×D represent the matrix data of the input LSTM network, H represents the number of rows, W represents the number of columns, and D represents the depth of the matrix. Corresponding to the EEG dataset, H is the data length of each channel collected, W is the number of channels for collecting data, and D is the number of experiments. ^{x (d)} ∈ ^RH×W represents the feature data of the dth input cell to the LSTM. When the new input tensor appears at H×W different candidate positions on x ^(d) , we design H×W positive templates.

Τ ⁺ = {T _1,1 ,T _1,2 ......, _TH,W } represents the ideal excitation feature for positive sampling; similarly, the negative template Τ ^- = {T _1,1 ,T _1,2 ......,TH _,W } is also used to describe the features of negative sampling. The evaluation function of the neural network is expressed as

f(x) _c = p(y = c|x)(4.6)

Where c represents the label category. In order to make f(x) _c as large as possible, its inverse should be minimized and avoid being close to 0 in the case of multi-classification. This is the loss function

loss(f(x),y)＝-Σ _c 1 _(y＝c) logf(x) _c ＝-logf(x) _y ＝-lnf(x) _y (4.7)

This expression essentially represents negative mutual information, so it can be expressed as:

Where X represents all input tensors in the training set, the prior probability is a constant 1, and p(x ^(d) |T) is the posterior probability. This loss value ensures that the feature data of the dth input to the cell represents a certain input tensor and is processed in the LSTM way without gradient diffusion. Because the final output of our LSTM model is the probability of the result belonging to a certain category (assuming a total of k categories) calculated by the softmax equation,

At this point, the prediction of the LSTM model is as described, and the decision mode of the fully connected layer as the i-th input can be roughly described as follows:

in Indicates that each gate processing in the model is mapped to convolution processing, where It can be calculated by gradient back propagation. The fully connected layer contained in the LSTM model is essentially the same as the fully connected layer in CNN, so a decision tree can also be constructed to extract the decision mode encoded in the last fully connected layer and construct the following decision tree, from the top node to the terminal node: Each node c of the decision tree represents a common decision mode, and adjacent nodes are merged to extend sub-nodes. The values of these nodes come from the weight values. The matrix in Figure 10 is the weight value matrix of the LSTM network.

In Figure 10, the source of the node value is the weight matrix of the neural network (such as the two matrices in the lower right corner of the figure). We construct an initial tree T1, in which the top node takes all the positive samples g of the gradient as the terminal node, thus forming the leftmost tree T1 in Figure 10. Then, every two nodes such as c and c' are merged to obtain a new node p, and c and c' become the child nodes of p. Repeat the above operation to merge, and finally modify the initial tree to the final decision tree, as shown in Figure 10.

However, unlike CNN, the w of LSTM involves multiple places (marked with w in the figure), as shown in Figure 11.

Next, as long as w is mapped to the phrases in the grammar, the conversion from decision tree to grammar tree can be achieved. Since the deep learning model can produce different training results by adjusting parameters, different grammar trees can be generated. Through the generated multiple grammar trees, the grammar G[s] for a certain EEG data can be gradually improved. At this point, the semantic analysis and interpretation of the sampled EEG signal is completed (the outline process is shown in Figure 12).

Specific embodiment 1

Decoding EEG data

The EEG dataset used is the dataset of the International BCI Competition.

The overall flow chart 3 shows that the first step is to perform noise reduction processing, input the original EEG information matrix collected by the electrodes, and after preprocessing the EEG data of indefinite length to generate output labels, the decoding module needs to process the preprocessed EEG data to obtain a discrete "label sequence" matrix, where each label has a corresponding semantic block, as shown in Formula 4.2. Therefore, the alignment relationship between the input feature data and the output label is uncertain, allowing the model to predict labels at any point in the input sequence. When processing data, the machine learning module will extract, reduce dimensionality, and filter local features.

Each matrix element of the mother matrix in Figure 8 is the result of feature extraction, dimensionality reduction and filtering of local features (and the elements in each matrix element are the elements of _M2 defined in Definition 1. Feature matrix alphabet in Section 4.2.2. After these elements are processed by the _fM->a function described in Definition 2, the elements in the semantic alphabet A described in Definition 2 are generated. These elements in A are the "label sequence" described in the first sentence of the first paragraph of this section. The reason why it is called a sequence is that these elements are composed of characters or strings. These meaningful characters or strings are semantic units that can support grammar, syntax and semantic paradigm processing). It can be a variety of different methods, such as wavelet, average, maximum, power spectrum, filter. For example, for the CNN network model, the matrix is equivalent to α2 in Figure 7, and the matrix elements are equivalent to the gray blocks in α2; the initial partition size and starting position of these local features can be either initial values without special meaning or customized according to the known BCI paradigm. If the initial domain is random, it can be updated based on the final semantic analysis results and evaluation results. At this point, for the decoding model based on the CNN network, the label sequence, the feature data corresponding to the label sequence, and the relationship between the two are obtained.

If it is based on the LSTM network model, the matrix corresponds to the label sequence matrix (if the wavelet transform method is used to generate the input data for the CNN network, the matrix elements correspond to the coefficients after wavelet decomposition). The label sequence is a sparse matrix. Assuming that the batch_size we generate is 32 samples, each sample is a digital string of length 1 to 30, then a (32, 30) matrix is generated. The matrix has non-zero elements with numbers and zero elements without numbers. Moreover, the label is of indefinite length, and the non-zero elements are randomly distributed. The label stores the position information while storing the digital string. So far, for the decoding model based on the LSTM network, the label sequence, the feature data corresponding to the label sequence, the relationship between the two, and the sequence length data are obtained. The sequence length is one-dimensional data, which can be expressed as: [max_time_step,…,max_time_step], its length is batch_size, and the specific value is max_time_step.

In addition, the source of each matrix element value in the above mother matrix can come directly from the feature information after feature extraction, dimensionality reduction and filtering of local features, and the element information in the mother matrix can also come from the weight matrix. In comparison, the latter has a greater impact on the output than the original feature. For the image set under the same node in the decision tree, the weight value has a universal property, which explains a group of images rather than a single image, so it is more advantageous to use the weight value to form the value in the feature mother matrix; in addition, the information from the original feature is related to a specific image and cannot be used as a universal attribute for a group of similar images.

Figure 8 shows the process of forming a matrix alphabet and labeling the EEG potential sequence. In the figure, a represents the original EEG signal waveform, b represents the input sample matrix mapped into a picture (part), c represents the mother matrix of formula (4.2), which can be directly derived from the feature information after dimensionality reduction or from the weight matrix, and d represents the feature matrix described in Section 4.2.5. Sub-figure c shows that each matrix element consists of 4 members, involving two channels c1 and c2. The number of channels involved can be designed as needed. For example, it can be composed of only a partial sequence of one channel, or it can be composed of partial sequences of two or even more channels.

FIG8 shows that discrete labels are stored in a matrix, and the number of rows of the matrix is related to the number of data streams. The elements of the matrix are represented by characters or strings, and all elements belong to the same alphabet, which is composed of non-repeating elements. The elements of the alphabet are defined by a character set, which can be an ASCII character set, a double-byte character set DBCS, or a Unicode character set.

In order to evaluate the probability that the current input EEG signal sequence belongs to the corresponding label in A, it is necessary to add a corresponding processing unit to the softmax output layer of the machine learning model, and this unit must be able to estimate the probability of the occurrence of unknown meanings, noise, and separation area labels. All this information gives the joint conditional probability of all labels at all time steps. Then the conditional probability of any label sequence can be found by summing the corresponding joint probabilities.

For a certain EEG signal input sequence, after processing, the output unit n at time t represents the probability of an observation at time t belonging to label k in A after being processed by the LSTM model. Given an input sequence x of length L and a training set Tr, the following probability evaluation is _distributed over π, ^π∈AL , where ^AL is the set of sequences of length L over the alphabet A, A＝ _Vn∪Vt

π is a random variable whose value can take any sign in A. Since there is a lot of background noise in EEG signals, as well as signals that are strongly related to personal characteristics such as the somatosensory cortex and psychological activities, one of the characteristics of these signals is the presence of small-scale repetitions. Therefore, there are multiple mappings from feature matrices to labels on A, T: _AM →A. Since these mapping paths are mutually exclusive, the conditional probability of a label can be calculated by adding the probabilities of all paths on T that are mapped to it.

So far, we have defined the conditional probability of the possible label sequence of the input EEG data to A. The subsequent processing is consistent with the existing RNN/LSTM algorithm processing.

For the labeled label (label is a label sequence) in A (or vocabulary V), assume that the forward variable δ _t (s) is the sum of the probabilities of all path prefixes of length t, which are mapped from T to s/2 length prefixes; the reverse variable θ _t (s) is the sum of the probabilities of all path suffixes starting from t, which are mapped from T to s/2 length suffixes of label label. Therefore, a series of derivations are performed on the labeled EEG information based on a similar derivation process, and the final result is:

Among them, v is the label variable, x is the time series of the input EEG signal of a given length, refers to the position of label k in v', while v' refers to the value of v for given conditions s and t. The product of the forward and backward variables is the total probability of all paths passing through v _'s .

Therefore, for the outputs a ₁ , a ₂ , … _ak … of the LSTM network, we can correspondingly obtain p(C ₁ |x), p(C ₂ |x), p(C _k |x), that is, the probability that the output category is C ₁ , C ₂ …C _k given the input.

After the network is trained, it will label an unknown input sequence x by selecting the most likely label l. Searching for this label is the decoding task. The decoding algorithm can use the aforementioned prefix information to decode, relying on the modified forward variables to efficiently calculate the probability of continuous expansion of the prefix label. The search decoding is a best-first search, forming a label tree as shown in Figure 9, where the child nodes of a given prefix (label) node share the prefix. In Figure b, the search ends when the label pr is more likely than any prefix (the probability of the end symbol e is the largest).

The search tree and label tree are also decision trees, and a syntax tree can be formed by defining a grammar G. For the same batch of data, a random forest can be formed by further adopting the bagging strategy and the random strategy.

At this point, the decoding process is completed, and the association between the original input EEG data and various semantic logical entities ( _AM |A|semantic flow label|decision tree|grammar syntax tree) is completed, realizing the semantic paradigm interpretation of EEG data using various machine learning algorithms, especially neural network algorithms.

Specific embodiment 2

Semantic Paradigm Process Example

Given the time series information of the EEG signals of a subject's left and right hand movements, according to the grammar G constructed above, can s deduce the results from G?

Algorithm idea: The time series information of the EEG signal is processed by the neural network to form a reduced-dimensional matrix, and the matrix information is decomposed into two parts, one part corresponds to the identity and action information, and the other part corresponds to the left hand (or right hand); thus, a sentence s to be analyzed is formed, and starting from the starting symbol of G, a sentence t is randomly derived, and t and s are compared.

This is the sentence derived from the start symbol, parsed from top to bottom, corresponding to the top-down construction order of the parse tree.

Let S represent a sentence, → represent deduction, and the grammar G is as follows:

Non-terminal symbol: {N, V}

Terminal symbol: {s, r, l, p}

Start symbol: S

In the above grammar, | represents "or".

Data processing and semantic sentence derivation process:

Assume that [the original information matrix] is the EEG information of three channels collected from the subjects. This information is processed by the machine learning model network. The information is filtered and reduced in dimension, and the result is expressed by the matrix of the following formula (4.12):

When performing grammar analysis, it can be written in one line: 1a2b345678902a2b345678913a2b34567892

in:

1a2b34567890 corresponds to the information of channel 1 (C1T)

2a2b34567891 corresponds to the information of channel 2 (C2T)

3a2b34567892 corresponds to the information of channel 3 (C3T)

Since all signals can be ultimately mapped to two elements: "identity information + press", "right (left)", the information in the matrix can be formally divided into two parts, one representing "identity information + press", and the other representing "right (left)". Identity information includes background feature signals related to individuals such as contingent negative variation (CNV) and event-related desynchronization (ERD). Therefore, the information in the matrix is divided into two parts, as shown in Figure 13:

The part framed in red represents "identity information + press", and the last column represents "right", so the part framed in red corresponds to the terminal symbols s and p in the derivation. Corresponding to the terminal symbol r.

Since EEG signals contain a certain degree of randomness, the positions of these two parts of information also have a certain degree of randomness, and the values also fluctuate. For example, in the feature matrix obtained by analysis (Figure 14), the information position corresponding to the terminal symbol r can be sandwiched between other signals, as shown in the red part of Figure 15.

After converting EEG information into information with clear semantics, clear semantics are added to the grammar, as follows:

S→N V N

→s p r (4.13)

Formula (4.13) is the newly formed grammar. With the grammar, we can construct a syntax tree, and then form sentence patterns for the newly collected EEG signals, determine their phrases, and understand their semantics.

The action corresponding to equation (4.13) is actually the action of pressing a key, which can be captured by the EEGLAB tool. This event is triggered by the key pressing action, which is significantly different from the waveforms before and after, and has obvious semantic features.

Since EEG signals may contain noise, the new test signal obtained after dimensionality reduction may not be accurately matched, just like the difference between Figure 15 and formula (4.12). Therefore, there needs to be a way to match the information with the trained information, and to be able to identify it probabilistically when it is indeed impossible to match it completely. Moreover, these processes are already existing technologies of neural networks, so the prediction function of neural networks corresponds exactly to the relevant processing flow of the grammatical analysis tree in semantic analysis.

Since CNN can automatically learn the most effective filter for a specific problem, the CNN model is used to obtain the result, as shown in Figure 16. Then, the effective original data is checked based on the visualized graph and data to find the feature data.

The points in Figure 16 represent the intermediate feature data after dimensionality reduction. The combination of these data can be effectively classified. We can check the original feature data based on the mapping table. In this way, it is easy to find the feature data template of a subject.

Specific embodiment 3

A method for constructing a syntax tree is implemented, wherein the process of constructing a syntax tree is to form a feature alphabet according to the semantic relationship results of the current experimental analysis of the input information, and the relationship between the key feature information blocks after dimensionality reduction, and then form a semantic alphabet through the feature alphabet to form a feature grammar, form a feature generation formula set (a non-empty finite set composed of generation formulas or rules), and then form a feature syntax tree. Sentence patterns are constructed according to the feature syntax tree so that the algorithm can use it in subsequent processing. With the newly formed grammar and feature syntax tree, sentence patterns are formed for the newly collected EEG signals, their phrases, handles and grammar are determined, and the semantics therein is understood.

According to the method, the characteristics also include:

Feature information forms a feature alphabet: All input information is processed by the function into a feature matrix M ₁ . Each element of M ₁ is also a matrix M ₂ . The elements of M ₂ are represented by binary or hexadecimal values. All elements of M ₁ belong to the same feature matrix alphabet, which consists of non-repeating elements.

According to the method, the characteristics also include:

The further division of feature information into groups with different semantics and conversion of each group into a feature string refers to the formation of a semantic alphabet from the feature strings. The semantic alphabet means that all elements of the input information _M2 can be semantically mapped into specific symbols by the function _fM->a at the same time. These symbols can be elements in a character set recognized by a computer system. These symbols, operators and separators form a finite non-empty set A, in which each element is a character that can be recognized (such as a computer system) or a string of multiple characters. Therefore, within the scope of discussion in this article, it can also be represented by V. That is to say, the semantic alphabet and the semantic vocabulary are regarded as the same and are referred to as the alphabet for short.

Subsequent features can refer to the other 10 definitions mentioned above.

Specific embodiment 4

A corresponding processing device can be made by this method, and the device includes:

The first processing module is used to process the EEG signal data to obtain feature information;

The second processing module is used to further divide the feature information into groups with different semantics, and convert each group into a feature string;

The third processing module is used to convert the EEG signal information into corresponding decision trees and syntax trees through a machine learning model;

The fourth processing module is used to form a grammar G[s] from the generated multiple syntax trees.

The fifth processing module is used to write each obtained characteristic character string into a production formula in the form of grammar to form a grammar G[s]; that is to say: construct and modify the grammar tree through experiments. The processing is to form the production formula of the grammar, the characteristic signal grammar, and then the characteristic grammar tree according to the semantic relationship results of the current experimental analysis of the input information and the relationship between the key characteristic information blocks after dimensionality reduction. Construct a sentence pattern according to the characteristic grammar tree so that the algorithm can be used in subsequent processing. With the newly formed grammar and characteristic grammar tree, the sentence pattern of the newly collected EEG signal is formed, its phrases, handles and grammar are determined, and the semantics therein is understood.

Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims (16)

1. A method for processing grammar, syntax and semantic paradigms of EEG signals, comprising: Processing the EEG signal data to obtain feature information, wherein the EEG signal includes a scalp EEG signal, a cortical ECoG signal, and a magnetoencephalogram signal; Divide the feature information into groups with different semantics, and convert each group into a feature string; Forming a feature alphabet according to the feature information; specifically comprising: forming a feature alphabet according to the semantic relationship result of the current experimental analysis of the input information and the relationship between the key feature information blocks after dimensionality reduction; Processing the EEG signal information to obtain a syntax tree; specifically, the process includes: forming a semantic alphabet through a feature alphabet, forming a feature grammar, forming a feature generation set, wherein the feature generation set is a non-empty finite set consisting of generation formulas or rules, and then forming a syntax tree; The generated syntax trees form a grammar G[s], where the number of syntax trees can be 1 or a value greater than 1. 2. The method according to claim 1, characterized in that it comprises: Convert EEG signal information into corresponding decision trees through machine learning models; Convert the decision tree into a syntax tree. 3. The method according to claim 1, characterized in that it comprises: The processing of EEG signals includes decoding the data, that is, the raw EEG signal data collected is input, and the decoding module processes the pre-processed EEG data to obtain a discrete "label sequence" matrix, and each label has a corresponding semantic block, as shown in Formula 1. When processing the data, the machine learning module extracts, reduces the dimension and filters the local features; 4. The decoding method according to claim 3, characterized in that it comprises: The specific implementation of the decoding process is divided into CNN network mode, CTC+LSTM network mode, other RNN mode, wavelet transform mode, and various machine learning modes; for the CNN network mode, the matrix is shown in Formula 1; thus, a label sequence and feature data corresponding to the label sequence, as well as the relationship between the two are obtained. If it is based on the CTC+LSTM network mode, the matrix corresponds to the label sequence matrix; thus, a label sequence and feature data corresponding to the label sequence, the relationship between the two, and sequence length data are obtained; the sequence length is one-dimensional data, and its expression is: [max_time_step,…,max_time_step], and its length is batch_size, and the specific value is max_time_step, or, the output formed by wavelet transform constitutes a feature matrix. 5. The method according to claim 3, characterized in that it comprises: In the mother matrix of Formula 1, the source of each matrix element can come directly from the feature information after feature extraction, dimensionality reduction and filtering of local features. The element information in the mother matrix can also come from the weight matrix in the machine learning network or deep learning neural network model. 6. The method according to claim 2, characterized in that it comprises: The machine learning models include convolutional neural network (CNN) models, long short-term memory (LSTM) models, other RNN models, wavelet transform models, and various machine learning models. 7. The method according to claim 3 is characterized in that it includes: the decoding of the EEG signal data is to prepare for the subsequent semantic analysis and processing, the purpose of which is to process the data into an ordered form with semantic features and labels according to different levels, and to obtain corresponding decision trees and syntax trees at the same time. The semantic analysis and processing requires mapping the obtained labeled features to the decision tree and syntax tree. 8. The method according to claim 1, comprising: The EEG signal data is processed to obtain feature information, and a feature alphabet is formed based on the feature information: all input information is processed by the function into a feature matrix M ₁ , and each element of M ₁ is also a matrix M ₂ . The elements of M ₂ are represented by binary or hexadecimal numerical values. All elements of M ₁ belong to the same feature alphabet, which is composed of non-repeating elements. 9. The method according to claim 8, characterized in that it comprises: The feature information is further divided into groups with different semantics, and each group is converted into a feature string, which means that the feature strings constitute a semantic alphabet. The semantic alphabet means that all elements of the input information _M2 can be semantically converted and mapped into specific symbols by the function _fM->a at the same time. These symbols can be elements in the character set recognized by the computer system. These symbols and operators and separators constitute a finite non-empty set A, in which each element is a recognized character, or a string composed of multiple characters, represented by V. That is to say, the semantic alphabet and the semantic vocabulary are regarded as the same, and are referred to as the alphabet for short. 10. The method according to claim 1, characterized in that it comprises: Generate sentence patterns and phrases according to the syntax tree for high-level grammar, syntax and semantic analysis. 11. A method for constructing a syntax tree, characterized in that the process of constructing the syntax tree is to form a feature alphabet according to the semantic relationship results of the current experimental analysis of the input information and the relationship between the key feature information blocks after dimensionality reduction, and then form a semantic alphabet through the feature alphabet to form a feature grammar and a feature generation formula set, wherein the feature generation formula set is a non-empty finite set composed of generation formulas or rules, and then form a syntax tree; construct sentence patterns according to the syntax tree so that the algorithm can be used in subsequent processing; with the feature grammar and the syntax tree, form sentence patterns for the newly collected EEG signals, determine their phrases, handles and grammar, and understand the semantics therein. 12. The method according to claim 11, characterized in that it comprises: The EEG signal data is processed to obtain feature information, and a feature alphabet is formed based on the feature information: all input information is processed by the function into a feature matrix M ₁ , and each element of M ₁ is also a matrix M ₂ . The elements of M ₂ are represented by values in binary or hexadecimal form. All elements of M ₁ belong to the same feature alphabet, which consists of non-repeating elements. 13. The method according to claim 12, characterized in that it comprises: The feature information is further divided into groups with different semantics, and each group is converted into a feature string, which means that the feature strings constitute a semantic alphabet. The semantic alphabet means that all elements of the input information _M2 can be semantically converted and mapped into specific symbols by the function _fM->a at the same time. These symbols can be elements in the character set recognized by the computer system. These symbols and operators and separators constitute a finite non-empty set A, in which each element is a recognized character, or a string composed of multiple characters, represented by V. That is to say, the semantic alphabet and the semantic vocabulary are regarded as the same, and are referred to as the alphabet for short. 14. A method for processing electroencephalogram signals, comprising: Preprocessing the collected EEG signals, and decoding the preprocessed EEG signals through a decoding module, obtaining a discrete "label sequence" matrix after processing the preprocessed EEG data through the decoding module, extracting, reducing and filtering local features through a machine learning module, and obtaining feature information by processing the EEG signals, wherein the EEG signals include scalp EEG signals, cerebral cortical EEG signals, and magnetoencephalogram signals; The EEG signal information is processed to obtain a syntax tree, and a grammar G[s] is formed from the syntax tree, specifically including: forming a semantic alphabet through a feature alphabet, forming a feature grammar, forming a feature generation formula set, wherein the feature generation formula set is a non-empty finite set composed of generation formulas or rules, and then forming a syntax tree. 15. The method according to claim 14, characterized in that In the “label sequence” matrix, each label has a corresponding semantic block, as shown in Formula 1; In the mother matrix of Formula 1, the source of each matrix element comes directly from the feature information after feature extraction, dimensionality reduction and filtering of local features, or from the weight matrix in the machine learning network or deep learning neural network model; The specific implementation of the decoding process is divided into various machine learning modes such as CNN network mode, CTC+LSTM network mode, other RNN mode, and wavelet transform mode; Based on the CNN network model, the matrix is shown in Formula 1; thus, the label sequence, the feature data corresponding to the label sequence, and the relationship between the two are obtained; Based on the CTC+LSTM network model, the matrix corresponds to the label sequence matrix; the label sequence, the feature data corresponding to the label sequence, the relationship between the two, and the sequence length data are obtained; the sequence length is one-dimensional data, and its expression is: [max_time_step,…,max_time_step], its length is batch_size, the specific value is max_time_step, and it can also be the output formed by wavelet transform to form a feature matrix. 16. The method according to claim 14, characterized in that Before the EEG signal information is processed to obtain the syntax tree, the feature information is divided into groups with different semantics, and each group is converted into a feature string; a feature alphabet is formed according to the feature information; The processing of the EEG signal information to obtain a syntax tree includes: converting the EEG signal information into a corresponding decision tree through a machine learning model; converting the decision tree into a syntax tree; The machine learning models include convolutional neural network (CNN) models, long short-term memory (LSTM) models, other RNN models, wavelet transform models, and various machine learning models; The decoding of EEG signal data is to prepare for the subsequent semantic analysis and processing, the purpose of which is to process the data into an ordered form with semantic features and labels according to different levels, and to obtain the corresponding decision tree and syntax tree at the same time; the semantic analysis and processing requires mapping the obtained labeled features with the decision tree and syntax tree; Feature information forms a feature alphabet: All input information is processed by the function into a feature matrix M1. Each element of M1 is also a matrix M2. The elements of M2 are represented by binary or hexadecimal values. All elements of M1 belong to the same feature alphabet, which consists of non-repeating elements. The feature information is further divided into groups with different semantics, and each group is converted into a feature string, which means that the feature string constitutes a semantic alphabet. The semantic alphabet means that all elements of the input information M2 can be semantically converted and mapped into specific symbols by the function fM->a at the same time. These symbols can be elements in the character set recognized by the computer system. These symbols and operators and separators constitute a finite non-empty set A, in which each element is a recognizable character or a string composed of multiple characters. Therefore, within the scope of discussion in this article, it can also be represented by V, that is, the semantic alphabet and the semantic vocabulary are regarded as the same, and are referred to as the alphabet for short; Generate sentence patterns and phrases according to the syntax tree for high-level grammar, syntax and semantic analysis.