CN1331080C - Virtual keyboard and robot control system by brain electric signal - Google Patents

Abstract

This device relates to a virtual keyboard and robot control system in the field of biomedical engineering. Its structure is composed of a special hardware platform and a control program. The hardware platform is composed of a PC, a brain-computer interface controller, a robot, and peripheral electrical appliances. The computer is connected with the brain-computer interface controller through the serial port, and the EEG signal realizes the control of the virtual keyboard and the robot through the central processing unit of the computer. People with disabilities; the software system has a high recognition rate and can realize the function of a virtual keyboard, so that different EEG signals can be expressed in words, and an active way is adopted when the virtual keyboard outputs text, so that the visual system of the subject will not be affected by any external The stimulation makes the subjects more comfortable. The virtual keyboard can output not only English, but also Chinese. The system can realize the control of the robot by EEG signals, and make the robot provide services for people according to the EEG signals of thinking.

Description

EEG signal-based virtual keyboard robot control system and method

Technical field

The invention belongs to the technical field of biomedical engineering, and relates to a virtual keyboard and a robot control system, which combines the electrophysiology (electroencephalogram signal) of the biological nervous system with a computer and a robot in the whole system, and specifically relates to the electrophysiology of the brain, database , robotics, digital signal analysis and processing and control methods and other technical fields.

Background technique

At present, some foreign scholars are researching the brain-computer interface (Brain-Computer Interface) technology based on EEG in the laboratory. The main purpose is to serve as a barrier subsidy for the disabled. There are IEEE Trans Rehab Eng 2000; 8:190-192), P300 brain potential (references are IEEE Trans Rehab Eng 2000; 8:174-179), mu or beta wave brain potentials (references are Trans Rehab Eng 2000; 8: 203-204) to control the computer and surrounding electronic equipment; the function of the virtual keyboard output text is also used in a passive way, that is, there must be specific light to stimulate the visual system of the subject, and only English can be output.

Contents of the invention

In order to solve the insufficiency of the current brain-computer interface technology, the purpose of the present invention is to provide a virtual keyboard robot control system and method based on EEG signals. A set of unique systems is established, which uses the _C3 and the evoked potential at _C4 to realize the control of the virtual keyboard and the robot, and when the virtual keyboard outputs text, it adopts an active method, which makes the visual system of the subject do not need to be stimulated by any external, so that the subject The tester is more freely, and the present invention can realize following purpose:

Not only can use the virtual keyboard to output English, but also can output Chinese;

Use the EEG signal induced by specific thinking to realize the control of the switching circuit, the control of the cursor on the screen, and the control of the voice system, so that the disabled who cannot speak can express their inner world through language;

Use the EEG signal induced by specific thinking to control the behavior of the robot (make the robot move in a certain direction, let the robot sing, etc.);

Use the EEG signal induced by specific thinking to output Chinese text and English text on the computer display (virtual keyboard typing);

The image on the computer monitor is enlarged and reduced by using the EEG signal induced by specific thinking.

In short, the virtual keyboard and robot control system based on EEG signals is an electromechanical device for disabled people (especially paralyzed patients), that is, it can be used by people with normal thinking but motor dysfunction to express their inner world to the outside world. Communicate and control the external environment, such as expressing to the outside world the idea of eating, outdoor activities or listening to music, and controlling the external environment such as radios and electric fans, so as to achieve a certain degree of self-care in life.

It can also provide healthy people with an environment to control the outside world in a special environment, such as helping people to control the internal equipment in the area outside the high-radiation area, and it can also provide people with a new service method, such as through EEG. Signals command robots to help people do things, etc.

Technical scheme of the present invention is realized like this:

It is a device that expresses the inner world and controls the surrounding environment based on the EEG signals induced by specific thinking. The device is composed of a special hardware platform and control method. The hardware platform consists of a PC, a brain-computer interface controller, Robots, electric lights and other peripheral electrical appliances are composed of four parts. The connection is that the PC is respectively connected with the robot and the brain-computer interface controller through the serial interface. The input end of the controller is connected with the brain conduction electrode, and the PC is connected with the external device display , the brain-computer interface controller is connected with lamps and other electrical appliances, and the brain conduction electrodes are connected with the subject (as shown in Figure 1).

The EEG signal is collected by the electrodes and sent to the brain-computer interface controller. The brain-computer interface controller performs preprocessing such as filtering and amplification on the EEG signal, and the processed EEG data is transmitted to the computer in real time through the serial port. After the computer receives the preprocessed EEG data, it analyzes the received EEG data in real time, extracts characteristic signals from it, and uses it to judge the thinking activities of the brain and realize corresponding control actions, that is, to display the output results of the virtual keyboard through the display ; Voice expression can be performed through the sound card; control instructions can also be sent to the robot through the serial port of the computer, so that the robot can realize some predetermined actions and singing according to the specified control method; the instructions can also be sent back to the interface controller, Control peripheral lights and surrounding electrical appliances, etc.

The system hardware brain-computer interface controller: mainly consists of five parts, namely: input protection part, pre-amplification part, signal isolation part, post-stage signal amplification and filtering part, and microcomputer Inter 80C196KC acquisition control part (as shown in the figure) 2), the structure and working principle of each part are as follows:

Input protection part: This part is mainly composed of a Zener diode and an input protection resistor, and also has a certain protection function at the input end of the input preamplifier INA128;

Pre-amplification part: This part has two functional modules, a total of three parts, which are composed of INA128 and LM324 respectively. The amplification of the first half adopts the differential amplification method to improve the common mode rejection ratio of the input signal, and the second half is used for amplification. The negative feedback amplifier circuit, in order to effectively suppress the power frequency interference, is equipped with a double T network filter in the middle of the two-pole amplifier to avoid the signal overflow of the power frequency signal during the amplification process, and its gain is adjustable around 80dB;

Signal isolation part; the isolation stage is composed of TLP521-2. By setting its input current size, the isolation of the signal 1:1 amplification is realized. Due to the addition of the isolation stage, the front-end signal measurement is electrically isolated from the back-end to improve System security, but also improve the anti-interference performance of the signal;

Signal amplification and filtering part: the amplifier uses LM324 and OP07, and the front part is a fourth-order low-pass filter, which is composed of LM324. The rear is connected to a negative feedback amplifier circuit, and finally a level adjustment circuit, whose gain is adjustable around 20dB.

This system (as shown in Figure 3) sends the EEG characteristic signal extracted by the external computer to the central processing unit through the serial port, and the central processing unit uses the input EEG characteristic signal as an image induced by the upward or downward movement of the eyeball. The EEG signal is used to judge, and is responsible for the execution of the control method, to realize the control of the motor and the speaker. The motor drive is used to drive the DC motor, drive the wheels, and realize the walking of the robot; the audio amplifier is used to amplify the audio signal and provide it to the speaker. Realize the voice of the robot, the main program and subroutines are kept in the external memory.

The method of EEG signal to robot control is realized by computer control method:

The computer control method of electroencephalogram signal to robot control comprises the following steps (control method flowchart as shown in Figure 4):

After the control system is started, first check the settings of the corresponding bits of the serial communication control register 1 (SCCR1), serial communication control register 2 (SCCR2), and baud rate register (BAUD) inside the microcontroller, and set them to the values required by the control system. state, realize the initialization of the single-chip microcomputer (4010);

Then check the state of the serial communication status register (SCSR) to judge whether the characteristic EEG data is received, if the data is received (4020), the result is saved in the Cmd variable (4030), for later judgment, if at this time Existing process start (4040), just earlier the original process is shut down (4050), then judge the content of Cdm variable;

Corresponding to different commands, different processes (4070, 4090, 4110) are started, and the actions of singing, turning around and moving forward are performed, and the control steps are looped back to wait for the input of the serial port again.

The computer control method for identifying EEG signals is composed of an overall control flow chart, and the control method includes the following parts (as shown in Figure 5):

The control method includes: user interface design device, data input device, data storage device, EEG data processing and display, printing device, English and Chinese display device, image size control and small ball movement on the screen control device.

The control method includes the following steps:

When the system starts to run, it first judges whether it is online analysis or offline analysis (5020, 5040). If it is online analysis, first initialize the serial port, then receive data, and then process the received data accordingly to obtain corresponding EEG data ( 5030);

If it is an offline analysis, first open the corresponding data file, then read the data, then perform corresponding processing on the obtained data, and finally obtain the corresponding EEG data (5050);

Then the obtained EEG data is smoothed (5060) to remove the burrs in the EEG waveform; then the EEG data is filtered (5070) to remove the high-frequency interference in the EEG signal for the next step Make a judgment (5090);

Then carry out the sampling of the characteristic waveform (5080), this step can be carried out many times, to obtain the best judgment result (5090);

After the sampling is completed, according to the characteristics obtained by sampling, make a corresponding judgment (5090), and then correct the judgment made (5100), and correct the false judgment;

The above steps can achieve the following functions;

1) Control the movement of the ball on the screen to make it move up or down, and there will be a corresponding sound prompt (5110);

2) Control the size of the image on the screen, which can be enlarged or reduced, and there will be a corresponding sound prompt (5130)

3) Coding the judgment, when the code reaches the specified length, the code is retrieved, and then the English or Chinese characters corresponding to the code are displayed (5150, 5160);

4) Controlling the behavior of the robot (5120);

5) Control the switches of the surrounding electric appliances (5140).

In this control method, the recognition algorithm for the EEG signal is the core part of the system. In this part, it is necessary to judge the two characteristic waveforms according to the obtained characteristics after standard characteristic sampling, correct the misjudgment, and complete the data processing. output.

The recognition algorithm control method of the EEG signal includes the following steps (the flow chart of the control method is shown in FIG. 6 ).

First carry out sampling of the standard characteristic waveform (6010), if the sampling effect is not good, re-sampling (6020) can also be carried out, so that the effect of judgment can reach the best;

Then perform feature extraction (6040) on the EEG data entering the judgment area, and perform judgments on feature 1 and feature 2 according to the obtained standard characteristic waveforms (6050, 6080), and then correct the judgment (6110) to remove errors. judgments and minor adjustments to correct judgments;

If the data processing ends (6120), the core judgment can be exited, if not, the control step returns and re-samples (6020).

The control method of the EEG signal controlling the output of Chinese characters on the virtual keyboard is:

First select the database: create a database named bciword.mdb, and create a table named bciword in it. In the table, create a bci field with a type of text, and a word field with a type of text. Then input the Chinese characters into the word field, and use the Chinese input method to input the corresponding code into the bci field. At this time, it should be noted that the content in the bci field should be unique, as the query standard, so that a special The Chinese character library of the input method.

Secondly, the digital code is formed: In the system, 26 English letters are encoded in binary, and each letter occupies five digits. 00001 represents a, 00010 represents b, etc. In addition, we use 00000 to represent R to indicate the start of query execution. 11110 represents B, which means to delete a Chinese character.

This control method comprises the following steps (control method flowchart as shown in Figure 7):

When the control method starts to run, first connect to the ODBC data source, here we call the custom ConfigAccess() function to automatically connect, and then judge the Chinese characters to be output according to the EEG signal. When the EEG signal is input (7010), first judge whether the signal is 1 or 0 (7020), and then add the signal value to the end of the m_charindexstring variable (7030 or 7040). When there are 5 consecutive signal inputs, you can use the custom The GetStrFromIndex() function finds out its corresponding value (7050, 7060). If the corresponding value is a lowercase letter, add the value to the end of the m_strchoose variable (7080). When the obtained value is "R" (7090), that is, the continuously input EEG signal is 00000, then find out the corresponding value in the database according to m_strchoose The Chinese character (7120) is displayed; when the obtained value is "B", that is, the continuously input EEG signal is 11110, then delete the last Chinese character; when the obtained value is "D", that is, the continuously input EEG signal Signal is 11011, then delete last lowercase letter, after typing a Chinese character or a lowercase letter, corresponding variable initialization (7140), so that query next time.

The binary encoding on our system looks like this:

a 00001 q 10001

b 00010 r 10010

c 00011 s 10011

d 00100 t 10100

e 00101 u 10101

f 00110 v 10110

g 00111 w 10111

h 01000 x 11000

i 01001 y 11001

j 01010 z 11010

k 01011 delete a letter 11011

l 01100 delete a Chinese character 11110

m 01101 start query 00000

n 01110 Space 11111

o 01111

p 10000

Advantages of the present invention: in EEG signal acquisition, only two leads (C3-A1, C4-A1) can be used to obtain the required signal; the controlled hardware system has stable performance and is suitable for disabled persons; The computer-based software system has a high recognition rate. The system can realize the function of a virtual keyboard, so that different EEG signals can be expressed in words, and the virtual keyboard uses an active way to output text, which makes the subjects The visual system does not need any external stimuli, which makes the subjects more comfortable, not only can use the virtual keyboard to output English, but also can output Chinese; Electric signals enable robots to provide some services to people.

Description of drawings

Fig. 1 is the virtual keyboard based on electroencephalogram signal of the present invention and robot control system hardware connection block diagram;

Fig. 2 is the electrical schematic diagram of the brain-computer interface controller of the present invention;

Fig. 3 is the structural block diagram of the robot hardware of EEG signal control;

Fig. 4 is a flow chart of the method for controlling the EEG signal in the robot;

Fig. 5 is a flow chart of the overall control method of the present invention;

Fig. 6 is the identification algorithm flowchart of core EEG signal of the present invention;

Fig. 7 is a flowchart of the output of Chinese characters on the virtual keyboard controlled by EEG signals according to the present invention.

Detailed ways

Firstly, brain conduction electrodes are installed on the subject's head, and after a few minutes of training, the subject can be officially operated. Subjects can open eyes, eyes can also be closed, it all depends on the subject's own wishes. The subjects use their thinking to think about eyeballs up or down to generate EEG signals to complete different functions. If they want to output Chinese or English text on the computer monitor, they can use EEG signals to call out the corresponding codes (such as 00001 Output "a", 00110 0001110101 output "go") to complete the output of different text (such as "Hello", "I want to eat", etc.).

For example, the input method of "go" is fcu, the code of f in the system is 00110, the code of c is 00011, the code of u is 10101, plus 00000 means start query. When the EEG signal 00110 00011 1010100000 is continuously input, the word "Go" can be typed on the screen.

The corresponding content can also be played by the voice system; if you want to control the behavior of the robot, you can use the EEG signal to call up the corresponding code to make the robot move or sing; If you want your eyeballs to go up many times; if you want to make the image on the computer monitor shrink continuously, you can use your thinking to control your eyeballs to go down many times. If you want to control the electrical switch, you can use the EEG signal to call out the corresponding code to turn the electrical switch on or off. In short, according to the requirements of the subjects, the following functions can be realized: output Chinese and English text on the virtual keyboard on the computer monitor, control the behavior of the robot, control the opening and closing of surrounding electrical appliances (such as lights, TVs, air conditioners and electric fans, etc.), zoom in Or reduce the size of an image on a computer monitor, express your inner thoughts with a voice system, and more.

Claims (5)

1, a kind of virtual keyboard robot control system based on EEG signal, it is characterized in that this system is made up of four parts of a PC, brain-computer interface controller, robot, electric light and surrounding electrical equipment, and its connection is PC through serial The line interface is respectively connected with the robot and the brain-computer interface controller, and the input end of the controller is connected with the brain conduction electrode, and at the same time, the PC is connected with the external device display, and the brain-computer interface controller is connected with the lamp and other electrical appliances, and the brain conduction electrode Connect with the subject, and control the robot through a computer. The brain-computer interface controller includes an input protection part, a pre-amplification part, a signal isolation part, a post-stage signal amplification and filtering part, and a microcomputer Inter 80C196KC Acquisition control part, the pre-amplification part: there are two functional modules, a total of three parts, respectively composed of INA128 and LM324, the amplification of the first half adopts differential amplification to improve the common mode rejection ratio of the input signal, the second half The negative feedback amplifying circuit used for internal amplification, in the middle of the two-pole amplifier, is equipped with a double T network filter, whose gain is adjustable around 80dB; signal isolation part: the isolation stage is composed of TLP521-2, by setting its input current, the signal 1:1 amplification isolation; the amplifier uses LM324 and OP07, the front part is a fourth-order low-pass filter composed of LM324, the rear part is connected to a negative feedback amplifier circuit, and the last is a level adjustment circuit, its gain is around 20dB adjustable. 2. According to the EEG signal-based virtual keyboard robot control system of claim 1, it is characterized in that the EEG signal induced by the imaginary eyeball upward/downward motion measured by the present invention adopts C3-A1 and C4-A1 two-way signal to measure. 3, a kind of EEG signal to the method for robot control, it is characterized in that the computer control method of EEG signal in the robot comprises the following steps: After the control system is started, first check the settings of the corresponding bits of the serial communication control register 1 (SCCR1), serial communication control register 2 (SCCR2), and baud rate register (BAUD) inside the microcontroller, and set them to the values required by the control system. state, realize the initialization of the single-chip microcomputer (4010); Then check the state of the serial communication status register (SCSR) to judge whether the characteristic EEG data is received, if the data is received (4020), the result is saved in the Cmd variable (4030), for later judgment, if at this time Existing process start (4040), just earlier the original process is shut down (4050), then judge the content of Cdm variable; Corresponding to different commands, different processes (4070, 4090, 4110) are started, and the actions of singing, turning around and moving forward are performed, and the control steps are looped back to wait for the input of the serial port again. 4. The control method according to claim 3, wherein the computer control method for identifying EEG signals comprises the following steps: When the system starts to run, it first judges whether it is online analysis or offline analysis (5020, 5040). If it is online analysis, first initialize the serial port, then receive data, and then process the received data accordingly to obtain corresponding EEG data ( 5030); If it is an offline analysis, first open the corresponding data file, then read the data, then perform corresponding processing on the obtained data, and finally obtain the corresponding EEG data (5050); Then the obtained EEG data is smoothed (5060) to remove the burrs in the EEG waveform; then the EEG data is filtered (5070) to remove the high-frequency interference in the EEG signal for the next step Make a judgment (5090); Then carry out the sampling of the characteristic waveform (5080), this step can be carried out many times, to obtain the best judgment result (5090); After the sampling is completed, a corresponding judgment is made according to the characteristics obtained by the sampling (5090), and then the judgment made is corrected (5100) to correct the false judgment. 5. The control method according to claim 3, characterized in that the control method for EEG signal recognition comprises the following steps First carry out sampling of the standard characteristic waveform (6010), if the sampling effect is not good, re-sampling (6020) can also be carried out, so that the effect of judgment can reach the best; Then perform feature extraction (6040) on the EEG data entering the judgment area, and perform judgments on feature 1 and feature 2 according to the obtained standard characteristic waveforms (6050, 6080), and then correct the judgment (6110) to remove errors. judgments and minor adjustments to correct judgments; If the data processing ends (6120), the core judgment can be exited, if not, the control step returns and re-samples (6020).

Images