ITPD20080261A1 - MAN INTERFACE SYSTEM - MACHINE FOR REHABILITATION PURPOSE AND REHABILITATION METHOD - Google Patents

Description

DESCRIPTION

The present invention relates to a human-machine interface apparatus, in particular for rehabilitation purposes, as well as a relative rehabilitation method.

The invention is particularly suitable to be applied on patients suffering from total paralysis and therefore unable to satisfy their own needs independently.

Interfaces between the human brain and a processing unit, called BCI (brain computer interface), are known in the art, which have been developed to allow patients with brain lesions to communicate with the surrounding environment, also through the use of robotic devices.

In particular, the BCIs are not based on the use of the patientâ € ™ s muscular activity, but use particular brain waves emitted by the patientâ € ™ s brain to control the robotic devices that act as physical actuators.

These waves are detected by special sensors, known in the art, positioned on the patient's head and converted into electrical signals. Several techniques have been developed in the known art to process these signals and recognize the commands generated by the patient. In most cases these commands are given to software running on a computer near the patient. There are few examples in the known art in which these commands are sent to a robot.

The solutions of the known art provide for the use of a graphical interface which usually includes a plurality of options or targets, depicting actions requested by the patient (such as for example the desire to drink, eat or read).

The interface also includes a randomly moving cursor on the interface display, near the targets.

When the user sees the cursor flashing in the direction of the target of interest, the latter emits brain waves which are intercepted by the sensors and sent to the actuators responsible for carrying out the requested action.

The solutions of the known art have some disadvantages.

In particular, the known solutions allow the patient to control a robot but have limits in terms of neurorehabilitation and social integration of the patient since the latter is unable to identify with the robot in order to consider and perceive the robot as a extension of one's perceptions.

The problem of the present invention is that of realizing an apparatus which solves the drawbacks mentioned with reference to the known art.

These drawbacks and limitations are solved by an apparatus according to claim 1 and by a rehabilitation method according to claim 14.

Other embodiments of the present invention are described in the subsequent claims.

Further characteristics and advantages of the present invention will be more understandable from the following description of its preferred and non-limiting embodiment examples, in which:

Figure 1 represents a schematic view of an apparatus according to an embodiment of the present invention;

Figure 2 is an enlarged schematic view of the detail II of Figure 1, in a first operating condition;

Figure 3 is an enlarged schematic view of the detail II of Figure 1, in a second operating condition.

The elements or parts of elements in common between the embodiments described below will be indicated with the same numerical references.

With reference to the aforementioned figures, the numeral 4 globally indicates a human-machine interface apparatus for rehabilitation purposes.

According to an embodiment, the apparatus comprises at least one sensor 8 suitable for being applied to the head of a patient and for receiving the emission of brain waves from the patient.

The apparatus 4 also comprises at least one interface 12 (BCI) equipped with a processing unit 14.

Preferably, interface 12 is associated with a display or video 16 on which some actions or targets 18 of the patient are represented, such as for example the desire to eat (Figures 2-3). It is evident that it is possible to represent on the display 16 any other action or desire, not necessarily linked to the primary needs of the patient.

On the display 16 of the interface 12 there is preferably a cursor that moves according to the choice of the patient. The cursor 20 is for example represented in the form of a dot (figures 2-3). In other words, the four arrows 21 in the four cardinal directions light up alternately in a random manner. When the patient recognizes that the arrow corresponding to his intention to move lights up, the brain signal recognized by the BCI is generated. At this point the cursor 20 is moved in the direction identified by the BCI.

According to an embodiment, the interface also comprises sound reproduction means 24, such as for example loudspeakers, suitable for reproducing and returning to the patient the sound signals coming from the surrounding or external environment, as explained below.

The apparatus 4 further comprises at least one robot 28, preferably of the holonomic type.

Advantageously, said robot is equipped with its own handling means 32, ie with motor means 32 capable of allowing the robot to be moved, under the guidance and control of the interface 12.

According to a preferred embodiment, the holonomic robot has a hexagonal structure provided with omnidirectional wheels, preferably three omnidirectional wheels.

Thanks to this configuration, the robot 28 has the possibility to reach any position on the surface without having to change its angle. The odometry control is preferably carried out in real time so as to allow the recognition of the position of the robot 28 at any time.

Advantageously, said robot 28 is equipped with sensory means 36 suitable for reproducing the senses such as sight and hearing by sending these sensations directly to the user, through the interface 12.

According to an embodiment, these sensory means 36 comprise a video camera 40, for filming scenarios and situations, and / or a microphone 44 for picking up and transmitting the sounds of environments.

According to a preferred embodiment, the sensory means 36 comprise an omnidirectional camera 40 which therefore allows a 360-degree view. Said camera 40 therefore allows to reproduce a circular image which is able to cover all the space around the robot 28.

This image is sent, preferably via a wireless â € wirelessâ € ™ connection to the processing unit 14 of the interface 12 (BCI).

Preferably, the image coming from the robot 28 is superimposed on the graphic representation of the display 16, in the least deleterious way possible for the patient's concentration.

In other words, the robot 28, and the camera 40 supported by it, is almost always stationary and moves in accordance with the cursor 20 on the screen 16; in this way the moments in which the image is in motion are limited in time and therefore the concentration of the subject is not affected.

According to a possible embodiment, it is envisaged to disconnect the acquisition of the interface 12, ie of the cerebral signals, during the movement of the robot 28.

According to an embodiment, as the robot 28 approaches a target 18, the latter represented on the screen 16, it enlarges (Figure 3).

In this way the patient receives feedback on the choice made.

Advantageously, the sensory means 36 of the robot 28 are operationally connected to the interface 12, so as to transmit the sensations acquired from the environment directly to the patient.

In this way, the patient is able to perceive the concept of telepresence and to project himself virtually, but in an "involving" way, into the surrounding environment.

The operation of the apparatus according to the invention according to some possible embodiments will now be described.

Preliminarily, some details are provided regarding the type of robot 28 actually used in some experimental tests.

In particular, it is possible to use a holonomic robot 28, preferably with a hexagonal structure provided with three omnidirectional wheels. In this way the robot 28 has the possibility to reach any position on the surface without having to change its angle.

This solution facilitates both the integration with the display 16 used by the interface 12 which is based on the movement of a circular figure on the screen itself, and avoids the burden on the patient of having to control the rotation of the robot 28 on his own axis.

Odometry control is entrusted to an engine card connected to the on-board computer through the serial port. According to one embodiment, a mini computer is mounted on the robot 28, for example with PC104 architecture. Since the mass unit is mounted on a mobile support, it is preferable to use a Compact Flash instead of a traditional hard disk. The motherboard is also preferably equipped with an integrated ethernet port. Modules have been added to the motherboard, such as a framegrubber for video capture, an audio module and a PCMCI module for the Wireless card. The robot 28 is also equipped with an omnidirectional camera 40 equipped with a hyperbolic mirror.

The movement system is based on cyclical writing / reading of the instruction on the motor card, through the serial port. In this way the robot 28 can check the odometry of the motor vehicles 32 almost in real time and know its position in the environment at any time. At the same time, the frames acquired by the camera 40 are sent via the TCP-IP connection to the display 16.

The interface 12 used provides for a step-by-step movement of the actuator or robot 28 until the desired target 18 is reached.

It is a type of movement in a pre-established environment whose characteristics are different from a movement that can be carried out in a free space. Wanting to configure robot 28 as a physical actuator mirroring the virtual actuator present on screen 16, it too must faithfully replicate this type of movement step by step until target 18 is reached. Having to confirm your choice for a number preset of times, it allows the patient to have greater control over the actuator 28 in order to reach the desired target 18.

Alternatively, if it is of interest to use the robot 28 in an undetermined space, with a free movement, the solution is to leave the patient with the task of directing the robot 28 towards the desired goal through the high-level commands â € œrightâ €, â € œto the rightâ €, â € œto the leftâ €, â € œbackwardsâ € which correspond to the four cardinal directions of interface 16.

In this type of use of the robot 28 with the interface 12 all the additional modules that allow the telepresence of the patient are placed.

First of all, the omnidirectional camera 40, which through an appropriate image processing can provide an undistorted image of what the robot 28 sees.

This image is integrated with the display 16 of the interface 12 and allows the patient to control the robot 28 remotely, to watch what is happening in other rooms and to recognize any people out of sight.

The microphone 44 and the speakers on the robot add other important features to the patient's telepresence.

According to an embodiment, the interaction between robot 28 and interface 12 is based on a Client-Server system by means of a TCP-IP connection between the two.

This allows you to control the robot and receive its perceptions even through a TCP-IP network such as the Internet. In this way the robot may not even be in the same room or building as the patient, but it may be anywhere in the world where an Internet connection is set up. The use of a standard such as TCP-IP allows the patient to connect to other robots connected to the Internet (sending commands and receiving perceptions) as long as these are equipped with the appropriate communication protocol defined below.

The processing unit 14 on which the BCI 12 is present acts as a server while the robot 28 as a client.

After each stimulus of the interface, the signal acquired and processed by the BCI is converted into a command to be sent to the robot 28 which, once executed, responds to the server with the outcome of the action. In this way there is a two-way connection between client and server, ie between robot 28 and BCI 12, which allows the actuator to faithfully replicate the commands originally sent by the patient.

Both the client and the server preferably rely on a finite state machine. The two machines interconnected with each other exchange commands and information relating to movement for the duration of the session. Preferably, it is possible to establish an ad hoc protocol for exchanging data between BCI 12 and robot 28.

This protocol is for example asymmetric between client and server. The server sends a data string 5 bytes long, while the client always replies with a single byte.

The string sent by the server is determined as follows: - Byte Character Meaning

- 1 S Start character

- 2 X Interface Id

- 3 X Direction (1,2,3,4)

- 4 X Go / Stop

- 5 E End character

Valid commands sent by the client are for example: - A The robot is ready and positioned

- R The command was correct and was executed - N The command was correct but was not executed - X The command was incorrect

- C The robot closes the connection

The correctness of the data received is entrusted both to the TCP protocol and to an internal control of the client.

Before processing the command, the client takes care of verifying its validity. It checks the length of the string received (5 characters), the correctness of the first and last characters (encoded respectively as â € œSâ € œ and â € œEâ € œ) and the type of command received. In this way a complete reliability of the protocol is guaranteed. The physical connection between BCI 12 and computer 14 preferably takes place via a VPN (Virtual Private Network). It allows you to connect two parties using a public transmission system such as the internet. The VPN network uses an authentication and encryption method so that the data sent is not intercepted by unauthorized users.

In this way it is highly unlikely that someone from the outside will be able to connect to the robot 28, guaranteeing a very high level of safety both for the patient and for the structures in which the robot 28 operates.

The client-side program present on the robot 28 bases its operation on a finite state machine. The states are used for the various movement functions and for dialogue with the server. When the client program loads, it is its job to try to connect to the server. Once connected, the robot 28 goes into a listening phase, until the server sends the session start command. Once this command has been sent, the patient is in direct control of the robot 28.

Once the session has started, the client enters a cycle whose main phases are: Waiting for command ∠’Command execution − Sending action result to the server.

Each time a command has been executed with a positive or negative outcome, robot 28 goes back to the â € ̃Waiting commandâ € ™ phase, ready for a new order. The cycle is interrupted when the end-of-session command arrives from the server. The execution phase of the command is in turn divided into three distinct parts. Initially the client performs an integrity check on the received command, if it is a valid command proceed to the next step, otherwise a warning is sent to the server and the server itself will take care to send the same command again. If the command is valid, the client will determine if it is a movement command, if so it will start communicating with the motor card. Once the action has been performed, the result will be sent to the server and the client will go back to the â € ̃Waiting for commandâ € ™ phase. The execution of the action by the robot takes place with a continuous feedback linked to the odometry of the motor means 32. The command sent by the server activates the movement of the robot in the desired direction. Preferably, every 200 ms the robot 28 checks the distance traveled by means of the odometry of the motor means 32: if the robot 28 has traveled a distance equal to the predefined step it will stop, otherwise it will continue to advance until the distance is reached pre-established.

Like the client program, the server also uses a finite state machine. The server for connection and management of the robot 28 is fully integrated with the BCI 12 program. The signals acquired and processed by the BCI 12 are managed by the server and sent in the form of commands to the robot 28. The operation of the server can be identified with three main phases: sending the command, waiting for execution, receiving the action result. The management of the signal coming from the patient is entrusted to the BCI 12 software. Once the presence or absence of a P300 brain wave has been determined, using the appropriate sensors, the BCI 12 determines to which direction of movement the cognitive potential corresponds by means of a synchronization with the stimuli of the display 16.

This directional command is sent to the robot 28 with the predetermined coding. Robot 28 will perform the action and reply to the server about the outcome. Depending on the result received, the server will be ready for a new command to be sent or will send the same again. When the server program is loaded it will listen on a specific address and port, waiting for a connection from the client. Once the client is connected, we will proceed to the phase of sending the session start command. At this point the server will manage and interpret the information relating to the movement provided by the robot 28. Robot 28 and BCI 12 are thus connected in a bi-univocal way. This system allows both a direct control by the patient of the movement of the robot 28 and an optimal management of any exceptions due to movement problems of the robot 28.

As already described, in order to fully develop the concept of patient telepresence through the robot, it is necessary that the robot 28 is equipped with other characteristics besides movement. Thanks to the appropriately processed images provided by the omnidirectional camera 40 present on the robot, the patient has the possibility to look at the entire environment surrounding the robot from a distance, thus being able to have both a feedback on the movement and a perception of what happens in another place. In parallel, but following the same concept, the robot 28 is equipped with at least one microphone 44 and speakers connected to the video card. Preferably, the display 16 is also equipped with speakers 48 so as to send the sounds picked up by the robot microphone 44 directly to the patient.

The program that manages the passage of data from the robot 28 to the computer 14 used by the patient was created specifically and is based on audio / video streaming.

The image acquired by the omnidirectional camera 40 is filtered with specific algorithms and modified in order to eliminate the distortions due to the 360 degree vision.

This processing phase is carried out on each frame acquired by the camera. Once processed, the frame is compressed into suitable formats, for example in MPEG4 format, to lighten the load on the network.

The audio is compressed into suitable formats, such as MPEG, and sent together with the images to the patient's computer. With the addition of audio and video features, the patient is therefore in total control of the robot.

It has the ability to move it and interact through it with other people. This system therefore allows the realization of the concept of telepresence for a bedridden subject.

The integration and interaction between BCI and robotic actuator opens up new possibilities and new avenues both in the field of neurorehabilitation and in the improvement of the life of completely paralyzed patients.

As can be appreciated from what has been described, the apparatus and the method according to the invention allow to overcome the drawbacks presented in the known art.

In particular, the apparatus and the method allow the subject to reach a sort of telepresence, thanks to the aid of the robot equipped with a camera.

As illustrated above, telepresence passes through three key points which are the interaction between the interface and the actuator, the movement of the actuator and the senses (sensory means) of the actuator.

The interaction between the interface and the actuator is based on the precise superimposition of the virtual image that the patient sees on the interface screen with the actions performed by the robot. Therefore the subject can interact with a simplified interface adapted to his characteristics, but at the same time knowing that every action that takes place on the screen will be faithfully replicated by the robot in a physical space.

The movement of the actuator is essential since being able to move the robot through only a cognitive potential related to concentration allows the subject to be estranged, at least in part, from the condition of immobility and gives him again the possibility of action.

Furthermore, the duplication of the senses through the robot, such as sight and hearing, allow the patient to see and hear through the robot itself and therefore to interact socially and express their needs.

Telepresence allows the subject to overcome the drawbacks and limitations of the known art in terms of neurorehabilitation and social integration of the patient, since the latter is able to identify with the robot in order to consider and perceive the robot as an extension of their perceptions.

In other words, the integration between interface and robot guarantees to a completely paralyzed subject the possibility to move, for example, inside his own home, to be able to report his needs and to be able to listen, talk and see other people in others rooms.

Telepresence also allows the patient to leave their home and, through a remote connection with a robot, visit for example museums or structures in other cities.

A person skilled in the art, in order to satisfy contingent and specific needs, will be able to make numerous modifications and variations to the apparatus and method described above, all however contained within the scope of the invention as defined by the following claims.

Claims (20)

CLAIMS 1. Man-machine interface apparatus for rehabilitation purposes (4), comprising - at least one sensor (8) suitable for picking up the electrical signals generated by the brain activity of a patient, - an interface (12) equipped with a screen (16) which represents at least one target (18) for the patient, the interface (12) being operatively connected to said at least one sensor (8) and being equipped with a processing (14), suitable for interpreting brain electrical signals and recognizing the commands given by the patient, - an actuator or robot (28) operatively connected to said interface (12) so as to be operable by the interface (12), characterized by the fact that - said robot (28) is equipped with its own handling means (32) which can be operated from said interface (12) and comprises sensory means (36) suitable for picking up visual and sound signals from the environment, said sensory means (36) being operationally connected to the interface (12) in order to transmit the visual and sound signals to the patient, through the interface (12). 2. Apparatus (4) according to claim 1, wherein said moving means (32) comprise motor means capable of allowing the movement of the robot (28) under the guidance and control of the interface (12). Apparatus (4) according to claim 1 or 2, wherein the robot (28) is of the holonomic type and comprises omnidirectional wheels. 4. Apparatus (4) according to claim 1, 2 or 3, in which said processing unit (14) controls the odometry of the robot (28) in real time so as to allow the recognition of the position of the robot ( 28) at any time. 5. Apparatus (4) according to any one of the preceding claims, wherein said sensory means (36) comprise at least one camera (40) operatively connected with the interface (12) so as to send the images detected from the environment onto called display (16) of the interface (12). 6. Apparatus (4) according to claim 5, wherein said camera (40) is omnidirectional so as to allow a 360-degree view covering all the space around the robot (28) and to reproduce a circular image of this space . 7. Apparatus (4) according to claim 5 or 6, wherein said camera (40) is connected to the interface (12) and, through this, to the display (16), by means of a wireless connection. Apparatus (4) according to any one of the preceding claims, wherein said sensory means (36) comprise at least one microphone (44) suitable for picking up and transmitting ambient sounds to said interface (12). Apparatus (4) according to claim 8, wherein the display (16) is equipped with at least one loudspeaker (48) so as to send the sounds picked up by the microphone (44) of the robot (28) directly to the patient. Apparatus (4) according to claim 8 or 9, wherein said at least one microphone (44) is connected to the interface (12) by means of a wireless connection. Apparatus (4) according to any one of the preceding claims, wherein the display (16) represents a plurality of actions or targets (18) arranged in a circular arrangement. Apparatus (4) according to any one of the preceding claims, in which on the display (16) of the interface (12) there is a cursor (20) which moves randomly so as to point cyclically on said targets ( 18). Apparatus (4) according to any one of the preceding claims, in which the targets (18) represented on the screen (16) are enlarged according to the position of the robot (28) with respect to the real target, so as to provide feedback to the patient on the choice made. Method of rehabilitating a patient by means of an apparatus (4) according to any one of claims 1 to 13, comprising the steps of: - detect the brain waves of a patient using at least one sensor (8), - operatively connect said sensors (8) to an interface (12) equipped with a processing unit (14) and a display (16) which represents a plurality of actions or targets (18) for the patient, - command a robot or actuator (28), operationally connected to the interface (12), based on the patient's requests detected through said sensors (8), characterized by the fact that - the method comprises the step of detecting by sensory means (36,40,44), associated with the robot (28), images and / or sounds of the environment surrounding the robot (28) and sending these signals to the patient through said interface (12). 15. Method according to claim 14, comprising the step of carrying out in real time the control of the odometry of the robot (28), by means of said processing unit (14), so as to allow the recognition of the position of the robot (28) at any time and to send images and sounds in real time to the patient. 16. Method according to claim 14 or 15, comprising the step of acquiring images of the environment surrounding the robot (28) by means of an omnidirectional camera (40), operatively connected to the interface (12), so as to send a circular image of the environment surrounding the robot (28) on said interface display (16) (12). 17. Method according to claim 14, 15 or 16, comprising the step of acquiring the sounds of the environment surrounding the robot (28) through at least one microphone (44), transmitting them to the interface (12) and reproducing them to the patient by means of at least one loudspeaker (48). 18. Method according to claim 16, comprising the step of representing on the display (16) a plurality of actions or targets (18) arranged according to a circular arrangement, said arrangement being superimposable on the circular image sent to the display (16) by the camera (40) omnidirectional. 19. Method according to any one of claims 14 to 18, comprising the step of making the arrows corresponding to the four cardinal directions (21) flash randomly on the display (16) of the interface (12) so as to cyclically highlight said target (18). Method according to any one of claims 14 to 19, comprising the step of enlarging the targets (18) represented on the screen (16) as a function of the instantaneous position of the robot (28) with respect to the real target, so as to provide feedback to the patient about the choice made.