KR20230054522A - Augmented reality rehabilitation training system applied with hand gesture recognition improvement technology - Google Patents

Abstract

The present invention utilizes infrared image-based hand motion recognition technology and voice recognition technology using a lip motion controller to induce interest in patients in augmented reality rehabilitation training, while greatly improving hand motion recognition accuracy and training efficiency. Augmented reality with hand motion recognition enhancement technology applied It is about the rehabilitation training system.

Description

Augmented reality rehabilitation training system applied with hand gesture recognition improvement technology}

The present invention relates to a rehabilitation training system, and more particularly, utilizes infrared image-based hand motion recognition technology and voice recognition technology using a leap motion controller to arouse patients' interest in augmented reality rehabilitation training while significantly improving hand motion recognition accuracy and training efficiency. It is about an augmented reality rehabilitation training system to which hand motion recognition improvement technology is applied.

Various studies in the field of cognitive rehabilitation have revealed that the brain can be restored through repetitive, intensive, and task-oriented training. Common.

Therefore, it is important for therapists to find ways to encourage patients to actively participate in rehabilitation programs, as motivating patients is an important factor in rehabilitation success and often affects rehabilitation outcomes.

However, the biggest difficulty these treatment methods face in the market is that they inevitably require guidance from rehabilitation experts or require high treatment costs.

Recently, as interest in and research on promising technologies for the 4th Industrial Revolution have been active, Metaverse technology has been in the limelight, and its effects have been proven through continuous research on media, culture, medical care, and industry as well as daily life. . Intervention of cognitive rehabilitation using metaverse-based digital computer contents is expected to provide a solution to the difficulties experienced due to the aforementioned cost or lack of experts.

In particular, it has been reported that in rehabilitation medicine, efforts to include metaverse elements in computerized contents for rehabilitation training can help patients not lose interest or increase training effects, and in the future, artificial intelligence will be used to maximize these training effects. Reinforcement of the input mechanism using , and achievements in system research and development are attracting attention.

Recently, virtual reality is trying to maximize its functionality by designing, developing, and constructing internal contents suitable for virtual reality without including an external auxiliary device belonging to hardware in an output mechanism or an input mechanism. These efforts represent a situation where the need for overall research, such as software processing of the input mechanism, is increasing, escaping from the implementation of the metaverse that relied only on external auxiliary devices, strengthening the content and development standards of suitable dedicated content.

However, the input mechanism of metaverse-based rehabilitation contents currently being developed uses ready-made controllers such as touch screens (a) and joysticks (b), or their usage is far from actual daily life motions and is not intuitive. To solve this, it is necessary to motivate participation in content through improvement of input mechanism.

In addition, having to learn how to use a controller for each product every time you try to perform computer contents for the purpose of rehabilitation is a burden on both the patient and the therapist, and in the case of elderly patients who mainly use cognitive rehabilitation, You may quickly lose interest because you are not familiar with the operation method of an input mechanism such as a joystick.

The qualifications necessary to apply content mediation methods in clinical practice are accessibility and simplicity, and the process of improving the subjects to actively participate in rehabilitation with interesting content is a great help in motivating patients in rehabilitation treatment through content. can give

Republic of Korea Patent No. 10-1519808 (2015.05.06)

The present invention was created based on the above background and needs, and an object of the present invention is to induce interest in patients in augmented reality rehabilitation training by utilizing infrared image-based hand motion recognition technology and voice recognition technology using a Leap Motion controller, but to improve hand motion To provide an augmented reality rehabilitation training system applied with hand motion recognition enhancement technology that greatly improves recognition accuracy and training efficiency.

For the above purpose, the present invention provides interconnection between a lower frame having a fixing part formed therein, an upper frame positioned above the lower frame and having a cradle formed therein, and an operating area for hand motion between the lower frame and the upper frame. Body portion consisting of a vertical frame to; an image detection module installed in the fixing part and capturing a hand motion in the motion area; an output module coupled to the cradle and having a display unit outputting an image and an audio output unit outputting a sound source; A storage unit for storing guide voices for rehabilitation, rehabilitation training contents, content images corresponding to the contents, a user interface, and content sound sources, a motion recognition unit for recognizing hand motions captured by the image sensing module, and recognized hand motions A control module including an image processing unit that applies to a content image and outputs it, an evaluation unit that evaluates the recognized hand motion in response to the contents of the content, and a feedback unit that outputs the evaluation result through video or audio; It is characterized by consisting of.

At this time, it is preferable that the motion recognition unit tracks the movement of the hand through a lip motion controller, reflects the 3D position value and rotation value of the hand model in a 3D virtual environment, and obtains a relative angle value for each finger segment. .

In addition, a voice input module having a voice input unit for receiving a voice input from a user and a voice recognition unit for analyzing the input voice and converting it into text information; Further, the control module may further include a voice interface unit for selecting and progressing content through the text information.

In addition, the rehabilitation training content outputs a score, subtitles, and an interactive target object while playing a sound source, and when the user claps his hands in accordance with the interactive target object, the hand position and hand clap motion are recognized to obtain a score along with sound effects. It can be configured to give.

In addition, in the rehabilitation training content, a plurality of animal characters are given as objects, and the selected object is changed along with a guide voice, and the user activates voice recognition through a set hand motion, and then the correct answer matches the voice recognition result input. If you do, it may be configured to give a score along with a sound effect.

In addition, in the rehabilitation training content, a plurality of positions are presented with a name tag as a separate garbage collection item, and after randomly generating an object to be collected separately on a virtual desk, a hand motion of moving the designated position by grabbing the object according to the voice guide The hand model corresponding to is output together, and if the attributes of the given name tag and the trash object match, it may be regarded as a correct answer and a score may be given along with a sound effect.

In addition, the rehabilitation training content presents a set number of objects and boxes, outputs a commanding voice message to find an object related to the given object and put it in the box, and when an operation of picking up the selected object and properly putting it into the box is performed, a voice message is produced. It can be configured to repeat the process of assigning a score with feedback and moving on to the next problem with voice feedback if an incorrect action is made.

According to the present invention, it is possible to solve the discrepancy felt during usage or use while maintaining the patient's interest by using a software method, and also accurately recognize hand gestures.

1 is a block diagram showing the configuration and connection relationship according to an embodiment of the present invention;
2 is a perspective view showing an external appearance according to an embodiment of the present invention;
3 is a flowchart of detection of hand clapping and grasping motions according to an embodiment of the present invention;
4 is a hierarchical structure for hand gesture recognition according to an embodiment of the present invention;
5 is a progress scenario of content 1 according to an embodiment of the present invention;
6 is a design and goal of content 1 according to an embodiment of the present invention;
7 is a system design layout of content 1 according to an embodiment of the present invention;
8 is a system internal screenshot of content 1 according to an embodiment of the present invention;
9 is a progress scenario of content 2 according to an embodiment of the present invention;
10 is a design and goal of content 2 according to an embodiment of the present invention;
11 is a system design layout of content 2 according to an embodiment of the present invention;
12 is a system internal screenshot of content 2 according to an embodiment of the present invention;
13 is a progress scenario of content 3 according to an embodiment of the present invention;
14 is a design and goal of content 3 according to an embodiment of the present invention;
15 is a system design layout of content 3 according to an embodiment of the present invention;
16 is a system internal screenshot of content 3 according to an embodiment of the present invention;
17 is a progress scenario of content 4 according to an embodiment of the present invention;
18 is a design and goal of content 4 according to an embodiment of the present invention;
19 is a system design layout of content 4 according to an embodiment of the present invention;
20 is a system internal screenshot of content 4 according to an embodiment of the present invention;
21 is a dictionary definition of a hand gesture to be measured according to an experimental example of the present invention;
22 is a comparison of segment angle acquisition methods according to object angles according to an experimental example of the present invention;
23 is an overview of CNN model blocks for the experiment of the present invention;
24 is a model layer definition and structure according to an experimental example of the present invention;
25 is a measurement table showing hand gesture recognition accuracy for each hand gesture using a local transform euler;
26 is a measurement table showing recognition accuracy for each hand gesture using Quaternion to Euler;
27 is a measurement table showing recognition accuracy for each angle using a local transform euler;
28 is a measurement table showing accuracy for each angle using Quaternion to euler;
29 is a model re-recognition reaction rate test results according to an experimental example of the present invention,
30 is a confusion matrix according to the hand gesture recognition accuracy test according to the present invention.

Hereinafter, the configuration of the augmented reality rehabilitation training system to which the hand gesture recognition enhancement technology of the present invention is applied will be described in detail with reference to the accompanying drawings.

Figure 1 is a block diagram showing the configuration and connection relationship according to an embodiment of the present invention, Figure 2 is a perspective view showing the external appearance according to an embodiment of the present invention, the present invention is a body part (100) constituting the frame of the system as a main component ) Including a configuration for augmented reality rehabilitation training, it consists of an image detection module 140, a voice input module 150, an output module 160, and a control module 170.

First, the body portion 100 includes a lower frame 110 having a fixing part formed therein, an upper frame 120 located above the lower frame 110 and having a cradle formed thereon, the lower frame 110 and the upper frame It consists of vertical frames 130 interconnected so that an operating area 131 for hand motion is formed between 120.

The lower frame 110 has a box shape with an upper side open, and a fixing part 111 for installing an image sensing module 140 to be described later is formed at the inner center, and vertical frames 130 are coupled to both sides to move upward. In addition to constructing a work station corresponding to the operation area 131, the upper frame 120 on which the holder 121 is formed is connected to the upper end of the vertical frame 130. Durability must be guaranteed when connecting each frame, and it is supported using a group-shaped clamp and additionally fixed with bolts on the side.

In addition, the lower frame 110 is separately provided with a cable hole for passing a USB cable for connecting the image sensing module 140 to a PC, that is, a control module.

The image sensing module 140 is installed in the fixing part and captures hand motions in the motion area. In an embodiment of the present invention, the image sensing module 140 is configured using a leap motion controller do.

The Leap Motion Controller is an infrared camera module for recognizing hand motions and is specialized in tracking the angle and movement of a hand and the degree of bending of a finger with a single device. In the embodiment of the present invention, it was produced using the Leap Motion 4.2 Orion SDK, and information was acquired and mapped from Unity using the desktop top connection method. It is a small USB device placed on a desktop and has proven remarkably good tracking accuracy. The standard deviation is less than 1 mm, and although the camera cannot see the occluded parts of the hand, the algorithm can detect some typical hand motions, providing useful tracking on its own, and is a very good size and weight for laboratory applications. It has characteristics suitable for the present invention because it takes a method of use.

The voice input module 150 is composed of a voice input unit 151 that receives a voice input from a user based on a microphone and a voice recognition unit 152 that analyzes the input voice and converts it into text information.

In the embodiment of the present invention, Google speech to text API was used for Korean speech recognition. The Google speech to text API provides hints for customizing speech recognition to convert subject terms and less common words to text, improve the accuracy of text conversion for specific words or phrases, and use classes to recognize speech as digits. to address, year, currency, etc., and applies Google's state-of-the-art Automatic Speech Recognition (ASR) deep learning neural network algorithm to provide a better user experience for products through voice commands and convert content in real time or from stored files. It has the advantage of being able to convert to text.

The output module 160 is coupled to the holder 121 and is composed of a display unit 161 outputting an image and an audio output unit 162 outputting a sound source. That is, the display unit 161 corresponding to the monitor and the sound output unit 162 corresponding to the speaker are fixed to the holder 121, and the image sensing module 140 is in the form of a box located below the upper frame 120. It is located in the lower frame 110 of the internal fixing part 111 and is configured to capture hand movements in the operation area (work station) composed of left and right vertical frames.

The control module 170 is connected to the image sensing module 140, the voice input module 150, and the output module 160 to perform rehabilitation training through rehabilitation training contents, and has a PC or equivalent function. It is composed of a terminal, and the detailed configuration includes a storage unit 171, a motion recognition unit 172, a voice interface unit 173, an image processing unit 174, an evaluation unit 175, and a feedback unit 176. provide

The storage unit 171, as a memory or storage medium, stores guide voices for rehabilitation training, rehabilitation training contents, content images corresponding to the contents, user interface, and contents sound sources, and rehabilitation training through images, sound sources, and augmented reality. Various contents and related data can be stored.

The motion recognition unit 172 is a component that recognizes a hand motion captured by the image sensing module. As mentioned above, since the image sensing module 140 is configured through the lip motion controller, in the present invention, the motion recognition unit 172 tracks the movement of the hand through the lip motion controller to create a hand model in a 3D virtual environment. It reflects the 3D position value and rotation value of , and obtains the relative angle value for each finger segment.

3 shows a flow chart for detecting a hand clapping motion and a grasping motion according to an embodiment of the present invention.

In the present invention, the detection of the grasping motion and the detection of the hand clap motion are sub-methods of the interaction algorithm for picking up objects, and generate a binarization variable receiving the detection of the grasping motion and a trigger receiving the detection of the hand clap motion.

The Leap Motion equipment tracks hand movements and returns the 3D position and rotation values of the hand model to the 3D virtual environment. This 3D hand model is a 3D object that includes location and rotation information for a total of 20 individual objects corresponding to 4 joints per finger, enabling flexible interaction with objects in the same scene.

Separately, labeling for recognizing and classifying hand shapes for hand gestures and activation functions, which are feedback for interactions according to hand gestures, are not provided. Only limited hand motion labels such as Pinch, Grab, and Palm direction are provided. Therefore, in order to implement actions such as collision parameter, rotational speed detection, and left/right snapping, it is possible to perform functions such as pinching a virtual object or zooming through left/right snapping and pinching only through adding and implementing functions.

In addition, Leap Motion itself only provides model implementation, but separate labeling and definition functions are additionally required for hand motion recognition. For example, when implementing a 3D model of a hand closing or opening to pick up an object, rendering the picking model can be solved by finding and mapping the location of the joint through Leap Motion, but the object and In order to interact, the collision check and the motion accompanying the collision operation must be implemented and specified.

For this reason, the palm part and each fingertip joint are added with a colliding body having a physical quantity that is a medium of collision, which is hereinafter referred to as a collision determination box. By adding a collision determination box, it is configured to proceed with a collision check with a collision determination box possessed by an interactive object or a constituent object of virtual reality.

By adding a fixed joint component, the point of contact between the collision detection box included in the hand model and other interaction objects can be set as the origin of the interaction object, which acts as a parent object for location information and acts as an object. You can make it look like pick up and move. The sequence of operation according to this principle is to first add a collision determination box to the hand 3D model, and check the tag variable of the interaction object when the collision determination box of the hand 3D model collides with the collision determination box of another interaction object.

A tag variable is a variable such as a kind of label, and performs a function of returning object identification information in the form of a string. At this time, if the tag variable containing the identified identification information is classified as an object that can be grasped, check whether the thumb and forefinger are at a certain angle or more, and if the angle is more than a certain angle, a fixed joint component, which is a component that fixes the skeleton inside the modeling of the 3D object, is created. It operates in a way to configure a contact point, and when there is a fixed joint component, if the angle between the thumb and index finger falls below a certain angle, the fixed joint component can be released.

As a result, detection of grasping motion and detection of hand clapping motion can be expressed as a binarization variable, and detection of hand clapping motion is processed through a trigger variable that returns to a standby state after measurement. The difference between a trigger variable and a binary variable is that the trigger variable passes the occurrence value to the event handler and destroys it, and when the function operation is completed, it enters the waiting state and operates again, but the binary variable maintains the status return value. For motion detection, it is possible to process an event once only at the moment when a hand is clapped by using a trigger variable.

Existing methods for detecting a grasping motion detect using a pinch motion, so it is preferable to grasp using the thumb and forefinger. In the case of actual hand-holding motion, the action point and the fulcrum point must exist, and this was adjusted so that when holding a small object with one finger, when the inner angle is less than 90 degrees when in contact with two or more hand objects, the holding motion occurs. The hand object is classified as a tag, and the count is increased by 1 when an object with a hand tag is contacted through an execution function in an object that can be grabbed, and the count is decreased by 1 when the hand object whose count was increased immediately before is out of the way.

And when the count is 2 or more, based on the angle of the hand object that increased the first count, if the interior angle with the hand object that increased the count later is 90 degrees or less, it is determined as a grabbing motion, and if collision calculation continues at this time, force is applied Therefore, since the hand object may show a shaking phenomenon due to animation in the wrong direction, disable the physics calculation rigid body component of the grippable object and create a fixed joint component so that it can be fixed to the hand object. Even if the physics calculation rigid body is lost, collision calculation can proceed, so there is no problem in decrementing the count when the collision is out of the way.

The detection of the hand clap motion can be implemented in a much simpler way. By adding a collision calculation unit capable of collision calculation to the palm model, a sound effect is reproduced only when a collision starts. In some cases, stability can be increased by resetting a flag when the collision is out of the way.

In addition, the present invention implements a method for collecting hand angles so that the grasping motion detection and hand clap motion detection for hand motion labeling can receive and process the angles of each finger, so that all possible hand motions can be recognized. . This is possible by obtaining and processing relative angles based on the palm model, and pre-specified hand motions were acquired through label inspection.

The angle of the finger segment can be obtained as an Euler angle for the local object, and is used to intuitively receive rotation and interpolation about the (x, y, z) axis of the segment after recognizing and rendering the hand model. For example, if the Euler angles are (γ, β, α) in the order of the (x, y, z) axes, it is first rotated by γ around the x-axis, and then the rotated axis (x', y', z') rotates by β about the y' axis of Then, it rotates by α based on the z'' axis of the last rotated axis (x'', y'', z''). At this time, the Euler angle expresses the sum of all rotations with one rotation, and when expressed as a matrix, it can be expressed as in [Equation 1].

[Equation 1]

However, Euler angles, as we commonly know, may not be able to implement the correct rotation angle because the axis is lost due to the occurrence of the gimbal lock phenomenon caused by the hierarchy. Design the hierarchy and receive the quaternion rotation angle By coming, the full segment angle can be obtained without loss of axis.

Distinguishing hand gestures can be implemented by properly specifying and measuring the origin of rotation for each finger.

To this end, the fingertip joint must be set to inherit the 3D information of the upper joint, and a series of standardized object structures that are finally connected to the palm model must be configured and measured. If the modeling is not configured according to the above object structure, the order of 3D information inheritance of the skeleton of the 3D model must be set using a program such as 3D Max or Maya. can be saved

4 is a hierarchical structure for hand gesture recognition according to an embodiment of the present invention. By obtaining relative angle values for each finger segment through this operation, the angle value of a complete object can be obtained regardless of the rotation of the wrist or the angle of the upper arm. be able to Basically, because of the gimbal lock effect of Euler angles, unless the hierarchical structure is set as above, the finger segment cannot rotate more than 90 degrees in the z-axis with respect to the upper segment.

It is possible to secure the stability of the axis and the angle through the hierarchical structure, and through this, it is possible to safely acquire the angle for each segment, so that it is possible to recognize various hand motions. Accurately, since it is possible to obtain angles for all finger segments, all hand motions defined through labeling can be distinguished. The method obtained through quaternion rotation has the disadvantage that as one axis rotates, the rotation value of the other axis is also adjusted through rotation calculation, making it difficult to accurately track and inferring an intuitive angle. It is judged to be inefficient in obtaining the angle of the segment than the hierarchical design for transformation.

The voice interface unit 173 is configured to select and proceed with content through the text information, and in the case of input means such as a keyboard keyboard, mouse, or touch screen, it can reduce the accessibility of rehabilitation patients, thereby enabling voice input. Based on this, it is configured so that motion input and content progress are made.

The image processing unit 174 is configured to create an augmented reality environment, and outputs an augmented reality image through image processing of applying, that is, merging, the hand motion recognized through the motion recognition unit to the content image.

The evaluation unit 175 is a component that evaluates the recognized hand gestures in correspondence with the contents of the content, and in particular, evaluates whether a certain hand gesture was made correctly or at the correct timing.

The feedback unit 176 outputs the evaluation result of the evaluation unit through video or audio so that the user can recognize whether the correct operation has been performed or whether the operation or timing needs to be corrected.

In the present invention, four rehabilitation contents are presented and described as examples.

In the first embodiment, the rehabilitation training content outputs a score, subtitles, and an interactive target object while reproducing a sound source. can be configured to score together.

5 is a progress scenario of content 1 according to an embodiment of the present invention, FIG. 6 is a design and goal of content 1 according to an embodiment of the present invention, FIG. 7 is a system design layout of content 1 according to an embodiment of the present invention, 8 is a system internal screen shot of content 1 according to an embodiment of the present invention.

The first augmented reality-based hand motion recognition rehabilitation content using Leap Motion developed in the present invention is training content aimed at attention training and management function training. is to train

For the plausibility of the content, motivation is given through a scenario in which you have to clap your hands while outputting songs and videos.

It is characterized by implementing rehabilitation contents using hand movements that are frequently used in daily life. It is a training that intentionally elicits a pre-learned motor response.

In the proceeding method, when a song is selected on the initial screen that provides a total of three song options and the start button is pressed, the UI configured as shown in FIGS. 7 to 8 is displayed. The UI includes a text view for displaying scores and subtitles, and random videos are played in the background, and interactive target objects are displayed on the left or right in line with the music.

When the user moves his hand to the interactive target object displayed on the screen and claps his hand, the position of his hand and hand clap motion are recognized and points are acquired along with sound effects.

In the second embodiment, the rehabilitation training content is given as an object, changes the selected object with a guide voice, activates voice recognition through a hand motion set by the user, and then inputs the voice recognition result and If the correct answer matches, it may be configured to give a score along with a sound effect.

9 is a progress scenario of content 2 according to an embodiment of the present invention, FIG. 10 is a design and goal of content 2 according to an embodiment of the present invention, FIG. 11 is a system design layout of content 2 according to an embodiment of the present invention, 12 is a system internal screen shot of content 2 according to an embodiment of the present invention.

Content 2 is short-term concept memory training content, virtual reality-based speech recognition rehabilitation content using Google speech to text API, and training content aimed at attention training and memory training.

One object is set to move randomly after a certain amount of time has elapsed in the state where the virtual object is placed. We have a scenario in which the name of the moved object can be selected from among the options presented.

Content 2 is characterized by implementing controllable rehabilitation content using voice recognition, and remembers information that was delivered in the past visually and perceptually on the display with the goal of verbal recall training and attention training among cognitive rehabilitation activity items investigated in advance. It is a training to acquire the changed information by concentrating on the screen information of the display and selectively derive it.

In the UI, a total of four 3D animal characters are displayed, including a score display part and moving objects that may require a little attention and memory activity. When the content starts, one of the four animals moves along with a guide voice asking you to remember that one animal is running away, and after a while, activate voice recognition by clench your fist to see which animal has moved, and then say the correct answer. Open your fist again and output the input voice recognition result. At this time, through the UI, three options including the correct animal appear. If the voice recognition result matches the correct answer, a score is obtained along with a sound effect.

In the third embodiment, in the rehabilitation training content, a plurality of positions are presented with name tags as items for separate garbage collection, and objects to be collected are randomly generated on a virtual desk, and then the object is grabbed according to the voice guide and the designated position is set. The hand model corresponding to the hand motion of moving the object is output together, and when the properties of the given name tag and the garbage object match, it can be configured to be considered as the correct answer and given a score along with a sound effect.

13 is a progress scenario of content 3 according to an embodiment of the present invention, FIG. 14 is a design and goal of content 3 according to an embodiment of the present invention, FIG. 15 is a system design layout of content 3 according to an embodiment of the present invention, 16 is a system internal screen shot of content 3 according to an embodiment of the present invention.

Content 3 is the third virtual reality-based hand motion recognition rehabilitation content using Leap Motion, and is training content aimed at composition and conceptualization training, which trains the process of classifying listed objects according to the previously presented classification items.

We have a scenario in which garbage is sorted and collected according to appropriate classification. Content 3 is a program that can model a hand using a Leap Motion Controller and render content, and can perform an object picking operation through the detection of the grabbing motion introduced in the above algorithm. Three goal positions are presented along with name tags of plastic, can, and glass, and objects for separate collection are randomly created on the imaginary desk in front.

As for the progress method, when the content starts, a guide voice is heard to distribute the garbage to the appropriate trash can, an object is created on the desk, and when the hand model is moved to make a fist, the object is picked up and moved inside the VR content.

You can use it to bring the item to the appropriate trash can and put the item into the trash can by opening your fist and placing the item. If the name tag attached to the trash can and the properties of the trash object match, it is considered a correct answer and points are awarded along with sound effects.

In the fourth embodiment, the rehabilitation training content presents a set number of objects and boxes, outputs a commanding voice message to find an object related to the given object and put it in the box, and performs an operation of picking up the selected object and putting it into the box appropriately. If done, it may be configured to give a score along with voice feedback and to repeat the process of moving on to the next problem along with voice feedback if an incorrect action is made.

17 is a progress scenario of content 4 according to an embodiment of the present invention, FIG. 18 is a design and goal of content 4 according to an embodiment of the present invention, FIG. 19 is a system design layout of content 4 according to an embodiment of the present invention, 20 is a system internal screen shot of content 4 according to an embodiment of the present invention.

Contents The 4-person composition and conceptualization training content is the fourth virtual reality-based hand motion recognition rehabilitation content using Leap Motion, and is training content aimed at composition and conceptualization training. The transfer operation must be performed.

Content 4 is the fourth virtual reality-based hand motion recognition rehabilitation content using Leap Motion, and it is training content aimed at constructive training and conceptualization training among previously investigated cognitive rehabilitation activity items. After receiving information visually and perceptually on the display, a process of finding a correlation with the categories of objects presented earlier is required.

When the training starts, a picture is presented in front of you, and a voice message is output to tell you to find a related object among the objects in front of you and put it in the box. After listening to this, if the object is picked up and put properly in the box, the score goes up along with voice feedback, and if it is put incorrectly, the process of moving on to the next problem with voice feedback is configured to be repeated a total of 10 times.

<Experimental example>

Hand motion recognition accuracy and Euler transformation angle when quaternion operation is used without conversion to examine the actual effectiveness of the input mechanism such as algorithm through segment hierarchy and grasping motion detection improvement mentioned in the use of the above rehabilitation training system

and the hand motion accuracy of the algorithm applying the hierarchical structure was measured, and the recognition accuracy was measured and compared according to each case.

In addition, in order to test the effectiveness of rehabilitation program contents, the post-processing speed of detection of hand clapping motions and the reaction speed of voice recognition according to the detection of grasping motions were measured.

In addition, in the process of recognizing and labeling hand gestures, an experiment was conducted to measure how high the accuracy of hand gesture recognition was by using a spectrogram analysis technique using CNN.

In Experiment 1, the hand gesture recognition accuracy was evaluated by repeating each hand gesture 30 times to determine how accurately the user's intention for the previously labeled hand gesture and the actual recognized hand gesture were recognized.

In Experiment 2, the time taken for the disabled hand model to return after detecting the hand clap motion was measured 50 times and the average value was confirmed.

Experiment 3 was divided into 3-1 and 3-2, and as the specific motion was selected in the existing invention using SVM, five motions with high accuracy were selected and conducted based on the same criteria.

In Experiment 3-1, data was collected 10 times for 5 operations, and data having 1,000 rows was input by receiving one row every 0.1 second for 10 seconds per time.

In Experiment 3-2, data was collected 10 times for 5 motions, and one row was input every 0.1 second for 1 second per time, and data having 10 rows was input. Then, after training the CNN model, the accuracy was confirmed through pretest.

- Experiment 1: Hand gesture recognition accuracy test

A total of eight hand gestures were predefined, and the angle of the finger segment was input to distinguish the hand gestures so that the target hand gestures could be displayed on the upper right UI. In addition, the current hand gesture is displayed on the upper left UI of the screen, and if the indicated and appropriate hand gesture is taken within 5 seconds, the score count of the UI located in the upper right corner rises, so that it is possible to check how many appropriate hand gestures have been taken. If an appropriate hand gesture is taken within 5 seconds, the count rises, or if the suggested hand gesture is not taken for 5 seconds, a notification sounds and the number related to the number of experiments in the upper right corner rises.

At this time, if the proper hand motion is not presented within the time limit of 5 seconds, only the number of times increases and the score does not increase. 21 is a pre-definition of a hand motion to be measured according to an experimental example of the present invention, and experiments were conducted 30 times for each motion, and the used motion was intended to be a predefined motion as shown in FIG. progressed

At this time, based on the image acquired through the camera, the accuracy tends to be recognized differently depending on the relative angle of the wrist, so the change in recognition rate according to the angle of the palm to the camera was analyzed for each motion and the difference was analyzed in a table.

In order to more clearly confirm the difference between other angle acquisition methods and the local Euler angle, which is an angle acquisition method in the present invention, the rotation rate of the second index finger is detected in four ways for an open hand.

The four methods applied at this time can be defined as follows. The global Euler angle follows the matrix operation, but when applying the hierarchical structure, it is the Euler angle obtained from the origin by setting the criterion as the origin. It is raw data on Unity that has been converted to solve the problem it causes. The local Euler angle is the result value of the technique dealt with in the present invention obtained by applying the hierarchical structure and performing matrix operation, and the quaternion Euler angle conversion function result value (quaternion to euler) also performs matrix operation, Unity converts the quaternion's internal representation of rotation to Euler angles, so that the value read back to represent a given rotation using Euler angles can be quite different from the value you assigned.

In other words, it operates by performing matrix operation internally, but when it is returned to use it, it is output as a value without performing matrix operation.

22 is comparative data on the segment angle acquisition method according to the object angle according to the experimental example of the present invention, and detects and compares the rotation rate of the second knuckle of the index finger for the open hand in four ways while performing vertical axis rotation for each of the four methods. .

The left side is all measured at an angle of 0 degrees, and the right side is measured at an angle of 360 degrees. At this time, the global Euler angle, local Euler angle, and rotation variable values have no difference between angle values between 0 and 360 degrees, but the quaternion's Euler angle conversion function result values show inconsistent results between 0 degrees and 360 degrees.

This is related to the gimbal lock phenomenon described above, and when the rotation axis is lost as it rotates, correction is not made, and considering that the value is not constant, the quaternion's Euler angle conversion function result value is an angle acquisition method for hand gesture recognition. can be judged unfavorable.

In the case of the rotation variable value, the rotation angle conversion to compensate for the problem of gimbal lock is well done, and the difference between 360 degree rotation and 0 degree rotation appears to be very small.

Even in the case of the hand model recognition algorithm to which the existing SVM model is applied, it has been studied that hardware improvement is essential for the above problems. In the field, the experiment was conducted by limiting only specific angles and specific hand movements.

In the present invention, as a method for confirming these limitations and equally applying the experimental conditions, it is possible to evaluate the accuracy for each hand motion and the accuracy for each angle to determine if there is a difference, and to convert which of the four conversion methods to a spectrogram This is an experiment to select whether to use as preprocessing raw data.

- Experiment 2: Model Recognition Response Speed Test

The second experiment is an experiment to verify the effectiveness by checking the time taken to recognize the model after hand clapping again when the hand model is recognized as a lump after the hand clapping motion is recognized and both hands are recognized as one hand and the one hand model is lost. .

The detection of the hand clap motion was operated as a trigger, and then the activation variable, the activation state inside the scene, was checked, and the rendering state of the hand model was returned 50 times, and the average time required for the maximum and minimum required times was checked. The detection of the hand clap motion was induced for a total of 50 times, and when it was lost after that, the case of recognizing the hand model again was measured in milliseconds (ms). The time it took to complete the game can be displayed at the top of the UI on the left.

- Experiment 3: Hand motion classification using CNN-based spectrogram

In order to apply the hand motion classification technique through the CNN technique and compare it with the SVM to confirm the improvement in accuracy, a model as shown in the figure below is designed and trained.

23 is an overview of CNN model blocks for the experiment of the present invention, and FIG. 24 is a model layer definition and structure according to an experimental example of the present invention. The model was designed with reference to mnist, the most widely used CNN model, and three convolution layers , 3 max pooling layers, and 2 dense layers are used, and the detailed structure of each layer is configured as shown in FIG. 24.

The data used for learning were Experiment 3-1, in which 10 data were measured for each motion, and 1000 data were measured for the five hand motions A, C, D, E, and F, which were measured with relatively high accuracy at the local Euler angle. Divided into Experiment 3-2, a simulation program was configured and operated to collect these data.

Basically, the minimum time unit inside VR content is influenced by the performance of the computer, and 0.04 seconds is set as the basic minimum time unit. It is possible to perform more than that if necessary, but it can reduce the stability of the system. Considering the basic minimum time unit that operates every 0.04 seconds, when a predetermined action is taken within the Unity VR content every 0.1 second and the data collection button is pressed data was collected. Therefore, in the simulation of collecting 1,000 data, data was collected every 0.1 second, and one hand gesture was instructed through the display, and after 1 second, data was collected every 0.1 second for a total of 100 seconds and recorded as a csv file.

After 100 seconds, the file was saved, data collection was completed on the display, and data collection was performed in such a way that the user waited until the next button was pressed.

The method of collecting 10 data was conducted to check how much difference in accuracy occurs when there is little data. After 1 second of collection, the end of data collection was indicated on the display.

By repeating this method, 100 data were collected for each hand motion in Experiments 3-1 and 3-2 for a total of 5 hand motions, and a total of 1,000 data of 500 each were collected. The experiment was divided and divided into 90% (450 pieces) of training set and 10% (50 pieces) of test set with 500 individual data in both Experiments 3-1 and 3-2 for learning and testing. proceeded.

<Experiment result>

- Experiment 1: Hand gesture recognition accuracy test

The recognition rate was the highest when the palm was facing the camera, and when the angle was 0 degrees, it was 84.58% at 90 degrees and 86.25% at 135 degrees, and the highest accuracy was 98.75% at 0 degrees and 315 degrees. When the palm was in front of the camera at 98.75%, the accuracy was the highest for all motions.

25 is a measurement table showing hand gesture recognition accuracy for each hand gesture using a local transform euler.

Based on the local transform euler, the recognition results were measured 30 times while changing the angle for 8 hand gestures. This showed a small standard deviation of 30±0, 29.37±0.91, and 29.37±0.91 for hand movements D, E, and H, and showed an overall accuracy of over 80%. Except for hand movements D, E, and H, the standard deviation was relatively high and the accuracy was measured relatively high.

26 is a measurement table showing recognition accuracy for each hand gesture using quaternion to euler.

Based on the quaternion's Euler angle conversion function result value, the recognition results were measured 330 times while changing the angle for 8 hand gestures. Although the overall accuracy was over 70%, the average accuracy for all angles was lower than the local Euler angle.

27 is a measurement table showing recognition accuracy for each angle using a local transform euler.

Table 10 shows the 30 recognition results shown as hand motion changes for 8 angle changes based on the local transform euler. This showed a small standard deviation of 29.62±1.06 and 29.62±0.74 for the angles of 0 degree and 315 degrees, and showed an overall accuracy of over 85%. Except for the angles of 0 degree and 315 degrees, the standard deviation was relatively high and the accuracy was measured relatively high.

28 is a measurement table showing accuracy for each angle using Quaternion to euler, and is a recognition result of 30 times represented by hand motion change for 8 angle changes based on Quaternion to euler.

Although the overall accuracy was over 50%, the standard deviation and average accuracy for all angles were measured at a lower level than the local transform euler.

In addition, when comparing 45 degrees and 315 degrees, when the palm is at a 45 degree angle with the front, 45 degrees tended to be slightly less accurate than 315 degrees, 98.75% at 95%. The recognition rate was good even when the little finger was covered, but the recognition rate of the degree of bending tended to be inferior when the thumb was covered.

As for the accuracy of each motion, the accuracy of motions that can be recognized only with the degree of bending of the fingers, such as motions D and H, was measured to be high, and it is difficult to guess what shape the finger is in cases where the motion of the finger cannot be specified when viewed from the back of the hand, such as B. The accuracy decreased according to the experience, and this showed that the recognition rate fell more when viewed from the side than when the back of the hand was facing the camera at 180 degrees. Likewise, C and G showed accuracy of 28 to 30 times when measured at 45 degrees and 315 degrees, which are 45 degrees apart from 0 degrees, where the palm faces the camera, and 26 and 27 times when measured at 180 degrees. Max C is 21 when used on the side

Episodes and G showed a drop in the recognition rate until episode 19.

When the accuracy change according to the angle was measured, the accuracy of 90% or more was satisfied at the angle of 0 degree and 315 degree. Since it is a method using a single depth camera, it is possible to prove that it is vulnerable to angles. We were able to confirm the physical limitations of the hardware presented in the hand gesture recognition algorithm using the existing SVM model. For this reason, the present invention, which decided to test the software algorithm, performed the same experimental conditions as the previous studies that recognized hand gestures using the existing SVM. For this purpose, the angle was limited to 0 degrees in future experiments 3-1 and 3-2.

- Experiment 2: Model Recognition Response Speed Test

The maximum required time was 0.9 seconds and the minimum required time was 0.512 seconds, showing the result value recognized after the hand model disappeared within an average of 0.727 seconds. At this time, when the SDK function related to activation of hand model rendering was disabled to prevent loss, the recognition rate tended to decrease. In this case, the maximum required time was 1.395 seconds and the minimum required time was 0.704 seconds, which significantly increased the time required for re-recognition as a whole. When hand clapping motion was detected without using the hand model rendering activation function, the hand model was recovered within an average of 0.727 seconds.

29 is a model recognition reaction speed test result according to an experimental example of the present invention, showing the reaction speed (number of times on the x-axis, time (second) on the y-axis) for hand model recognition after detecting a hand clap motion.

- Experiment 3: Hand gesture recognition accuracy test

In Experiment 3-1, 10 rows of data collected for 1 second for each hand motion were collected 100 times for 5 hand motions A, C, D, E, and F for a total of 500 data, and then 90% (450 data) ) was used for learning and when 10% (50 pieces) were used for testing, the test result was measured as 100%.

During Experiment 3-2, 1,000 rows of data collected for 1 second for each hand gesture were collected 100 times for 5 hand gestures A, C, D, E, and F for a total of 500 data, and then 90% (450 dog) was used for learning and 10% (50 pieces) was used for testing, and the test result was measured as 100%.

30 is a confusion matrix (X-axis: actual, Y-axis: predicted) according to the hand gesture recognition accuracy test according to the present invention, and the results of Experiment 3-2 are shown as a confusion matrix for the items for which the test result and the correct answer were correct. .

According to these experimental results, the CNN technique presented in the present invention is more accurate than the hand motion recognition method using SVM and is advantageous to the surrounding background change, so it can be expected to be usefully applied to the augmented reality-based cognitive rehabilitation training system.

The rights of the present invention are defined by what is described in the claims, not limited to the embodiments described above, and that those skilled in the art can make various modifications and adaptations within the scope of rights described in the claims. It is self-evident.

100: body part 110: lower frame
111: fixing part 120: upper frame
121: cradle 130: vertical frame
131: operation area 140: image detection module
150: voice input module 151: voice input unit
152: voice recognition unit 160: output module
161: display unit 162: sound output unit
170: control module 171: storage unit
172: motion recognition unit 173: voice interface unit
174: image processing unit 175: evaluation unit
176: feedback unit

Claims (7)

A lower frame 110 having a fixing part 111 formed therein, an upper frame 120 located above the lower frame 110 and having a cradle 121 formed thereon, the lower frame 110 and the upper frame 120 ) The body portion 100 composed of vertical frames 130 interconnected so that an operation area 131 for hand motion is formed between them;
An image sensing module 140 installed on the fixing part 111 and capturing hand motions in the motion area;
an output module 160 coupled to the cradle 121 and having a display unit 161 outputting an image and an audio output unit 162 outputting a sound source;
A storage unit 171 for storing guide voices for rehabilitation training, rehabilitation training contents, content images corresponding to the contents, a user interface, and contents sound sources, and a motion recognition unit recognizing hand gestures captured by the image detection module 140. 172, an image processing unit 174 that applies the recognized hand motion to a content image and outputs it, an evaluation unit 175 that evaluates the recognized hand motion corresponding to the contents of the content, and converts the evaluation result into video or audio. Control module 170 having a feedback unit 176 to be output through; Augmented reality rehabilitation training system, characterized in that consisting of.
According to claim 1,
The motion recognition unit 172,
Augmented reality rehabilitation training characterized by tracking the movement of the hand through a lip motion controller, reflecting the 3-dimensional position value and rotation value of the hand model in a 3-dimensional virtual environment, and obtaining the relative angle value for each finger segment system.
According to claim 1,
a voice input module 150 having a voice input unit 151 that receives voice input from a user and a voice recognition unit 152 that analyzes and converts the input voice into text information; Including more,
The control module 170,
Augmented reality rehabilitation training system further comprising a voice interface unit 173 for selecting and proceeding with content through the text information.
According to claim 3,
The rehabilitation training content,
Augmentation characterized in that, along with playing the sound source, scores and subtitles and interactive target objects are output, and when the user claps his hands in line with the interactive target objects, the hand position and hand clap motion are recognized and a score is given along with sound effects. A real rehabilitation training system.
According to claim 3,
The rehabilitation training content,
A plurality of animal characters are given as objects, and the selected object is changed along with a guide voice. After activating voice recognition through a hand gesture set by the user, if the correct answer matches the input voice recognition result, a score is given along with a sound effect. Augmented reality rehabilitation training system, characterized in that.
According to claim 3,
The rehabilitation training content,
Multiple positions are presented with name tags as items for separate garbage collection, and after randomly generating objects to be collected separately on a virtual desk, a hand model corresponding to the hand motion of moving the designated position by catching the object according to the guidance voice is output together. And, if the attributes of the given name tag and the trash object match, it is considered as the correct answer and a score is given with sound effects. Augmented reality rehabilitation training system.
According to claim 3,
The rehabilitation training content,
A number of set objects and boxes are presented, and a voice message is output to instruct to find objects related to the given object and put them in the box. Augmented reality rehabilitation training system characterized by repeating the process of moving to the next problem with voice feedback when the action is made.

Images