CN116027683A - Data glove-based simulation method, device and system and storage medium - Google Patents

Abstract

The invention relates to the technical field of simulation and discloses a simulation method, a device, a system and a storage medium based on data gloves, wherein a feedback module is used for monitoring a user or a plurality of users in real time in the simulation process.

Description

Simulation method, device, system and storage medium based on data glove

technical field

The invention relates to the technical field of simulation, in particular to a data glove-based simulation method, device, system and storage medium.

Background technique

Simulation is shape simulation, operation simulation, and visual experience simulation. Real car models or other models of aircraft and spaceships with equal proportions are used as the control platform for participants. VR technology (virtual reality technology) is used to make the Participants have an immersive experience of a technology, mainly used for simulated driving, training, demonstration, teaching, training; military simulation, command, virtual battlefield; architectural vision and urban planning, data glove is a multi-purpose The virtual reality hardware of different modes, through software programming, can perform actions such as grabbing, moving, and rotating objects in the virtual scene, and can also use its multi-modality as a tool to control scene roaming. The emergence of data gloves provides a new means of interaction for the virtual reality system. The product has been able to detect the bending of the finger, and use the magnetic positioning sensor to accurately locate the position of the hand in three-dimensional space. This combination of finger bending test The data glove used in the space positioning test is called "real glove", which can provide users with a very real and natural three-dimensional interaction method.

With the advent of the era of artificial intelligence, gesture control based on data gloves is more and more respected by the public. Correspondingly, various products based on data gloves have emerged as the times require. However, there is no gesture control using data gloves on the market. Surgical operation simulation system, through the data glove-based surgical operation simulation system, allows doctors and students to have a sense of reality in the virtual world and the real world to actually operate human samples. However, the data glove and the computer are generally connected through the serial port This greatly limits the participation of multiple users, and it is impossible to truly realize the intelligence of the simulation through the data glove, and it is not convenient to repair the system problems encountered by the data glove during the simulation process in real time, so as to ensure the intelligence degree of expression.

Contents of the invention

The present invention provides the following technical solution: a data glove-based simulation method, device, system, and storage medium. The step S3 also includes acquiring tag information of the data glove, and searching for a corresponding target virtual hand according to the tag information.

By adopting the above technical solution; the present invention obtains the connection data of the data glove, and switches the current connection mode to a parallel mode when the number of connections is greater than the preset threshold, so that multiple users can simultaneously view the three-dimensional human body model Improve the communication efficiency of the data glove to the simulation operation, monitor the system problems of the user or multiple users in the simulation operation process in real time through the feedback module, monitor the quality of the simulation in real time, and analyze the monitored system problems process, and formulate a response plan, so as to solve the system problems that arise, and then enable users or multiple users to perform simulation operations stably, improve the actual user experience of users, and facilitate the finding of loopholes in the simulation system, and improve The simulation simulates the system.

Preferably, the acquiring the tag information of the data glove, and searching for the corresponding target virtual hand according to the tag information specifically includes: acquiring the tag information of the data glove, and verifying the tag information, and the verification result is When the verification is successful, the corresponding target virtual hand is searched according to the tag information.

By adopting the above-mentioned technical solution and by obtaining tag information, it is convenient to quickly find the target virtual hand, which is convenient for users to use.

Preferably, the connection mode of the data glove is switched to a parallel mode, and before creating corresponding virtual hands according to the number of connections, the method further includes obtaining the API interface of the data glove, and connecting all the data gloves in real time through the API interface. The data glove, so that the data glove obtains the number of connections, and sends the number of connections to the processing device.

By adopting the above-mentioned technical solution, the real-time connection of the data glove through the API interface is convenient for switching between the parallel mode and the serial mode according to actual needs.

Preferably, the step S3 further includes acquiring the force feedback information sent by the data glove, and operating the virtual hand according to the force feedback information, so as to realize the simulation of the three-dimensional human body model by multiple users,

By adopting the above-mentioned technical solution and by feeding back force information to the simulation simulation system, it is convenient to grasp the use situation of the user.

Preferably, the step S3 further includes obtaining the positioning information sent by the data glove, searching for the target position corresponding to the three-dimensional human body model according to the positioning information, and operating the target position through the virtual hand, thereby Realize multi-user simulation of three-dimensional human body model.

By adopting the above technical solution, it is convenient to quickly find the target position corresponding to the three-dimensional human body model through the positioning information.

Preferably, the feedback module is used to monitor the user or multiple users in real time during the simulation process, and receive system problems such as system freezes and flashbacks encountered by the user or multiple users during the simulation process, so The processing module is used to analyze and process the feedback information data and formulate corresponding solutions after receiving the system problem information fed back by the feedback module, so as to deal with the problems that occur during the simulation process of users or multiple users. The application module is used for fusing and applying the plan information data formulated by the processing unit to the simulation program in the memory after receiving the response plan formulated by the processing module.

By adopting the above-mentioned technical solution; through the feedback module, real-time monitoring of the system problems of users or multiple users during the simulation operation process, real-time monitoring of the quality of the simulation simulation, analysis and processing of the monitored system problems, and formulation of countermeasures, thus Solve the system problems that appear, and then enable users or multiple users to perform simulation operations stably, improve the actual use experience of users, and facilitate the finding of loopholes in the simulation system, and improve the simulation system.

Preferably, the processing device includes: a memory, a processor, and a data glove-based simulation program stored on the memory and operable on the processor, and the data glove-based simulation program is configured to realize The steps of the simulation simulation method based on data glove according to any one of claims 1 to 7.

By adopting the above-mentioned technical scheme; through the above-mentioned processing device, it is convenient to realize the simulated operation of the data glove.

Preferably, the data glove-based surgical operation simulation system includes: a data glove, a processing device, and a display device, the data glove is connected to the processing device, and the processing device is connected to the display device.

By adopting the above technical solution, the processing device facilitates switching between the parallel mode and the serial mode.

Preferably, a data glove-based simulation program is stored on the storage medium, and when the data glove-based simulation program is executed by a processor, the data glove-based simulation program according to any one of claims 1 to 7 is realized. Simulate the steps of the simulation method.

Preferably, the simulation method based on the data glove comprises the following main steps;

S1: The processing device sends the simulation modeling instruction to the display device in response to the simulation modeling instruction input by the user, so that the display device creates a three-dimensional human body model according to the simulation modeling instruction;

S2: Obtain the access information of the data glove, extract the number of connections in the access information, and when the number of connections is greater than the preset threshold, switch the connection mode with the data glove to a parallel mode, and in the connection When the number is less than the preset threshold, the connection mode with the data glove is switched to a serial mode, and corresponding virtual hands are created according to the number of connections;

S3: Obtain the simulated operation data of the 3D human body model sent by the data glove in parallel or serially, and operate the virtual hand according to the simulated operation data, so as to realize multi-user control of the 3D human body model simulation simulation;

S4: In the process of simulation, the feedback module receives the user's problems such as freezes and faults, and transmits the problems to the processing module;

S5: After the processing module receives the information fed back by the feedback unit, the processing module analyzes and handles the problems that arise, formulates a solution, and transmits the formulated solution to the application module;

S6: After the application module receives the information of the countermeasure prepared by the processing module, it integrates the information and data of the countermeasure into the simulation program in the memory, so as to repair the feedback problem.

Beneficial effect

Compared with the prior art, the present invention provides a data glove-based simulation method, device, system and storage medium, which have the following beneficial effects:

In the present invention, by acquiring the connection data of the data glove, when the number of connections is greater than the preset threshold, the current connection mode is switched to a parallel mode, so that multiple users can operate on the three-dimensional human body model at the same time, and the data glove can be improved. For the communication efficiency of simulation operation, monitor the system problems of users or multiple users in the process of simulation operation in real time through the feedback module, monitor the quality of simulation in real time, analyze and deal with the monitored system problems, and formulate countermeasures , so as to solve the system problems that arise, and then enable users or multiple users to stably perform simulation operations, improve the actual use experience of users, and facilitate the finding of loopholes in the simulation system, and improve the simulation system.

Description of drawings

Fig. 1 is a schematic diagram of the structure of the processing device of the hardware operating environment of the simulation method, device, system and storage medium based on the data glove of the present invention;

Fig. 2 is a schematic diagram of a parallel process of the simulation method, device, system and storage medium based on the data glove of the present invention;

Fig. 3 is a schematic diagram of the parallel and serial part of the simulation method, device, system and storage medium based on the data glove of the present invention;

Fig. 4 is a schematic diagram of the tag information flow of the data glove-based simulation method, device, system and storage medium of the present invention;

5 is a schematic flow diagram of the API and positioning information of the simulation method, device, system and storage medium based on the data glove of the present invention;

FIG. 6 is a schematic diagram of the device structure of the data glove-based simulation method, device, system and storage medium of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Embodiment one;

Please refer to Figures 1, 2 and 3, the processing device sends the simulation modeling command to the display device in response to the simulation modeling command input by the user, so that the display device creates a three-dimensional human body model according to the simulation modeling command, and obtains the connection of the data glove Input information, extract the number of connections in the access information, switch the connection mode with the data glove to parallel mode when the number of connections is greater than the preset threshold, and switch the connection mode with the data glove when the number of connections is less than the preset threshold It is a serial method, and the corresponding virtual hands are created according to the number of connections, and the simulated operation data of the three-dimensional human body model sent by the data glove is obtained in parallel or serially, and the virtual hands are operated according to the simulated operation data, so as to realize multi-user For the simulation of the three-dimensional human body model, the present invention switches the current connection mode to a parallel mode when the number of connections is greater than the preset threshold by obtaining the connection data of the data glove, so that multiple users can simultaneously control the three-dimensional human body model. Operation, improve the communication efficiency of the data glove to the simulation operation, monitor the system problems of users or multi-users during the simulation operation in real time through the feedback module, monitor the quality of the simulation in real time, and analyze and process the monitored system problems , and formulate a response plan, so as to solve the system problems that arise, and then enable users or multiple users to perform simulation operations stably, improve the user's actual experience, and facilitate the finding of loopholes in the simulation system, and improve the simulation system .

Further, the processing device may include: a processor, such as a CPU, a communication bus, a user interface, a network interface, and a memory, wherein the communication bus is used to implement connection and communication between these components. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), and an optional user interface may also include a standard wired interface and a wireless interface. Optionally, the network interface may include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory can be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory can optionally also be a storage device independent of the aforementioned processor;

Those skilled in the art can understand that the structure of the processing device does not constitute a limitation on the processing device, and may include more or less components than shown in the figure, or combine some components, or arrange different components;

The network interface is mainly used to connect to the server and perform data communication with the server; the user interface is mainly used to connect to the user terminal and perform data communication with the terminal; the processing device of the present invention calls the simulation program based on the data glove stored in the memory through the processor, and Do the following:

In response to the simulation modeling instruction input by the user, the simulation modeling instruction is sent to the display device, so that the display device creates a three-dimensional human body model according to the simulation modeling instruction.

Further, the processing device may be a repeater, or other devices with data processing functions, which is not limited in this embodiment. In this embodiment, the repeater is used as an example for illustration, and the repeater connection display Device and data gloves;

The simulation modeling instruction can be the start work instruction input by the user through the operation interface, or can be input by the user in other ways. This embodiment does not limit this. In this embodiment, the user enters the instruction through the operation interface as an example. illustrate;

In the specific implementation, the user clicks through the modeling control on the operation interface, the processing device receives the modeling instruction, extracts the type of the modeling object in the modeling instruction, and according to the type of the modeling object input by the user, stores Search the target object corresponding to the modeled target object type in the area, and send the target object to the display device for three-dimensional display.

Embodiment two;

Please refer to Fig. 4 and 5, propose a kind of simulation system of surgery based on data glove, the simulation system based on data glove shown, the simulation system of surgery based on data glove comprises: data glove, processing device and The display device, the data glove is connected to the processing device, and the processing device is connected to the display device.

Obtain the label information of the data glove, and find the corresponding target virtual hand according to the label information, specifically including: obtain the label information of the data glove, verify the label information, and when the verification result is successful, search for the corresponding target virtual hand according to the label information , by obtaining tag information, it is convenient to quickly find the target virtual hand, which is convenient for users to use.

Further, before the connection mode of the data glove is switched to a parallel mode, and before creating corresponding virtual hands according to the number of connections, the method also includes obtaining the API interface of the data glove, and connecting the data glove in real time through the API interface, so that the data glove can obtain the number of connections, And the number of connections is sent to the processing device, and the data gloves are connected in real time through the API interface to facilitate switching between parallel and serial modes according to actual needs.

Further, step S3 also includes obtaining the force feedback information sent by the data glove, and operating the virtual hand according to the force feedback information, so as to realize the simulation of the three-dimensional human body model by multiple users. By feeding back the force information to the simulation simulation system, it is convenient Master the usage of users.

Further, step S3 also includes obtaining the positioning information sent by the data glove, searching for the target position corresponding to the three-dimensional human body model according to the positioning information, and operating the target position through the virtual hand, so as to realize the simulation of the three-dimensional human body model by multiple users, It is convenient to quickly find the target position corresponding to the three-dimensional human body model through the positioning information.

Embodiment three;

Please refer to Figures 1, 2 and 3. The feedback module is used for real-time monitoring of users or multiple users during the simulation process, and to receive system freezes, flashbacks, etc. encountered by users or multiple users during the simulation process System problem, the processing module is used to analyze and process the feedback information data and formulate corresponding solutions after receiving the system problem information fed back by the feedback module, so as to deal with the problems that occur in the simulation process of users or multiple users, The application module is used to fuse and apply the plan information data formulated by the processing unit to the simulation program in the memory after receiving the response plan formulated by the processing module, and monitor the system in real time by the user or multiple users during the simulation operation through the feedback module problems, monitor the quality of simulation in real time, analyze and process the monitored system problems, and formulate countermeasures, so as to solve the system problems that arise, and then enable users or multiple users to perform simulation operations stably, and improve the user's practical experience. Use experience, and it is convenient to find the loopholes in the simulation system and improve the simulation system.

Embodiment four;

Referring to Fig. 1, the processing device includes: a memory, a processor, and a data glove-based simulation program stored on the memory and operable on the processor, the data glove-based simulation program is configured to implement claims 1 to 7 In any one of the steps of the data glove-based simulation method, the simulation operation of the data glove is facilitated by the above-mentioned processing device.

Further, the processing device can generate control instructions from the simulated operation data, and operate the virtual hand through the control instructions, thereby realizing the simulation of the three-dimensional human body model;

In the specific implementation, open the VR or AR or MR application through the operation interface, and each user puts on the data glove and wears the VR or AR or MR head-mounted display device, which is generally a HTCVive supporting device. If the application is an AR or MR application You need to open the AR glasses or the application (Application, APP) on the mobile phone. After opening, the user can see his own hand in the virtual world corresponding to the real world, and the user can pick up the surgical instrument in the virtual world with his hand through the data glove. Start manipulating the human body model and organ model in the virtual world, so as to realize the simulation of the three-dimensional human body model.

Embodiment five;

Please refer to Fig. 6, the simulation system of surgery based on data gloves includes: data gloves, processing device and display device, the data glove is connected to the processing device, the processing device is connected to the display device, and the parallel mode or serial mode can be easily switched through the processing device.

Further, the display device can be a virtual reality (VirtualReality, VR), augmented reality (AugmentedReality, AR) or mediated reality (MediatedReality, MR) head-mounted display device, and can also be a device that realizes the same or similar functions for other subjects, This embodiment does not limit this. In this embodiment, the VR head-mounted display device matched with HTCVive is used as an example for illustration;

In the specific implementation, three-dimensional (3Dimensions, 3D) modeling software such as Autodesk3dsMax is used to establish a 3D model such as Unity3D or UE platform, and software such as Unity3D or UE is used to develop a VR application based on an immersive experience such as HTCVive. The 3D model is a human body model, various organ models, and a virtual doctor or student's hand model based on medical data.

Further, a data glove-based simulation program is stored on the storage medium, and when the data glove-based simulation program is executed by the processor, the steps of the data glove-based simulation method according to any one of claims 1 to 7 are realized.

Working principle: When in use, the processing device sends the simulation modeling command to the display device in response to the simulation modeling command input by the user, so that the display device creates a three-dimensional human body model according to the simulation modeling command, and obtains the access information of the data glove , extract the number of connections in the access information, switch the connection mode with the data glove to parallel mode when the number of connections is greater than the preset threshold, and switch the connection mode with the data glove to serial mode when the number of connections is less than the preset threshold Create corresponding virtual hands according to the number of connections, obtain the simulated operation data of the 3D human body model sent by the data glove in parallel or serially, and operate the virtual hands according to the simulated operation data, so as to realize multi-user control of the 3D The simulation of the three-dimensional human body model, by obtaining the connection data of the data glove, when the number of connections is greater than the preset threshold, the current connection mode is switched to a parallel mode, so that multiple users can operate the three-dimensional human body model at the same time and improve the data quality. The communication efficiency of the glove to the simulation operation, during the simulation process, the feedback module receives the user's stuck, fault and other problems, and transmits the problems to the processing module, and the processing module receives the information fed back by the feedback unit , the processing module analyzes and handles the problems that arise, formulates a solution, and transmits the solution to the application module. After receiving the information of the solution formulated by the processing module, the application module integrates the information and data of the solution into the In the simulation program in the memory, the feedback problems are repaired, and the system problems that occur during the simulation operation of the user or multiple users are monitored in real time through the feedback module, the quality of the simulation is monitored in real time, and the monitored system problems Carry out analysis and processing, and formulate countermeasures, so as to solve the system problems that arise, and then enable users or multiple users to perform simulation operations stably, improve the actual user experience of users, and facilitate the finding of loopholes in the simulation system, and improve Simulate the analog system.

It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is a relationship between these entities or operations. There is no such actual relationship or order between them. Furthermore, the term "comprises", "comprises" or any other variation thereof is intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus comprising a set of elements includes not only those elements, but also includes elements not expressly listed. other elements of or also include elements inherent in such a process, method, article, or device. Without further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article or apparatus comprising said element.

Although the embodiments of the present invention have been shown and described, those skilled in the art can understand that various changes, modifications and substitutions can be made to these embodiments without departing from the principle and spirit of the present invention. and modifications, the scope of the invention is defined by the appended claims and their equivalents.

Claims (10)

1. The simulation simulation method based on the data glove is characterized in that: comprising the following main steps; S1: The processing device sends the simulation modeling instruction to the display device in response to the simulation modeling instruction input by the user, so that the display device creates a three-dimensional human body model according to the simulation modeling instruction; S2: Obtain the access information of the data glove, extract the number of connections in the access information, and when the number of connections is greater than the preset threshold, switch the connection mode with the data glove to a parallel mode, and in the connection When the number is less than the preset threshold, the connection mode with the data glove is switched to a serial mode, and corresponding virtual hands are created according to the number of connections; S3: Obtain the simulated operation data of the 3D human body model sent by the data glove in parallel or serially, and operate the virtual hand according to the simulated operation data, so as to realize multi-user control of the 3D human body model simulation simulation; S4: In the process of simulation, the feedback module receives the user's problems such as freezes and faults, and transmits the problems to the processing module; S5: After the processing module receives the information fed back by the feedback unit, the processing module analyzes and handles the problems that arise, formulates a solution, and transmits the formulated solution to the application module; S6: After the application module receives the information of the countermeasure prepared by the processing module, it integrates the information and data of the countermeasure into the simulation program in the memory, so as to repair the feedback problem. 2. The simulation method based on data glove according to claim 1, characterized in that: said step S3 further comprises acquiring tag information of said data glove, and searching for a corresponding target virtual hand according to said tag information. 3. The simulation method based on data glove according to claim 2, characterized in that: acquiring the tag information of the data glove, and searching for the corresponding target virtual hand according to the tag information, specifically includes: acquiring the The tag information of the data glove, the tag information is verified, and when the verification result is successful, the corresponding target virtual hand is searched according to the tag information. 4. The simulation method based on data glove according to any one of claims 1-3, characterized in that: the connection mode of the data glove is switched to parallel mode, and the corresponding virtual hand is created according to the number of connections , the method further includes acquiring an API interface of the data glove, and connecting the data glove in real time through the API interface, so that the data glove acquires a number of connections, and sends the number of connections to the processing device. 5. The simulation method based on the data glove according to claim 1, characterized in that: said step S3 also includes acquiring the force feedback information sent by the data glove, and performing an operation on the virtual hand according to the force feedback information. Operation, so as to realize the simulation of three-dimensional human body model by multiple users. 6. The simulation method based on the data glove according to claim 1, characterized in that: the step S3 also includes obtaining the positioning information sent by the data glove, and searching for the corresponding three-dimensional human body model according to the positioning information. The target position is operated by the virtual hand, so as to realize the simulation of the three-dimensional human body model by multiple users. 7. The simulation method based on data gloves according to claim 1, characterized in that: the feedback module is used for real-time monitoring of the user or multiple users during the simulation process, and receives the user or multiple users in the simulation process For system problems such as system freezes and flashbacks encountered during the simulation process, the processing module is used to analyze and process the feedback information data and formulate corresponding solutions after receiving the system problem information fed back by the feedback module , to deal with the problems of the user or multiple users during the simulation process, the application module is used to fuse and apply the solution information data formulated by the processing unit to the simulation program in the memory after receiving the response scheme formulated by the processing module. 8. The simulation device based on the data glove according to claim 1, characterized in that: the processing device comprises: a memory, a processor, and a data-based data storage system that is stored on the memory and can run on the processor. A glove simulation program, the data glove-based simulation program is configured to implement the steps of the data glove-based simulation method according to any one of claims 1-7. 9. The simulation system based on the data glove according to claim 1, characterized in that: the simulation system of the surgery based on the data glove comprises: a data glove, a processing device and a display device, and the data glove is connected to the The processing device is connected to the display device. 10. The simulation storage medium based on data glove according to claim 1, characterized in that: a simulation program based on data glove is stored on the storage medium, and the simulation program based on data glove is executed by a processor When realizing the steps of the simulation simulation method based on the data glove according to any one of claims 1 to 7.

Images