TWI868019B - Operation training system for ultrasound and operation training method for ultrasound - Google Patents

Abstract

An operation training system for ultrasound and an operation training method for ultrasound are proposed. The operation training system includes a sensing device, a data processing device and a display device. The sensing device is configured to sense a hand movement signal and a voice signal. The data processing device analyzes the hand movement signal to perform virtual ultrasound detection. The data processing device identifies one of a plurality of ultrasound data to be played and generates a view judgment result and a disease judgment result. The data processing device analyzes the voice signal to retrieve at least one keyword, and generates answer content corresponding to the voice signal based on the at least one keyword. Thus, the effect of simulation training of ultrasonic operation is improved.

Description

Ultrasound operation training system and ultrasound operation training method

The present disclosure relates to an operation training system and an operation training method, and in particular to an ultrasound operation training system and an ultrasound operation training method.

Professional ultrasound inspectors must undergo rigorous training to know the organs or parts to be inspected, and to move or rotate the ultrasound probe appropriately to obtain ultrasound images at specific locations or angles, which can help to correctly judge the condition of the organs or parts. Furthermore, they must be able to judge whether the organs or parts are normal or abnormal through ultrasound images.

As is known, ultrasound operation training is to use a dummy or real person to conduct ultrasound testing training. However, in ultrasound testing of children, for example, it is not easy to make the children lie still on a bed or chair to cooperate with the training, and there is still room for improvement.

In view of this, how to develop an ultrasound operation training system and ultrasound operation training method has become the goal pursued by relevant academics/professionals.

To solve the above problems, the present disclosure provides an ultrasonic operation training system and an ultrasonic operation training method, which can effectively perform simulation training through the configuration of the system.

According to an embodiment of the present disclosure, an ultrasound operation training system is provided, comprising a sensing device, a data processing device and a display device. The sensing device is used to sense a hand of a user to generate a hand motion signal, and capture a speech of the user to generate a speech signal. The data processing device is signal-connected to the sensing device and comprises a display generation module, an analysis module, a recognition module and an interaction module. The display generation module is used to generate a virtual scene, which comprises a virtual human body, a virtual hand, a virtual probe and a virtual ultrasound display. The analysis module signal is connected to the display generation module and is used to analyze the hand movement signal. According to the hand movement signal, the virtual hand is controlled to move the virtual probe in the virtual scene and perform virtual ultrasound detection, so that the display generation module selects one of the subjects to be played from the plurality of ultrasound data, and switches another subject to be played from these ultrasound data according to the scanning angle of the virtual probe. The identification module signal is connected to the display generation module and the analysis module, and is used to identify the subject to be played and generate a cross-sectional judgment result and a disease judgment result, and store the cross-sectional judgment result and the disease judgment result in a question-answering model. The interactive module is signal-connected to the display generation module and the analysis module, and is used to analyze the voice signal to capture at least one keyword, and to generate a dialogue content corresponding to the voice signal according to the question-answer model. The display device is signal-connected to the display generation module, and is used to display the virtual scene and play the dialogue content.

Other embodiments of the aforementioned implementation method are as follows: the ultrasound operation training system further includes a database, which includes a plurality of ultrasound section samples and a plurality of ultrasound disease samples corresponding to different organs. The recognition module extracts at least one feature of the person to be played, compares it with these ultrasound section samples to generate a section judgment result, and compares it with these ultrasound disease samples to generate a disease judgment result.

Other implementation examples of the aforementioned implementation method are as follows: the database further includes a question-answering model, which receives at least one keyword and generates answer content based on the at least one keyword.

Other embodiments of the aforementioned implementation method are as follows: the ultrasound operation training system further includes a database, which includes at least one ultrasound file, and the at least one ultrasound file is divided into these ultrasound data, each ultrasound data is an ultrasound dynamic segment or an ultrasound static frame, and the data processing device further includes an ultrasound data receiving module for receiving these ultrasound data.

Other embodiments of the aforementioned embodiment are as follows: the display device further includes a wearable portion, which is used to connect and support a wearable virtual reality display and is used for a user to wear on a head, and the sensing device includes a head position sensor, which is located in the wearable portion to sense the user's head to emit a head movement signal and a voice signal.

Other embodiments of the aforementioned embodiment are as follows: the virtual scene further includes at least one button, and the at least one button is used to be triggered to change a state of the virtual human body in the virtual scene, freeze the image of the virtual ultrasound display, switch a scan area prompt type on the virtual human body, or change the scan theme.

According to another embodiment of the present disclosure, an ultrasound operation training method is provided, comprising a virtual scene display step, a sensing step, an ultrasound data display step, a recognition step, and a question-answer feedback step. In the virtual scene display step, a display generation module of a data processing device of an ultrasound operation training system generates a virtual scene, wherein the virtual scene comprises a virtual human body, a virtual hand, a virtual probe, and a virtual ultrasound display, and the virtual scene is displayed on a display device of the ultrasound operation training system. In the sensing step, a sensing device senses a hand of a user to generate a hand motion signal, and captures a speech of the user to generate a speech signal. In the ultrasonic data display step, an analysis module of the data processing device analyzes the hand motion signal, controls the virtual hand to move the virtual probe in the virtual scene and performs virtual ultrasonic detection according to the hand motion signal, and enables the display generation module to select one of the ultrasonic data to be played, and switches another one of the ultrasonic data to be played according to the scanning angle of the virtual probe. In the recognition step, a recognition module of the data processing device recognizes the player to generate a face judgment result and a disease judgment result, and stores the face judgment result and the disease judgment result in a question-answering model. In the question-answering feedback step, an interactive module of the data processing device analyzes the voice signal to capture at least one keyword, and generates a dialogue content corresponding to the voice signal according to the question-answering model, and plays the dialogue content on the display device.

Other embodiments of the above-mentioned implementation are as follows: The ultrasound operation training method further includes an adjustment switching step. In the adjustment switching step, the virtual scene further displays at least one button, the user moves the other hand, the sensing device senses the other hand to send the other hand movement signal, and the analysis module determines whether the other virtual hand in the virtual scene touches the at least one button according to the other hand movement signal, so that the display generation module changes a state of the virtual human body in the virtual scene, freezes the screen of the virtual ultrasound display, switches a scanning area prompt type on the virtual human body, or changes the scanning theme.

Other embodiments of the aforementioned implementation are as follows: In the recognition step, the recognition module extracts at least one feature of the person to be played, compares it with a plurality of ultrasound section samples in a database to generate a section judgment result, and compares it with a plurality of ultrasound disease samples in the database to generate a disease judgment result. Among them, these ultrasound section samples and these ultrasound disease samples in the database correspond to different organs.

Other implementation examples of the aforementioned implementation method are as follows: In the question-answer feedback step, the question-answer model receives at least one keyword and generates answer content based on the at least one keyword.

The following will describe embodiments of the present disclosure with reference to the drawings. For the sake of clarity, many practical details will be described together in the following description. However, the reader should understand that these practical details should not be used to limit the present disclosure. In other words, in some embodiments of the present disclosure, these practical details are not necessary. In addition, in order to simplify the drawings, some commonly used structures and components will be shown in the drawings in a simple schematic manner; and repeated components may be represented by the same number or similar number.

In addition, the terms "first", "second", "third", etc. in this article are only used to describe different elements or components, and do not limit the elements/components themselves. Therefore, the first element/component can also be renamed as the second element/component. Moreover, the combination of elements/components/mechanisms/modules in this article is not a generally known, conventional or familiar combination in this field. Whether the elements/components/mechanisms/modules themselves are known cannot be used to determine whether their combination relationship is easy to be completed by ordinary knowledge in the technical field.

Please refer to FIG. 1, FIG. 2 and FIG. 3, wherein FIG. 1 is a schematic diagram of an ultrasonic operation training system 100 and a user U1 according to a first embodiment of the present disclosure; FIG. 2 is a block diagram of the ultrasonic operation training system 100 of FIG. 1; and FIG. 3 is a virtual scene P of the ultrasonic operation training system 100 of FIG. 1. The ultrasonic operation training system 100 includes a sensing device 110, a data processing device 120 and a display device 130.

The sensing device 110 is used to sense a hand U11 of the user U1 to generate a hand motion signal. The data processing device 120 is signal-connected to the sensing device 110 and includes an analysis module 123 and a display generation module 121. The analysis module 123 is used to analyze the hand motion signal. The display generation module 121 is signal-connected to the analysis module 123 and is used to generate a virtual scene P. The virtual scene P includes a virtual human body P2, a virtual hand P11, a virtual probe P5, and a virtual ultrasound display P3. The display device 130 is signal-connected to the display generation module 121 and is used to display the virtual scene P. The virtual hand P11 moves the virtual probe P5 in the virtual scene P and performs virtual ultrasound detection according to the hand motion signal. The analysis module 123 selects one of the ultrasound data to be played from the plurality of ultrasound data according to the hand motion signal, and causes the display generation module 121 to display the one of the ultrasound data to be played on the virtual ultrasound display P3. When the analysis module 123 determines that the hand U11 of the user U1 has moved or rotated according to the hand motion signal, another object to be played is selected from the ultrasound data according to the scanning angle of the virtual probe P5, and the display generation module 121 plays the aforementioned another object to be played to the virtual ultrasound display P3 to switch the screen of the virtual ultrasound display P3 accordingly.

Thus, the virtual scene P generated by the data processing device 120 can simulate the actual detection scene, and the sensing device 110 can sense the hand U11 of the user U1 to allow the user U1 to move the virtual hand P11 in the virtual scene P, and the ultrasound data to be played can be adjusted according to the movement of the hand U11, thereby increasing the realism of the simulation training. The details of the ultrasound operation training system 100 will be described in detail later.

The ultrasound operation training system 100 may further include a database 140, which includes at least one ultrasound file, wherein the at least one ultrasound file is divided into the plurality of ultrasound data, each of which is an ultrasound dynamic segment or an ultrasound static frame. The data processing device 120 may further include an ultrasound data receiving module 122 for receiving the plurality of ultrasound data.

Specifically, the data processing device 120 may be an electronic device such as a smart device, a virtual reality processor or a mixed reality processor, etc. The electronic device may include, for example, a central processing unit for performing calculations, a random access memory for generating temporary information by random calculations, and a memory unit such as a hard disk, etc. The data processing device 120 may be programmed to form a display generation module 121, an ultrasonic data receiving module 122, and an analysis module 123 to execute instructions and corresponding functions. In this embodiment, the data processing device 120 is illustrated as a virtual reality processor. The database 140 may be located in a server or a computer, and may be connected to the data processing device 120 via a wireless or wired manner, so as to transmit the ultrasound data to the data processing device 120 .

The number of ultrasound files can be multiple, which are dynamic images recorded during the actual operation of ultrasound detection. The ultrasound files can be classified according to the needs, such as by organ classification, or by normal, abnormal, etc., and can be further classified by disease. In addition, the ultrasound file can be cut according to the needs, such as dividing each complete dynamic image into multiple ultrasound dynamic segments and/or ultrasound static frames according to the probe position and/or angle, so each ultrasound file can be divided into multiple ultrasound dynamic segments and/or ultrasound static frames. In this embodiment, the ultrasound file may be segmented by the server before being transmitted to the ultrasound data receiving module 122. In other embodiments, the server may transmit the ultrasound file to the data processing device, which may segment the file and then transmit the file to the ultrasound data receiving module. The present invention is not limited to this.

The display device 130 may include at least one of a wearable virtual reality display 131, a wearable mixed reality display 132, and a projection display 133. Specifically, as shown in FIG. 1 , the wearable virtual reality display 131 may have a head mounted display (HMD) structure, so the display device 130 may further include a wearable portion 134, which is used to connect and support the wearable virtual reality display 131 and is used for the user U1 to wear on a head U13, and the data processing device 120 may also be disposed on the wearable portion 134. The wearable mixed reality display 132 may have a transparent, semi-transparent or see-through near-eye display structure, and may also be connected to another wearable portion. In one embodiment, the display device may only include a wearable virtual reality display and a projection display, so that the virtual scene can be displayed on the wearable virtual reality display and the projection display at the same time, which can facilitate other personnel to observe the user's training status.

The sensing device 110 may include a hand position sensor 111, and the hand position sensor 111 has a handle structure for the hand U11 to hold. In addition, the sensing device 110 may further include a head position sensor 112, and the head position sensor 112 may be located in the wearable portion 134 to sense the head U13 of the user U1 to send a head movement signal. In addition, the sensing device 110 may further include a reference point position sensor 113.

Specifically, the hand position sensor 111 may include an inertial sensing element and a data transmission element. The inertial sensing element senses the movement of the hand U11 (and the hand U12 mentioned later), such as displacement or rotation. Displacement refers to linear displacement, such as longitudinal displacement and lateral displacement. The data transmission element forms a hand movement signal based on the sensing result and transmits it back to the data processing device 120. After analysis by the analysis module 123, the display generation module 121 changes the state of the virtual hand P11 in the virtual scene P based on the hand movement signal. The hand position sensor 111 may further include a tactile feedback module, which can provide feedback to the user U1 in the form of vibration or the like. In other embodiments, the hand position sensor may have a glove structure for the hand to wear, and is not limited to the above. The structure of the head position sensor 112 is similar to that of the hand position sensor 111 and may also include an inertial sensing element and a data transmission element, etc., to sense the position or rotation of the head U13 of the user U1, and to control the change of the angle or position of the virtual scene P.

The reference point position sensor 113 may be, for example, mounted on a bracket and connected to the data processing device 120 by signal. Furthermore, it may also be connected to the hand position sensor 111 and the head position sensor 112 by signal to confirm the specific position of each part of the user U1 in the real space, and facilitate the integration of virtual and real, but it is not limited to this. In other embodiments, the sensing device may include a camera and a plurality of patches. The patches may, for example, include two-dimensional codes and may be attached to the parts of the user to be sensed and tracked, and the positions of each part may be confirmed through image recognition. Alternatively, the sensing device may include an infrared light detector, an infrared light emitter, and a reflective patch. The reflective patch is used to be attached to the part of the user to be sensed and tracked, and the infrared light detector and the infrared light emitter confirm the position. The content of the present disclosure is not limited to this.

In the embodiments of Figures 1 to 3, the sensing device 110 can also sense the other hand U12 of the user U1, for example, by using another hand position sensor 111 to sense the hand U12 to send out another hand movement signal, so that the virtual scene P generated by the display generation module 121 can further include another virtual hand P12, and control the movement of the virtual hand P12 in the virtual scene P according to the movement of the hand U12.

As shown in FIG. 3 , the virtual scene P includes a virtual human body P2, virtual hands P11, P12, a virtual ultrasound display P3, and a virtual probe P5. The virtual hands P11, P12 will make corresponding displacements or rotations according to the movements of the hands U11, U12. The virtual scene P may also include an ultrasound machine, other equipment or instruments in the ward, but is not limited to this. The virtual scene P may further include at least one button P4, which is used to be triggered to change a state of the virtual human body P2 in the virtual scene P, freeze the screen of the virtual ultrasound display P3, switch a scan area prompt type on the virtual human body P2, or change the scan theme. Specifically, the number of buttons P4 may be plural, and texts corresponding to the above functions such as "human body display mode change", "freeze scan screen", "switch display scan area", "heart scan" and "abdomen scan" may be displayed on different buttons P4. During training, the virtual hand P12 can be controlled to trigger the button P4 to select a heart scan or an abdominal scan, and there is no prompt in the default state of the initial scan. If necessary, the user U1 can control the virtual hand P12 to press the "switch display scan area" button P4 to display the prompt of the scan area on the human body. The prompt can be divided into no prompt, longitudinal scan display and horizontal scan display, which can be switched by controlling the virtual hand P12 to press the button P4. In addition, a scanning direction D1 including a long axis and three short axes can also be displayed on the virtual human body P2. The scanning direction D1 is located on the window corresponding to the scanned window and can be used as a reference for the user U1. It should be particularly noted that the ultrasound scanning in this case can be applied to the scanning of various organs in the human body, not limited to the organs mentioned above.

The user U1 can control the virtual hand P11 to move the virtual probe P5. In the initial state, there is no data on the screen of the virtual ultrasound display P3. When the user U1 operates the virtual probe P5 to make the angle or position correct, the virtual ultrasound display P3 will display the corresponding ultrasound data. Moreover, when the user U1 continues to move or rotate the hand U11, for example, when the movement exceeds a movement threshold, the hand U11 is determined to be moving. At this time, the ultrasound data displayed on the virtual ultrasound display P3 should be switched accordingly, that is, the original ultrasound dynamic segment and/or ultrasound static frame is switched to another ultrasound dynamic segment and/or ultrasound static frame. Similarly, when the hand U11 of the user U1 rotates beyond a rotation threshold, the hand U11 is determined to be rotated, and the ultrasound data displayed on the virtual ultrasound display P3 should be switched accordingly. The movement threshold may be, for example, 15 mm, and the rotation threshold may be, for example, 10 degrees, and may be between 8 degrees and 17 degrees, or between 10 degrees and 20 degrees. When the user U1 uses the virtual probe P5 to perform an ultrasound scan, the user U1 may use the other hand U12 to control the virtual hand P12, such as pressing the button P4 to freeze the scan screen, but is not limited thereto.

Please refer to FIG. 4, wherein FIG. 4 is a block flow chart of an ultrasonic operation training method S200 according to the second embodiment of the present disclosure. The ultrasonic operation training method S200 includes a virtual scene display step S210, a sensing step S220, and an ultrasonic data display step S230. The details of the ultrasonic operation training method S200 will be described below in conjunction with the ultrasonic operation training system 100 of FIGS. 1 to 3.

In the virtual scene display step S210, the display generation module 121 of the data processing device 120 of the ultrasound operation training system 100 generates a virtual scene P. The virtual scene P includes a virtual human body P2, a virtual hand P11, a virtual probe P5 and a virtual ultrasound display P3. The virtual scene P is displayed on the display device 130 of the ultrasound operation training system 100.

In the sensing step S220, the sensing device 110 senses the hand U11 of the user U1 to generate a hand motion signal. The analysis module 123 of the data processing device 120 analyzes the hand motion signal and controls the virtual hand P11 to move the virtual probe P5 in the virtual scene P and perform virtual ultrasound detection according to the hand motion signal.

In the ultrasound data display step S230, the display generation module 121 selects one to be played from the plurality of ultrasound data according to the hand motion signal, and displays the one to be played on the virtual ultrasound display P3. When the display generation module 121 determines that the hand U11 of the user U1 moves or rotates according to the hand motion signal, another to be played is selected from the plurality of ultrasound data according to the scanning angle of the virtual probe P5, and the display generation module 121 plays the other to be played to the virtual ultrasound display P3 to switch the screen of the virtual ultrasound display P3 accordingly.

Specifically, in the virtual scene display step S210, the display generation module 121 generates a virtual scene P, and the virtual scene P can be displayed by the wearable virtual reality display 131 of the display device 130, so that the user U1 can perform virtual reality ultrasound detection training.

In the sensing step S220 , the hand position sensor 111 senses the motion state of the hand U11 , such as movement or rotation, so that the virtual hand P11 generates corresponding motion in the virtual scene P to control the virtual probe P5 .

In the ultrasound data display step S230, a real ultrasound detection is simulated, so when the virtual probe P5 moves to the corresponding position, the virtual ultrasound display P3 in the virtual scene P will display the corresponding ultrasound data, and when the virtual probe P5 moves or rotates, the screen switches. Therefore, the analysis module 123 determines whether the hand U11 of the user U1 moves or rotates according to the hand movement signal, and can select the corresponding ultrasound data to be played, so that the display generation module 121 can switch the screen.

In addition, in the ultrasonic data display step S230, the sensing device 110 includes a hand position sensor 111, and the hand position sensor 111 has a handle structure for the hand U11 to hold. When the data processing device 120 determines that the position and angle of the hand U11 are correct based on the hand motion signal, the data processing device 120 causes the hand position sensor 111 to vibrate. That is to say, initially, there may be no image on the virtual ultrasound display P3. When the user U1 controls the hand U11 so that the virtual hand P11 manipulates the virtual probe P5 to the correct or specified position and angle, an image will appear on the virtual ultrasound display P3, and the data processing device 120 may include a feedback module, which sends instructions to activate the tactile feedback module of the hand position sensor 111, thereby generating vibration to let the user U1 know that the operation is correct.

As shown in FIG. 4 , the ultrasound operation training method S200 may further include an adjustment switching step S240, the virtual scene P further displays at least one button P4, the user U1 moves the other hand U12, the sensing device 110 senses the other hand U12 to send another hand motion signal, and the analysis module 123 determines whether the other virtual hand P12 in the virtual scene P touches the at least one button P4 according to the other hand motion signal, so that the display generation module 121 changes the state of the virtual human body P2 in the virtual scene P, freezes the screen of the virtual ultrasound display P3, switches the scanning area prompt type on the virtual human body P2, or changes the scanning theme.

Specifically, the number of buttons P4 is plural, so when the virtual hand P12 in the virtual scene P touches the button P4 showing "Human body display mode change" to change the human body display mode, a state of the virtual human body P2 in the virtual scene P can be changed, such as switching the human skin, viewing the abdomen, viewing the heart, etc. When the virtual hand P12 in the virtual scene P touches the button P4 showing "Heart scan" or "Abdomen scan" to select to perform a heart scan or an abdomen scan, the selected scan theme can be changed.

Please refer to FIG. 5, which shows a flow chart of the steps of the ultrasound operation training method S200 of the embodiment of FIG. 4. Initially, the user U1 can wear the wearable part 134 to place the wearable mixed reality display 132 in front of the eyes, and hold the hand position sensor 111 such as a handle structure, and in step S01, the wearable mixed reality display 132 of the display device 130 can display the virtual scene P, so that the user U1 can see the virtual human body P2, virtual hands P11, P12, virtual probe P5 and virtual ultrasound display P3 in the virtual scene P.

In step S02, the hands U11 and U12 of the user U1 are detected, and the virtual hands P11 and P12 are moved in the virtual scene P according to the movements of the hands U11 and U12. In step S03, it is first confirmed whether the user U1 moves the hand U12 to make the virtual hand P12 press the button P4 to confirm whether to change the organ tissue display mode. If so, the process proceeds to step S04 to switch the human body display mode to human skin, see-through abdomen, see-through heart, etc. If not, the process proceeds to step S05 to confirm the scanning theme selected or switched by the user U1.

Next, the process proceeds to step S06. According to the scanning theme selected or switched by the user U1, a correct probe placement position and angle can be preset. Therefore, it can be confirmed whether the angle and position of the virtual probe P5 controlled by the user U1 through the virtual hand P11 are correct. If not, the process proceeds to step S07, and the display generation module 121 does not display the screen. On the contrary, if it is correct, the process proceeds to step S08, and the display generation module 121 displays the corresponding screen, and the hand position sensor 111 vibrates through the feedback module to indicate that the operation is correct. The virtual ultrasound display P3 can repeatedly display about 2 seconds of ultrasound dynamic clips, or a single ultrasound static frame, until the screen is switched.

Afterwards, enter step S09 and step S12 to continuously detect the hand U11 of the user U1 to confirm whether the hand U11 moves beyond the displacement threshold or rotates beyond the rotation threshold. If not, enter step S11 and the display generation module 121 maintains the same display screen on the virtual ultrasound display P3. Otherwise, enter step S10 to change the display screen.

It should be particularly noted that the above step flow is only an example, and steps S03 and S05 can be judged or performed at any time during the operation, and are not limited to the above sequence. In addition, when performing the ultrasound operation training method S200, the ultrasound data to be displayed can be pre-set, for example, a set of ultrasound data corresponding to the heart and a set of ultrasound data corresponding to the abdomen are selected in advance in the database 140, and then the ultrasound data corresponding to the heart or the ultrasound data corresponding to the abdomen are played according to the selection of the user U1, and the ultrasound dynamic segment and/or ultrasound static frame in the set of ultrasound data to be played is determined according to the hand U11 of the user U1. However, in other embodiments, multiple sets of ultrasound data corresponding to the heart and multiple sets of ultrasound data corresponding to the abdomen may be selected in advance and then randomly played, but the present invention is not limited thereto. In one embodiment, the data processing device requests the database to transmit the corresponding ultrasound data based on the user's selection of "heart scan" or "abdomen scan"; however, in another embodiment, the data processing device may have stored at least one set of ultrasound data corresponding to the heart and at least one set of ultrasound data corresponding to the abdomen in advance, but the present invention is not limited thereto.

In addition, the user U1 can learn to recognize the positions of various organs by moving the virtual probe P5 in the virtual scene P, and can more easily identify various internal organs by switching the human body display mode to see-through abdomen and see-through heart, etc., so that the skin and muscle tissue can be transparent. Furthermore, the scanning area prompt on the human body is switched to assist in learning the scanning position.

In addition, the ultrasound operation training method S200 is also used for testing, so a specific case in the database 140 can be selected as a test subject, that is, the aforementioned ultrasound data is selected from a specific case in the database 140, and this is used as a test subject for testing. For example, the test subject can be selected by an examiner or by artificial intelligence in an electronic device. The test subject can be, for example, organ position, angle, symptom confirmation, etc. The examiner can observe the operation of the user U1 or judge whether it is correct through the projection display 133. In other embodiments, if the data processing device is an intelligent device, the virtual scene can also be displayed on the computer screen, but it is not limited to this.

Please refer to FIG. 1, FIG. 2 and FIG. 6, FIG. 6 shows a block diagram of an ultrasound operation training system 300 according to the third embodiment of the present disclosure. The ultrasound operation training system 300 includes a sensing device 310, a data processing device 320, a display device 330 and a database 340. The sensing device 310 includes a hand position sensor 311, a head position sensor 312 and a reference point position sensor 313. The data processing device 320 includes a display generation module 321, an ultrasound data receiving module 322 and an analysis module 323. The display device 330 includes a wearable virtual reality display 331, a wearable mixed reality display 332, a projection display 333 and a wearable part (not shown). In the third embodiment, the hand position sensor 311 and the reference point position sensor 313 of the sensing device 310, the display generation module 321, the ultrasonic data receiving module 322 and the analysis module 323 of the data processing device 320, and the display device 330 are respectively the same as the hand position sensor 111 and the reference point position sensor 113, the display generation module 121, the ultrasonic data receiving module 122 and the analysis module 123 of the data processing device 120, and the display device 130 of the aforementioned first embodiment, and are not further described here.

The difference between the third embodiment and the first embodiment is that the head position sensor 312 of the sensing device 310 further includes capturing a speech voice (not shown) of the user U1 to send out a voice signal. The data processing device 320 further includes a recognition module 324 and an interaction module 325. In addition, the database 340 further includes a question-answering model (not shown) and a plurality of ultrasound section samples and a plurality of ultrasound disease samples corresponding to different organs.

The recognition module 324 and the interactive module 325 are signal-connected to the display generation module 321 and the analysis module 323. The recognition module 324 is used to recognize the person to be played and generate a section judgment result and a disease judgment result, and store the section judgment result and the disease judgment result in the question-answering model. The recognition module 324 extracts at least one feature of the person to be played, compares it with the ultrasound section sample in the database 340 to generate the section judgment result, and compares it with the ultrasound disease sample in the database 340 to generate the disease judgment result.

In detail, referring to FIG. 1, FIG. 3 and FIG. 6, when the user U1 scans the organs of the virtual human body P2, the recognition module 324 can automatically perform image recognition on the person to be played displayed on the virtual ultrasound display P3 through the convolutional neural network (CNN), and confirm the corresponding organs and scanned sections of the person to be played, and at the same time confirm whether the organs have diseases.

The interactive module 325 is used to analyze the voice signal and capture at least one keyword, and calculate and generate a response content corresponding to the voice signal according to the question-answer model. The question-answer model receives at least one keyword, and calculates and generates the response content according to at least one keyword. In detail, the user U1 asks the interactive module 325 questions about the ultrasonic operation in a spoken voice, and the interactive module 325 can extract keywords from the voice signal of the spoken voice, so that the question-answer model searches from the question-answer set through artificial intelligence to generate the response content, and automatically plays the response content in the form of voice through the display device 330 to reply to the user U1. In this way, when the user U1 performs the simulated operation, the virtual teacher interacts with the user U1 and plays a role of guidance or reminder. In addition, in other possible embodiments, the user may use voice to control the display generation module to modify the virtual probe angle or control buttons in the virtual scene, but the present disclosure is not limited thereto.

For example, when the user U1 is scanning the heart of the virtual human body P2, the recognition module 324 will first identify the user to be played and generate a section judgment result of "parasternal long axis view" and a disease judgment result of "no abnormality", and store it in the question-answering model. When user U1 asks via voice, "Which section does the current heart scan correspond to?", the interactive module 325 can correspondingly capture the keywords "heart" and "section", and the question-answer model generates the answer "The current heart scan corresponds to the longitudinal plane of the left edge of the sternum" based on the keywords and the aforementioned section judgment results, and automatically replies to the user U1 via voice via the display device 330.

Please refer to FIG. 1, FIG. 6 and FIG. 7, wherein FIG. 7 is a block flow chart of the ultrasonic operation training method S400 according to the fourth embodiment of the present disclosure. The ultrasonic operation training system 300 is configured to implement the ultrasonic operation training method S400, and it must be noted that the ultrasonic operation training method S400 of the present disclosure is not limited to being implemented by the ultrasonic operation training system 300 provided by the third embodiment of the present disclosure. In the fourth embodiment, the ultrasound operation training method S400 includes a virtual scene display step S410, a sensing step S420, an ultrasound data display step S430, an adjustment switching step S440, a recognition step S450, and a question-answer feedback step S460. The virtual scene display step S410, the ultrasound data display step S430, and the adjustment switching step S440 are respectively the same as the virtual scene display step S210, the ultrasound data display step S230, and the adjustment switching step S240 of the second embodiment, and are not further described herein.

In the sensing step S420, the sensing device 310 senses the hand U11 of the user U1 to send out a hand movement signal, and captures the user U1's speech voice (not separately labeled) to send out a voice signal. In the recognition step S450, the recognition module 324 recognizes the person to be played and generates a section judgment result and a disease judgment result, and stores the section judgment result and the disease judgment result in the question-answering model. The recognition module 324 extracts at least one feature of the person to be played, compares it with a plurality of ultrasound section samples in the database 340 to generate a section judgment result, and compares it with a plurality of ultrasound disease samples in the database 340 to generate a disease judgment result.

In the question-answer feedback step S460, the interactive module 325 analyzes the voice signal and captures at least one keyword. The interactive module 325 receives the at least one keyword through the question-answer model and generates a dialogue content corresponding to the voice signal. The display device 330 plays the dialogue content.

As can be seen from the above embodiments, the present disclosure has the following advantages: First, the virtual scene generated by the data processing device can simulate the actual detection scene, and the sensing device senses the user's hand to allow the user to move the virtual hand in the virtual scene, and the ultrasound data to be played can be adjusted according to the movement of the hand, which can increase the authenticity of the simulation training. Second, the interactive module can interact with the user in the role of a virtual teacher and play a role of guidance or reminder when the user performs the simulation operation, which can improve the effect of the simulation training.

Although the contents of this disclosure have been disclosed as above by way of embodiments, they are not intended to limit the contents of this disclosure. Anyone skilled in the art can make various changes and modifications without departing from the spirit and scope of the contents of this disclosure. Therefore, the protection scope of the contents of this disclosure shall be subject to the scope defined by the attached patent application.

100,300: Ultrasonic operation training system 110,310: Sensing device 111,311: Hand position sensor 112,312: Head position sensor 113,313: Reference point position sensor 120,320: Data processing device 121,321: Display generation module 122,322: Ultrasonic data receiving module 123,323: Analysis module 130,330: Display device 131,331: Wearable virtual reality display 132,332: Wearable mixed reality display 133,333: Projection display 134: Wearable unit 140,340: Database 324: Recognition module 325: Interactive module D1: Scanning direction P: Virtual scene P11, P12: Virtual hand P2: Virtual human body P3: Virtual ultrasound display P4: Button P5: Virtual probe S01, S02, S03, S04, S05, S06, S07, S08, S09, S10, S11, S12: Steps S200, S400: Ultrasound operation training method S210, S410: Virtual scene display step S220, S420: Sensing step S230, S430: Ultrasound data display step S240, S440: Adjustment and switching step S450: Identification step S460: Question and answer feedback step U1: User U11, U12: Hands U13: Head

FIG. 1 is a schematic diagram of an ultrasonic operation training system and a user according to the first embodiment of the present disclosure; FIG. 2 is a block diagram of the ultrasonic operation training system of FIG. 1; FIG. 3 is a virtual scene of the ultrasonic operation training system of FIG. 1; FIG. 4 is a block flow chart of an ultrasonic operation training method according to the second embodiment of the present disclosure; FIG. 5 is a step flow chart of the ultrasonic operation training method of the embodiment of FIG. 4; FIG. 6 is a block diagram of an ultrasonic operation training system according to the third embodiment of the present disclosure; and FIG. 7 is a block flow chart of an ultrasonic operation training method according to the fourth embodiment of the present disclosure.

300: Ultrasound operation training system

310:Sensor device

311:Hand position sensor

312: Head position sensor

313: Reference point position sensor

320: Data processing device

321: Display generated module

322: Ultrasound data receiving module

323:Analysis module

324: Identification module

325:Interactive module

330: Display device

331: Wearable virtual reality display

332: Wearable mixed reality display

333: Projection display

340: Database

Claims (10)

An ultrasonic operation training system comprises: A sensing device for sensing a hand of a user to generate a hand movement signal, and capturing a speech of the user to generate a speech signal; A data processing device, signal-connected to the sensing device and comprising: A display generation module for generating a virtual scene, wherein the virtual scene comprises a virtual human body, a virtual hand, a virtual probe and a virtual ultrasonic display; An analysis module, which is connected to the display generation module by signal and is used to analyze the hand motion signal, and controls the virtual hand to move the virtual probe in the virtual scene and perform virtual ultrasound detection according to the hand motion signal, so that the display generation module selects one of the subjects to be played from the plurality of ultrasound data, and switches another subject to be played from the ultrasound data according to the scanning angle of the virtual probe; An identification module, which is connected to the display generation module and the analysis module by signal and is used to identify the subject to be played and generate a section judgment result and a disease judgment result, and store the section judgment result and the disease judgment result in a question-answering model; and An interactive module, signal-connected to the display generation module and the analysis module, and used to analyze the voice signal to capture at least one keyword, and generate a dialogue content corresponding to the voice signal according to the question-answer model; and A display device, signal-connected to the display generation module, and used to display the virtual scene and play the dialogue content. The ultrasound operation training system as described in claim 1 further includes a database, which includes a plurality of ultrasound section samples and a plurality of ultrasound disease samples corresponding to different organs. The recognition module extracts at least one feature of the person to be played, compares it with the ultrasound section samples to generate the section judgment result, and compares it with the ultrasound disease samples to generate the disease judgment result. An ultrasound operation training system as described in claim 2, wherein the database further includes the question-answer model, which receives the at least one keyword and generates the answer content based on the at least one keyword. The ultrasound operation training system as described in claim 1 further includes a database, which includes at least one ultrasound file, and the at least one ultrasound file is divided into a plurality of ultrasound data, each of which is an ultrasound dynamic segment or an ultrasound static frame. The data processing device further includes an ultrasound data receiving module for receiving the ultrasound data. An ultrasonic operation training system as described in claim 1, wherein the display device further includes a wearable portion, which is used to connect and support a wearable virtual reality display and is used for the user to wear it on the head, and the sensing device includes a head position sensor, which is located on the wearable portion to sense the head of the user to emit a head movement signal and the voice signal. An ultrasound operation training system as described in claim 1, wherein the virtual scene further includes at least one button, and the at least one button is used to be triggered to change a state of the virtual human body in the virtual scene, freeze the image of the virtual ultrasound display, switch a scan area prompt type on the virtual human body, or change a scan theme. A method for ultrasonic operation training comprises: A virtual scene display step, a display generation module of a data processing device of an ultrasonic operation training system generates a virtual scene, the virtual scene includes a virtual human body, a virtual hand, a virtual probe and a virtual ultrasonic display, and the virtual scene is displayed on a display device of the ultrasonic operation training system; A sensing step, a sensing device senses a hand of a user to generate a hand movement signal, and captures a speech of the user to generate a voice signal; An ultrasound data display step, an analysis module of the data processing device analyzes the hand motion signal, controls the virtual hand to move the virtual probe in the virtual scene and performs virtual ultrasound detection according to the hand motion signal, so that the display generation module selects one of the objects to be played from the plurality of ultrasound data, and switches another object to be played from the ultrasound data according to the scanning angle of the virtual probe; An identification step, an identification module of the data processing device identifies the object to be played and generates a section judgment result and a disease judgment result, and stores the section judgment result and the disease judgment result in a question-answering model; and In a question-and-answer feedback step, an interactive module of the data processing device analyzes the voice signal to capture at least one keyword, generates a dialogue content corresponding to the voice signal according to the question-and-answer model, and plays the dialogue content on the display device. The ultrasound operation training method as described in claim 7 further comprises: an adjustment switching step, wherein the virtual scene further displays at least one button, the user moves the other hand, the sensing device senses the other hand to send another hand motion signal, and the analysis module determines whether the other virtual hand in the virtual scene touches the at least one button according to the other hand motion signal, so that the display generation module changes a state of the virtual human body in the virtual scene, freezes the screen of the virtual ultrasound display, switches a scan area prompt type on the virtual human body, or changes the scan theme. The ultrasound operation training method as described in claim 7, wherein, in the identification step, the identification module extracts at least one feature of the person to be played, compares it with a plurality of ultrasound section samples in a database to generate the section judgment result, and compares it with a plurality of ultrasound disease samples in the database to generate the disease judgment result; wherein the ultrasound section samples and the ultrasound disease samples in the database correspond to different organs. An ultrasonic operation training method as described in claim 7, wherein, in the question-answer feedback step, the question-answer model receives the at least one keyword and generates the answer content based on the at least one keyword.

Images