TWI837015B - Process for rendering real-time ultrasound images used in virtual reality - Google Patents

Abstract

The present invention relates to a process for rendering real-time ultrasound images used in virtual reality. Firstly, various echo bounce coefficients are generated by ultrasonic scanning, based on a pre-defined three-dimensional (3D) organ model, for creation of images with gray-scale pixels, wherein the higher the echo bounce coefficients, the higher the relative pixel intensity value for an image, so as to obtain relative echo bounce images formed as a 3D organ is ultrasonically scanned. Secondly, the echo bounce images are treated by edge highlight enhancement for obtainment of edge enhanced images. Thirdly, a mask having graininess and diffused light is prepared to overlay the edge enhanced images for obtainment of images with diffused light and graininess. Fourthly, the images with diffused light and graininess can further overlay an upper skin texture simulated background image to form an upper skin texture image. Lastly, the upper skin texture image is treated by sectoring and cut by use of a sector mask, so as further to obtain a lifelike real-time ultrasonic scanned image that can be used in virtual reality.

Description

Rendering method for real-time ultrasound images for virtual reality

The present invention relates to a method for rendering an image, and more particularly to a method for rendering a real-time ultrasonic image for virtual reality.

With the rapid development of science and medical technology, humans can use more sophisticated instruments to diagnose and analyze diseases. In addition to improving the accuracy of diagnosis, it can also detect diseases at an early stage and improve treatment effects.

Among them, ultrasound examination is a medical imaging diagnostic technology that uses high-frequency sound waves and has no radiation concerns. Therefore, it is a safe and non-invasive diagnosis and treatment method; common abdominal ultrasound examination can help doctors in the Department of Gastroenterology easily Observing the health status of abdominal organs (such as liver, gallbladder, etc.) is a great tool for gastrointestinal physicians in clinical diagnosis and treatment. However, the accuracy of lesion identification in abdominal ultrasound examination has a considerable relationship with the experience of the operator. Although in the existing technology, AI (artificial intelligence) can Smart, artificial intelligence) finds lesions from ultrasound scan images through appropriate learning technology. However, there are still many uncertain factors in the process of judging lesions, which affect the accuracy of the judgment; therefore, AI's lesion judgment is only for reference, and ultimately must be left to the physician's judgment in clinical practice. Doctors must undergo a considerable degree of ultrasound scanning training before they can make correct judgments on lesions during diagnosis and treatment.

With the rapid development of information technology, the application of virtual reality is also booming. For example, medicine, military, education, tourism, games, etc. all have traces of its application. In the medical field, there are already quite a lot of applications, such as surgical simulation training, ultrasound scanning simulation training, behavioral therapy, body rehabilitation, etc.

The current technology for simulating ultrasonic scanning images in virtual reality is to first obtain actual ultrasonic images with medical ultrasonic imaging equipment, then cooperate with the spatial positioning device to obtain spatial coordinates, and then establish an ultrasonic image data set, and use nuclear magnetic resonance to Establish a three-dimensional visualization of the human body structure based on contrast imaging technology, provide the scanning reference position of any cut plane, and then calculate and display the ultrasound image corresponding to the cut plane in real time from the database (see Xiao Huahong, "3D Visualization and Image Reference Ultrasound Virtual Reality System", Master's Thesis, Institute of Medical Engineering, National Yang-Ming University, 2008. <https://hdl.handle.net/11296/s8gm6g>).

However, the technology disclosed above requires the use of medical ultrasound imaging equipment to obtain a large number of actual ultrasound images in advance and the establishment of a large number of ultrasound image data sets, in order to subsequently display the ultrasound of the corresponding section when the virtual probe is scanned at any angle. The process of creating sonic images is quite cumbersome, time-consuming and labor-intensive, and requires a large amount of memory space to store these images and related spatial coordinate data.

In addition, because the type, location, and size of the lesions are all established in advance, the training model is fixed and lacks change, which affects the effectiveness of the training.

Furthermore, if you want to change the type, location, or size of the lesion, you must re-establish the ultrasound image data set, which is a very troublesome process.

Now, in view of the above situation, the inventor has developed the present invention to provide a method for rendering real-time ultrasonic images for virtual reality.

The main purpose of the present invention is to provide a method for rendering real-time ultrasound images in virtual reality. The main purpose is to enable real-time rendering when scanning human organs with ultrasound in virtual reality. The image appears as realistic as a real ultrasound scan image.

Another purpose of the present invention is to set a lesion on a 3D organ model in virtual reality, and to give a preset pixel according to the location and material of the lesion during ultrasound scanning, so that the echo rebound image includes the lesion image, providing users with a case study of finding lesions through ultrasound in virtual reality.

The above-mentioned objects of the present invention are achieved by the following technologies:

A rendering method for real-time ultrasound images in virtual reality, including the following steps:

To construct an echo-reflection image, a transparent rendering shader is applied to a 3D organ model defined by software and expressed with an alpha value, thereby generating a corresponding organ echo-reflection image; wherein, the coloring principle of the transparent rendering shader is based on When ultrasound scans a 3D organ model, the echo reflection coefficients are different due to different materials, positions, and overlaps. According to the different echo reflection coefficients, different grayscale pixels are given to the corresponding image. The stronger the echo reflection coefficient, the corresponding The whiter the pixels of the image are, the darker they are on the contrary, so as to construct a corresponding echo reflection image formed when the ultrasound scans the 3D organ;

Constructing an edge enhanced image, passing the echo reflection image through an edge highlight enhancement procedure to obtain an edge enhanced image;

Form an image with scattered light and graininess, design a scattered light background, and overlap the scattered light background and the edge-enhanced image so that a top-down scattered light is superimposed on the edge-enhanced image, and then The edge-enhanced image is superimposed on the scattered light background through a radial blur process to generate noise and graininess caused by simulated diffuse light, so that an image with scattered light and graininess is formed by overlaying the image;

To form an image with the texture of the upper skin, a background image is designed to simulate the texture of the upper skin obtained by ultrasonic scanning of the human body, and the image with scattered light and granularity is overlapped with the background image, so that the image with scattered light and granularity has the effect of the texture of the upper skin produced by ultrasonic scanning, thereby forming an image with the texture of the upper skin;

Generate an ultrasonic scan image, fan the image with upper skin texture, and design a fan-shaped mask to crop the fanned image with upper skin texture. After cropping, the image with upper skin texture is obtained in virtual reality. Ultrasound scans instant images of human body organs.

The method for rendering real-time ultrasound images for virtual reality as described above, wherein the 3D organ model defined by the software also includes at least one lesion.

The rendering method of real-time ultrasound images for virtual reality as described above, wherein the edge highlight enhancement process is to obtain the echo reflection image by convolution operation.

In the rendering method of real-time ultrasound images for virtual reality as described above, the scattered light background is obtained by converting the pixel coordinates of a rectangular body with a sphere inside into a polar coordinate system (r, θ), and then calculating the total opacity of each pixel rolled up, and then inverting to obtain the scattered light background simulating the ultrasound shadow effect.

The method for rendering real-time ultrasound images in virtual reality as described above, wherein the radial blur procedure is to obtain the adjacent pixel coordinates (r) of the image after the edge-enhanced image overlaps the scattered light background ,θ)=(r,θ±l/2) average value.

As described above, the method for rendering real-time ultrasound images in virtual reality, wherein the background image is a black image that is filtered by a tanh function to obtain an upper white image, and then the upper white image is obtained. The image is noisy, that is, the background image is noisy and has a white upper part.

S1: Construct echo reflection image

S2: Construct edge enhancement image

S3: Forming images with scattered light and granularity

S4: Creates an image with the texture of upper skin

S5: Generate ultrasound scan image

A: Liver cyst

B: Stones

1: Hand lever

2:Pillow

Figure 1: A flowchart of the processing steps of the method for rendering real-time ultrasound images in virtual reality according to the present invention.

The second picture is a schematic diagram of the abdominal 3D organ model rendered by a transparent rendering shader and represented by alpha value, forming a perspective organ echo reflection image.

The third figure: A rectangular body is used as a human body and a sphere is used as an organ to illustrate the edge-enhanced image obtained after the edge highlight is enhanced.

Figure 4: Reveals the schematic diagram of the scattered light background obtained by converting the pixel coordinates of the echo reflection image into a polar coordinate system, then calculating the opacity of each pixel in the polar coordinate system and convolving it upward, and then inverting it.

Figure 5: A schematic diagram showing the image with scattered light and graininess obtained by overlaying the edge-enhanced image on the scattered light background and processing it with a radial blurring process.

Figure 6: A schematic diagram showing a black image that is filtered through a tanh function to obtain an image with a white upper portion, and then the image with a white upper portion is noised to generate a background image with noise and a white upper portion.

Figure 7: A schematic diagram of an image with the texture of the upper skin layer formed by overlaying the image with scattered light and granularity on the background image.

Figure 8: A schematic diagram of real-time ultrasonic scanning images after fanning the upper skin texture image and then cutting it with a fan mask to form a fan shape.

Figure 9: A schematic diagram showing the final ultrasonic scanning image produced after designing liver cyst lesions on the liver model.

Figure 10: Figure 10 shows the final ultrasound scan image after designing the stone lesion on the gallbladder model.

Figure 11: A user entering a virtual reality scene of ultrasound scanning shows a user holding the hand control lever in reverse and touching the surface of a pillow in a real environment.

In order to have a more complete and clear disclosure of the technical content, the purpose of the invention and the effects achieved by the present invention, they are described in detail below, and please refer to the disclosed drawings and drawing numbers:

Please refer to the first figure, which is a flowchart of the processing steps of the rendering method of the real-time ultrasonic image for virtual reality of the present invention.

First, in order to obtain an ultrasound image for ultrasonic scanning of human organs in virtual reality, a 3D organ model must be defined through software. However, the modeling of the 3D organ model is not a feature of this case, and software 3D modeling is an existing technology, such as 3ds Max, which is commonly used for 3D modeling of the human body or organs.

The rendering method of real-time ultrasound images for virtual reality of the present invention, in sequence It includes the following processing steps: constructing an echo reflection image S1, constructing an edge-enhanced image S2, forming an image with scattered light and graininess S3, forming an image with upper skin texture S4, and generating an ultrasonic scanning image S5; among which:

In step S1 of constructing echo reflection image, a transparent rendering shader is applied to a 3D organ model defined by software, and is expressed by alpha value to generate an organ echo reflection image with a sense of perspective; wherein, the transparent rendering shader renders the 3D organ model according to the principle of coloring the 3D organ model, which is to give different grayscale pixels according to the different echo reflection coefficients generated by the overlap, material and position of the 3D organ model when the 3D organ model is scanned by ultrasound. When the echo reflection coefficient is stronger, the pixel is whiter, and vice versa, the pixel is blacker, so as to construct an echo reflection image with a sense of perspective corresponding to the formation of the 3D organ when the ultrasound scan is simulated. The present invention discloses abdominal organs as an example in the drawings, especially the liver and gallbladder. As shown in the second figure, the abdominal 3D organ model is rendered by a transparent rendering shader and represented by an alpha value, forming a schematic diagram of the organ echo reflection image with a sense of perspective.

Among them, the following rules can be summarized for internal organs and ultrasound scanning imaging:

(1) Liquid and air have almost no echo, so the ultrasonic scanning image is dark.

(2) When moving from one tissue to another, if the material difference is large, ultrasound scanning imaging will produce edge specular reflections.

(3) The uneven surface of the organ will cause sound wave diffusion, resulting in noise and granularity in the ultrasound scan image.

(4) Every time the sound wave passes through a layer of reflective material, its intensity weakens, and a shadow is produced in the ultrasound scanning image.

(5) The tissue closest to the sensor has the strongest echo, and the intensity gradually weakens as the penetration goes deeper (attenuation).

The transparent rendering shader of the present invention renders the 3D organ model based on the above-mentioned characteristics of ultrasound scanning imaging and gives different grayscale pixels according to the different echo reflection coefficients generated by the overlap, material and position of the 3D organ model.

In the step S2 of constructing the edge-enhanced image, the echo reflection image obtained in the previous step is passed through an edge highlight enhancement process to obtain an edge-enhanced image. Among them, the edge highlight enhancement program is obtained by convolving the echo reflection image. Specifically, it is assumed that the source of the ultrasound is above the imaging, so the color change of each pixel and the pixel above it is sampled to form an intensity map, and then the resolution is reduced to blur it for easier observation. In order to facilitate the explanation of the processing process of the present invention below, the human body is simplified into a rectangular body and the organs are simplified into spherical bodies to simulate the following steps. Please refer to the third picture, which is a perspective echo reflection image processed by the step S1 of constructing an echo reflection image of a rectangular body (i.e., the human body) and a spherical body (i.e., an organ), and then the image is enhanced through edge highlighting. , a schematic diagram of the obtained edge enhancement image.

In step S3 of forming an image with scattered light and granularity, a scattered light background is first designed. The design method of the scattered light background is to convert the pixel coordinates of the echo reflection image obtained in step 1 into a polar coordinate system (r, θ), then calculate the opacity of each pixel in the polar coordinate system, and then perform an upward convolution operation. After inversion, a scattered light background simulating the ultrasonic shadow effect can be obtained. Then, the obtained scattered light background and the edge enhanced image formed in the previous step are overlapped with each other, so that a scattered light from top to bottom is superimposed on the edge enhanced image, and then the image after the edge enhanced image is overlapped with the scattered light background is passed through a radial blurring process, so that the image generates noise and granularity caused by simulating diffuse light, and thus an image with scattered light and granularity is formed. The radial blurring process takes the average value of the neighboring pixel coordinates (r, θ) = (r, θ ± 1/2) and outputs the radial blurring effect, so that the edge-enhanced image is superimposed on the scattered light background to produce noise and graininess. As shown in the fourth figure, it is a schematic diagram of the scattered light background obtained after the pixel coordinates of the echo reflection image processed by step 1 are converted into a polar coordinate system, and then the opacity of each pixel in the polar coordinate system is calculated and the upward convolution operation is performed, and the inversion is obtained; the fifth figure is a schematic diagram of the image with scattered light and graininess obtained after the edge-enhanced image is superimposed on the scattered light background and processed by the radial blurring process.

In step S4 of forming an image with upper skin texture, a background image is designed to simulate the upper skin texture obtained by ultrasonic scanning of a human body, and the image with scattered light and granularity obtained in the previous step is overlapped with the background image, so that the image with scattered light and granularity has the effect of the upper skin texture produced by ultrasonic scanning, thereby forming an image with upper skin texture. Among them, the design method of the background image is to take a black image and pass it through a tanh function filter to obtain an upper white image, and then the upper white image is noised, so as to obtain a background image with noise and upper white, as shown in Figure 6. Please refer to Figure 7, which is a schematic diagram of the image with upper skin texture formed by overlaying the image with scattered light and graininess in Figure 5 with the background image designed in this step.

In step S5 of generating an ultrasonic scan image, the image with upper skin texture is sectorized, and a sector mask is designed. Then, the sectorized image with upper skin texture is sector-cut using the sector mask. After cutting, a sector-shaped real-time ultrasonic scan image of a human organ is obtained. The eighth figure shows a schematic diagram of a real-time ultrasonic scan image formed by sectorizing the image with upper skin texture in the seventh figure and then cutting it with a sector mask to form a sector-shaped real-time ultrasonic scan image.

Through the above processing steps, when operating ultrasound to scan human organs in virtual reality, an image that is no different from the ultrasound image produced by physical ultrasound scanning of human organs can be obtained, providing a way to scan human organs without real data. Achieve realistic operation simulation technology, allowing users to conduct organ theory and ultrasound scanning training in a less-cost virtual reality, instead of using expensive ultrasound equipment and actual human bodies for training.

In addition, in order to further train students to identify diseases, at least one lesion can be designed on the 3D organ model defined by the software, and after the steps of constructing an echo reflection image, constructing an edge enhancement image, forming an image with scattered light and granularity, forming an image with upper skin texture, and generating an ultrasound scan image, the 3D organ model with the lesion displays the lesion on the organ in the generated ultrasound scan image. For example, the ninth figure shows the final ultrasound scan image after designing a liver cyst lesion on the liver model. The component symbol A in the figure represents the liver cyst found under the ultrasound scan image. The tenth figure shows the final ultrasound scan image after designing a gallbladder lesion. The component symbol B in the figure represents the gallstone found under the ultrasound scan image. In addition, the position of the lesion can be moved at any time. It only needs to adjust the position directly on the 3D organ model defined by the software. The operation is quite simple. Therefore, a variety of implementation examples can be provided for students to practice and test.

Furthermore, when operating the virtual reality system for ultrasonic organ scanning developed by the method of the present invention, in order to make the hand operating rod more consistent with the actual holding state of the ultrasonic probe, the user is required to hold the hand operating rod in an upside-down manner. At the same time, in order to simulate the elastic effect of the ultrasonic probe contacting the human body, a pillow is provided for the user to hold the hand operating rod and contact it, instead of operating the hand operating rod in the air. See Figure 11, which is a schematic diagram of an ultrasonic scanning simulation in which a user who enters the virtual reality scene of ultrasonic scanning holds the hand operating rod 1 in reverse and contacts the surface of the pillow 2 in the actual environment.

The above examples are only some of the embodiments of the present invention and are not intended to limit the present invention. Any slight changes and modifications based on the creative spirit and characteristics of the present invention should also be included in the scope of this patent.

To sum up, the embodiments of the present invention can indeed achieve the expected effects, and the specific technical means disclosed have not only not been seen in similar products, but have also not been disclosed before the application, and they are fully in compliance with the patent law. According to the regulations and requirements, if you submit an application for an invention patent in accordance with the law, it will be very convenient for you to be reviewed and granted a patent.

S1: Construct echo reflection image

S2: Construct edge-enhanced images

S3: Form images with scattered light and graininess

S4: Form an image with upper skin texture

S5: Generate ultrasound scan image

Claims (8)

A rendering method for real-time ultrasound images in virtual reality, including the following steps: To construct an echo-reflection image, a transparent rendering shader is applied to a 3D organ model defined by software and expressed with an alpha value, thereby generating a corresponding organ echo-reflection image; wherein, the coloring principle of the transparent rendering shader is based on When ultrasound scans a 3D organ model, the echo reflection coefficients are different due to different materials, positions, and overlapping states. According to the different echo reflection coefficients, different grayscale pixels are given to the corresponding image. The stronger the echo reflection coefficient, the corresponding The whiter the pixels of the image are, the darker they are on the contrary, so as to construct a corresponding echo reflection image that is formed when the ultrasound scans the 3D organ; Construct an edge-enhanced image, and pass the echo reflection image through an edge highlight enhancement process to obtain an edge-enhanced image; To form an image with scattered light and granularity, a scattered light background is designed, and the scattered light background and the edge-enhanced image are overlapped, so that a scattered light from top to bottom is superimposed on the edge-enhanced image, and then the image after the edge-enhanced image is overlapped on the scattered light background is subjected to a radial blurring process to generate noise and granularity caused by simulated diffuse light, so as to form an image with scattered light and granularity by overlapping; To form an image with the texture of the upper skin, a background image is designed to simulate the texture of the upper skin obtained by ultrasonic scanning of the human body, and the image with scattered light and granularity is overlapped with the background image, so that the image with scattered light and granularity has the effect of the texture of the upper skin produced by ultrasonic scanning, thereby forming an image with the texture of the upper skin; Generate an ultrasonic scanning image, fan out the image with upper skin texture, and design a fan-shaped mask to cut the fan-shaped image with upper skin texture. After cutting, a real-time image of a human organ scanned by ultrasound in a virtual reality is obtained. As described in claim 1, the rendering method of real-time ultrasound images for virtual reality, wherein the 3D organ model defined by the software also includes at least one lesion, and the shape and/or position of the lesion can change at any time. A method for rendering real-time ultrasound images for virtual reality as described in claim 1 or 2, wherein the edge highlight enhancement process is performed by convolution operation on the echo reflection image. The rendering method of real-time ultrasound images for virtual reality as described in claim 3, wherein the pixel coordinates of the echo reflection image are converted into a polar coordinate system (r, θ), and then the opacity of each pixel is calculated and convolution is performed upward, and after inversion, the scattered light background simulating the ultrasound shadow effect is obtained; the radial blurring process is to take the average value of the neighboring pixel coordinates (r, θ) = (r, θ ± l/2) of the image after the edge enhancement image is overlapped with the scattered light background. The method for rendering real-time ultrasound images for virtual reality as described in claim 4, wherein the background image is a black image that is filtered by a tanh function to obtain an upper white image, and then the background image is The upper part of the white image is noisy, that is, the background image with noise and the upper part of the image is white. A method for rendering real-time ultrasound images for virtual reality as described in claim 1 or 2, wherein the scattered light background is obtained by converting the pixel coordinates of the echo reflection image into a polar coordinate system (r, θ), calculating the opacity of each pixel and performing an upward convolution operation, and obtaining the scattered light background simulating the ultrasound shadow effect after inversion; the radial blurring procedure is to take the average value of the neighboring pixel coordinates (r, θ) = (r, θ ± l/2) of the image after the edge-enhanced image is overlaid with the scattered light background. As described in claim 6, the rendering method of real-time ultrasound images for virtual reality, wherein the background image is a black image that is filtered through a tanh function to obtain an image with a white upper portion, and then the image with a white upper portion is noised to obtain the background image with noise and a white upper portion. A method for rendering real-time ultrasound images for virtual reality as described in claim 1 or 2, wherein the background image is a black image that is filtered through a tanh function to obtain an image with a white top, and then the image with a white top is noised to obtain the background image with noise and a white top.

Images