WO2025204551A1 - Information processing device, information processing method, and program - Google Patents

Abstract

The present technology pertains to an information processing device, an information processing method, and a program that make it possible for an operator to accurately perform catheter shaping. The information processing device of the present technology comprises a display control unit that causes a display device capable of displaying a 3D image comprising an image for left-eye and an image for right-eye having parallax with respect to each other to display error information indicating a difference between a target shape serving as a target for catheter shaping and an actual shape of a catheter, and a 3D model referred to by a user during catheter shaping. The present technology can be applied, for example, to a catheter shaping support system.

Description

Information processing device, information processing method, and program

This technology relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that enable surgeons to perform catheter shaping with high precision.

Catheter shaping, in which the surgeon shapes the catheter (tip) to fit the shape of the patient's blood vessels, is commonly performed.

Traditionally, surgeons would refer to a 3D vascular model displayed on a 2D monitor to shape the catheter into the desired shape, so catheter shaping required a high level of spatial awareness on the part of the surgeon. A 3D vascular model is a 3D model that shows the three-dimensional shape of the patient's blood vessels.

For example, Non-Patent Document 1 proposes technology to assist surgeons in catheter shaping. The technology proposed in Non-Patent Document 1 allows the surgeon to set the contact points between the inside of the blood vessel wall and the catheter, which are important in catheter shaping, using an operating pen, and then displays a 3D model of the target shape of the catheter generated based on the set contact points on a 2D monitor, thereby promoting understanding of the target shape.

Nagano Yoshitaka, Miyaji Shigeru, Izumi Takatsugu, and Oshima Tomoki, "Development of Shape Design Application for Catheters for Neuroendovascular Treatment," Design Studies Research (2020), Japanese Society for the Science of Design, pp. 242-243

Whether the catheter has been shaped to the target shape is determined by the surgeon, and the accuracy of this judgment depends on the surgeon's level of skill. For surgeons who are not skilled in catheter shaping, it is difficult to accurately determine the deviation between the target shape and the shape of the actual catheter.

This technology was developed in light of these circumstances, allowing surgeons to perform catheter shaping with precision.

An information processing device according to one aspect of this technology includes a display control unit that displays error information indicating the difference between the target shape of catheter shaping and the shape of the actual catheter, as well as a 3D model referenced by the user during catheter shaping, on a display device capable of displaying 3D images consisting of left-eye and right-eye images with mutual parallax.

An information processing method according to one aspect of this technology displays error information indicating the difference between the target shape of catheter shaping and the shape of the actual catheter, as well as a 3D model referenced by the user during catheter shaping, on a display device capable of displaying 3D images consisting of left-eye and right-eye images with mutual parallax.

A program according to one aspect of this technology causes a computer to execute a process that displays error information indicating the difference between the target shape of catheter shaping and the shape of the actual catheter, as well as a 3D model referenced by the user during catheter shaping, on a display device capable of displaying 3D images consisting of left-eye and right-eye images with mutual parallax.

In one aspect of this technology, control is performed to display error information indicating the difference between the target shape for catheter shaping and the shape of the actual catheter, as well as a 3D model referenced by the user during catheter shaping, on a display device capable of displaying 3D images consisting of left-eye and right-eye images with mutual parallax.

FIG. 1 is a block diagram illustrating an example configuration of a catheter shaping assistance system according to an embodiment of the present technology. FIG. 1 is a diagram showing an example of a display surface of a spatial reproduction display. FIG. 10 is a first diagram showing another example of the display surface of the spatial reproduction display. FIG. 2 is a second diagram showing another example of the display surface of the spatial reproduction display. FIG. 10 is a diagram illustrating a display example of a 3D model. FIG. 10 is a diagram showing an example of the placement position of a 3D model relative to an actual catheter. FIG. 10 is a diagram illustrating an example of a screen displayed on a display device. FIG. 10 is a diagram illustrating details of error information. 10 is a flowchart illustrating processing performed by an information processing device. 10A and 10B are diagrams for explaining details of the process of acquiring the shape of the catheter and the process of calculating the error from the target shape. FIG. 10 is a diagram illustrating another example of the configuration of the catheter shaping support system. 10A and 10B are diagrams illustrating an example of a method by which an information processing device acquires the three-dimensional shape of an actual catheter. FIG. 10 is a diagram showing another display example of a 3D model. FIG. 10 is a diagram showing another example of the placement position of the 3D model relative to the actual catheter. 10 is a flowchart illustrating processing performed by an information processing device when a 3D model is placed at a position different from that of an actual catheter. FIG. 1 is a diagram showing an example of the size of an actual catheter and a 3D blood vessel model. 1A and 1B are diagrams illustrating application examples of a display device according to the present technology. FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer.

Hereinafter, embodiments of the present technology will be described in the following order.
1. Embodiment of the present technology 2. Modification

1. Embodiments of the present technology
Fig. 1 is a block diagram showing an example of the configuration of a catheter shaping support system according to an embodiment of the present technology. The catheter shaping support system in Fig. 1 is a system that supports catheter shaping, in which a surgeon shapes a catheter (a distal end portion thereof) to match the shape of a patient's blood vessel.

The catheter shaping support system in Figure 1 is composed of a DICOM (Digital Imaging and Communications in Medicine) server 1, a display device 2, and an information processing device 3.

DICOM Server 1 is a server that manages DICOM data such as CT (Computed Tomography) images and MRI (Magnetic Resonance Imaging) images taken before catheter treatment. DICOM data includes size information that indicates the size of the subject (part of the patient's body) shown in the CT image, MRI image, etc.

Display device 2 displays a 3D image consisting of a left-eye image and a right-eye image with parallax between them, allowing the user to visually recognize virtual 3D objects placed in a specified space as three-dimensional objects. Display device 2 is, for example, a naked-eye 3D display (spatial reproduction display) that can present 3D images without the use of dedicated eyewear.

The display device 2 is equipped with an imaging unit 11 that is used, for example, to detect the surgeon's pupils and the catheter. The display device 2 is installed, for example, so that the display surface 12 is perpendicular to the horizontal plane in real space, or so that the display surface 12 faces diagonally upward relative to the horizontal plane in real space.

The information processing device 3 is a device that controls the entire catheter shaping support system of this technology, and is, for example, a computer with an information processing circuit equipped with a CPU, memory, etc. For example, the information processing device 3 controls the display on the display device 2, thereby presenting a 3D model to be referenced by the surgeon (user) during catheter shaping. Here, a 3D model showing the three-dimensional shape of a part of the patient's body, for example a 3D vascular model that is a 3D model showing the three-dimensional shape of the surface or cross-section of the patient's blood vessels, is presented.

The information processing device 3 is composed of an image processing unit 21, a 3D processing unit 22, an error calculation unit 23, a superimposition processing unit 24, and a display control unit 25.

The image processing unit 21 acquires the captured image taken by the image capture unit 11 of the display device 2.

The image processing unit 21 is composed of a pupil detection unit 31 and a catheter detection unit 32.

The pupil detection unit 31 detects the pupils of the surgeon from the captured image captured by the imaging unit 11, and detects the surgeon's viewpoint. Here, the viewpoint includes the viewpoint position, line of sight, and the distance between the eyes. The pupil detection unit 31 supplies the detection result of the surgeon's viewpoint to the 3D processing unit 22.

The catheter detection unit 32 detects the actual catheter formed by the surgeon from the image captured by the imaging unit 11 and obtains the shape of the actual catheter. The catheter detection unit 32 supplies information indicating the shape of the actual catheter to the error calculation unit 23.

The 3D processing unit 22 acquires DICOM data from the DICOM server 1 and generates, for example, a 3D blood vessel model based on the DICOM data. The 3D processing unit 22 calculates the target shape for catheter shaping based on the 3D blood vessel model and generates a 3D catheter model, which is a 3D model that indicates the target shape. Note that information indicating the target shape may also be acquired from another device (for example, the DICOM server 1).

The 3D processing unit 22 places the 3D blood vessel model and the 3D catheter model in a virtual space. The 3D processing unit 22 generates a 3D image by rendering the virtual space based on the surgeon's viewpoint detected by the pupil detection unit 31. For example, the 3D processing unit 22 generates a 3D image so that the 3D blood vessel model and the 3D catheter model are visible appropriately from the user's viewpoint. Here, the placement and orientation of the 3D blood vessel model and the 3D catheter model are controlled according to the operations input by the surgeon using an input device (not shown) or gestures.

The 3D processing unit 22 supplies the generated 3D catheter model to the error calculation unit 23, and supplies the generated 3D image to the superposition processing unit 24.

The error calculation unit 23 calculates the difference between the target shape represented by the 3D catheter model supplied from the 3D processing unit 22 and the shape of the actual catheter detected by the catheter detection unit 32. The error calculation unit 23 generates error information to provide visual feedback to the surgeon of the difference between the target shape and the shape of the actual catheter. The error calculation unit 23 supplies the error information to the superposition processing unit 24.

The superimposition processing unit 24 superimposes the error information supplied from the error calculation unit 23 onto the 3D image supplied from the 3D processing unit 22. The superimposition processing unit 24 supplies the 3D image with the error information superimposed to the display control unit 25.

The display control unit 25 controls the display by the display device 2. Specifically, the display control unit 25 supplies the 3D image supplied from the superimposition processing unit 24 to the display device 2, and displays the left-eye image and right-eye image that make up the 3D image. The left-eye image is delivered to the user's left eye, and the right-eye image is delivered to the user's right eye, thereby presenting, for example, a 3D blood vessel model, a 3D catheter model, and error information to the user.

The display device 2 and the information processing device 3 may also be configured as a single device.

Next, we will explain the spatial reproduction display used as the display device 2 using Figures 2 to 4.

In recent years, naked-eye 3D displays have been proposed as spatial reproduction displays that can display content in 3D without the need for specialized eyewear. Such spatial reproduction displays can display images that are shifted horizontally for each viewing angle, allowing the user to perceive depth due to the shift (parallax) between the image for the left eye and the image for the right eye.

Figure 2 shows an example of the display surface of a spatial reproduction display.

As described above, the spatial reproduction display is positioned, for example, so that the display surface 12 faces diagonally upward relative to the horizontal plane in real space. In this case, the spatial reproduction display reproduces a virtual space VW that intersects with the display surface 12, in other words, has areas both in front of and behind the display surface 12 when viewed from the surgeon's viewpoint V.

3D objects placed in the virtual space VW are displayed so that, for example, some of them appear to protrude from the display surface 12. The spatial reproduction display can express virtual depth by, for example, delivering different images to the user's left and right eyes, giving the user the sensation that the 3D objects are present in real space. By positioning the display surface 12 so that it faces diagonally upward relative to the horizontal plane in real space, the spatial reproduction display can easily reproduce the virtual space VW.

Figures 3 and 4 show other examples of the display surface of a spatial reproduction display.

As shown in Figures 3 and 4, the spatial reproduction display may be positioned, for example, so that the display surface 12 is perpendicular to a horizontal plane in real space.

In this case, as shown in Figure 3, for example, the 3D object O11 is placed near the display surface 12 (the front part of the 3D object O11 is placed on the front side of the display surface 12, and the back part is placed on the back side of the display surface 12).

Furthermore, as shown in Figure 4, for example, the entire 3D object O11 is placed on the front side of the display surface 12.

Figure 5 shows an example of a 3D model display.

As shown in Figure 5, the 3D catheter model M1 is displayed so that the entire model appears to pop out from the display surface 12 of the display device 2. Based on the size information contained in the DICOM data, the information processing device 3 displays the life-size 3D blood vessel model and 3D catheter model so that they are perceived in the optimal position according to the specifications of the display device 2.

For example, the surgeon brings the actual catheter C1 close to the displayed 3D catheter model M1 and shapes the actual catheter C1 to match the shape of the 3D catheter model M1.

Figure 6 shows an example of the placement position of the 3D model relative to the actual catheter.

As mentioned above, the surgeon performs catheter shaping by bringing the actual catheter C1 close to the 3D catheter model M1, so as shown in Figure 6, the 3D catheter model M1 is placed in approximately the same position as the actual catheter C1.

Figure 7 shows an example of a screen displayed on the display device 2.

When the surgeon performs catheter shaping, the display device 2 displays a target shape presentation screen, for example, as shown in Figure 7A. In the following, the surface visible from the imaging unit 11 (the surface reflected in the captured image) is referred to as the back surface of the actual catheter.

On the left side of the target shape presentation screen, for example, a 3D blood vessel model M11 and a 3D catheter model M12 are displayed, arranged in virtual space with their front surfaces (the surface opposite the back surface) facing the front of the display surface 12. On the target shape presentation screen, the point of contact Po1 between the blood vessel wall of the 3D blood vessel model M11 and the 3D catheter model M12 is indicated by, for example, a hollow dot.

Furthermore, on the right side of the target shape presentation screen, a 3D blood vessel model M11 and a 3D catheter model M12 are displayed, for example, rotated 90 degrees horizontally from a state in which their front faces face the front of the display surface 12 and placed in virtual space. On the target shape presentation screen, points of contact Po2 to Po4 between the blood vessel wall of the 3D blood vessel model M11 and the 3D catheter model M12 are indicated, for example, by hollow dots.

In this way, the target shape presentation screen presents the front and side shapes of the 3D blood vessel model M11 and the 3D catheter model M12 side by side to the surgeon. The surgeon shapes the actual catheter (not shown) while looking at the target shape presentation screen.

After catheter shaping is performed, the display device 2 displays a feedback screen, such as that shown in Figure 7B.

On the left side of the feedback screen, a 3D blood vessel model M11 and a 3D catheter model M12 are displayed, arranged in virtual space with their front surfaces facing the front of the display surface 12, just like the target shape presentation screen.

Furthermore, on the right side of the feedback screen, error information E1 and catheter image P1 are superimposed on a 3D blood vessel model M11 and a 3D catheter model M12, which are placed in virtual space with their front surfaces facing the front of the display surface 12. The catheter image P1 is an image obtained by cutting out the area showing the actual catheter from an image captured by the imaging unit 11 of the actual catheter after catheter shaping.

Figure 8 is a diagram explaining the details of error information E1.

In the example of Figure 8, the error information E1 is information (color map) that indicates, by color, the magnitude of the difference between the target shape (the shape of the front surface of the 3D catheter model M12) and the shape of the front surface of the actual catheter. For example, the error information E1 is displayed so that the catheter image P1 is outlined in a color that corresponds to the magnitude of the difference between the target shape and the shape of the actual catheter. In the error information E1 of Figure 8, the greater the difference, the darker the color.

By simply looking at the error information E1, the surgeon can see that the error from the target shape is particularly large in the central part of the catheter, shown circled in gray in Figure 8.

Traditionally, surgeons would refer to a 3D vascular model displayed on a 2D monitor while shaping the actual catheter into the desired shape, so catheter shaping required a high level of spatial awareness on the part of the surgeon.

In the catheter shaping support system of this technology, a 3D blood vessel model and a 3D catheter model are displayed on a display device 2 capable of displaying 3D images, allowing the surgeon to intuitively grasp the target shape.

Conventionally, whether or not the actual catheter has been shaped to the target shape is determined by the surgeon, and the accuracy of this judgment depends on the surgeon's level of skill. For surgeons who are not skilled in catheter shaping, it is difficult to accurately determine the deviation between the target shape and the shape of the actual catheter.

With this technology's catheter shaping system, the surgeon can view the error information and intuitively grasp the error between the target shape and the actual catheter shape. By switching between the target shape presentation screen and the feedback screen while shaping the catheter, the surgeon can easily shape the catheter to the target shape.

Because surgeons can easily shape catheters to the desired shape, this catheter shaping support system can contribute to shortening the time required for catheter shaping. Because surgeons can accurately shape catheters, this catheter shaping support system can contribute to improving patient safety during catheter treatment.

The feedback screen quantitatively displays the accuracy of catheter shaping, making this catheter shaping support system suitable for educational and training purposes.

In addition to the imaging unit 11 mounted on the display device 2, a camera capable of capturing images of the actual catheter from the side may be connected to the information processing device 3. In this case, error information indicating the difference between the target shape (the shape of the side of the 3D catheter model M12) and the shape of the side of the actual catheter is generated based on the image captured by the camera. This error information is displayed, for example, on the feedback screen, superimposed on the 3D blood vessel model and 3D catheter model arranged in virtual space with their sides facing the front of the display surface 12.

The surgeon can switch between displaying front-side error information and side-side error information on the feedback screen as needed.

Next, the processing performed by the information processing device 3 will be described with reference to the flowchart in Figure 9.

In step S1, the 3D processing unit 22 acquires DICOM data from the DICOM server 1. The 3D processing unit 22 generates a 3D blood vessel model and a 3D catheter model based on the DICOM data.

In step S2, the pupil detection unit 31 detects the pupils of the surgeon from the image captured by the imaging unit 11 and detects the surgeon's viewpoint.

In step S3, the display control unit 25 displays the 3D models (3D blood vessel model and 3D catheter model) on the display device 2. Specifically, the 3D processing unit 22 generates a 3D image by rendering the virtual space in which the 3D models are placed, based on the surgeon's viewpoint detected by the pupil detection unit 31, and the display control unit 25 displays the 3D image generated by the 3D processing unit 22 on the display device 2. Here, a target shape presentation screen is displayed on the display device 2.

The surgeon performs catheter shaping while viewing the 3D model displayed on display device 2.

In step S4, the catheter detection unit 32 detects the actual catheter from the image captured by the imaging unit 11 and acquires the shape of the actual catheter.

In step S5, the error calculation unit 23 calculates the difference (error) between the target shape and the shape of the actual catheter detected by the catheter detection unit 32, and generates error information indicating this difference.

Figure 10 is a diagram explaining the details of the process for obtaining the catheter shape and the process for calculating the error from the target shape.

First, as shown in A of Figure 10, the catheter detection unit 32 performs image processing to cut out (extract) the area in which the catheter C1 appears from the captured image P11 of the actual catheter C1, thereby generating a catheter image P12. The catheter detection unit 32 treats the shape of this catheter image P12 as the shape of the front surface of the actual catheter C1. In reality, the imaging unit 11 captures the back surface of the actual catheter C1, so the catheter detection unit 32 flips the left and right of the catheter image P12.

Next, the error calculation unit 23 aligns the catheter image P12 with the 3D catheter model M1 viewed from the front side. Specifically, as shown in FIG. 10B, the error calculation unit 23 detects feature points Po11 and Po12 from the catheter image P12 and the 3D catheter model M1, respectively, and aligns the catheter image P12 with the 3D catheter model M1 based on the feature points Po11 and Po12. Here, parts with characteristic shapes, such as points of contact with the blood vessel wall, are detected as feature points.

Note that alignment markers may be added to the actual catheter C1. In this case, the error calculation unit 23 can align the catheter image P12 and the 3D catheter model M1 based on the markers.

Next, the error calculation unit 23 calculates the difference between the shape of the catheter image P12 and the shape of the 3D catheter model M1 as the difference between the shape of the front surface of the actual catheter C1 and the target shape.

Returning to the explanation of Figure 9, in step S6, the display control unit 25 causes the display device 2 to display the error information. Specifically, the superimposition processing unit 24 superimposes the error information on the 3D image, and the display control unit 25 causes the display device 2 to display the 3D image on which the error information has been superimposed. Here, a feedback screen is displayed on the display device 2.

In step S7, the error calculation unit 23 determines whether the error between the target shape and the actual catheter shape is within an acceptable range.

If it is determined in step S7 that the error between the target shape and the actual catheter shape is not within the allowable range, processing returns to step S4, and subsequent processing is performed. In other words, the target shape presentation screen is displayed, and the surgeon performs catheter shaping again.

On the other hand, if it is determined in step S7 that the error between the target shape and the actual catheter shape is within the allowable range, processing ends.

As described above, the information processing device 3 of the present technology controls the display device 2, which is capable of displaying 3D images, to display error information indicating the difference between the target shape for catheter shaping and the shape of the actual catheter, as well as a 3D model referenced by the user during catheter shaping. By performing catheter shaping while viewing the 3D model and error information displayed on the display device 2, the surgeon can accurately shape the actual catheter to the target shape.

2. Modified Examples
Example of Using a Back Camera and a Side Camera FIG. 11 is a diagram showing another example of the configuration of a catheter shaping support system.

As shown in Figure 11, the back side of the actual catheter may be photographed by a back side camera 11A mounted on the display device 2, and the side side of the actual catheter may be photographed by a side side camera 11B provided separately outside the display device 2.

In this case, on the left side of the feedback screen, a catheter image P1 and front surface error information E1 are displayed superimposed on a 3D blood vessel model M11 and a 3D catheter model M12, which are placed in virtual space with their front surfaces facing the front of the display surface 12. The catheter image P1 is an image showing the shape of the front surface of an actual catheter. The front surface error information E1 is information indicating the magnitude of the difference between the target shape (the shape of the front surface of the 3D catheter model M12) and the shape of the front surface of the actual catheter.

Also, on the right side of the feedback screen, a catheter image P2 and side error information E2 are displayed superimposed on a 3D blood vessel model M11 and a 3D catheter model M12, which are arranged in virtual space with their sides facing the front of the display surface 12. The catheter image P2 is an image showing the shape of the side of the actual catheter. The side error information E2 is information indicating the magnitude of the difference between the target shape (the shape of the side of the 3D catheter model M12) and the shape of the side of the actual catheter.

Example of Acquiring a Three-Dimensional Shape of an Actual Catheter FIG. 12 is a diagram showing an example of a method in which the information processing device 3 acquires a three-dimensional shape of an actual catheter.

As shown by the arrow in Figure 12, the imaging unit 11 captures multiple images while the surgeon rotates the actual catheter C1 once horizontally in front of the imaging unit 11. The multiple images are images of various surfaces of the actual catheter C1.

The catheter detection unit 32 of the information processing device 3 generates a 3D model that shows the three-dimensional shape of the actual catheter C1 based on multiple images. Since the shape of the actual catheter C1 as seen from all viewpoints, not just the front and side shapes of the actual catheter C1, can be obtained from the 3D model, the catheter shaping support system can present the surgeon with any deviations from the target shape as seen from various viewpoints.

Example of rotating a 3D model according to the orientation of an actual catheter FIG. 13 is a diagram showing another example of displaying a 3D model.

As shown by the solid arrow in Figure 13, when the surgeon rotates the actual catheter C1, the information processing device 3 can rotate the 3D blood vessel model M51 in the same way as the actual catheter C1, as shown by the dashed arrow in Figure 13. Note that the information processing device 3 can rotate not only the 3D blood vessel model, but also the 3D catheter model in the same way as the actual catheter C1.

When rotating the 3D vascular model M51 according to the orientation of the actual catheter C1, the information processing device 3 detects the movement of the surgeon's hand, for example, by performing hand tracking based on the captured image captured by the imaging unit 11. The information processing device 3 may also detect the movement of the surgeon's hand based on sensor data from an IMU (Inertial Measurement Unit) mounted on a device worn on the surgeon's finger, for example.

The information processing device 3 estimates the orientation of the actual catheter C1 based on the detection results of the surgeon's hand movement, and controls the orientation of the 3D vascular model M51 according to the estimated orientation of the actual catheter C1.

- Example of placing a 3D model in approximately the same position as the actual catheter Instead of the surgeon bringing the actual catheter model close to a 3D catheter model, the information processing device 3 can estimate the positional relationship between the actual catheter and the display device 2 (the position of the actual catheter) and place the 3D catheter model in approximately the same position as the actual catheter.

In this case, the information processing device 3 estimates the positional relationship between the actual catheter and the display device 2, for example, by performing three-dimensional position estimation based on the captured image (RGB image) captured by the imaging unit 11, or by using a depth camera or stereo camera other than the imaging unit 11. The information processing device 3 controls the placement position of the 3D catheter model and the like in the virtual space based on the result of estimating the positional relationship between the actual catheter and the display device 2.

Example of arranging a 3D model at a position different from that of an actual catheter FIG. 14 is a diagram showing another example of the position at which a 3D model is arranged relative to an actual catheter.

As shown in Figure 14, a 3D model such as a 3D blood vessel model M51 may be placed in a position different from the actual catheter C1.

In this case, the information processing device 3 estimates the positional relationship between the surgeon's viewpoint V and the actual catheter C1, and controls the display size of the 3D blood vessel model M51 and the like according to the distance between the surgeon's viewpoint V and the actual catheter C1. Specifically, the information processing device 3 controls the display size of the 3D model so that the size of the actual catheter C1 and the size of the 3D catheter model appear to be the same from the surgeon's perspective. For example, the closer the surgeon brings the actual catheter C1 to their eye, the larger the 3D blood vessel model M51 will be displayed. Here, the placement position of the 3D model is fixed.

The information processing device 3 estimates the positional relationship between the surgeon's viewpoint V and the actual catheter C1, for example, by performing three-dimensional position estimation based on the captured image (RGB image) captured by the imaging unit 11, or by using a depth camera or stereo camera other than the imaging unit 11.

With reference to the flowchart in Figure 15, the processing performed by the information processing device 3 when placing a 3D model in a position different from that of the actual catheter will be described.

In step S21, the 3D processing unit 22 acquires DICOM data from the DICOM server 1. The 3D processing unit 22 generates a 3D blood vessel model and a 3D catheter model based on the DICOM data.

In step S22, the pupil detection unit 31 detects the pupil of the surgeon from the image captured by the imaging unit 11 and detects the surgeon's viewpoint. Furthermore, the image processing unit 21 estimates the positional relationship between the surgeon's viewpoint and the actual catheter.

In step S23, the display control unit 25 displays the 3D models (3D blood vessel model and 3D catheter model) on the display device 2. The 3D models are displayed at a display size based on the positional relationship between the surgeon's viewpoint and the actual catheter. Here, a target shape presentation screen is displayed on the display device 2.

The surgeon performs catheter shaping while viewing the 3D model displayed on display device 2.

In step S24, the catheter detection unit 32 detects the actual catheter from the image captured by the imaging unit 11 and acquires the shape of the actual catheter.

In step S25, the catheter detection unit 32 estimates the distance between the actual catheter and the imaging unit 11 (display device 2). For example, the catheter detection unit 32 estimates the distance between the actual catheter and the imaging unit 11 by performing three-dimensional position estimation based on the captured image (RGB image) captured by the imaging unit 11, or by using a depth camera or stereo camera other than the imaging unit 11.

In step S26, the error calculation unit 23 calculates the difference (error) between the target shape and the shape of the actual catheter, and generates error information indicating this difference. Specifically, the error calculation unit 23 estimates the size of the actual catheter based on the distance to the actual catheter detected by the catheter detection unit 32, and corrects the size of the catheter image based on the estimated size of the actual catheter. The error calculation unit 23 aligns the catheter image with the 3D catheter model based on the feature points, and calculates the difference between the target shape (the shape of the 3D catheter model) and the shape of the actual catheter.

In step S27, the display control unit 25 displays the error information on the display device 2. Here, a feedback screen is displayed on the display device 2.

In step S28, the error calculation unit 23 determines whether the error between the target shape and the actual catheter shape is within an acceptable range.

If it is determined in step S28 that the error between the target shape and the actual catheter shape is not within the allowable range, processing returns to step S24, and subsequent processing is performed. In other words, the target shape presentation screen is displayed, and the surgeon performs catheter shaping again.

On the other hand, if it is determined in step S28 that the error between the target shape and the actual catheter shape is within the allowable range, processing ends.

As described above, the 3D model may be placed in a different position from the actual catheter.

An example of displaying a 3D model so that the 3D model appears larger than the actual catheter when viewed from the operator. FIG. 16 is a diagram showing an example of the sizes of an actual catheter and a 3D blood vessel model.

As shown in FIG. 16, a 3D model such as a 3D blood vessel model M51 may be displayed so that it appears larger to the surgeon than the actual catheter C1. In this case, the surgeon can freely change the display size (magnification) of the 3D model by operating a keyboard or the like. In other words, the information processing device 3 controls the display size of the 3D model in response to operations by the surgeon.

Application Example of Display Device FIG. 17 is a diagram showing an application example of the display device 2 of the present technology.

As shown in Figure 17A, the display device 2 of the present technology can be applied to a naked-eye 3D display 2A. As shown in Figure 17B, the display device 2 of the present technology can be applied to a 3D monitor 2B that can present 3D images using dedicated eyewear (polarized glasses). As shown in Figure 17C, the display device 2 of the present technology can be applied to a head-mounted display 2C that supports XR (Extended Reality).

In addition, when the display device 2 of the present technology is applied to the naked-eye 3D display 2A, the surgeon does not need to wear anything, so it is desirable that the display device 2 of the present technology be configured as the naked-eye 3D display 2A.

Regarding the computer, the above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

Figure 18 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes using a program.

The CPU (Central Processing Unit) 501, ROM (Read Only Memory) 502, and RAM (Random Access Memory) 503 are interconnected by a bus 504.

Further connected to the bus 504 is an input/output interface 505. Connected to the input/output interface 505 are an input unit 506 consisting of a keyboard, mouse, etc., and an output unit 507 consisting of a display, speakers, etc. Also connected to the input/output interface 505 are a storage unit 508 consisting of a hard disk or non-volatile memory, a communication unit 509 consisting of a network interface, etc., and a drive 510 that drives removable media 511.

In a computer configured as described above, the CPU 501 performs the above-described series of processes by, for example, loading a program stored in the storage unit 508 into the RAM 503 via the input/output interface 505 and bus 504 and executing it.

The programs executed by the CPU 501 are stored on removable media 511, or are provided via wired or wireless transmission media such as a local area network, the Internet, or digital broadcasting, and are installed in the storage unit 508.

The program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or a program in which processing is performed in parallel or at the required timing, such as when called.

In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

The effects described in this specification are merely examples and are not limiting, and other effects may also occur.

The embodiments of this technology are not limited to the above-described embodiments, and various modifications are possible without departing from the spirit of this technology.

For example, this technology can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices over a network.

Furthermore, each step described in the above flowchart can be performed by a single device, or can be shared and executed by multiple devices.

Furthermore, if one step includes multiple processes, the multiple processes included in that one step can be executed by one device, or they can be shared and executed by multiple devices.

Example of configuration combinations The present technology can also be configured as follows.

(1)
An information processing device comprising: a display control unit that displays, on a display device capable of displaying 3D images consisting of a left-eye image and a right-eye image having parallax therebetween, error information indicating the difference between the target shape of catheter shaping and the shape of the actual catheter, and a 3D model referenced by the user during said catheter shaping.
(2)
The information processing device described in (1), wherein the 3D model includes at least one of a 3D model showing the three-dimensional shape of the target shape and a 3D model showing the shape of a part of the patient's body.
(3)
The information processing device according to (1) or (2), wherein the display control unit causes the display device to display the error information superimposed on the 3D model.
(4)
The information processing device according to any one of (1) to (3), wherein the error information is information indicating the magnitude of the difference between the target shape and the shape of the actual catheter using a color.
(5)
The information processing device according to (4), wherein a catheter image of the actual catheter is displayed on the display device together with the error information and the 3D model.
(6)
The information processing device according to (5), wherein the catheter image is captured by an imaging unit mounted on the display device.
(7)
The information processing device according to (5) or (6), wherein the display control unit causes the display device to display the error information so as to outline the catheter image with a color according to the magnitude of the difference between the target shape and the shape of the actual catheter.
(8)
The information processing device according to any one of (1) to (7), further comprising a calculation unit that calculates a difference between the target shape and the shape of the actual catheter based on a catheter image obtained by photographing the actual catheter.
(9)
The information processing device described in (8), wherein the calculation unit calculates the difference between the target shape and the shape of the actual catheter based on at least one of the catheter image captured by an imaging unit mounted on the display device and the catheter image captured by an imaging unit provided separately outside the display device.
(10)
The information processing device according to (8), further comprising an imaging unit that images the actual catheter.
(11)
The information processing device according to any one of (1) to (10), further comprising the display device.
(12)
The information processing device according to any one of (1) to (11), wherein the display control unit controls an orientation of the 3D model depending on an orientation of the actual catheter.
(13)
The information processing device according to any one of (1) to (12), wherein the display control unit controls the placement position of the 3D model in a virtual space based on the position of the actual catheter.
(14)
The information processing device according to any one of (1) to (12), wherein the display control unit controls a display size of the 3D model according to a positional relationship between the user and the actual catheter.
(15)
The information processing device according to any one of (1) to (12), wherein the display control unit controls a display size of the 3D model in response to an operation by the user.
(16)
An information processing method that displays error information indicating the difference between the target shape of catheter shaping and the shape of the actual catheter, as well as a 3D model referenced by the user during catheter shaping, on a display device capable of displaying 3D images consisting of a left-eye image and a right-eye image with mutual parallax.
(17)
On the computer,
A program for executing a process to display error information indicating the difference between the target shape of catheter shaping and the shape of the actual catheter, and a 3D model referenced by the user during said catheter shaping, on a display device capable of displaying 3D images consisting of a left-eye image and a right-eye image with mutual parallax.

1 DICOM server, 2 Display device, 3 Information processing device, 11 Imaging unit, 12 Display surface, 21 Image processing unit, 22 3D processing unit, 23 Error calculation unit, 24 Superposition processing unit, 25 Display control unit, 31 Pupil detection unit, 32 Catheter detection unit

Claims (17)

An information processing device comprising: a display control unit that displays, on a display device capable of displaying 3D images consisting of a left-eye image and a right-eye image having parallax therebetween, error information indicating the difference between the target shape of catheter shaping and the shape of the actual catheter, and a 3D model referenced by the user during said catheter shaping. The information processing device according to claim 1 , wherein the 3D model includes at least one of a 3D model showing a three-dimensional shape of the target shape and a 3D model showing a shape of a part of a patient's living body. The information processing device according to claim 1 , wherein the display control unit causes the display device to display the error information superimposed on the 3D model. The information processing device according to claim 1 , wherein the error information is information that indicates the magnitude of the difference between the target shape and the shape of the actual catheter using a color. The information processing device according to claim 4 , wherein a catheter image of the actual catheter is displayed on the display device together with the error information and the 3D model. The information processing device according to claim 5 , wherein the catheter image is captured by an imaging unit mounted on the display device. The information processing device according to claim 5 , wherein the display control unit causes the display device to display the error information such that the catheter image is outlined in a color according to the magnitude of the difference between the target shape and the shape of the actual catheter. The information processing device according to claim 1 , further comprising a calculation unit that calculates a difference between the target shape and the shape of the actual catheter based on a catheter image obtained by photographing the actual catheter. The information processing device according to claim 8, wherein the calculation unit calculates a difference between the target shape and the shape of the actual catheter based on at least one of the catheter image captured by an imaging unit mounted on the display device and the catheter image captured by an imaging unit provided separately outside the display device. The information processing device according to claim 8 , further comprising an imaging unit that images the actual catheter. The information processing device according to claim 1 , further comprising the display device. The information processing device according to claim 1 , wherein the display control unit controls the orientation of the 3D model in accordance with the orientation of the actual catheter. The information processing device according to claim 1 , wherein the display control unit controls a placement position of the 3D model in the virtual space based on a position of the actual catheter. The information processing device according to claim 1 , wherein the display control unit controls a display size of the 3D model in accordance with a positional relationship between the user and the actual catheter. The information processing device according to claim 1 , wherein the display control unit controls a display size of the 3D model in response to an operation by the user. An information processing method that displays error information indicating the difference between the target shape of catheter shaping and the shape of the actual catheter, as well as a 3D model referenced by the user during catheter shaping, on a display device capable of displaying 3D images consisting of a left-eye image and a right-eye image with mutual parallax. On the computer,
A program for executing a process to display error information indicating the difference between the target shape of catheter shaping and the shape of the actual catheter, and a 3D model referenced by the user during said catheter shaping, on a display device capable of displaying 3D images consisting of a left-eye image and a right-eye image with mutual parallax.