HK1227126B - Dynamic and interactive navigation in a surgical environment - Google Patents

Description

Dynamic and interactive navigation in surgical environments

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application Serial No. 61/975,330, filed April 4, 2014, and incorporated herein by reference.

background

The present application generally relates to a system and method for displaying surgical procedures in a surgical environment to aid in surgery. More specifically, the present application relates to a system and method for converting static/still medical images of an actual surgical patient into dynamic and interactive images that interact with medical tools (such as, for example, surgical tools, probes, and/or implantable medical devices) by coupling a tissue dynamics model to the patient-specific image during the surgical procedure. The interaction with the medical tool has enabled proximity warning, which is a mechanism for providing continuous indication and warning of the tool's proximity to specific anatomical structures.

Surgeons lack surgical tools that would provide them with a realistic visual image of a surgical patient, wherein real physical tissue properties are displayed in a 3D display along with medical tools (e.g., surgical tools, probes, and/or implantable medical devices). A surgical tool comprising: (i) a realistic, "lifelike" 2D and/or 3D display of a patient-specific surgical region (e.g., an aneurysm); (ii) modeling of the local patient-specific surgical region geometry and physical properties; (iii) an interface that enables manipulation of the patient-specific surgical model region and displays surgical actions such as cutting, shifting, and clamping; and (iv) an interface that provides feedback cues to the surgeon and gives real-time 3D navigation that warns the surgeon of the proximity of the tool to specific anatomical structures.

Overview

An exemplary system and method are generally provided for converting a medical image of a specific patient into a high-resolution 3D dynamic and interactive image that interacts with medical tools including medical devices by coupling a model of tissue dynamics and tool properties to the patient-specific image to simulate a medical procedure in an accurate and dynamic manner. The method includes tools for adding and/or adjusting dynamic tissue images and the ability to draw and add geometric shapes on the dynamic tissue images. The system imports a 3D surgical plan (craniotomy, head position, approach, etc.). The surgeon establishes multiple views, rotates the navigation image and interacts with the navigation image to see behind the lesion and key structures. The surgeon can make structures such as tumors, blood vessels and tissue transparent to improve visualization and be able to see behind the lesion. The system can issue warnings about the proximity of tools to specific anatomical structures.

In addition, there are several exemplary embodiments, including but not limited to a modeling system for performing a medical procedure on a specific patient, comprising: a display; an image generator comprising dedicated software executed on a computer system for generating a dynamic tissue image for display on the display, the generation for displaying on the display tissue that realistically represents the corresponding actual biological tissue of the specific patient; a user tool generator comprising dedicated software executed on the computer system for generating a tool model of a user tool for dynamically interacting with the dynamic tissue image through manipulation provided by user input for display on the display; and a user interface of the computer system, the user interface being configured to allow a user to adjust the dynamic tissue image displayed on the display by adding or modifying features of the tissue to assist in surgical operations. The modeling system is configured for use in an operating room during a medical procedure on a specific patient.

Further provided is a modeling system for performing a surgical procedure, comprising: a touch screen display; a database for storing a library of multiple models of different organs and/or tissues, wherein the database is further configured to store medical images of a specific patient, wherein the modeling system is configured to establish a case supporting the procedure before the surgical procedure by creating a model for the specific patient using the patient's medical images; a user interface for selecting the case from a plurality of such cases for loading into the modeling system; an image generator comprising dedicated software executed on a computer system for generating dynamic tissue images based on the selected case for display on the display, the generating software for displaying the selected case on the modeling system; The display displays tissue that realistically represents the corresponding actual biological tissue of a specific patient; a user tool generator, which includes dedicated software executed on a computer system for generating a tool model of a user tool for dynamically interacting with the dynamic tissue image through manipulation provided by user input for display on the display; a user interface of the computer system, which is configured to receive input from the user to configure the modeling system through the touch screen display; and an interface, which is connected to an interface of an external surgical system or tool present in the operating room to receive data from the external surgical system or tool for generating the dynamic tissue image for display consistent with the operation of the external surgical system or tool.

The above modeling system may be configured for use in an operating room during a surgical procedure on a particular patient.

Additionally provided is a method of using a modeling system (such as described herein) to support a medical procedure, comprising the steps of:

Providing a computer system configured for use in an operating room;

providing a display connected to the computer system;

obtaining patient image information regarding biological tissue of a particular patient for storage in the computer system;

generating, using dedicated software executed on a computer system, a dynamic image of biological tissue of a specific patient for display on the display, the generating using the patient image information so that the dynamic tissue image is displayed on the display as a faithful representation of the corresponding actual tissue of the specific patient;

generating, using the computer system, a user tool model for dynamically interacting with the dynamic tissue image through manipulations input by a user for display on the display;

using user input of the computer system to adjust the dynamic tissue image displayed on the display by adding or modifying features of the tissue for display so as to adjust anatomical structures located in actual biological tissue of a particular patient; and

A realistic simulation of at least a portion of a surgical procedure performed on a particular patient is generated using the computer system for display on the display, showing interaction between a dynamic tissue image and a user tool model based on user input.

Additionally provided is a method of using a modeling system (such as described herein) to support a medical procedure, comprising the steps of:

Providing a computer system configured for use in an operating room;

providing a 3D display connected to the computer system;

obtaining patient image information about biological tissue of a specific patient;

establishing a case supporting the surgical procedure prior to the surgical procedure by creating a model for the specific patient using medical images of the patient configured to generate dynamic images of biological tissue of the specific patient;

generating, using dedicated software executed on a computer system, a dynamic image of biological tissue of a specific patient using the case for display on the display, the generating using the patient image information so that the dynamic tissue image is displayed on the display as a realistic representation of the corresponding actual tissue of the specific patient;

generating, using the computer system, a user tool model for dynamically interacting with the dynamic tissue image through manipulations input by a user for display on the display;

using user input of the computer system to adjust a dynamic tissue image displayed on the display by adding or modifying features of the tissue for display so as to adjust anatomical structures located in actual biological tissue of a particular patient;

generating, using the computer system, a realistic simulation of at least a portion of a surgical procedure performed on a particular patient for display on the display, showing interaction between a dynamic tissue image and a user tool model based on user input;

receiving data from an external surgical system used during the surgical procedure, the data being used by the simulation tool to ensure that the dynamic image of the biological tissue is consistent with the operation of the external surgical system; and

Providing the ability to alert a user about the proximity of one or more surgical tools to specific anatomical structures of a patient.

The present invention also provides additional exemplary embodiments, some but not all of which are described in more detail below.

BRIEF DESCRIPTION OF THE DRAWINGS

The features and advantages of the embodiments of the present invention described herein will become apparent to those skilled in the art upon reading the following description and referring to the accompanying drawings, in which:

FIG1 is a block diagram illustrating an exemplary system architecture and interface for using the SNAP surgical tool disclosed herein;

FIG1A is a block diagram illustrating exemplary SNAP tool components for a mobile cart embodiment;

FIG1B is a diagrammatic representation of exemplary SNAP tool components for a mobile cart embodiment;

FIG1C is a block diagram illustrating an exemplary software interface for an exemplary SNAP Mobile Care implementation;

Figures 2-11 and 12-18 are screenshots of example display and/or control interfaces for an example SNAP tool; and

11A is a block diagram illustrating an exemplary fusion processing pipeline method for combining differently segmented tissue datasets using the SNAP tool.

Detailed Description of Exemplary Embodiments

The terms used herein are for the purpose of describing exemplary embodiments only and are not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms as well. It will be further understood that the terms "comprises" and/or "comprising", "including", "having", and "containing" when used in this specification specify the presence of the stated features, integers, steps, operations, elements, and/or parts, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and/or groups thereof.

This application incorporates all teachings of U.S. patent application Ser. No. 14/008,917, filed Sep. 30, 2013, and PCT application Ser. No. PCT/US13/42654, filed May 24, 2013, which provide design information for the improvements discussed herein.

The disclosed exemplary Surgical Approach Graphics Tool (SNAP) integrates with operating room technology to provide advanced 3D capabilities and augmented reality, allowing surgeons to enhance their surgical execution and preparation. For example, the SNAP tool provides neurosurgeons with virtual reality guidance to determine the safest and most effective approach to remove brain tumors and treat vascular lesions, among other uses.

SNAP imports 3D plans of the craniotomy, head position, and approach to the lesion, such as for keyhole and other minimally invasive techniques. SNAP allows the surgeon to visualize the intended surgeon's eye view in advance.

Utilizing the SNAP tool, surgeons can execute their surgical plans in the operating room while utilizing actual CT/MRI (and other) scans of a specific patient, allowing for enhanced accuracy and efficiency. SNAP also provides innovative features that allow surgeons to see arteries and other critical structures behind in rotatable 3D, which can be modified to make the image more useful to the surgeon. For example, SNAP provides the ability to rotate the image or make it semi-transparent to help the surgeon visualize the procedure. SNAP utilizes advanced imaging technology that allows surgeons to perform realistic "walk-throughs" of "patient-specific" surgical procedures. The tool provides preparation support outside the operating room and can also be used to take a pre-planned path into the operating room itself, which the surgeon (and his or her staff) will use during the procedure.

SNAP obtains tracking coordinates of surgical tools, navigation probes, microscope focal points, and the like by connecting to an OR's intraoperative tracking and navigation system. SNAP provides a 3D navigation mode that provides enhanced situational awareness. SNAP can receive images or tracking/navigation information from any of these surgical tools configured to collect such information, and the SNAP system can use this information to display high-resolution images to the surgeon corresponding to the received information. For example, the SNAP image can track the tool's position in the displayed image or update the image based on visual information provided by the tool, for example.

The SNAP proximity warning system operates in a manner similar to the Ground Proximity Warning System (GPWS) and the Airborne Collision Avoidance System (ACAS), the Ground Collision Avoidance System (TCAS), and other similar systems in aircraft that indicate and warn the crew of approaches and/or maneuvers that may result in proximity to the ground and other obstacles. SNAP proximity warning system operation consists of the following major phases:

-SNAP proximity warning system automatically marks anatomical structures that the surgeon needs to avoid. Such anatomical structures may include fiber trajectories, nerves, blood vessels, arteries, etc.

The SNAP proximity warning system allows for the manual placement of markers within a 3D or 2D navigation scene. These markers can mark obstacles and anatomical structures to avoid or identify targets for the surgeon to navigate to. Each marker placed can be labeled with a specific color, shape, etc.

- The indication of the warning of the SNAP proximity warning system may be visual (eg, color change), audible (sound), and others.

-SNAP allows the creation of tracks. By marking an entry point and then associating this entry point with an above marker/target, SNAP creates a track that allows navigation from the entry point to the target.

-SNAP Path Planner allows you to connect several markers, goals, and entry points and create a path. Multiple paths can be created. A path can be a desired route to follow or paths to avoid.

-Similar to an aircraft instrument landing system (ILS), SNAP provides visual graphical guidance to the surgeon. As long as the surgeon maintains his movements within the guided markings, he will accurately get from point A to point B (from entry point to target).

The tool provides an opportunity for institutions (e.g., hospitals) and their respective surgeons to reduce surgical errors, lower the amount of surgical waste and associated costs, reduce operating room time, and minimize the high risk nature of the procedure. The tool provides an opportunity to maintain the high quality of neurosurgery training and to conduct teaching outside the operating room: Halstedian training of surgical skills relies on a large number of cases and almost endless residency time in the hospital. Recent developments have forced a rethinking of the Halstedian system. Among the many recent pressures on the Halstedian system are: limited working hours, increased crowd supervision, and a decrease in surgical experience.

Training in the use of the tool can reduce the need for subsequent procedures and adjustments. For example, when the tool is used in an aneurysm procedure, using the tool can reduce the need to adjust or replace the aneurysm clip. Adjustment and replacement of the clip can often result in prolonged temporary occlusion and an overall longer procedure time. This can increase the overall risk of the procedure.

It will be understood by those skilled in the art that the exemplary embodiments disclosed herein may be implemented as or may generally utilize methods, systems, computer program products, or combinations thereof. Thus, any of the embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.) for execution on hardware, or an embodiment combining software and hardware aspects, which may generally be referred to herein as a "system." Furthermore, any of the embodiments may take the form of a computer program product on a computer-usable storage medium having computer-usable program code embodied on the medium.

For example, the features disclosed herein can be implemented using a networked computer system 1 set up in a hospital environment (such as an operating room) as shown in FIG1 . This system can be set up in a surgical environment where a surgeon 20 is supported by various surgical staff 22 to operate on a patient 5. Such a system 1 integrates one or more servers (e.g., PCs) 10A-10n that access data from one or more databases 15A-15n that are networked together using a computer network 12. The system will execute proprietary software that provides the functions and other features described herein. One or more computers 20 can be used to interface with various surgical tools, such as surgical probes/cameras 32, other surgical tools 34, and/or other devices 36 connected to the computers 20, using one or more computer buses or networks 30 as interfaces between the computers 18 and the tools. It should be noted that in some cases, the computers 18, servers 10, and databases 15 can all be housed in a single server platform.

The system is connected to a high-resolution 3D display 40 on which the surgeon can monitor the operation and activity of the various tools 32, 34, and 36. In some cases, the display may not have 3D capabilities.

The system is configured with patient-specific parameters 7 including patient imaging details, including image preparation from previously acquired available CT and MRI images of the patient, and other information about the simulation model, such as patient age, gender, etc. (some or all of this information may be obtained from external entities such as medical databases, laboratories, or other sources). The system uses tissue information parameters describing tissue and organ characteristics obtained from the system database. The system can be configured to interact with one or more external entities 60 via a communication network 50, such as the Internet, as needed.

Any suitable computer-usable (computer-readable) medium may be used to store software for execution on one or more computers to implement the disclosed processes and store the disclosed data and information. A computer-usable or computer-readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. More specific examples of computer-readable media (not an exhaustive list) would include the following: an electrical connection having one or more conductors, a tangible medium such as a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a compact disc read-only memory (CDROM), or other tangible optical or magnetic storage device; or a transmission medium such as an Internet or intranet-enabled medium. It should be noted that a computer-usable or computer-readable medium may even include another medium from which the program may be electronically retrieved, for example, by optical or magnetic scanning, and then compiled, interpreted, or otherwise processed as necessary, and then stored in any acceptable type of computer memory.

In the context of this document, a computer-usable or computer-readable medium can be any medium that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution system, platform, apparatus, or device, which may include any suitable computer (or computer system) including one or more programmable or special-purpose processors/controllers. A computer-usable medium may include a propagated data signal in baseband or as part of a carrier wave with computer-usable program code embodied therein. The computer-usable program code may be transmitted using any suitable medium, including but not limited to the Internet, wire, fiber optic cable, radio frequency (RF), or other means.

Computer program code for carrying out operations of the exemplary embodiments may be written by conventional means in any computer language including, but not limited to, interpreted or event-driven languages such as BASIC, Lisp, VBA, or VBScript, or GUI implementations such as Visual Basic, compiled programming languages such as FORTRAN, COBOL, or Pascal, object-oriented, scripting, or non-scripting programming languages such as Java, JavaScript, Perl, Smalltalk, C++, Object Pascal, etc., artificial intelligence languages such as Prolog, real-time embedded languages such as Ada, or even more direct or simplified programming using ladder logic, assembler languages, or direct programming using an appropriate machine language.

Computer program instructions may be stored or otherwise loaded in a computer-readable medium memory that may direct a computing device or system (such as depicted by exemplary system 1 of FIG. 1 ), or other programmable data processing apparatus, to function in a particular manner such that the instructions stored in the computer-readable memory produce an article of manufacture that includes instruction means for implementing the functions/acts specified herein.

Software includes the special-purpose computer program instruction being performed by being provided to execution device or component, and described execution device or component may include general-purpose computer, special-purpose computer or controller or other programmable data processing equipment or component, the processor of custom device described herein, makes the instruction of special-purpose computer program create the means for realizing the function/effect specified herein when being executed.Therefore, the computer program instruction of custom software is used for causing the series of operations that will be performed on execution device or component or other programmable device to produce computer-implemented processing, makes instruction perform the step for realizing the function/effect specified in the present disclosure on computer or other programmable device.These steps or actions can be combined with the step or action realized by operator or mankind and the step or action provided by other components or equipment to carry out any number of exemplary embodiments of the present invention.As required, customized software can also use various commercially available software, such as computer operating system, database platform (for example, MySQL) or other COTS software.

For an exemplary system called the "Surgical Navigation Advanced Platform" (SNAP), medical images of actual patients are converted into dynamic, interactive 3D scenes. This dynamic and interactive image/model creates a new and original standard for medical images and has many applications.

SNAP provides surgeons (neurosurgeons, etc.) with unique virtual reality guidance, for example, to determine the safest and most effective way to remove tumors (brain, etc.) and treat malformations such as vascular malformations. The "Surgical Navigation Advanced Platform" (SNAP) can be used as a standalone system or application of a surgical navigation system, or in conjunction with a third-party navigation system for the same types of procedures that the third-party navigation system is used for. These procedures include, but are not limited to, brain, spinal, and ear, nose, and throat (ENT).

SNAP allows surgeons to analyze and plan a specific patient case before surgery, and then bring the plan into the operating room (OR) and use it in conjunction with the navigation system during surgery. SNAP then renders the navigation data into advanced, interactive, high-quality 3D images with multiple viewpoints.

SNAP is actually an image-guided surgical system that includes medical imaging devices and presents real-time and dynamic multiple-line-of-sight views (from different/multiple angles) of the surgical procedure. The images include scanned medical images (based on scans such as CT, MRI, ultrasound, X-ray, etc.) and surgical instruments. It can also include real-time video and models based on video-generated microscopes or other sources. SNAP provides surgeons with real-time 3D interactive guidance images. The orientation of anatomical structures (i.e., head, brain, knee, shoulder, etc.) is marked and pre-aligned in the physical/patient and scanned medical images (CT, MRI, ultrasound, X-ray, etc.); therefore, the orientation of the scanned medical images and the patient's actual anatomical structures at the time of surgery are synchronized and aligned.

Furthermore, the aforementioned pre-aligned markers provide a spherical reference for tracking the surgical instruments and the OR microscope (and/or surgeon's head) and thus allowing the surgical instrument image/model to be presented spatially relative to the scanned medical image.

The patient's 2D or 3D anatomy and the position and orientation of the surgical instruments are synchronized in real time and presented to the surgeon with the real-time position and orientation of the instruments and markers in space relative to the anatomy.

The SNAP system is capable of preparing cases with multiple scan datasets. A built-in "Fusion" mode allows the user to select one dataset to be the primary dataset and add secondary datasets that will be aligned ("fused") to the primary scan dataset.

The SNAP system has a unique clipping feature. Any Plane IG Cube Clipping is a feature that allows users to "clip" a 3D model from any desired angle, essentially cutting into the model and removing a section to expose the interior. The clipping plane is the plane at which the 3D model is "clipped." The plane is defined by two variables: the plane normal (a vector) and the plane position (the point in space through which the plane passes).

Furthermore, the SNAP system knows to make any planar IG cube clip the 3D moving elements in the scene. Since the cube clipping planes are defined by normals and positions, we can use moving elements in the scene to define planes for the user. These elements include: navigation probes, 3D controllers (Omni), channels, IG viewpoints (eye cameras), etc.

Another feature is the transfer function. The SNAP system has a unique ability to display "tissue-specific intensity." Initial dataset slices are acquired and stacked to reconstruct a pixel cube, or what we call a voxel cube. The 3D model is a cubic volume of voxels. A transfer function is used to map each voxel intensity value to a color and opacity. This way, we control tissue intensity and enable the surgeon to see things they normally wouldn't be able to. This innovative feature allows the surgeon to see arteries and other critical structures from behind, something that was previously impossible.

SNAP can present the model in one or more windows on the same screen or on multiple screens. In an example of SNAP's features and applications, multiple features can be activated side by side on a screen.

A typical system configuration includes the following major components: (1) a system-mounted cart for mobility; (2) a medical-grade isolation transformer; (3) a personal computer (or server) running a Microsoft Windows 7 operating system; (4) a high-end nVIDIA graphics adapter for high-quality graphics; (5) a 27" or 32" full HD touchscreen display; (6) a medical-grade keyboard and mouse; and (7) the SNAP software application for implementing the features described herein. Such a system is illustrated by the block diagram of FIG1A , where an exemplary mobile cart-mounted SNAP system 70 including a touchscreen monitor 71 is configured with a PC 72 and a power system 73, all of which can be provided in an operating room. The exemplary system 70 is connected to a navigation system 75 provided in the operating room, such as an exemplary image-guided surgery (IGS) system, from which the SNAP system 70 can receive data so that the SNAP system 70 can display high-resolution, true 3D images that follow the operation of the navigation system, thereby effectively enhancing the operation and display capabilities of the navigation system using the SNAP high-resolution imaging capabilities based on images of the specific patient undergoing surgery.

FIG1B illustrates an exemplary cart-mounted portion of the system 70 of FIG1A , showing a primary high-resolution display 71 with a supplemental display 104 mounted on a stand 101, along with a keyboard 102, a computer 72, and a mobile platform 107, all configured for use in an operating room environment. The power and navigation systems are not shown in this figure. If necessary, this configuration can be networked to one or more additional servers using, for example, Ethernet.

The Surgical Navigation Advanced Platform (SNAP) is intended for use as a software interface and image segmentation system to transfer imaging information from CT or MR medical scanners to output files. A tissue segmentation window is provided to edit and update tissue segmentation to prepare cases. Changes to tissue segmentation are reflected in the 3D image, and the results can be saved as part of the case file. It is also intended for use as preoperative and intraoperative software to simulate/evaluate surgical treatment options. The Surgical Navigation Advanced Platform (SNAP) is a preoperative and intraoperative tool for simulating/evaluating surgical treatment options.

The system will typically provide EMC immunity suitable for use in an operating room and will use touchscreen operation for navigation, typing, and image manipulation. The system can store individual patient cases using case files, which can be loaded into the system as needed. Surgeons can create cases from scratch using scanned information (e.g., MR or CT DIACOM image data files) and patient data for a specific patient. These cases can be edited and updated as needed. An editing window can be used to edit and manipulate images and files.

Generic models of various organs and tissues can be provided that can be overlaid with patient-specific models based on patient imaging or other diagnostic tools or experimental input. Thus, for organs or other features of non-specific interest, the system can use a generic model (e.g., the eye or other organs) where patient-specific information is not required for specialized treatment.

The Advanced Surgical Navigation Platform displays a patient-specific dynamic and interactive 3D model with real-time navigation data. While performing a navigation session, tools are available to verify the accuracy of the SNAP navigation pointer position (set on SNAP's high-resolution display) by pointing and touching visible structures on the patient (i.e., nose tip, earlobe) and verifying that the pointer on the SNAP screen points to the same location in the 3D model.

FIG1C shows an example of the main processing routine driven by the software modules that generate the images provided by the SNAP tool. The image generator 110 generates a real tissue image for display on the display 160 by using information stored on the database 115 (such as generic tissue images, patient-specific images, etc.). The image generator 110 assigns a visual representation of each segmentation (shading texture, etc.), and the mechanical properties and other modeling features will provide a real image.

Similarly, the user tool generator will generate a real image of the surgical tool, which is displayed to dynamically interact with the tissue image generated by the image generator. In a surgical environment, the tool image displayed on the display 160 may represent an actual surgical tool interface 140, a representation of which may be generated for display by, for example, the image generator 110 or the user tool generator 120. The surgical tool interface may dynamically interact with the tissue image generated by the image generator 110. Likewise, specifications and details about the tool image may be stored in the database 115. Note that because the display 160 may be a touch screen display, the display 160 may serve as the user interface 130 along with other devices such as a keyboard, mouse, or other input devices.

The SNAP tool also provides an external system interface 150 that allows the system to interface with other external systems that provide navigation or other functions to the surgeon, such that the SNAP tool can generate images consistent with the external system's output based on the operation of the external system, such as mirrored navigation or updated imaging. The SNAP tool can then appropriately update its displayed image to provide the surgeon with an integrated view in the high-resolution 3D image that interacts with the graphical tool.

Once the surgeon selects surgical tools and other objects, they are integrated into the virtual surgical scene displayed by the display 160 and become integral elements of the simulation, including real visual features and mechanical and operational properties applied to each of those selected items, e.g., displayed scissors have real mechanical properties and will cut as real scissors would, and an aneurysm clip will block blood flow when placed at a blood vessel. In this way, the displayed tools interact with the tissue model in a realistic manner, but in a way that the surgeon can manipulate to provide viewpoints that are not possible in the real world, such as making multiple features transparent, rotating the image, reversing the procedure, etc.

The interactive image/scene displayed to the surgeon is composed of elements, which are both volume rendering elements and surface rendering elements. In addition, each element, volume or surface interacts with one or more elements that are volumes. The interactions between elements include, but are not limited to, physical interactions such as collision models that are implemented to represent the interactions between elements caused by the movement and/or reshaping of the elements that replicate the actual physical movement of the elements based on physical conditions such as pressure, element material (elasticity, viscosity, etc.) and collision conditions such as collision angle and element orientation.

The rendering process equations may take into account all lighting, shading, and shadow effect phenomena and produce a final output stream incorporating all visual elements.

Anatomical structures created using the tissue rendering or magic tissue wand algorithms and integrated with the scanned image are now an integrated part of the image. For example, a vascular anatomy that was initially partial and then complete after applying the magic tissue rendering and tissue wand algorithms will become a complete anatomy with a structure that is a combination of the original scanned image and the newly created structures. In addition, a control (checkbox) allows the newly created structures to be selected and toggled on (showing the newly created structures) or off (hiding the newly created structures). Additionally, options are provided for rendering the newly created structures using volumetric and/or network/polygon rendering/reconstruction rendering.

The developed algorithms and software tools provide an interface for users to draw any geometric shape or freehand shape in two or three dimensions (e.g., line, circle, caudal, sphere, etc.). The area included/enclosed/captured within the geometric shape (two or three dimensions) is defined as a "marker region." The user then has the ability to define and assign any visual characteristics and any mechanical properties to the "marker region."

Visual Properties: Color/Transparency/Shading - Newly created structures drawn using Magic Tissue, the Tissue Wand algorithm, or marked regions can be rendered with any selected visual properties, including a color that can be selected from a library of available colors and a transparency that can be selected on any scale from 0 to 100. Furthermore, the shading and shadowing properties of the newly created structures can be modified by adjusting the properties of the virtual light source. These properties include: spherical position in space, light color, light intensity, aspect ratio, virtual source geometry, etc.

Mechanical Properties - Newly created structures using tissue rendering, the magic tissue wand algorithm, or marked areas can be assigned mechanical property characteristics. That is, a mechanical model of a specific tissue can be coupled to the newly created structure, and thus the newly created structure will carry such mechanical properties and will react dynamically and statically according to those mechanical properties. For example, if "soft tissue" mechanical properties are assigned to the newly created structure, then the newly created structure will react according to the soft tissue. For example, when the newly created structure is pushed by a virtual surgical instrument, it will squeeze and reshape according to the applied force and the tissue mechanical model. In addition, the interaction between the newly created structure and other newly created structures, the interaction between the originally scanned structure and the newly created structure, and the interaction between the newly created structure and the originally scanned structure is seamless. The mechanical property coefficients (stiffness, elasticity, etc.) of any anatomical structure can be adjusted by the user to create tailored mechanical behavior.

Real-time tracking and feedback - a system for tracking actual surgical instruments during surgery. The tracking system transmits the spatial position and coordinates of the orientation and position of the surgical instruments relative to the actual anatomical structure (e.g., a specific spot on the patient's head). The position and orientation of the instrument are then sent to the surgical simulation system. Feedback is provided to the surgeon based on the patient-specific simulation and the position and orientation of the instrument. An example of such feedback could be that the system generates feedback to the surgeon on the type of tissue he is dissecting and warns the surgeon in case he is dissecting healthy brain tissue instead of a tumor. Another example is that after the surgeon applies the instrument to the actual anatomical structure (e.g., an aneurysm clip applied to an aneurysm on an actual patient), the system allows the surgeon to rotate the simulation image/model, which is accurately oriented to the actual anatomical structure based on tracking, and observe and evaluate the position and effectiveness of the placed implant.

This tracking and feedback of the actual instrument can be accomplished in a variety of ways, such as by using a video system to track the position and movement of the instrument and patient features. Alternatively (or in addition to video tracking), the surgical instrument can be modified to enable tracking, such as by using GPS, accelerometers, magnetic detection, or other position and motion detection devices and methods. For example, such modified instruments can use WiFi, Bluetooth, MICS, wired USB, RF communication, or other communication methods (e.g., via surgical tool interface 140 of FIG. 1C ) to communicate with the SNAP tool.

FIG2 shows an image of a virtual view 210 associated with the tip of a surgical instrument 201: this is an enhanced example, as if a virtual camera were attached to the tip of the instrument 201. This endoscopic view 210 provides the ability to represent what is seen on the SNAP plane as if the probe tip had a camera (simulating the probe as a "virtual endoscope"), as well as the ability to rehearse, plan, evaluate, and simulate endoscopic or other procedures. The surgeon can manipulate the image using a touchscreen toolbar 220 and by touching the model on the screen to rotate and zoom; thus, FIG1 shows an example of a virtual camera view.

The microscope view is the view from the orientation of the surgical operating microscope or the view from the surgeon's eye. Figure 3 is an example of a microscope view 230 and a key structures view 235 side by side. This view can be limited to the view through the craniotomy, which allows the surgeon to simulate navigation and approach to the lesion despite the limited access to the craniotomy. In addition, minimally invasive or keyhole approaches or endoscopic approaches can be simulated and evaluated, allowing the surgeon to try, evaluate and decide which approach will provide the most desired result (e.g., maximum tumor resection) with the least risk (e.g., an approach that does not risk hitting blood vessels on the way to the lesion). Thereafter, by loading the plan, the surgeon can drill/create the decided craniotomy/approach on the patient's head with the help of SNAP, and then actually navigate and perform the surgery with the help of SNAP through the approach selected above.

4 shows a posterior view of a lesion 255 (here an aneurysm) that can also be provided, for example, so that the surgeon can "freeze" the pointer/marker or tracking of the instrument 250 in order to explore the image in the system to further clarify the orientation of the instrument 250 within the anatomy. Freezing the pointer in place and manipulating the image (rotating, focusing, changing segmentation) allows the surgeon to view the aneurysm from behind to, for example, allow the surgeon to see whether the clip 250 is being applied correctly.

The ability to display multiple viewpoints provides the ability to view anatomical images and surgical instrument models simultaneously from multiple viewpoints in real time, and you can rotate, offset, and focus each image independently. Figure 5 shows a microscope view 260 and a sagittal view 265 side by side. Figure 6 shows a microscope view and a coronal view side by side. This allows the surgeon to navigate through his chosen approach and see 3D navigation images of structures that the surgeon wants to avoid but cannot be seen under a microscope or traditional tracking navigation systems (for example, blood vessels hidden inside or behind a tumor). SNAP shows views that are related to the microscope view (the surgeon's eye view) and are related to seeing views that the surgeon cannot see in a synchronized manner - views from behind the lesion.

Controlling the segmentation allows each viewpoint to be shown individually; for example, one image can be segmented to show only blood vessels while another image can be segmented to show only soft tissue. For example, FIG7 shows multiple segmented viewpoints. By manipulating the tissue segmentation, we can clearly see the aneurysm neck 240 juxtaposed with the normal viewpoint image 245. Independent and simultaneous views from different vantage points are provided to enhance visualization. Independent segmentation can be used to improve visualization of the aneurysm neck and other critical structures.

Different modalities (CT and or MRI) can be presented in each image/angle, for example one image can be from MRI and another image can be from CT of the same patient. Figure 8 shows different segmentations side by side, showing a CTA image 280 and an MRI image 285. Overlapping images can be shown where one image can be combined from multiple modalities; for example MRI layered/overlaid on CT. Figure 9 shows an overlay of CT and MRI images combined with different left segmentations 290 and right segmentations 295. When creating a historical case study, patient specific still images (snapshots) or video files generated during the actual surgical procedure or during the diagnostic process can be stored in the patient case folder. The SNAP software supports all standard Windows bitmap image file formats, such as bmp, jpeg, png and gif, and supports all standard Windows video file formats, such as mpeg and avi.

Overlay images with fade-in/fade-out modalities can also be provided, where one modality can be faded in and out in the combined image; for example, a CT and MRI stacked/overlay image with CT faded out or an overlay image with segmentation control on each modality; for example, a CT and MRI stacked/overlay image with CT segmented to show only vessels. FIG10 shows an overlay with transparency, 10% CTA, 100% MRI. FIG11 shows an overlay with transparency, 30% CTA, 100% MRI. FIG11A shows an exemplary fusion processing pipeline method using the SNAP tool for variously taking tissue datasets 301, 302 and independently defining and segmenting them 303, 304 to combine them 350 so as to present a combined image 307 with a transparency specified by a user 308.

Enhanced proximity and orientation indications allow for the calculation and generation of indications of the proximity of surgical instruments and/or pointers/markers to specific anatomical structures. In FIG12 , for example, a virtual clamp 310 is shown at the tip of a navigation probe for assessment of an aneurysm neck. The navigation image can be frozen and rotated to examine the clamp from multiple viewpoints, such as a channel view 311 and a rotation view 312. Markers placed on the virtual clamp allow for measurement of the neck and fitting of the clamp.

The SNAP tool makes it easier to select the size of the craniotomy and also allows a person to better identify some important landmarks. The three-dimensional representation of the tumor gives a superior view of the tumor border, veins, and sinuses to ensure that the surgeon can stay away from these structures when the craniotomy is actually started. Figure 13 shows a craniotomy view 310 and a vein view 315 side by side.

Based on the calculation of the proximity of the surgical instrument and/or pointer/marker to a specific anatomical structure, the surgeon can generate a risk threshold associated with the proximity and orientation indication. In addition, the measurement of distances is supported by virtual markers in the marker pointer, for example, a virtual lateral extension to the navigation probe is presented to measure the remaining tumor and the distance to the artery. FIG14 shows an example of measurement using a zoom probe 320, 325 using a virtual lateral extension, while FIG15 shows an example of determining aneurysm size measurement using virtual markers 330, 335, where the measurement results are displayed in window 337.

The system allows for "cubic clipping" to be performed, which independently trims a 3D image from any desired plane. A plane is defined by two variables, the plane normal (a vector) and the plane position (the point in space the plane passes through). This "cubic clipping" can be performed independently for each image. Figure 16 shows different independent clippings, while Figure 17 shows different clippings on different modalities.

The SNAP tool has the ability to draw tumors. For example, a surgeon can use drawing for case preparation and for all features during editing. This can be turned on and off during preparation or surgery. Figure 18 shows an example of drawing tissue. "Case Preparation" in SNAP is used to analyze and view DICOM objects residing in a folder on a local drive and generate a 3D scene in a proprietary format that is used as input to the navigation dialog. This involves converting a 2D collection of DICOM images into a 3D volume model with properties completed using the SNAP tool.

By measuring the distance in the marking pointer, the virtual marker of the implant position. SNAP combines modeling of implants (such as aneurysm clips or pedicle screws) and the use of augmented reality to guide surgery for accurate implant placement. For example, the plan for screw placement is performed with SNAP and the screw model (3D or 2D) is placed/positioned on top of the patient's 3D/2D model (CT, MRI, etc.). The surgeon can then navigate the actual screw in the operating room and SNAP will calculate the actual placement of the screw currently in the navigation compared to the pre-planned placement. Feedback from SNAP can be provided from the image/model of the screw being navigated, and the image/model of the screw will turn green only when the actual placement of the screw matches the planned placement. Measurements of distance, angle, orientation, etc. can be displayed in real time, indicating the actual screw position and its orientation (or any other pathway, implant, marker, probe in the navigation scene) relative to the planned placement position of the screw (or any other reference point).

The SNAP navigation images can be rotated, offset, focused, etc. independently for each image; the surgeon's eye view can be independently rotated, offset, focused to other images/views (e.g., view behind the lesion, etc.).

Change the transparency and color of each segmentation range. SNAP allows the user to render any tissue type or element within the navigation scene in any specific color and any specific transparency.

Using SNAP

The SNAP tool can be used by surgeons both in pre-operative preparation and during the actual surgical procedure. For example, the SNAP can be used to support an arteriovenous malformation nidus (Latin for "nest") - an AVM nidus.

Prepare-SNAP can be used to analyze and view DICOM objects residing in a folder on a local drive, and can be used to generate a 3D scene in a proprietary format to be used as input to the navigation session. This requires converting a 2D collection of DICOM images into a 3D volume model with attributes. The first thing to be selected is the preferred case in the library in order to load it (making it the current case) and then activate the navigation session mode by clicking the Navigation button.

Plan - Define the volume of interest (VOI), and the tissue deformation VOI indicates "living" tissue that uses tissue mechanical properties to realistically react when interacting with surgical tools/probes during surgery. Axial, coronal, and sagittal slider controls are used to define the tissue deformation volume of interest (VOI) cube that will be used. Subsequently, interaction and manipulation of the image on the SNAP touch screen is completed to plan the approach angle and required exposure range to adequately visualize the draining veins and understand the position and orientation of the arterial feeder relative to the AVM nidus. As part of case preparation, SNAP allows the surgeon to use the image generator to display the volume of interest (VOI) window to define images, such as for defining the tissue deformation volume for simulation. The button then allows the surgeon to display the VOI window to define the characteristics of the VOI.

Axial, coronal, and sagittal slider controls are set in SNAP to define a cube that represents the volume of interest (VOI) used by surgical simulation applications. Adjusting the high and low boundary axial sliders defines the top and bottom boundaries of the VOI. Adjusting the high and low boundary coronal sliders defines the front and back boundaries of the VOI. Adjusting the high and low boundary sagittal sliders defines the left and right boundaries of the VOI. The slider values for each axis are set in the high and low boundary fields.

The Visualization - Tissue Segmentation window can be used to modify the tissue-specific intensity range applied to the patient data. The Tissue Rendering Threshold (Segmentation Defaults) feature can be used, or the segmentation can be modified. By doing so, the segmentation can be adjusted and customized to highlight the AVM nidus, draining veins, feeding arteries, and lateral sinus for optimal visualization, while making the skull and non-critical structures more transparent.

3D Navigation-SNAP can be integrated with traditional navigation systems, allowing one to simultaneously view the 2D collection of DICOM images (on the traditional navigation system) and the trajectory on the 3D volume model (on SNAP). With SNAP, surgeons can use 3D angiography to navigate and accurately target non-embolic feeding arteries on the surface of the AVM nidus.

Example Applications

As discussed above, SNAP can be used in a surgical setting to support surgical procedures on a patient. Here, we provide an exemplary scenario to illustrate its use during a craniotomy on an exemplary patient with a brain tumor.

Surgeons create and update patient cases by entering their patient's setup parameters, which include patient details that allow the system to upload relevant data for a specific patient. SNAP then loads the patient's CT, CTA, MRI and MRA images and other information about the simulation model, such as patient age, gender, etc.

The surgeon will see three planes on a traditional navigation system, which will be the axial, coronal and sagittal plans, and at the same time on the SNAP display the surgeon will see a three-dimensional image of the same patient's head. Thus, the SNAP display shows greater detail in a more realistic viewing manner.

The surgeon activates the Navigation Dialog mode by tapping the Navigation button on the SNAP touch screen. The system displays the Navigation Dialog windows—two separate IG camera windows (canvases) and the Navigation Quick Access window.

The surgeon will enable fusion mode, where SNAP will show a combined model of CTA overlaid with MRI and define the working mode, for example he will select combined mode, where both layers (CTA and MRI) are displayed on two canvases.

The surgeon will now use the transparency preset buttons and the left/right buttons on the display dialog window to set the desired transparency level for the slices (CTA and MRI).

On the left canvas, the surgeon selects one of four transfer functions. He begins changing the transparency by moving the "keys" on the transfer function dialog window. To change the "keys," he uses the system mouse. The surgeon manipulates the "keys" until he achieves the desired appearance, such as a visible tumor, borders, and veins.

On the right canvas the surgeon will select different transfer functions and he can see important landmarks, for example.

By touching the SNAP screen to rotate the model left and right and zoom in and out, the surgeon gets a better representation of the tumor in three dimensions and can determine how large the bone opening will be. The surgeon can truly understand the relationship between the tumor and where the blood vessels are located and how he will approach the tumor during the operation.

The surgeon will use the SNAP tool to mark important veins and the desired bone opening in the skull.

The surgeon will use the SNAP navigation pointer to estimate the path length from the skull to the tumor. He will place the pointer tip over the penetration point on the skull and will select the virtual trajectory pointer from the navigation pointer type menu on the SNAP tool.

On the SNAP display, the surgeon will see a pointer with a trajectory extension that moves in real time according to his movements. By giving the appropriate direction, he will obtain the penetration length.

The surgeon has the option to freeze the data from the navigation system. In this case, the incoming data is ignored and the pointer remains at its last position before clicking the Freeze button. The surgeon can use SNAP to turn the model left and right and zoom in and out to get a better representation of the path and surrounding key parts.

Next, the surgeon will begin the craniotomy based on the information he sees on SNAP. After opening the craniotomy, the surgeon begins cutting along the pathway, and by using the navigation pointer he can follow on SNAP to the right path, he can estimate the distance to the tumor and how close he is to the sinuses and veins.

Use instances close to your application.

A path is created to navigate toward the tumor through a keyhole craniotomy. A keyhole craniotomy is smaller and does not allow the surgeon to see many structures. Therefore, a path or trajectory is created to the target (tumor). Additionally, markers or paths will be made to mark areas to avoid.

During surgery, SNAP presents real-time 3D and 2D images of the navigation scene, showing the surgeon the position of their tools. The surgeon can then follow the navigation path. The surgeon can also modify the path, markers, trajectories, and the like. These modifications are made in a similar manner to how markers, targets, and the like were created. Markers, targets, and the like can be dragged to different locations, recolored, and the like.

During navigation, should the surgeon deviate from a specific marker, path, trajectory, etc. by a specific preset value (distance, angle, etc.), the warning will be eliminated. The specific preset value (distance, angle, etc.) can be modified.

Example of catheter placement for a third ventriculostomy. - The surgeon marks a point within the third ventriculostomy as the target. The surgeon marks the entry point. SNAP creates a trajectory and visual graphics to guide the surgeon in catheter placement.

Many other exemplary embodiments of the present invention can be provided by various combinations of the above-mentioned features. Although specific examples and embodiments have been used to describe the present invention above, it will be understood by those skilled in the art that various alternatives can be used, and equivalents can be substituted for the elements and/or steps described herein without departing from the intended scope of the present invention. Modifications may be required to adapt the present invention to specific circumstances or to specific needs without departing from the intended scope of the present invention. The present invention is not intended to be limited to the specific implementations and embodiments described herein, but the claims are given the broadest reasonable interpretation to cover all novel and non-obvious embodiments, whether literal or equivalent embodiments, disclosed or undisclosed.

Claims (40)

1. A modeling system for performing medical procedures on a specific patient, comprising: monitor; An image generator comprising specialized software executed on a computer system for generating dynamic tissue images for display on the display, the generation being used to display on the display the tissue realistically representing the corresponding actual biological tissue of a particular patient; A user tool generator, comprising specialized software executed on a computer system for generating tool models of user tools to dynamically interact with the dynamically organized image via manipulations provided by user input for display on the monitor; and The computer system's user interface, configured to provide user tools, allows the user to adjust a dynamic tissue image displayed on the monitor prior to a surgical procedure by adding or modifying features of the tissue to aid in the subsequent surgical operation. The added or modified features represent the mechanical properties of the tissue. The modeling system is configured for use in the operating room during the medical procedure for the specific patient. 2. The modeling system of claim 1, wherein the user interface includes a touchscreen display. 3. The modeling system of claim 1 or 2, wherein the user interface for adjusting the dynamic tissue image includes tools for providing the ability to draw any geometry on the dynamic tissue image. 4. The modeling system of claim 1 or 2, wherein the user interface for adjusting the dynamic tissue image includes tools for providing the ability to complete incomplete anatomical structures of the dynamic tissue image. 5. The modeling system of claim 1 or 2, wherein the user interface for adjusting the dynamic tissue image provides the ability to modify the texture, lighting, shadows, and/or shadow processing of a portion of the dynamic tissue image. 6. The modeling system of claim 1 or 2, wherein the dynamic tissue image comprises an image of an anatomical structure, and wherein the user interface comprises instruments for dynamically interacting with the anatomical structure. 7. The modeling system as described in claim 1 or 2, further comprising: A database, the database being used to store a library of multiple models of different organs and/or tissues; and A user interface is provided for selecting one model from the plurality of models for use with the user tool model to dynamically interact with the tissue image. 8. The modeling system of claim 1 or 2, wherein the user interface for adjusting the dynamic tissue image comprises a real surgical tool used by a surgeon to perform surgery on the particular patient. 9. The modeling system of claim 1 or 2, wherein the user interface for adjusting the dynamic tissue image includes tools for providing the ability to select a model of the tool and/or an element of the dynamic tissue image for removal from the displayed image. 10. The modeling system of claim 1 or 2, wherein the user interface for adjusting the dynamically organized image includes tools for providing the ability to reposition or rotate the object in the displayed image by selecting the object and manipulating the object to a desired position for display in the image. 11. The modeling system of claim 1 or 2, wherein the user interface for adjusting the dynamic tissue image includes tools for providing the ability to enhance and integrate anatomical structures into the dynamic tissue image. 12. The modeling system of claim 1 or 2, wherein the user interface for adjusting the dynamic tissue image includes tools for providing the ability to draw any geometry to be added to the dynamic tissue image. 13. The modeling system of claim 1 or 2, further comprising a camera for interacting with the image displayed on the display. 14. The modeling system of claim 1 or 2, wherein the modeling system is configured to have an interface connected to an interface of an external surgical system present in the operating room to receive data from the external surgical system for generating the dynamic tissue image for display in accordance with the operation of the external surgical system. 15. The modeling system of claim 14, wherein the external surgical system is a navigation system for use during the medical procedure, which is a surgical procedure performed in the operating room. 16. A method for using a modeling system to support a medical process, comprising the following steps: Provide a computer system configured for use in the operating room; Provide a display connected to the computer system; To obtain patient image information about the biological tissues of a specific patient for storage in the computer system; Dynamic tissue images of the biological tissues of the specific patient are generated using specialized software executed on the computer system for display on the monitor, wherein the generation uses the patient image information such that the dynamic tissue images are displayed on the monitor as a true representation of the corresponding actual tissues of the specific patient; The computer system is used to generate a user tool model for dynamically interacting with the dynamically organized image through user input for display on the monitor; User input using the computer system adjusts the dynamic tissue image displayed on the monitor prior to surgical procedures by adding or modifying features of the tissue to adjust the anatomical structure of the actual biological tissue located in the specific patient; the added or modified features represent the mechanical properties of the tissue; and The computer system is used to generate a realistic simulation of at least a portion of the medical procedure performed on the specific patient for display on the monitor, illustrating the interaction between the dynamic tissue image and the user tool model based on the user's input. 17. The method of claim 16, wherein the user tool model comprises a model of an actual surgical tool used by a surgeon during surgery on the particular patient. 18. The method of claim 17, wherein the surgical tool includes an image acquisition tool configured to receive images of tissue from the particular patient, such that the images displayed on the display are modified by manipulating the image acquisition tool. 19. The method of claim 16 or 17, wherein the adjustment includes drawing at least a portion of the tissue image. 20. The method of claim 16 or 17, wherein the adjustment includes making at least a portion of the tissue image transparent. 21. The method of claim 16 or 17, wherein the adjustment includes rotating the image on the display. 22. The method of claim 16 or 17, further comprising the step of: creating a case study supporting the medical procedure prior to the medical procedure by using a patient medical image of the particular patient configured to generate a dynamic tissue image of the biological tissue of the particular patient. 23. The method of claim 17, further comprising the step of: receiving data from an external surgical system used by the surgeon, the data being used by a simulation tool to ensure that the dynamic tissue image of the biological tissue is consistent with the operation of the external surgical system. 24. The method of claim 16 or 17, further comprising the step of: receiving data from a surgical navigation system for the medical procedure for generating a dynamic tissue image of the biological tissue consistent with the operation of the surgical navigation system, the medical procedure being a surgical procedure on the particular patient. 25. The method of claim 16 or 17, further comprising the step of: warning the user about one or more surgical instruments approaching specific anatomical structures of the patient. 26. A method for using a modeling system to support surgical procedures, comprising the following steps: Provide a computer system configured for use in the operating room; Provide a 3D display that is connected to the computer system; Obtain patient image information about the biological tissues of a specific patient; A case supporting the surgery is established prior to the surgery by creating a model of the specific patient using patient medical images configured to generate dynamic tissue images of the biological tissue of the specific patient; Using specialized software running on the computer system, the case is used to generate a dynamic tissue image of the biological tissue of the specific patient for display on the monitor, wherein the generation uses the patient image information such that the dynamic tissue image is displayed on the monitor as a true representation of the corresponding actual tissue of the specific patient; The computer system is used to generate a user tool model for dynamically interacting with the dynamically organized image through user input for display on the monitor; User input using the computer system adjusts the dynamic tissue image displayed on the monitor by adding or modifying features of the tissue to be displayed prior to surgical procedures, in order to adjust the anatomical structure in the actual biological tissue of the particular patient, the added or modified features reflecting the mechanical properties of the tissue; The computer system is used to generate a realistic simulation of at least a portion of the surgical procedure performed on the specific patient for display on the monitor, illustrating the interaction between the dynamic tissue image and the user tool model based on the user's input; Data is received from an external surgical system used during the surgical procedure, the data being used by a simulation tool to ensure that the dynamic tissue images of the biological tissue are consistent with the operation of the external surgical system; and Provides the user with the ability to warn them about one or more surgical instruments approaching specific anatomical structures of the patient. 27. The method of claim 26, wherein the adjustment includes drawing at least a portion of the tissue image, making at least a portion of the tissue image transparent, and/or rotating the image on the display. 28. A modeling system for performing surgical procedures, comprising: Touchscreen display; The database is a library for storing multiple models of different organs and/or tissues, and the database is also configured to store medical images of specific patients. The modeling system is configured to establish a case supporting the surgical procedure prior to the procedure by creating a model of the specific patient using the patient's medical images; A user interface for selecting a case from a plurality of such cases for loading into the modeling system; An image generator comprising dedicated software executed on a computer system for generating dynamic tissue images based on a selected case for display on the display, the generation being used to display on the display the tissues realistically representing the corresponding actual biological tissues of the particular patient; A user tool generator includes dedicated software executed on a computer system for generating tool models of user tools for dynamically interacting with the dynamically organized image through manipulations provided by user input for display on the monitor. The user interface of the computer system, configured to receive input from the user, adjust the dynamic tissue image displayed on the monitor by adding or modifying features of the tissue to be displayed prior to surgical procedures, the added or modified features representing the mechanical properties of the tissue, and configure the modeling system via the touchscreen display; and An interface, which connects to an external surgical system or tool present in the operating room, is used to receive data from the external surgical system or tool to generate the dynamic tissue images for display in accordance with the operation of the external surgical system or tool. The modeling system is configured for use in the operating room during surgery on the specific patient. 29. The modeling system of claim 28, wherein the user interface for adjusting the dynamic tissue image includes tools for providing the ability to draw any geometry on the dynamic tissue image. 30. The modeling system of claim 28 or 29, wherein the user interface for adjusting the dynamic tissue image includes tools for providing the ability to complete incomplete anatomical structures of the dynamic tissue image. 31. The modeling system of claim 28 or 29, wherein the user interface for adjusting the dynamic tissue image provides the ability to modify the texture, lighting, shadows, and/or shadow processing of a portion of the dynamic tissue image. 32. The modeling system of claim 28 or 29, wherein the dynamic tissue image comprises an image of an anatomical structure, and wherein the user interface comprises instruments for dynamically interacting with the anatomical structure. 33. The modeling system of claim 28 or 29, wherein the user interface for adjusting the dynamic tissue image comprises actual surgical tools used by a surgeon for the specific patient. 34. The modeling system of claim 28 or 29, wherein the user interface for adjusting the dynamic tissue image includes tools for providing the ability to select a model of the tool and/or an element of the dynamic tissue image for removal from the displayed image. 35. The modeling system of claim 28 or 29, wherein the user interface for adjusting the dynamically organized image includes tools for providing the ability to reposition or rotate the object in the displayed image by selecting the object and manipulating the object to a desired position for display in the image. 36. The modeling system of claim 28 or 29, wherein the user interface for adjusting the dynamic tissue image includes tools for providing the ability to enhance and integrate anatomical structures into the dynamic tissue image. 37. The modeling system of claim 28 or 29, wherein the user interface for adjusting the dynamic tissue image includes tools for providing the ability to draw geometry to be added to the dynamic tissue image. 38. The modeling system of claim 28 or 29, further comprising a camera for interacting with the image displayed on the display. 39. The modeling system of claim 28 or 29, wherein the external surgical system is a navigation system for use during the surgical procedure. 40. The modeling system of claim 33, wherein the modeling system is configured to provide the surgeon with warnings about one or more surgical instruments approaching specific anatomical structures of the patient.