WO2025224080A1 - Method for generating an augmented image using a medical visualisation system, medical visualisation system, and computer program product - Google Patents

Abstract

The invention relates to a method for generating an augmented image (AA) using a medical visualisation system (4), comprising the following steps: a. receiving at least one image signal which has been generated by at least one image capturing device (5) of a surgical microscope (10) and which represents an image (A1) of an examination region (1); b. dividing the image (A1) into a foreground region (V) and a background region (H); c. receiving at least one signal which represents or encodes supplementary information (ZI) for augmentation; d. generating the augmented image (AA) by i. superimposing the background region (H) of the image of the examination region (1), or a portion thereof, with the supplementary information (ZI), or ii. superimposing the image (A1) of the examination region (1), or a portion thereof, with the supplementary information (ZI), wherein the superimposition in the background region (H) and in the foreground region (V) is carried out according to superimposition modalities that are different from one another. The invention also relates to a medical visualisation system (4) and to a computer program product.

Description

Method for generating an augmented image using a medical device

Visualization system, medical visualization system and

Computer program product

The invention relates to a method for generating an augmented image using a medical visualization system, a medical visualization system and a computer program product.

Surgical microscopes are used, among other things, to prepare for and perform medical operations on a patient. Such surgical microscopes are used by a user, e.g., a surgeon or an assistant, during a procedure to provide a magnified view of an area of examination, particularly in or on the patient's surgical site. For this purpose, a surgical microscope may include an objective lens or lens system to produce a true optical image of the area of examination. The objective lens may include optical elements for beam guidance, shaping, and/or direction. An optical element may, in particular, be a lens.

Surgical microscopes are used in medical facilities, as well as in laboratories and industrial applications. Examples of medical applications include neurosurgery, ophthalmic surgery, otolaryngology (ENT), plastic or reconstructive surgery, and orthopedic surgery. This list is not exhaustive. Generally, they are used in all areas of surgery where a magnified, high-resolution view of the surgical field is required to perform precise procedures.

A distinction can be made between analog and digital surgical microscopes. Unlike digital surgical microscopes, analog surgical microscopes do not capture images that are then displayed, for example, on a screen to magnify the examination area. Instead, they offer the user a directly visual magnification of the examination area. Here, radiation reflected or scattered from the area of application passes through the objective lens into at least one beam path and to at least one output section, through or into which the user looks to visually perceive the radiation and thus also the typically magnified representation of the examination area. An exemplary embodiment of a The output section is a so-called eyepiece into or through which the user looks in order to optically perceive the examination area with at least one eye.

Digital surgical microscopes comprise at least one image acquisition device for microscopic imaging, which captures radiation in a beam path of the surgical microscope to generate a magnified image. This image can be displayed to the user or multiple users on one or more display devices, thus enabling high-resolution visualization. The image can be generated as a transmittable image signal that encodes or represents the image. Purely digital surgical microscopes, unlike analog surgical microscopes, do not have an output section for visually detectable radiation, specifically no eyepiece. The image signal can then be transmitted as a data signal, either wired or wirelessly. Digital surgical microscopes allow for the recording, storage, and further processing of images and videos. By applying image processing techniques, contrast, brightness, and other parameters can be adjusted to optimize the image quality of the generated images. Hybrid surgical microscopes can include at least one image acquisition device as well as at least one output section. For example, the radiation guided in a beam path of the operating microscope can be split with a beam splitter, whereby a first portion is directed to the output section and another portion is captured by the at least one image acquisition device.

Stereoscopic surgical microscopes are also well-known. These typically include two separate beam paths for beam guidance and provide the user with a depth perception of the examination area. The beams guided in the two beam paths can be visually detected by the user via output sections. Digital surgical microscopes alternatively or additionally include two image acquisition units, each capturing the beams in one of the beam paths to generate an image. Based on these two images, which can also be referred to as corresponding images, a three-dimensional image is then provided to the user via a suitable display device. The image acquisition units are components of a stereo (camera) system.

The operating microscope can form a medical visualization system, or the medical visualization system can include the operating microscope. The components of the medical visualization system described below may be components of the operating microscope or components designed differently from the operating microscope.

Another known method is the provision of an augmented representation of the examination area to a user. An augmented representation can, in particular, be a representation of the real examination area that is enhanced by computer, especially by adding or overlaying at least one virtual object and/or other additional information onto the representation of the real examination area. The augmented representation can be displayed to a user as an augmented image on a display device or provided in a visually perceptible manner via an output section.

Additional information can be provided in the form of data that represents or encodes a geometric description of a space, particularly a three-dimensional space, especially one containing objects arranged within it. Additional information can also be information generated from such data, for example, information produced by rendering. Rendering, or image synthesis, is the process of computer-implemented generation of a photorealistic or non-photorealistic image from a 2D or 3D model. Multiple models can be defined in a scene file, which contains objects in a defined language or data structure. The scene file can contain geometry, viewpoint, texture, lighting, and shading information that describes the virtual scene. The data contained in the scene file is then passed to a rendering program, which processes it and outputs it to a digital image or raster graphics file. A software application or component that performs the rendering is called a rendering engine, rendering system, graphics engine, or simply a renderer.

An important requirement for augmentation is that the viewer of the augmented image, especially a surgeon, is not disturbed or distracted by the augmentation during their work. In particular, it is desirable that a surgeon can operate ergonomically even when viewing an area of examination with superimposed augmentation. Therefore, it is especially important that a An object, such as a tumor, that lies spatially behind a surface of the examination area along a particular line of sight or viewing direction, must also be represented in an augmented image in such a way that it does not mistakenly superimpose objects located spatially in front of the surface, as this can impair the viewer's spatial perception. This can also distort the stereoscopic depth impression when viewed through a stereoscopic operating microscope. This can be particularly true if augmented objects are superimposed in a stereoscopically perceptible manner, as the resulting three-dimensional perception can be confusing for the viewer.

If other objects, such as surgical instruments, are within the field of view of the medical visualization system, augmentation can be problematic if it obscures the representation of these objects. This can disrupt the viewer's perception of information regarding the relative position of the object and the area under examination. Furthermore, it can be problematic if augmentation covers areas where tissue is depicted, as the viewer, especially the aforementioned surgeon, needs to clearly see the depicted tissue in order to perform surgical procedures.

US2010/295931 A1 relates to the technical field of medical navigation image output. Medical navigation is used in image-guided surgery and assists the surgeon in the optimal positioning of their instruments, for example, by referencing previously acquired image data of the patient. The treating physician thus has access to an image output, such as a monitor, on which they can see where their instrument or its functional component is located in relation to specific body regions of the patient.

US Patent 2015/221105 A1 discloses imaging systems, imaging devices, and imaging methods that merge portions of a multidimensional reconstructed image with multidimensional visualizations of at least a portion of a surgical site. The imaging systems can generate multidimensional reconstructed images based on preoperative image data. In a selected section of the visualization, the imaging systems can display a portion of the multidimensional reconstructed image. The document M. Allan et al., 2017 Robotic Instrument Segmentation Challenge, https://arxiv.org/abs/1902.06426, 2019 reveals a semantic segmentation.

The technical problem therefore arises of creating a method for generating an augmented image using a medical visualization system, a medical visualization system, and a computer program product that increase the display quality of the augmented image to improve the perception of a depicted examination area during augmentation and thus overcome at least one of the disadvantages explained above.

The solution to the technical problem is provided by the articles with the features of the independent claims. Further advantageous embodiments of the invention are described in the dependent claims.

A method for generating an augmented image using a medical visualization system is proposed. This system includes, in particular, an operating microscope or can be generated by the operating microscope itself.

Surgical microscopes and their technical features were briefly explained earlier. A surgical microscope can be, in particular, a stereo microscope. It can also be designed as an endoscope.

An operating microscope can comprise a microscope body. The objective lens described above can be integrated into or attached to the microscope body, particularly in a detachable manner. The objective lens can be fixed in position relative to the microscope body. In addition to the objective lens, the microscope body can also have or incorporate at least one beam path for microscopic imaging and/or other optical elements for beam guidance, shaping, and/or deflection. In analog and hybrid operating microscopes, the microscope body can include at least one mounting interface for attaching an output element, such as an eyepiece, in particular in a detachable manner. The microscope body can comprise or form a housing, or be arranged within a housing. Components of the operating microscope, such as an image acquisition device for microscopic imaging, are also included. The illustration shows that the parts can be located in or on the housing.

The medical visualization system can include a stand for mounting the operating microscope. The operating microscope, in particular the microscope body, can be mechanically attached to the stand. The stand is designed to allow movement of the operating microscope in space, in particular with at least one degree of freedom, preferably with six degrees of freedom, where one degree of freedom can be translational or rotational. Furthermore, the stand can include at least one drive unit for moving the operating microscope. Such a drive unit can, for example, be a servo motor. Naturally, the stand can also include means for transmitting force/torque, e.g., gear units. In particular, it is possible to control the at least one drive unit in such a way that the operating microscope performs a desired movement and thus a desired change of position in space, or assumes a desired position and/or orientation in space. For example, the at least one drive unit can be controlled in such a way that an optical axis of an objective lens of the operating microscope assumes a desired orientation. Furthermore, the at least one drive unit can be controlled such that a reference point of the operating microscope, e.g., a focal point, is positioned at a desired position in space. A target position can be specified by a user or another higher-level system. Methods for controlling the at least one drive unit as a function of a target position and a kinematic structure of the stand are known to those skilled in the art.

Furthermore, the medical visualization system can include one or more display devices for showing the images. The display device can be used to show two- or three-dimensional images. A three-dimensional image can, in particular, be or comprise a stereo image pair, wherein the images of this image pair are stereoscopic images. Typical display devices are screens, especially 3D screens, head-mounted displays (HMDs), or digital eyepieces, which can also be referred to as booms. Furthermore, the medical visualization system, in particular the

Operating microscope, including one or more of the following elements:

• at least one white light lighting device,

• at least one infrared lighting device,

• at least one fluorescence illumination device for exciting fluorescence radiation,

• at least one beam filter to provide excitation radiation with wavelengths from a broader spectrum, e.g. the spectrum of the white light illumination device,

• at least one fluorescence detection device for detecting fluorescence radiation,

• at least one filter device for filtering radiation from a broader spectrum, e.g. for detection by an image acquisition device for microscopic imaging,

• at least one image acquisition device of an optical position detection device, which can also be referred to as a surrounding camera,

• at least one gaze direction detection device,

• at least one position detection device for determining a pose, i.e. a position and/or orientation, at least of the operating microscope

• at least one input device for operation,

• at least one interface for data transmission to or from another system or facility,

• at least one device for determining depth information, in particular with regard to the elements arranged in the detection range of the operating microscope, which may be designed, for example, as a distance sensor,

• at least one storage device for storing signals and/or information, especially in a retrievable manner.

An image acquisition device may, in particular, include a CMOS or CCD sensor. The detection range of the ambient camera may fully or at least partially encompass the detection range of the surgical microscope. Alternatively, the detection range of the surgical microscope may fully or at least partially encompass the detection range of the ambient camera. In a fluorescence visualization mode, a filter device can be inserted into an observation beam path, providing the viewer with a filtered representation of the examination area. This radiation can be captured by the at least one image acquisition device for microscopic imaging. Alternatively, fluorescence radiation can also be captured by a separate acquisition device, such as a spectral camera. The fluorescence mode advantageously enables intraoperative tissue differentiation. Tumor tissue, in particular, can be visualized using fluorescence-based images. Nerve tissue, in particular, can be visualized using polarization-contrast-based images.

Medical visualization systems can be operated, for example, by manually controlling a component, particularly the operating microscope, or a corresponding input device; by voice control; by gesture control; by eye-tracking; by image-based control; or by other operating methods. The medical visualization system or the operating microscope may include the necessary components. Image-based control may, in particular, include the generation of operating or control signals by evaluating at least one image produced by an image acquisition device for microscopic imaging or by an image acquisition device of an optical position detection system.

Adjustable operating parameters of the medical visualization system or the surgical microscope can be formed by one or more of the following parameters:

• Magnification factor or zoom factor,

• Working distance or focus position,

• Detection range

• Light intensity,

• Illumination spectrum.

The proposed method comprises the following steps: a) Receiving at least one image signal, which is generated by at least one

The image acquisition device of the operating microscope was generated, and which is a The image represents a section of the area being examined. The image signal can be received via an interface of the medical visualization system. Specifically, the image signal can represent a two-dimensional image. The area being examined can be a region of a patient's body during surgery or a diagnostic examination.

The received image signal can preferably represent a white light image (VIS image) generated with visible radiation, i.e., radiation from a wavelength range between 360 nm and 830 nm. It is also conceivable that the image is provided as a fluorescence contrast image, generated by radiation with predetermined fluorescence-specific wavelengths or wavelength ranges, for example, wavelengths of 400 nm or 560 nm. The image can also be provided as a polarization contrast image, generated by radiation with a predetermined polarization. The image acquisition device of the surgical microscope can therefore be an image acquisition device for microscopic, i.e., magnified, imaging of the examination area, or it can be one of several different image acquisition devices. b) Dividing the image into a foreground and a background area. This division can also be referred to as segmentation. In particular, pixels or image areas of the image are assigned to either the foreground or the background area, or classified as either foreground or background. The division can be achieved, in particular, by depicting elements such as objects or structures in the foreground that are not intended to be overlaid by augmentation. These elements can be referred to as foreground elements. Foreground elements include, for example, instruments, especially surgical instruments, hands, or fingers, or sections thereof. Similarly, the background contains elements, also referred to as background elements, which can be overlaid by augmentation. Background elements include, in particular, tissue. However, background elements can also be instruments. In particular, so-called hybrid elements can exist, which can be either background or foreground elements depending on the scenario. An example of a hybrid element could be a swab, which is classified as a background element. This is particularly true when the swab is statically and/or unactuated within the area of investigation. However, the swab can also be classified as a foreground element, especially if it moves more than a predetermined amount and/or is actuated by a user. For the purposes of this invention, foreground elements are not limited to instruments. Furthermore, no object recognition is performed for the detection of foreground elements.

In particular, both parts of the image in which tissue is depicted and parts of the image in which an instrument is depicted can be classified as parts of the background area.

The division can be achieved, in particular, by creating an image mask that represents information about the foreground and background. Such an image mask will be explained in more detail below. The image mask can be represented or encoded by a transmittable signal. Of course, information about the division can also be provided in other ways.

It is particularly possible to identify foreground elements in an image, with the pixels in which the foreground elements are mapped being assigned to the foreground area. The remaining sub-areas can then be assigned to the background area. Alternatively, it is possible to identify background elements in an image, with the pixels in which the background elements are mapped being assigned to the background area. The remaining sub-areas can then be assigned to the foreground area. Similarly, both background and foreground elements can be identified. Pixels that represent neither a foreground nor a background can then be assigned to one of the areas, preferably the foreground area. c) Receiving at least one signal comprising additional information for augmentation. This signal, which is hereinafter referred to as the additional information signal, can be received via an interface of the medical visualization system. Exemplary additional information has already been explained at the outset. Preferably, the additional information signal represents a, in particular, two-dimensional image of the The examination area is provided based on information generated preoperatively or intraoperatively. The additional information signal can be generated by another component of the medical visualization system, such as another image acquisition device or a sensor. The additional information signal can also be retrieved from a storage device of the medical visualization system. It is also conceivable that the additional information signal is retrieved from a higher-level system, such as a network. The additional information signal represents or encodes information that is to be superimposed on the image of the examination area. d) Generating the augmented image by: i. Superimposing the background area or a part thereof with the additional information. This can also be referred to as background overlay. In other words, the overlay with additional information is performed exclusively in the background area or in a part of the background area. ii. Superimposing the image of the examination area or a part thereof with the additional information, whereby the overlay in the background and foreground areas is performed according to different overlay modalities. This can also be referred to as combined overlay.

Both options i. and ii. have in common that the overlay does not occur across the entire image, but rather involves separate or different treatment or overlay of the background and foreground areas. In particular, the background and foreground areas are treated differently for augmentation.

Unless otherwise stated, the descriptions in this disclosure apply to both background overlay and combined overlay. The generated augmented image can then be transmitted, in particular as an image signal, to a display device, which is then controlled to display the to output an image in a visually perceptible manner. In particular, a virtual (3D) image or an augmented (3D) image can contain visible and/or hidden objects or elements, which can then be represented in a visually perceptible way.

In the case of a stereo operating microscope, image signals generated by the two image acquisition units of the stereo system, representing corresponding images of the examination area, can be received. Each of these corresponding images can then be divided into a foreground and a background area. Furthermore, after receiving at least one signal containing additional information, augmented images (corresponding) can be generated by superimposing the additional information in the background area onto each of the images. It is also possible to receive an image-specific additional information signal for augmentation for each of the corresponding images, which is then used to generate the augmented image. For example, virtual image acquisition units can be used to generate an additional information signal representing a virtual image created from or by the additional information. The virtual image acquisition units can be optical models of the image acquisition units of the stereo system. This advantageously results in perspective-correct augmentation, particularly in a three-dimensional and consistent manner. A virtual image can encode a texture, especially with coloring and/or transparency information.

To generate the augmented image, the additional information can be introduced into the beam path, for example, by reflection. This information can be projected onto a projection element, such as a radiolucent disc, positioned in the beam path using a projection device of the operating microscope. The augmented image can then be generated by creating an image based on the beams into which the additional information has been introduced as described. Alternatively or additionally, the radiation representing the augmented image can also be provided by an output section for visual perception by a viewer. Alternatively, an augmented image can be generated by computer-aided enhancement of an image of the real examination area, particularly through image processing. In this process, additional information can be superimposed onto the image of the real examination area. With a stereoscopic operating microscope, it is possible to provide the user with two augmented representations. Generally, corresponding additional information can be introduced into each of the two beam paths of a stereoscopic operating microscope. For example, with digital stereoscopic operating microscopes, augmented images can be generated from the images produced by both image acquisition units. Thus, an augmented image with depth information—that is, an augmented three-dimensional representation—can be provided to the user on a suitable display device or via an output section.

Additional information displayed to a user through augmentation can include, in particular, preoperatively generated information, such as preoperatively generated data, which can also be used for surgical planning. Such preoperatively generated data can be, in particular, volumetric data. Volumetric data can be provided as a point cloud, a voxel-based representation, or a mesh-based representation. The additional information can also be provided, in particular, as a transmittable signal.

Preoperative data can be generated, for example, using computed tomography (CT) or magnetic resonance imaging (MRI). Other imaging techniques, particularly ultrasound, X-ray, fluorescence, SPECT (single-photon emission computed tomography), or PET (positron emission tomography), can also be used. Such augmentation allows, for example, a tumor object or its contours, generated based on preoperative information, to be superimposed onto a white light image.

Preoperative data generated using magnetic resonance imaging (MRI) can identify various tissue types, such as adipose tissue, muscle tissue, tumor tissue, as well as blood vessels and nerve pathways. Preoperative data generated using a computed tomography-based method can be used to visualize bony structures in particular.

Alternatively or additionally to using preoperatively generated information to provide the augmented image, intraoperative information—that is, information acquired or generated during treatment—can be used as supplementary information for generating the augmented image. For example, information can be collected and stored during surgery and then subsequently used to generate an augmented image. This is particularly advantageous when different visualization modalities are activated at different times. For instance, fluorescence information can be displayed in a fluorescence visualization modality or superimposed on a white light image.

The additional information can be assigned a reference coordinate system, which means that the additional information can also include spatial information. This reference coordinate system can also be called a world coordinate system.

Augmentation typically requires registration between the reference coordinate system of the additional information and a reference coordinate system of the medical visualization system, particularly the operating microscope or its image acquisition unit. This registration can be performed prior to augmentation. The registration establishes a spatial relationship between both the additional information and the image to a common reference coordinate system, especially for the information in the image generated by the operating microscope's image acquisition unit. This common reference coordinate system, hereinafter also referred to as the reference coordinate system, can be, in particular, the reference coordinate system of the additional information, the reference coordinate system of the medical visualization system, or a different reference coordinate system altogether.

Various methods can be used for registration, for example, model-based registration. In this process, features can be detected in an image that correspond to previously known features, e.g., geometric features. Features in the supplementary information correspond to the registration, which can then be determined based on these corresponding features. The registration can, for example, be determined in the form of a transformation matrix that includes a rotation and/or translation component. An exemplary model-based registration can be edge-based, where the corresponding features are formed, for example, by a property of at least one, preferably several, edges in both the image and the supplementary information. Topography-based registration is also possible, particularly if a topography can be determined, e.g., with a stereo system of an operating microscope. Topographic information can thus be determined in the at least one image, and corresponding features, points, or sections can then be detected in both the supplementary information and this topographic information, which can then be used to determine the registration.

The additional information can therefore be registered information. For the purposes of this invention, the property "registered" can mean that a spatial reference to the reference coordinate system is known, in particular in the form of a transformation matrix. A registered device can generate signals whose spatial reference to the reference coordinate system is known.

The additional information can be generated primarily through rendering, a process already described in the introduction. A virtual image can be created through rendering, which can then be used for augmentation and, for example, superimposed on a live-recorded image of the real-world area under investigation. The virtual image can also be provided as an image signal that encodes or represents the virtual image.

The virtual image can be generated using a virtual image acquisition device, which can be a mathematical or physical, and in particular computer-aided, optical model of an image acquisition device. Specifically, a computer-implemented calculation of the pixels of the virtual image can be performed. This virtual image depends, among other things, on parameters of the (modeled) image acquisition device. In particular, the virtual image can be used for microscopic imaging depending on the intrinsic parameters of the image acquisition device. can be generated, especially with these parameters. If corresponding images of a virtual stereo system are generated, these can additionally be generated for microscopic imaging depending on the extrinsic parameters of the two image acquisition devices, especially with these parameters.

In other words, when evaluating the model to generate virtual images, the parameters of the operating microscope's image acquisition device(s) used for microscopic imaging can be taken into account. This makes it possible to generate virtual images under the same conditions as real images.

The virtual image can also be generated depending on the pose, i.e., the position and/or orientation, of the (modeled) image acquisition device of the operating microscope. In particular, when evaluating the model to generate the virtual images, the pose of the operating microscope's image acquisition device(s) used for microscopic imaging can be taken into account, utilizing the registration information described above. By considering the registration information, it is possible, for example, to determine which pose of the virtual image acquisition device in the reference coordinate system of the additional information, which can also be referred to as the render coordinate system, corresponds to the actual pose of the (modeled) image acquisition device of the operating microscope, and this information can be used for the rendering process. In other words, a pose of at least one virtual image acquisition device can be identical to the pose of the modeled image acquisition device in the reference coordinate system. Thus, it is possible to generate a virtual image that corresponds to the modeled image acquisition device in terms of both parameters and acquisition pose. For example, an image of a tumor object to be superimposed can be generated by rendering and then transmitted as an image or video signal and used for augmentation.

For the provision of virtual images, it may be necessary to determine the current pose of the operating microscope, particularly the image acquisition unit. This pose can be determined using a position detection unit. Registration allows a relationship to be established between the reference coordinate system of the position detection unit and the previously described reference coordinate systems, especially the reference coordinate system of the additional information. This enables the pose of the The position of an operating microscope within a desired reference coordinate system, particularly the reference coordinate system, can be determined. Depending on the position of the operating microscope, the position of the objective's optical axis or the position of a focal point can be determined. If the operating microscope is mounted on a stand with at least one joint, its position can also be determined based on the joint position, which can be detected, for example, by a sensing device or sensor.

The position of the operating microscope and the additional information can define a previously described scene, i.e., a virtual spatial model that defines objects and their material properties, light sources, and the position and viewing direction of an observer, here the operating microscope.

Naturally, it is possible for the position detection device, or another position detection device, to also detect the pose of at least one other subject or object, or a part thereof. A subject can be, in particular, a user of the medical visualization system, such as someone observing the examination area or a display device. For example, it is conceivable to determine the pose of a body part of such a user, such as a hand, arm, or head. An object can be, in particular, another component of the medical visualization system, especially a display device. However, an object can also be an item that is not part of the medical visualization system, such as a piece of equipment like an operating table or a medical instrument. This makes it possible to determine the pose of the other subject or object within a desired reference coordinate system.

Such a position detection device can also be called a tracking system. A tracking system can be optical, electromagnetic, or operate in some other way. The tracking system can be marker-based, detecting active or passive markers. Markers can be attached to objects or subjects whose pose is to be detected by the tracking system. An optical tracking system can, in particular, include optically detectable markers. This is a system for monoscopic position detection. Here, the pose of an object can be determined by evaluating a two-dimensional image, specifically a single two-dimensional image. In particular, the position can be determined by evaluating the intensity values of pixels (picture elements) of the two-dimensional image.

It is further conceivable that the medical visualization system includes at least one image acquisition device of an optical position detection device, which may in particular be a component of the operating microscope. This can also be referred to as a peripheral camera and serves in particular for monoscopic position detection.

A tracking system can also be part of an input device, where, for example, gesture control or gaze direction control is performed depending on information generated by the tracking system.

In background overlay, additional information can be superimposed according to a predefined overlay modality. In particular, the additional information can be generated with a predefined rendering style.

Various rendering styles, from which the predefined rendering style can be selected, include, for example, transparent rendering, in particular alpha-blending-based rendering, animation-based rendering, texture-based rendering, or Fresnel-effect-based rendering. Another rendering style can be a wireframe rendering style, which only displays the edges or edge lines of an object. In such a rendering, no filled areas or textures are displayed. A rendering style can also be a style that sets at least one, preferably several or even all of the following effects: color, texture, transparency, animation, or Fresnel effect. With background overlay, the overlay with additional information can be applied exclusively to the background area or to a portion of the background area.

In combined overlay, the additional information is overlaid in a domain-specific manner. For example, the overlay in the background area can be with background-specific additional information according to a background-specific overlay modality, and the overlay in the foreground area with Foreground-specific additional information is displayed according to a foreground-specific overlay modality. This area-specific overlay ensures that additional information in the background is displayed differently in the augmented image than in the foreground. In other words, area-specific overlay can create an augmented image in which background-specific additional information is displayed according to a first visualization modality and foreground-specific additional information is displayed according to a second visualization modality that differs from the first. A visualization modality refers to the way in which an image is displayed for visual perception and/or what information is displayed in the image. In particular, a visualization modality can be used to display the image with predetermined color properties and/or predetermined transparency properties. It can also be used to display only modality-specific areas, properties, or features of a (virtual) object, such as only surfaces and/or borders and/or edges.

For area-specific overlays, background-specific and foreground-specific additional information can also be generated using different rendering styles. These different rendering styles can vary in the number of effects, their specific parameters, and/or the combination of effects.

The additional information mentioned can also be generated using the same rendering style, although this is executed differently for generating background-specific additional information than for generating foreground-specific additional information, in particular with different (render) parameters.

In combined overlay, the elements, especially pixels, of an image mask, each representing a pixel's position in the foreground or background, can also be assigned information about the background-specific overlay modality or the foreground-specific overlay modality. For example, the pixels of the image mask representing a background position can be assigned a first degree of transparency. For the overlay, and the pixels of the image mask that represent membership in the foreground area, a further transparency level for the overlay is assigned, whereby the further transparency level does not represent complete transparency but a higher transparency than the first transparency level.

In both background overlay and combined overlay, it is possible for an overlay modality in the edge regions of a foreground or background region, and/or in a transition region between a foreground and a background region, to differ from an overlay modality in the remaining portions of the foreground or background region. Various overlay modalities and their provision have been explained previously. For example, a transition region can include pixels whose distance to the nearest pixel in the foreground region and distance to the nearest pixel in the background region is less than a predetermined distance threshold. In particular, the edge- or transition region-specific overlay modality can change, especially depending on the distance to an edge/transition. For example, the degree of transparency can change, increasing especially towards the edge/transition. Furthermore, the additional overlay information in a transition region can be generated using a predetermined rendering style, such as the wireframe rendering style.

The method according to the invention advantageously enables improved display quality of an area of investigation represented by an augmented image, since, according to the background overlay, the overlay being applied exclusively in the background area reliably prevents contradictory perceptions of information by the viewer. This is particularly important because additional information, which, as explained above, lies behind a surface of the area of investigation, is not erroneously displayed in the foreground. In other words, this occlusion-correct display ensures correct 3D depth perception of augmented additional information. The combined overlay also results in improved display quality, as the selection of different overlay qualities ensures that a viewer can perceive an image with correct depth perception and still receive information about augmented additional information even in a foreground area. Unlike pure object recognition for detecting foreground elements, the proposed method offers the advantage of increasing the information content of the augmented image, since the assignment to foreground or background is not based on object recognition. Instead, it allows an object to be assigned to the foreground or background depending on the scenario.

In a further embodiment, at least two image signals are received, generated sequentially by the at least one image acquisition device of the operating microscope, each representing an image of the examination area. Furthermore, information about temporal changes in the image information as a function of these at least two images is determined, and the division of the image, in particular the image generated earlier or later, into foreground and background regions is performed based on this information. The change in the image information can be determined, in particular, as a function of the change itself or as the change between the at least two images, e.g., as the difference between the images or dependent on this difference.

Information about temporal changes in image information can preferably be information about the optical flow, which can be determined based on at least two images. The optical flow information represents motion information, specifically the apparent movement of brightness patterns in an image sequence. The optical flow of the described image sequence can, for example, be defined as a vector field representing the velocity of visible points in object space projected onto the image plane, particularly within the coordinate system of the image acquisition device. Specifically, each pixel of the image can be assigned a quantity representing the optical flow. In this case, a pixel can be assigned to the foreground if its assigned quantity is greater than a predetermined threshold. If the assigned quantity is less than or equal to the predetermined threshold, the pixel can be assigned to the background. This advantageously results in a simple and reliable division into foreground and background regions, enabling good image quality of the augmented image. As an alternative to determining information about optical flow, other methods can be used to determine information about temporal changes in image information, e.g., methods for determining a motion field. The motion field provides motion information, particularly 3D motion, for each pixel of an image. Specifically, the motion field can represent the true, especially three-dimensional, motion at each point that is mapped onto the 2D image.

In a further embodiment, depth information is determined for at least one image of the examination area, with the division depending on this depth information. Depth information can, for example, represent the distance of the element imaged at a pixel from a reference point, a reference line, or a reference surface. This distance can be determined along a reference direction. In particular, the distance can be determined within the reference coordinate system. For example, the reference surface can be oriented perpendicular to the optical axis of the operating microscope, with the reference direction being parallel to the optical axis. The reference surface can, for example, include an intersection point between the optical axis and an end lens of the operating microscope.

Depth information can be determined using the described device for determining depth information, for example, a distance sensor. Such a distance sensor could be, for example, an OCT distance sensor, a lidar sensor, a time-of-flight sensor, or a triangulation sensor such as a fringe projection sensor. Of course, distance sensors that generate distance information according to other physical principles can also be used.

The device for determining depth information can be registered, meaning the depth information can be registered information.

It is also possible to determine, in particular, registered reconstruction information from corresponding images of a stereo system, which represents a three-dimensional reconstruction of the detection area, wherein the Depth information can be determined based on this reconstruction information. For example, it is possible to detect a surface of the examination area in the three-dimensional reconstruction and then determine the distance from the previously described reference point, line, or surface. Furthermore, it is possible to determine depth information from preoperatively generated data. For instance, a skull surface or a dural surface can be detected in preoperative data, and then, as previously described, a distance from the previously described reference point, line, or surface can also be determined.

In particular, a depth map can be generated for the image of the area under investigation, assigning depth information to each pixel. A pixel can then be assigned to the foreground if its depth information is greater than a predetermined minimum threshold and/or less than a predetermined maximum threshold. Otherwise, the pixel can be assigned to the background. It is also possible to determine a statistical parameter for the depth information generated in this way, such as a mean or median of all depth information assigned to the pixels. If the depth information assigned to a pixel is greater than or less than the median or mean, this pixel can be assigned to the foreground; otherwise, it is assigned to the background. This also advantageously results in a simple and reliable division into foreground and background regions, which enables good image quality of the augmented image.

In a further embodiment, additional semantic information is determined for the image of the area under investigation, or at least a sub-area thereof, with the division being carried out depending on this additional semantic information. The additional semantic information can be information that is taken into account during the division in addition to other information, e.g., in addition to the previously described information about a temporal change in the image information and/or the depth information. In particular, in addition to information required for visually perceivable representation, such as color and/or transparency information, semantic information can be assigned to the image, especially to a pixel or a set of pixels. Semantic information can be, in particular, descriptive information about the type of depicted element, its relationship to other depicted elements, or to the environment. This information can be determined image-based, for example, using a semantic segmentation method. Additional semantic information can therefore be information about or derived from a semantic segmentation of a depicted scene, where the segmented scene can be divided into image areas, each assigned a class from a set of predefined classes. The set of predefined classes could, for example, include the class "fabric," the class "instrument," the class "cloth," the class "swab," and/or another class.

Semantic supplementary information can include, in particular, information that can be evaluated for the described division, i.e., division-relevant information. By evaluating semantic supplementary information, each pixel of the image can be classified as containing either a background or a foreground element. Specifically, the semantic supplementary information can differ from the information required for the visually perceptible representation of the image. Considering semantic supplementary information advantageously results in a very reliable division into foreground and background areas, which in turn improves the image quality.

In a further embodiment, contextual information is determined for the image of the area under investigation, or at least a sub-area thereof, with the division depending on this contextual information. Contextual information can, in particular, be information about a temporal or spatial context. Contextual information can also include information about the type of current user activity, the type of current operational phase, or the type of operation. In particular, the contextual information can differ from the information required for the visually perceptible representation of the image. By taking contextual information into account, a very reliable division into foreground and background areas is advantageously achieved, which in turn improves the image quality.

In another embodiment, the division is carried out using a model generated by machine learning. The term machine learning This encompasses or refers to methods for determining the model based on training data. For example, the model can be determined using supervised learning methods, where the training data (i.e., a training dataset) comprises input data and output data. Input data can consist of images representing the area under investigation or the corresponding image signals, while output data consists of the division of the respective image into foreground and background regions.

For example, input and output data for such training data can be generated by having a user manually select the foreground and background areas in the images. This allows the model to learn the relationship between images and their division into foreground and background. However, it is also conceivable that unsupervised learning methods could be used to determine the model.

Suitable mathematical algorithms for machine learning include: Decision tree-based methods, Ensemble methods (e.g., boosting, random forest)-based methods, Regression-based methods, Bayesian methods (e.g., Bayesian belief networks)-based methods, Kernel methods (e.g., support vector machines)-based methods, Instance (e.g., k-nearest neighbor)-based methods, Association rule learning-based methods, Boltzmann machine-based methods, Artificial neural networks (e.g., perceptron)-based methods, Deep learning (e.g., convolutional neural networks, stacked autoencoders)-based methods, Transformer-based methods, Dimensionality reduction-based methods, and Regularization methods-based methods.

After the model has been created, i.e., after the training phase, the parameterized model can be used in the so-called inference phase to generate the foreground and background partitioning from images. This results in a reliable and high-quality partitioning. The model can be trained once, preferably with a sufficiently large dataset (training phase), and then used as a model (inference phase).

In another embodiment, the partitioning is carried out using a neural network. For example, the neural network can be an autoencoder, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), a neural transformer network, or a combination of at least two of the aforementioned networks. Such a neural network, particularly one configured as an autoencoder, can be trained using the training data described above, after which the partitioning can be performed. The advantage of using an autoencoder for the neural network is that the computational effort required for the partitioning is low, allowing it to be performed reliably and quickly, especially by embedded systems.

Training a CNN advantageously reduces network complexity, making it suitable for devices with limited computing power. This applies to both the training and inference phases. Furthermore, the training time required for CNNs is short, particularly compared to LSTM networks, which also require comparatively higher computing power. However, LSTM network training is especially well-suited for time series analysis because its architecture incorporates temporal dependencies. This results in a high-quality partitioning.

In a further embodiment, input variables for the model-based partitioning include, in addition to the at least one image of the examination area or the corresponding image signal, at least one of the following pieces of information: a) depth information for the at least one image of the examination area, b) at least one image of the examination area generated temporally prior to the image of the examination area, c) an image corresponding to the image of the examination area, which was generated by another image acquisition device of the operating microscope, d) information for classifying imaged objects, e) information for classifying a user activity, f) information for classifying a surgical phase, g) information for classifying a surgical type.

If the operating microscope includes a stereo system with two image acquisition devices and the image of the examination area was generated by one of the image acquisition devices, then the corresponding function described above can be used. The image is generated by the remaining image acquisition unit of the stereo system. The information according to points d) and e) can constitute semantic (additional) information. The information according to points f) and g) can constitute contextual information. Thus, input variables for the model-based partitioning can be image information, e.g., in the form of a microscopic white-light image and, if applicable, an image according to point b) or c), and/or image processing information, e.g., information about a temporal change in the image information and/or information according to point a), and/or semantic (additional) information and/or contextual information.

Information for classifying depicted objects can be information about a class to which a depicted object is assigned. An example class could be "fabric" or "instrument".

Information for classifying user activity can be information about a class of user activity, for example, the class "cutting", "suction", "ablating", "coagulating", "retracting" or "swabbing".

Accordingly, information for the classification of an operational phase can be information about the class of the current operational phase, for example the class "opening", "exploration", "resection", "reconstruction", "wound closure", "wound healing", "planning of opening", "exposure".

The information regarding the classification of the surgical type can be a class of surgical type, for example, "neurosurgical surgery," "orthopedic surgery," or "ophthalmic surgery." Neurosurgical surgeries can also be classified into classes such as "tumor surgery," "vascular surgery," or "spinal surgery." Ophthalmic surgeries can also be classified into "anterior segment surgery" and "posterior segment surgery," with anterior segment surgery including procedures such as cataract treatment or Descemet membrane endothelial keratoplasty. Posterior segment surgery includes procedures such as (membrane) peeling or vitrectomy.

The depth information can be determined as previously explained. The classification information can be determined by a classification system, with the classification being particularly image-based, i.e., through evaluation. The classification information can be generated from at least one image, in particular from the image acquisition device of the operating microscope or from a different image acquisition device. The classification information can also be predetermined. It is also possible that the classification information is determined based on control signals generated for or within the medical visualization system. Such information can also be generated by operating an input device.

By taking into account one, several or all of the aforementioned information in addition to the at least one image of the area under investigation, it advantageously results in the reliability and accuracy of the division into foreground and background areas being improved, which in turn improves the user's perception of the augmented image.

In a further embodiment, a prediction of image positions in the foreground and/or background area, or a sub-area thereof, is performed, with the division occurring depending on a predicted image position. Through this prediction, the division into foreground and background areas for the image generated at the current time can be predicted from images generated at earlier times and then taken into account when generating the augmented image at the current time.

The division into foreground and background areas can also be predicted for a future image (i.e., a subsequent image) and then taken into account when generating the augmented image at that future time. Such a prediction of the division in a subsequent image can also consider the image generated at the current time.

In other words, prediction can be used to forecast the spatial division of an image into foreground and background areas. In particular, prediction can take into account the movement of an imaged element. Thus, prediction is preferably used to forecast the foreground or a part thereof, especially when it contains a moving foreground element such as an instrument, a glove, a finger, a suction cup, etc. For example, prediction can be used to forecast and/or take into account where the moving foreground object is moving or will move. In particular, it can also be used to predict the division of a subsequent image or sequence of subsequent images into foreground and background areas. As explained above, predicting the position of the foreground image can also predict the position of the background image, and vice versa, namely as the remaining portion of the image.

This advantageously enables further improved display quality. In particular, latencies during overlaying, such as those caused by segmentation, can be avoided or reduced, and real-time or near-real-time overlaying is possible. Such latency can arise especially when segmentation requires processing time. This ensures high display quality even with moving foreground elements that are not to be augmented or are to be augmented in a different way.

It is also possible that, based on the division of an image generated at an earlier time into foreground and background areas, the sub-area(s) of a subsequent image can be identified that need to be considered in the prediction, particularly for determining motion information. In other words, it is possible to determine from previously determined division results, which are available, for example, in the form of already generated image masks, which

• To analyze sub-areas of an image or subsequent image generated at the current time for prediction purposes, for example because the representation of moving elements is expected there, and/or

• Sub-areas of an image or subsequent image generated at the current time cannot be analyzed for prediction, for example because no representation of moving elements is expected there.

This can advantageously reduce the image processing effort required for prediction.

For prediction, several images relating to previous [data/data] can be used.

at points in time, i.e. before the point in time when a current image is created, Information about the image positions of the (partial) area was determined. This information can be stored in a retrievable format. It can include, in particular, information about the (image) position and/or the shape and/or size of the foreground and/or background area (or the selected sub-areas). This information can be determined, for example, based on the image masks that were defined for images generated at earlier times.

Then, based on this information, a prediction method can be used to forecast the image positions of a (sub)area in the image generated at the current time or in a subsequent image, for example, in the form of an image mask. For instance, based on the information, motion information for a (sub)area can be determined. This motion information can be information about a motion trajectory, such as the motion trajectory of a reference point (e.g., the geometric center of the (sub)area or a pixel), and/or information about motion parameters such as velocity. Then, based on this motion information, the image position in the current image and/or in a subsequent image (i.e., an image generated at a future time) can be predicted, for example, using an extrapolation method. Similarly, information about changes in shape and/or size can be determined.

It is possible that the division is carried out according to the predicted image position of the (sub)areas.

In general, it is also possible to fuse multiple pieces of partitioning information to determine the resulting partition. The set of partitioning information that can be fused can include the aforementioned prediction information, depth information, additional semantic information, contextual information, and information about temporal changes.

For example, it is possible that the prediction is based on the image masks, whereby, depending on image masks that were created at earlier times, i.e. before the time of the creation of an image mask for the current image, information about the coordinates of image points representing membership in the foreground area and/or of image points representing membership in the background area is determined in the image mask for the current image. In another embodiment, confidence information is determined when splitting the image, with the splitting and/or superimposition being confidence-dependent.

The respective confidence information can, in particular, represent the confidence level in assigning a pixel or image area to one of the areas foreground or background. In other words, this information can represent how accurate the corresponding assignment is, or with what certainty the assignment is determined. If the division is performed using a machine learning model (MLM), the confidence level can be (co-)determined during processing by the MLM.

The confidence information can be provided as a confidence value, which can take values from a predetermined range between a first limit representing maximum confidence and another limit representing minimum confidence.

For example, confidence-based partitioning can be achieved by assigning a pixel/image area to one of the areas only if the corresponding confidence value exceeds or falls below a predetermined confidence threshold. Otherwise, it can be assigned to the remaining area, or alternative methods can be used to assign these pixels/image areas to one of the areas.

In particular, pixels/image areas assigned to the background with a low confidence value can be reassigned to the foreground based on confidence, for example, if the confidence value for the background assignment is lower than a predetermined confidence threshold. This allows the foreground area to be enlarged based on confidence. This reduces the risk of additional information being erroneously superimposed in an area identified as the background with low confidence. This can advantageously improve the display quality.

It is also possible that pixels/image areas assigned to the foreground or background with a low confidence value may be assigned to a different area depending on the confidence level. A transition region can be assigned, for example, if a confidence value of the assignment represents a lower confidence level than a predetermined confidence threshold. This confidence-dependent assignment to the transition region preferably only occurs if at least one additional assignment criterion is met, for example, if the distance of the pixel/image area to the nearest foreground region, in particular to the nearest foreground pixel, is less than a predetermined distance threshold, and/or if the distance of the pixel/image area to the nearest background region, in particular to the nearest background pixel, is less than a predetermined distance threshold. If at least one of these additional criteria is not met, the assignment to the foreground region described above can still be made.

The predetermined confidence thresholds for assigning a result to the foreground region and the transition region can be the same or different. Preferably, the predetermined confidence threshold for assigning a result to the foreground region is lower than the threshold for assigning it to the transition region.

For example, confidence-based overlaying can be achieved by applying a confidence-based overlay modality. Overlaying according to various overlay modalities has already been explained. In particular, coloring and/or transparency properties can be set based on confidence. For example, the lower the confidence level, the higher the transparency, especially for pixels/image areas in the background. In other words, the additional information can be displayed in the relevant areas using a confidence-based visualization modality. Confidence-based rendering styles can also be applied to display the additional information. This allows viewers to easily assess the reliability of the overlay with additional information, which in turn improves the overall display quality.

Confidence values representing low confidence can be generated particularly when assigning pixels in blurred images or in blurred image areas, which can occur during image acquisition by the operating microscope, especially in cases of defocusing. This ensures good image quality even in such scenarios. Confidence-based overlay can improve image quality, particularly in border areas and/or transitions between foreground and background. If it is assumed that the assignment of pixels to one of the areas tends to occur with low confidence in border areas/transitions, a viewer can easily perceive such border areas/transitions and the reliability of the overlay there with additional information, which in turn also improves image quality.

In a further embodiment, after the image of the area under investigation has been divided into foreground and background areas, but before the superimposition, the size of the foreground area is changed, particularly at least with respect to a sub-area of the foreground. The size of the foreground area can be enlarged or reduced, in particular by assigning pixels from the background area surrounding a sub-area of the foreground area to the foreground, or by assigning pixels from a sub-area of the foreground area to the background. If the foreground area is enlarged, the background area is reduced. If the foreground area is reduced, the background area is enlarged accordingly. The change can be geometric; for example, a circular, ellipsoidal, rectangular, or square enlargement or reduction can be performed, whereby, for example, all pixels arranged in a geometric shape around a reference point of the sub-area are assigned to the foreground. Properties of the change, such as its geometric shape and/or the magnitude of the change, can be predetermined.

Alternatively or cumulatively, the size of the background area is changed, in particular at least with respect to a sub-area of the background area, especially by being enlarged or reduced. The explanations given for changing the foreground area apply accordingly.

Such a subsequent change in the size of the foreground and/or background area advantageously results in a further improved image quality for the augmented image, particularly in certain application-dependent scenarios. For example, it can be achieved that an image area containing a part in contact with the tissue, e.g., the tip of an instrument, is rendered more accurately. The area shown, classified as the foreground, is enlarged to allow the user to perceive the contacted tissue without augmentation. It is also possible to reduce the size of foreground sections containing little information. This further improves the display quality of the provided augmented image.

In a further embodiment, a sub-area of the foreground or background to be modified is determined by an object detection method. The sub-area to be modified can designate a sub-area of the foreground or background with respect to which the size of the foreground or background is changed. Alternatively, the sub-area to be modified can also designate a sub-area whose size is changed.

As explained above, the size of the foreground area can be increased or decreased, in particular by assigning pixels from the background area in a section adjacent to, or at least partially surrounding, a portion of the foreground area to the foreground, or by assigning pixels from a portion of the foreground area to the background. Similarly, the size of the background area can be increased or decreased, in particular by assigning pixels from the foreground area in a section adjacent to, or at least partially surrounding, a portion of the background area to the background, or by assigning pixels from a portion of the background area to the foreground.

The object recognition method can, in particular, be an image-based method, which assigns at least one pixel to an object by evaluating at least one image. Such methods are known to those skilled in the art. For example, it is possible to identify specific sections, such as the tip of an instrument, in the image using an object recognition method. The size of the foreground area can then be increased with respect to this section. In this case, the properties of the change can be object-dependent, with predetermined properties of the change being assigned to an object. This advantageously results in a reliable identification of a section, which in turn improves the previously described image quality. In a further embodiment, a sub-area to be modified is determined as a function of at least one optical parameter of the operating microscope. Optical parameters can be, in particular, the operating parameters described above. For example, an image area depicting foreground elements that are not arranged within a predetermined interval around a set working distance of the operating microscope can be defined as the sub-area of the foreground to be reduced in size. It can be assumed that such foreground elements are rendered blurry by the microscope, i.e., with insufficient image quality, where image quality can, in particular, represent image sharpness. It can be advantageous to enlarge or reduce such foreground sub-areas with insufficient image quality. In other words, an image area with insufficient image quality can be enlarged or reduced, where the image quality can, for example, depend on the degree of blurriness of the elements depicted in the image area.

By reducing the size of such a (foreground) area, an additional image area can be created for augmentation, but this additional image area only includes areas in which image information such as tissue or instruments are depicted with insufficient image quality.

Enlarging such a (foreground) area can prevent augmentation with sufficient image quality from occurring next to an area of insufficient image quality, for example, by creating a sharp image area next to a blurry one. This type of representation can advantageously reduce viewer confusion. Such enlargement can be particularly useful when the image quality is below a predetermined level.

It is also conceivable that augmented information, particularly in the vicinity of an image area with insufficient image quality, could be displayed with the same insufficient image quality. Such a display of augmented information is particularly possible if the image quality is better than or equal to a predetermined level. This also allows for the advantageous and reliable identification of sub-areas that need to be changed, which in turn improves the previously explained presentation quality.

In another embodiment, the change is carried out depending on a) a distance of the object or element depicted in the sub-area from a surface of the investigation area, b) sub-area-specific image information.

In particular, a property of the change can be chosen to depend on at least one of the aforementioned quantities. For example, the magnitude of the change can be positively or negatively correlated with the distance. The distance can be determined, in particular, depending on the depth information explained above. Of course, other methods of determining the distance are also conceivable.

Image information can be a property of the image and can be determined image-based. It can be, in particular, blur information, shadow information, or contrast information. Sub-area-specific image information can also be determined by a suitable device. In particular, this information can be determined image-based. For example, it is possible to perform a blur classification procedure, a shadow classification procedure, and/or a contrast classification procedure to identify image areas, especially in the foreground, that are blurred and/or shadowed by more than a predetermined degree and/or lack sufficient contrast. Such areas can then be reduced in size, or even completely removed from the foreground. It is also conceivable that such areas could be enlarged.

Image quality in an image area can also be determined depending on the image information, e.g., blur information, shadow information, and/or contrast information, whereby the magnification and/or the Reduction in size can then be done depending on the image quality. Reference can be made to the preceding explanations in this regard.

This has the advantageous effect of allowing even parts of the foreground to be used for augmentation, areas that would otherwise provide little information to the user. This, in turn, improves the display quality and increases the information content of the augmented image.

In a further embodiment, a sub-area to be modified and/or the change in size is determined by means of a model generated through machine learning, in particular by means of a neural network. Reference can be made to the preceding explanations regarding the division of the image into a foreground and a background area using a model. The input data for a training dataset can consist of images and the division of these images into foreground and background areas, or the corresponding image signals. The output data can be information about selected sub-areas and/or the changed division into foreground and background areas. For example, input and output data for such training data can be generated by a user manually selecting the sub-areas and/or the changes to the foreground and background areas. In this way, the model can learn the relationship between images and their division into foreground and background areas, as well as the subsequently changed division. This results in a reliable and high-quality determination of the sub-area to be modified and/or the change in size.

In a further embodiment, input variables for determining the sub-area to be modified and/or the change in size include at least one of the previously described pieces of information a) to g) for the model-based partitioning. This advantageously improves the reliability and accuracy of determining the sub-area to be modified and/or the change in size, which in turn improves the user's perception of the augmented image.

A medical visualization system is further proposed, in particular comprising or consisting of an operating microscope which includes at least one The medical visualization system comprises an interface for receiving an image signal from an image acquisition device for generating an image of an examination area and at least one evaluation unit. The medical visualization system, in particular the evaluation unit, is configured to perform a method according to one of the embodiments described in this disclosure. As explained above, the medical visualization system may, in particular, further comprise at least one of the following: a device for determining information for classifying imaged objects and/or user activity and/or a surgical phase and/or a surgical type; a device for determining the distance of an imaged object from a surface of the examination area; and a device for determining area-specific blur and/or shadowing information.

The evaluation unit can be configured as a computing unit or include one. A computing unit, in turn, can include at least one microcontroller and/or at least one integrated circuit, or be configured as such. The computing unit can, in particular, generate the virtual image. It is possible that the evaluation unit includes at least one graphics processing unit (GPU) for this purpose. The medical visualization system advantageously enables the execution of a method according to one of the embodiments described in this disclosure, with the advantages already explained.

A further proposal is a computer program product comprising a computer program, wherein the computer program includes software means for executing several or all steps, in particular steps b. and c., of the method according to one of the embodiments described in this disclosure, when the computer program is executed by or in a computer or an automation system. The computer or automation system may include the evaluation device described above. The computer program product may, in particular, include means for performing the rendering, i.e., a rendering engine. The computer program product advantageously enables the Implementation of a method according to one of the embodiments described in this disclosure, with the advantages already explained.

The invention is explained in more detail using exemplary embodiments. The figures show:

Fig. 1a shows an exemplary representation of a superimposition without a division into foreground and background areas according to the invention,

Fig. 1b shows another exemplary representation of an augmented image without a division into foreground and background areas according to the invention,

Fig. 2 shows a schematic flowchart of a method according to the invention,

Fig. 3 shows an exemplary representation of an augmented image produced using the method according to the invention.

Fig. 4a shows a schematic representation of an examination area and an instrument,

Fig. 4b is a schematic representation of an image of the scene depicted in Fig. 4a,

Fig. 4c is a schematic representation of information on optical flow,

Fig. 4d a schematic representation of an image mask,

Fig. 5a shows a schematic representation of an investigation area and a

Instruments,

Fig. 5b is a schematic representation of an image of the scene depicted in Fig. 5a,

Fig. 5c shows a schematic representation of depth information, Fig. 5d a schematic representation of an image mask,

Fig. 6 shows a schematic flowchart of a method according to the invention in a further embodiment,

Fig. 7 shows a schematic flowchart of a method according to the invention in a further embodiment,

Fig. 8a shows a schematic representation of an image mask,

Fig. 8b shows an augmented image generated on the basis of the image mask shown in Fig. 8a,

Fig. 9a shows a schematic image mask with a locally enlarged foreground area,

Fig. 9b shows an augmented image generated on the basis of the image mask shown in Fig. 9a,

Fig. 10a a schematic representation of an operation scene,

Fig. 10b shows a schematic representation of the additional information to be augmented,

Fig. 10c shows an exemplary image mask,

Fig. 10d shows an augmented image generated on the basis of the image mask shown in Fig. 10c,

Fig. 10e shows another exemplary image mask,

Fig. 10f shows an augmented image generated on the basis of the image mask shown in Fig. 10e,

Fig. 10g shows another exemplary image mask,

Fig. 10h shows an augmented image generated on the basis of the image mask shown in Fig. 10g, Fig. 11 shows a schematic block diagram of a medical visualization system according to the invention and

Fig. 12 shows an exemplary representation of an augmented image produced using the inventive method in a further embodiment.

In the following, identical reference symbols denote elements with the same or similar technical characteristics.

Fig. 1a shows an exemplary white-light image of an examination area 1 with instruments 2. A virtual object 3, for example a tumor object 3, is superimposed onto the image. This superimposition is done without considering areas in the image that are suitable for augmentation. It can be seen that the tumor object 3 obscures parts of an instrument 2 and the tissue.

Fig. 1b shows the same scene as Fig. 1a, except that the tumor object 3 is depicted as more transparent than in Fig. 1a, in particular as semi-transparent. Despite the now possible visibility of parts of the instrument 2 that were obscured by the tumor object 3 in Fig. 1a, the perception of the tissue areas and instrument sections obscured by the tumor object 3 is nevertheless impaired. Furthermore, the spatial impression arises that the tumor object 3 is hovering above the tissue, which makes a spatially accurate perception of the scene difficult for the viewer.

Fig. 2 shows a schematic flowchart of a method according to the invention for generating an augmented image AA using a medical visualization system 4 (see Fig. 11). In a receive step ES1, an image signal is received which was generated by at least one image acquisition device 5 for microscopic imaging (see, for example, Fig. 11) and which represents an image of an examination area 1. This image A1 is divided in a partitioning step AS into a foreground area V and a background area H (see, for example, Fig. 4d). Exemplary partitioning possibilities are explained below. In a further reception step ES2, at least one signal comprising additional information ZI for augmentation, also referred to as the additional information signal, is received. Figure 2 shows that the additional information signal is stored in a retrievable manner in a storage device 6, which can be part of the medical visualization system 4. However, it is also possible for the additional information ZI to be acquired intraoperatively and/or retrieved from a network via a suitable interface. In particular, the additional information signal can also represent an image, preferably an image of the same size as the image A1 generated by the image acquisition device 5. The image represented by the additional information signal can, in particular, be a virtual image generated from preoperatively generated additional information ZI. As already explained at the outset, such a virtual image can be generated with a virtual image acquisition device, which is an optical model of the image acquisition device 5.

In a generation step GS, the augmented image AA is then created by overlaying the additional information ZI onto the image A1 in the background area H or a part thereof.

In the partitioning step AS, an image mask M can be generated that encodes or represents information about the foreground region V and the background region H. The image mask M can be provided, in particular, as an image, especially a two-dimensional image, where the image size of the image mask M can correspond to the image size of the image A1. Each pixel of the image mask M can be classified as either a foreground region V or a background region H. The image mask M can, for example, be a binary image, where foreground region pixels are assigned the value 1 or 0, and background region pixels are assigned the remaining value.

In this case, the overlay of the additional information ZI can only occur for pixels of image A1 whose corresponding pixels are classified as background area pixels in the image mask M. A corresponding pixel can have the same pixel coordinates, which can refer to the same image coordinate systems. In particular, the overlay can be opaque or with a predetermined degree of transparency, especially semi-transparent. For example, an alpha blending method can be used for overlay. In this case, each background pixel of the image mask can be assigned an alpha value between 0 (inclusive) and 1 (inclusive), where the value 0 or 1 represents, for example, complete transparency and the value 1 or 0 represents complete opacity. If a background pixel in the image mask M is assigned an alpha value representing complete opacity, the corresponding pixel in the image A1 can be completely overlaid by additional information. If a background pixel in the image mask M is assigned an alpha value that represents neither complete transparency nor complete opacity, the corresponding pixel in the image A1 will not be completely overlaid by additional information. If an alpha value representing complete transparency is assigned to a background area pixel in the image mask M, the corresponding pixel in the image A1 cannot be overlaid with additional information.

It is also conceivable that at least one, but preferably several, foreground pixel(s) of the image mask are assigned an alpha value that does not represent complete transparency, where the minimum transparency level of all foreground pixels is greater than the maximum transparency level of all background pixels. Thus, an augmented image can be created in which an overlay in a foreground area is displayed more transparently than an overlay in a background area. In this way, a viewer can also perceive additional information in the foreground area; however, this information is less distracting due to the higher transparency.

It is possible to assign alpha values between the measure of complete transparency (exclusive) and the measure of complete opacity (inclusive) to all background area pixels, while foreground area pixels are assigned an alpha value with the measure of complete transparency.

The augmented image AA can then be transmitted to a display device 20 (see Fig. 11), in particular as an image signal, to present it to a viewer in a visually perceptible manner. Fig. 3 shows an exemplary augmented image AA, generated using the method described in Fig. 2. In the scene shown in Fig. 3, the additional information ZI represents the tumor object 3. It can be seen that, unlike the images shown in Fig. 1a and Fig. 1b, the tumor object 3 does not obscure the instruments 2. Thus, the instruments 2 are depicted in the foreground area V of image A1, while tissue areas of the examination area 1 are depicted in the background area H. The augmented image AA shown in Fig. 3 allows the viewer an improved perception of the scene and therefore offers a higher image quality.

Fig. 4a shows an exemplary representation of an instrument 2 and an examination area 1, in particular a surgical surface. Also shown is an image acquisition device 5 of an operating microscope.

Fig. 4b shows an image A1 generated by the image acquisition device 5. The instrument 2 and the examination area 1 are depicted therein, with the instrument 2 obscuring parts of the examination area 1.

Fig. 4c shows, by way of example and indicated by arrows, information on the optical flow in the image A1. This information on the optical flow can be generated by receiving at least two image signals, which were generated sequentially by the image acquisition device 5 and each represent an image A1 of the examination area 1. Methods for determining this information from the sequence of these images A1 are known to those skilled in the art. As indicated by arrows, a quantity representing the optical flow, which is greater than a predetermined threshold, is assigned to a sub-region of the image A1, into which the instrument 2 is imaged. The pixels of this sub-region can then be classified as the foreground region V, i.e., as foreground pixels, while the remaining pixels of the image A1 are classified as the background region H, i.e., as background pixels. The resulting image mask M, consisting of foreground pixels and background pixels, is shown in Fig. 4d. Fig. 5a shows an exemplary representation of an instrument 2 and an examination area 1, in particular a surgical surface. Also shown is an image acquisition device 5 of an operating microscope.

Fig. 5b shows an image A1 generated by the image acquisition device 5. The instrument 2 and the examination area 1 are depicted therein, with the instrument 2 obscuring parts of the examination area 1.

Fig. 5c shows an example of depth information in image A1. This information can be generated, for example, with a distance sensor that detects the distance of the instrument and the uncovered part of the examination area from, for example, a reference plane oriented perpendicular to an optical axis of the operating microscope and in which an intersection of the optical axis with an end glass of the operating microscope is located, along an optical axis of the operating microscope.

It is evident that distance information is assigned to a sub-area of image A1, in which instrument 2 is depicted, representing a distance that is smaller than the distance assigned to the sub-area of image A1 that depicts the unobstructed part of the examination area. In particular, this smaller distance is smaller than a predetermined distance.

The pixels of this instrument sub-area can then be classified as foreground area V, i.e., as foreground area pixels, while the remaining pixels of image A1 are classified as background area H, i.e., as background area pixels. The resulting image mask M, consisting of foreground area pixels and background area pixels, is shown in Fig. 5d.

Fig. 6 shows a schematic flowchart of a further embodiment of a method according to the invention. In contrast to the embodiment shown in Fig. 2, for the division step AS, in addition to the information necessary for the visually perceptible representation of the image A1, in particular color information, further information I is taken into account. Such further information I can be, in particular, additional semantic information or contextual information.

In particular, such semantic or contextual information can be the depth information for image A1 explained in relation to Fig. 5c. If the image acquisition device 5 with which image A1 was generated is part of a stereo system, the information I can also include information about a device related to the image. A1 represents a corresponding image generated by another image acquisition device 5b (see Fig. 11). Alternatively, the information I can represent at least one image of the investigation area 1 generated prior to image A1.

Preferably, the information I also includes information about a classification of objects depicted in image A1. Object recognition can be performed for this purpose, whereby the recognized objects are then classified using a classification procedure. However, this classification does not provide a classification in the foreground and background areas V and H; in particular, for at least one object class from the set of all object classes, there is no unambiguous assignment to the foreground area V or background area H. For example, objects of the class "cloths" or "swabs" can be assigned to the background area, especially if they are statically arranged, e.g., fixed in position relative to the surgical site or lying on the tissue of the surgical site. However, such objects of the class "cloths" or "swabs" can also be assigned to the foreground area, especially if they are not statically arranged, particularly because they are held or even moved by an instrument. An object or area of the class "Fabric" can also be assigned to the background area, particularly if there is no object or area of the class "Instrument" in that fabric area or in a predetermined area around it. Alternatively, an object or area of the class "Fabric" can also be assigned to the foreground area, particularly if there is an object or area of the class "Instrument" in that fabric area or in a predetermined area around it. In the latter case, it can be avoided that an augmentation would disturb a viewer when viewing areas with which an instrument is currently interacting.

The information can also include information about the classification of a user activity, an operation phase, or an operation type. These have already been explained previously.

It is possible that the partitioning step AS is performed with or by evaluating a model generated by machine learning, in particular by evaluating a neural network. An input variable of the The model can be at least one image A1 of the investigation area 1. Another input variable can be the information I explained above. An output variable of the model can be the image mask M explained previously.

Fig. 7 shows a schematic flowchart of a further embodiment of a method according to the invention. In contrast to the embodiment shown in Fig. 6, a modification step VS is performed after the partitioning step for generating the image mask M. In the modification step, a size of the foreground area V is changed, at least with respect to a sub-area of the foreground area V. As a result of the modification step VS, a modified image mask MV is provided. This corresponds to the image mask M, but includes more or fewer foreground pixels and thus correspondingly fewer or more background pixels compared to the image mask M. The dashed lines indicate that the partitioning step AS and/or the modification step VS can optionally be performed depending on the additional information I explained above.

Fig. 8a shows a schematic image mask M with three foreground elements, which correspond, for example, to image sections of the image A1, into which instruments 2 (see Fig. 3) are depicted.

Fig. 8b shows the augmented image AA, which is generated based on this image mask M. It can be seen that an augmented tumor object 3 does not cover the depicted instruments 2.

Fig. 9a shows an exemplary modified or altered image mask MV. In contrast to the image mask M shown in Fig. 8a, it is evident that the foreground area V has been enlarged in the region of a tip or free end of the instruments 2. In particular, the enlargement of the foreground area V was achieved by defining a circular area with a predetermined diameter around a reference point of the sub-area, in this case, for example, around the geometric center of an image mask section classified as an instrument tip, and classifying all image points of the modified image mask MV within this circular area as foreground image points. Fig. 9b shows the augmented image generated based on the modified image mask MV. In contrast to the augmented image AA shown in Fig. 8b, it is evident that a portion of the tissue around the instrument tips is not augmented by the tumor object 3. This allows the viewer to perceive the tissue actuated by the instruments without augmentation.

Fig. 10a shows an exemplary image A1, which was generated by an image acquisition device 5 of a medical visualization system 4 (see Fig. 11).

Fig. 10b shows an exemplary representation of a tumor object 3, which is represented by an additional information signal.

Fig. 10c shows an exemplary image mask M with foreground area V and background area H, which was created without the modification step VS shown in Fig. 7.

Fig. 10d shows the augmented image AA generated on the basis of this image mask M.

Fig. 10e shows an image mask MV modified compared to the image mask M shown in Fig. 10c, with a modified foreground and background area V, H. In the corresponding modification step VS (see Fig. 7), shadowed image areas vB of image A1 were detected, whereby the foreground area V of the image mask M shown in Fig. 10c was reduced by the image areas that are shadowed. Such shadowed image areas vB can be determined, for example, using shadow classification methods. One exemplary classification method compares the color values of pixels in image A1 with predetermined threshold values and classifies a pixel as a shadowed pixel depending on the comparison result.

Fig. 10f shows an augmented image AA, which was generated based on the modified image mask shown in Fig. 10e. It is evident that additional information ZI can also be superimposed on image A1 in the shaded image area vB.

Fig. 10g shows an image mask MV that has been modified in comparison to the image mask M shown in Fig. 10c, with a changed foreground and background area in terms of size. In the corresponding modification step VS (see Fig. 7), in addition to the shadowed image areas vB, blurred image sections uB of image A1 were detected, whereby the foreground area V of the image mask M shown in Fig. 10c was reduced by image areas that are blurred. Such blurred image areas uB can be determined, for example, using blur classification methods. An exemplary classification method for detecting blurred areas can include the application of Laplacian operators, in particular the application of Laplacian pyramids, and threshold operators.

Fig. 10h shows an augmented image AA, which was generated based on the modified image mask shown in Fig. 10g. It is evident that additional information ZI can also be superimposed on image A1 in the out-of-focus image area uB.

Fig. 11 shows a schematic block diagram of a medical visualization system 4 and an examination area 1. Also shown are an instrument 2 and a tumor object 3, which are arranged within the detection range of an operating microscope 10 of the medical visualization system 4. The tumor object 3 can be a hidden object.

Optional elements of the medical visualization system 4 are shown here with dashed lines. The medical visualization system 4 comprises at least one image acquisition device 5 for microscopic imaging of the examination area 1. The image acquisition device 5 can be part of the operating microscope 10, which can include an objective 19 with a lens. This operating microscope 10, in turn, can be configured as a stereo operating microscope, comprising a further image acquisition device 5b for microscopic imaging of the examination area 1, and the image acquisition devices 5 and 5b forming a stereo system. A storage device 6, in which additional information ZI can be stored, is also shown. The medical visualization system 4 further comprises an evaluation device 7, which can receive and evaluate an image A1 generated by the image acquisition device 5, wherein the image A1 can be transmitted as an image signal via an interface 21 to the evaluation device 7. The evaluation device 7 can, in particular, perform the division step AS and the generation step GS. The figure further shows that the medical visualization system 4 can include a device 8 for determining depth information. This device can, for example, be designed as a distance sensor or include one. Also shown is an interface 9 of the medical visualization system 4 for data transmission with other, especially higher-level, systems.

Furthermore, the medical visualization system can include 4:

• at least one white light lighting device 11 ,

• at least one infrared lighting device 12,

• at least one fluorescence illumination device 13 for excitation of fluorescence radiation,

• at least one fluorescence detection device 14 for detecting fluorescence radiation,

• at least one surround-view camera 15,

• at least one device 16 for detecting the gaze direction of a viewer,

• at least one tracking system 17,

• at least one input device 18 for operating or controlling the medical visualization system,

• at least one display device 20.

The elements of the medical visualization system 4 can be connected via data and/or signal technology.

Not shown are beam filters for providing excitation radiation with wavelengths from a broader spectrum, e.g. the spectrum of the white light illumination device, or for filtering radiation from a broader spectrum.

The environmental camera 15 can be part of a further tracking system, which serves in particular for the optical determination of the pose of instruments within the detection range of the environmental camera 15. The pose determination can be monoscopic. In particular, the determination can also be marker-based. Images from the environmental camera 15 can be evaluated, in particular, for object recognition in order to identify a sub-area with respect to which the size of the foreground area V and/or background area H is then changed. Fig. 12 shows an exemplary augmented image AA, which was generated essentially like the augmented image AA shown in Fig. 3. However, unlike the augmented image AA shown in Fig. 3, the superimposition of image A1 with the tumor object 3 occurs in both the foreground region V and the background region H (see Fig. 4d). The superimposition of the background region H with the tumor object 3 occurs according to a first superimposition modality, represented by line hatching, and the superimposition of the foreground region V with the tumor object 3 occurs according to a further superimposition modality, represented by dotted hatching. The superimposition modalities are thus different from each other.

It can be seen that, in contrast to the image shown in Fig. 3, the tumor object 3 overlays the instruments 2, which are shown in the foreground area V, differently than tissue areas which are shown in the background area H.

Reference symbol list

1 examination area

2 Instrument

3 Tumor object

4 medical visualization system

5 Image acquisition device of an operating microscope

5b further image acquisition device of the operating microscope

6 Storage setup

7 Evaluation unit

8 Device for determining depth information

9 Interface

10 Operating microscope

11 White light lighting device

12 Infrared lighting device

13 Fluorescence lighting device

14 Fluorescence detection device

15 Surround camera

16 Device for gaze direction detection

17 Tracking system

18 Input device

19 Lens

20 Display device

21 Interface

A1 image

M Image mask

MV modified image mask

I Information

ZI Additional Information

V Foreground area

H Background area

AA augmented image

ES1 receive step

AS division step

ES2 receive step GS production step

VS Change Step

Claims

Patent claims 1. Method for generating an augmented image (AA) by a medical visualization system (4), comprising the steps of: a. Receiving at least one image signal generated by at least one image acquisition device (5) of a surgical microscope (10) and representing an image (A1) of an examination area (1), b. Splitting the image (A1) into a foreground area (V) and a background area (H), c. Receiving at least one signal representing or encoding additional information (SI) for augmentation, d. Generating the augmented image (AA) by i. superimposing the background area (H) of the image (A1) of the examination area (1) or a part thereof with the additional information (SI), or ii. Overlaying the image (A1) of the investigation area (1) or a part thereof with the additional information (ZI), wherein the overlay in the background area (H) and in the foreground area (V) is carried out according to different overlay modalities. 2. Method according to claim 1, characterized in that the overlay with additional information (ZI) is carried out exclusively in the background area (H) or in a part of the background area (H). 3. Method according to claim 1 or 2, characterized in that at least two image signals are received which were generated successively by the at least one image acquisition device (5) of the operating microscope (10) and which each represent an image (A1) of the examination area (1), wherein information about a temporal change of the image information depending on these at least two images (A1) is determined and the division of the image (A1) takes place depending on this information. 4. Method according to claim 3, characterized in that the information about a temporal change in the image information is information about the optical flow. is. 5. Method according to one of the preceding claims, characterized in that depth information is determined for the at least one image (A1) of the investigation area (1) and the division is carried out depending on this depth information. 6. Method according to one of the preceding claims, characterized in that additional semantic information is determined for the image (A1) of the investigation area (1) or at least a sub-area thereof, wherein the division is carried out depending on this additional semantic information. 7. Method according to one of the preceding claims, characterized in that context information is determined for the image (A1) of the investigation area (1) or at least a sub-area thereof, wherein the division is carried out depending on this context information. 8. Method according to one of the preceding claims, characterized in that the division is carried out using a model generated by machine learning, preferably using a neural network. 9. Method according to claim 6, characterized in that input variables for the model-based partitioning, in addition to the at least one image (A1) of the investigation area (1), comprise at least one of the following information: a. depth information for the at least one image of the investigation area, b. at least one temporally prior image of the investigation area. Image of the examination area, c. an image corresponding to the image of the examination area, which was generated by another image acquisition device (5b) of the operating microscope (10), d. information for the classification of imaged objects, e. information for the classification of a user activity, f. information for the classification of an operation phase, g. information for the classification of an operation type. 10. Method according to one of the preceding claims, characterized in that a prediction of image positions of the foreground area (V) and/or the background area (H) is carried out, wherein the division is performed depending on a predicted image position. 11. Method according to one of the preceding claims, characterized in that confidence information is determined when splitting the image (A1), wherein the splitting and/or superimposition is confidence-dependent. 12. Method according to one of the preceding claims, characterized in that after the division of the image (A1) of the investigation area (1) into the foreground area (V) and the background area (H) a. a size of the foreground area (V) and/or b. a size of the background area (H) is changed. 13. Method according to claim 12, characterized in that a partial area of the foreground area (V) and/or the background area (H) to be changed is determined by a method for object recognition. 14. Method according to claim 12 or 13, characterized in that a partial area to be modified is determined depending on optical parameters of the operating microscope (10). 15. Method according to one of claims 12 to 14, characterized in that the change is carried out depending on a. a distance of the object depicted in the sub-area from a surface of the investigation area (1), b. a sub-area-specific image information. 16. Method according to one of claims 12 to 15, characterized in that a sub-area to be changed and/or the change in size is determined by means of a model which was generated by machine learning. 17. Method according to claim 16, characterized in that input variables for a model-based determination of the sub-area to be changed and/or the change in size comprise at least one of the following information: a. Depth information for the at least one image of the area under investigation, b. At least one image of the area under investigation generated prior to the image of the area under investigation, c. An image corresponding to the image of the area under investigation, which was generated by a further image acquisition device (5b) of the operating microscope (10), d. Information for classifying imaged objects, e. Information for classifying a user activity, f. Information for classifying a surgical phase, g. Information for classifying a surgical type. 18. Medical visualization system (4) comprising at least one interface (20) for receiving an image signal from an image acquisition device (5) for generating an image (A1) of an examination area (1) and at least one evaluation device (7), wherein the medical visualization system (4) is configured to perform a method comprising the steps according to any one of claims 1 to 17. 19. A computer program product comprising a computer program, wherein the computer program includes software means for performing several or all steps of the method according to any one of claims 1 to 17, when the computer program is executed by or in a computer or an automation system. ...