CN118212277B

CN118212277B - Organ-based multi-modal registration method and device

Info

Publication number: CN118212277B
Application number: CN202410626457.7A
Authority: CN
Inventors: 王杉杉; 徐孟程; 雷飞飞
Original assignee: Kaben Shenzhen Medical Equipment Co ltd
Current assignee: Kaben Shenzhen Medical Equipment Co ltd
Priority date: 2024-05-20
Filing date: 2024-05-20
Publication date: 2024-09-27
Anticipated expiration: 2044-05-20
Also published as: CN118212277A

Abstract

The application relates to a multi-mode registration method and a device based on organs, which are characterized in that after a target image sequence is acquired, an organ image and a three-dimensional model are obtained through organ identification and three-dimensional reconstruction, the relative positions of an object to be inspected and a magnetic field generator are confirmed, the acquisition guide based on the three-dimensional reconstruction model is displayed in a display area, a user controls an ultrasonic probe to acquire an ultrasonic image of the object to be inspected according to the acquisition guide, the conversion relation between an image coordinate system of the target image sequence and a magnetic field coordinate system of the magnetic field is obtained through algorithm calculation based on the relative positions of the object to be inspected and the magnetic field generator, and any coordinates in the coordinate systems corresponding to the two modes can be mutually converted based on the conversion relation, so that multi-mode image registration fusion is realized.

Description

Organ-based multi-modal registration method and device

Technical Field

The application relates to the field of multi-modal image registration, in particular to a multi-modal registration method and device based on organs.

Background

The medical multi-mode fusion system has the key steps that the registration process of a plurality of mode data is adopted, the process needs to unify coordinate systems of different images under different coordinate systems to ensure accurate fusion results, at present, the traditional image coordinate fusion mode generally needs an operator to control the movement of an ultrasonic probe according to a certain CT/MRI/PET image as a reference image, so that the ultrasonic imaging display content acquired by the ultrasonic probe is basically consistent with the content displayed by the selected reference image through manual judgment, the image information and the coordinate information of the currently acquired ultrasonic imaging are sent to a computer to be compared and calculated with the reference image, the steps are repeated for a plurality of times, and finally, organ registration is completed through matching of the ultrasonic image and the CT/MRI image, so that the registration of the coordinate systems of the two modes is further realized; the main flow registration mode is complicated in flow and long in time consumption, and also relies on experience of operators, so that operation experience in clinic is reduced, and an operation threshold is raised.

Disclosure of Invention

Based on the above, it is necessary to provide an organ-based multi-mode registration method and device for solving the technical problems of complicated registration process, long time consumption, higher operation threshold and the like in the prior art.

The first aspect of the present application provides an organ-based multimodal registration method comprising:

Acquiring a target image sequence of an inspected object, and carrying out target organ identification and three-dimensional reconstruction based on the target image sequence to acquire a target organ image sequence and a three-dimensional reconstruction model of the inspected object; displaying an organ registration operation interface in a first display area, wherein the organ registration operation interface comprises a magnetic field orientation selection control and an ultrasonic image acquisition control, the magnetic field orientation selection control is used for selecting a magnetic field orientation, and the magnetic field orientation is used for representing the relative position relation between a magnetic field generator and the inspected object; responsive to a triggering operation for the magnetic field orientation selection control, displaying a plurality of alternative magnetic field orientations at the organ registration operation interface, and determining a target magnetic field orientation from the plurality of alternative magnetic field orientations; responding to triggering operation of the ultrasonic image acquisition control, and displaying an acquisition guide interface for guiding an ultrasonic probe to acquire an ultrasonic image in a second display area, wherein the acquisition guide interface comprises an acquisition guide prompt based on the three-dimensional reconstruction model; the ultrasonic probe carries out ultrasonic sampling on the checked object based on the acquisition guide prompt, and an ultrasonic image of the checked object is obtained; and completing registration of the target image sequence and the ultrasonic image based on the target magnetic field position, the target organ image sequence and the ultrasonic image.

In one embodiment, the method further comprises, in response to a trigger operation for the magnetic field orientation selection control, displaying a plurality of alternative magnetic field orientations at the organ registration operation interface, and determining a target magnetic field orientation from the plurality of alternative magnetic field orientations, and further comprising: and responding to triggering operation for a posture selection control, displaying a plurality of alternative postures on the organ registration operation interface, and determining a target posture from the plurality of alternative postures, wherein the posture selection control is used for selecting the posture of the checked object, and the posture is used for representing the current posture of the checked object.

In one embodiment, the acquisition guidance cues include arrow-like trajectory guidance displayed on a target organ region of the three-dimensional reconstruction model for guiding movement of the ultrasound probe.

In one embodiment, the method for obtaining the target organ image sequence includes: displaying the target image sequence frame by frame on the organ registration operation interface; determining that a first frame of the target organ appears in the target image sequence in response to a first frame confirmation operation; determining a last frame of the target organ in the target image sequence in response to a tail frame confirmation operation; and acquiring all the target image sequences from the first frame to the last frame, and determining the target image sequences as target organ image sequences.

In one embodiment, the organ registration operation interface further includes a relative position display area for displaying a relative positional relationship of the magnetic field generator and the object under examination.

In one embodiment, in response to a trigger operation for the ultrasound image acquisition control, displaying an acquisition guidance interface for guiding an ultrasound probe to acquire ultrasound images in a second display area, the acquisition guidance interface including an acquisition guidance prompt based on the three-dimensional reconstruction model, comprising: responding to triggering operation for the ultrasonic image acquisition control, wherein the triggering operation comprises automatic acquisition triggering operation and manual acquisition triggering operation; displaying an ultrasonic image acquired by the ultrasonic probe in real time in a first subarea of the second display area, and displaying an acquisition guide interface for guiding the ultrasonic probe to acquire the ultrasonic image in a second subarea of the second display area, wherein the acquisition guide interface comprises an acquisition guide prompt based on the three-dimensional reconstruction model; and displaying acquisition contents on the organ registration operation interface, wherein the acquisition contents are determined by the automatic acquisition triggering operation and the manual acquisition triggering operation.

In one embodiment, when the triggering operation is an automatic acquisition triggering operation, displaying the acquisition content on the organ registration operation interface, wherein the acquisition content comprises acquisition time length and ultrasonic image frame number acquired by the ultrasonic probe.

In one embodiment, when the trigger operation is a manual acquisition trigger operation: displaying an acquisition control on the organ registration operation interface; responding to the triggering operation of the acquisition control, and acquiring an ultrasonic image currently acquired by the ultrasonic probe; displaying the acquisition content on the organ registration operation interface, wherein the acquisition content comprises ultrasonic images acquired by the ultrasonic probe, and the ultrasonic images are sequentially displayed on the organ registration operation interface according to an acquisition sequence.

In one embodiment, the registering the target image sequence with the ultrasound image based on the target magnetic field position, the target organ image sequence, and the ultrasound image comprises: determining a relative position relation between the magnetic field generator and the inspected object based on the target magnetic field azimuth, and determining an adjustment coefficient of a magnetic field coordinate system of the magnetic field generator and an image coordinate system of a target organ image sequence based on the relative position relation; identifying an ultrasonic image of a target organ contained in the ultrasonic image of the checked object as an ultrasonic sequence of the target organ; constructing an organ three-dimensional body corresponding to each sequence based on the target organ image sequence and the target organ ultrasonic sequence, and calculating the geometric center of the organ three-dimensional body corresponding to each sequence; rotating at least one of the magnetic field coordinate system and the image coordinate system based on the geometric center and the adjustment coefficient to enable the geometric center of the organ three-dimensional body corresponding to each sequence to coincide, and enabling the three axes of the magnetic field coordinate system and the image coordinate system to be the same in orientation; according to the organ three-dimensional body corresponding to each sequence, calculating to obtain a coordinate transformation matrix according to at least one of ICP algorithm registration, energy field algorithm registration and image similarity algorithm registration; based on the coordinate transformation matrix, mapping any coordinate in the magnetic field coordinate system and the image coordinate system to another coordinate system to finish registration of the magnetic field coordinate system and the image coordinate system

A second aspect of the present application provides an organ-based multimodal registration apparatus comprising: the image acquisition module is used for acquiring a target image sequence of the checked object, and carrying out target organ identification and three-dimensional reconstruction based on the target image sequence to acquire a target organ image sequence and a three-dimensional reconstruction model of the checked object; the first display module is used for displaying an organ registration operation interface in a first display area, wherein the organ registration operation interface comprises a magnetic field orientation selection control and an ultrasonic image acquisition control, the magnetic field orientation selection control is used for selecting a magnetic field orientation, and the magnetic field orientation is used for representing the relative position relation between a magnetic field generator and the inspected object; the magnetic field control response module is used for responding to the triggering operation of the magnetic field orientation selection control, displaying a plurality of alternative magnetic field orientations on the organ registration operation interface and determining a target magnetic field orientation from the plurality of alternative magnetic field orientations; the second display module is used for responding to the triggering operation of the ultrasonic image acquisition control, displaying an acquisition guide interface for guiding an ultrasonic probe to acquire an ultrasonic image in a second display area, wherein the acquisition guide interface comprises an acquisition guide prompt based on the three-dimensional reconstruction model; the ultrasonic acquisition module is used for carrying out ultrasonic sampling on the checked object based on the acquisition guide prompt through an ultrasonic probe to acquire an ultrasonic image of the checked object; and the registration module is used for completing registration of the target image sequence and the ultrasonic image based on the target magnetic field position, the target organ image sequence and the ultrasonic image.

According to the multi-mode registration method and device based on the organ, after the target image sequence is obtained, the corresponding organ image and the three-dimensional model are obtained through organ identification and three-dimensional reconstruction, the relative positions of the currently inspected object and the magnetic field generator are confirmed, the acquisition guide based on the three-dimensional reconstruction model is displayed in the display area, the user controls the ultrasonic probe to obtain the ultrasonic image of the inspected object according to the acquisition guide, the conversion relation between the image coordinate system of the target image sequence and the magnetic field coordinate system of the magnetic field is obtained through algorithm calculation based on the relative positions of the inspected object and the magnetic field generator, any coordinate can be mutually converted in the two coordinate systems based on the conversion relation, the corresponding image frame can be determined in the target image sequence based on the coordinate information of any ultrasonic image, and the ultrasonic image and the image frame are simultaneously displayed in the same display area, so that the multi-fusion display effect is achieved.

Drawings

FIG. 1 is a flow diagram of a method of organ-based multimodal registration in one embodiment;

FIG. 2 is a schematic diagram of a magnetic field orientation and position control in one embodiment;

FIG. 3 is a schematic diagram of an alternative magnetic field orientation and an alternative body position in one embodiment;

FIG. 4 is a schematic diagram of a second display area in one embodiment;

FIG. 5 is a schematic diagram of a head-to-tail frame of a target image sequence according to one embodiment;

FIG. 6 is a schematic diagram of an interface for ultrasound image acquisition completion in one embodiment;

FIG. 7 is a schematic diagram of automated acquisition in one embodiment;

FIG. 8 is a schematic diagram of manual acquisition in one embodiment;

FIG. 9 is a block diagram of a multi-modality registration device in one embodiment;

FIG. 10 is a schematic diagram of a multimodal fusion apparatus in another embodiment;

FIG. 11 is a block diagram of a computer device in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

The terms "comprises," "comprising," "includes," "including," or any other variation thereof, as used herein, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

The term "plurality" as used herein means two or more (including two) unless otherwise specifically defined.

The terms "first," "second," and the like, as used herein, are used for naming a distinction between similar objects, but the objects themselves are not limited by these terms, and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated, a particular order or primary and secondary relationships, etc. It is to be understood that these terms may be interchanged as appropriate without departing from the scope of the application. For example, the "first display area" may be described as the "second display area", and similarly, the "second display area" may be described as the "first display area".

It should be understood that, although the steps in the flowchart of fig. 1 are shown in sequence as indicated by the arrows, the steps are not necessarily performed in sequence as indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in fig. 1 may include multiple sub-steps or stages that are not necessarily performed at the same time, but may be performed at different times, nor do the order in which the sub-steps or stages are performed necessarily performed in sequence, but may be performed alternately or alternately with at least a portion of other steps or sub-steps of other steps.

The following terms are used herein.

Multimode fusion/ultrasonic fusion: the technology is a technology for combining US and CT/MRI/PET tomographic images and synchronously displaying the CT/MRI/PET images on the same screen in real time, the technology guides the CT/MRI/PET images into an ultrasonic system, mutually matches ultrasonic and CT/MRI tomographic images through various positioning modes, utilizes a magnetic field to position, realizes the linkage of real-time ultrasonic and CT/MRI volumetric images, can simultaneously have the characteristics of high resolution of CT/MRI imaging and the like, the real-time property of US, easy operation and display of any section, mutually supplements various imaging information and accurately positions focuses.

US: ultrasonic around, for short, US, is an ultrasonic technique, in which imaging information is obtained by scanning a human body by an ultrasonic technique, that is, an ultrasonic imaging technique, in which an ultrasonic sound beam is used to scan the human body, and reflected signals are received and processed to obtain images of organs in the body;

Target image sequence: the image sequence of the target organ is formed by slice images of a frame according to the examination item of a user, the arrangement sequence of each frame of slices is consistent with the sequence during scanning, the slice images are usually in CT/MRI/PET and other formats and are stored according to the DICOM standard, each frame of slice images comprises two-dimensional image information and three-dimensional coordinates under a DICOM coordinate system, and the coordinate system is the DICOM default coordinate system.

DICOM coordinate system: the coordinate system in DICOM is defined based on the patient: taking the center of the patient as an origin, and taking the direction of the left finger of the patient as the positive direction of the X axis; the direction behind the patient is the positive direction of the Y axis; the direction of the patient's head is the forward direction of the Z-axis, i.e., the DICOM coordinate system is the right hand coordinate system centered on the patient.

Fig. 1 is a schematic flow chart of an organ-based multi-mode registration method according to an embodiment of the present disclosure, where the method may be performed by a multi-mode registration device, where the device may be implemented by software and/or hardware, and may be generally integrated in an electronic device, where an integrated manner may be that a client is installed in a corresponding electronic device, and the client may be an application program for displaying a registration process, or may be a web client, or may be a sub-application running in an operating environment of a parent application, and the electronic device may be a mobile device such as a mobile phone, a smart watch, a tablet computer, or a personal digital assistant; other devices such as desktop computers; further, it may be a respective medical device such as an ultrasonic fusion device or the like.

In the following embodiments, optional features and examples are provided in each embodiment at the same time, and the features described in the embodiments may be combined to form multiple alternatives, and each numbered embodiment should not be considered as only one technical solution.

In one embodiment, as shown in fig. 1, there is provided an organ-based multi-modality registration method, the method comprising the steps of:

Step S101, acquiring a target image sequence of an inspected object, and carrying out target organ identification and three-dimensional reconstruction based on the target image sequence to acquire a target organ image sequence and a three-dimensional reconstruction model of the inspected object.

Acquiring a target image sequence of an object to be inspected, wherein the target image sequence obtained by scanning the instrument can be acquired through communication connection with the instrument for performing medical scanning or communication connection with an internal server of a hospital; in another implementation scenario, the target image sequence may be obtained by a flash memory medium, a mail, or the like, and the specific obtaining manner is not limited herein, and in one embodiment, the user performs an operation on an interactive interface of the device side, and selects and determines the target image sequence by clicking or touching an import button, or the like, so as to obtain the target image sequence.

The target image sequence, typically one of CT/MRI/PET, contains slice images of the target organ of the subject under examination.

Target organ identification refers to identifying the target organ and the outline thereof by means of algorithmic model identification or manual sketching and the like on a slice containing the target organ in a target image sequence.

And (3) three-dimensional reconstruction, namely, carrying out three-dimensional reconstruction on image information, coordinate information and the like of organs, tissues, bones, skin and the like of the checked object based on each slice in the target image sequence, supplementing and simulating a body part of the checked object which is not scanned through an algorithm model due to the interval between the scans of the slices, and finally obtaining a complete three-dimensional reconstruction model for reflecting the internal structure of the checked object.

And selecting the slices containing the target organ from the target organ image sequence based on target organ identification, and arranging the slices into the target organ image sequence according to the sequence.

Step S102, displaying an organ registration operation interface in a first display area, wherein the organ registration operation interface comprises a magnetic field orientation selection control and an ultrasonic image acquisition control, the magnetic field orientation selection control is used for selecting a magnetic field orientation, and the magnetic field orientation is used for representing the relative position relation between a magnetic field generator and the checked object.

The first display area may be a partial display area in one screen, or may refer to the entire area that can be displayed by one screen, and in one embodiment, the first display area and the second display area that appear hereinafter refer to two display screens that are independent of each other.

The organ registration operation interface refers to a display window for operation interaction of a user, and the display window can be displayed in a first display area in a full screen manner or in a partial area of the first display area.

The magnetic field orientation selection control refers to an interaction control set in the organ registration operation interface and used for clicking or touching by a user, and is used for selecting the relative position relation between the magnetic field generator and the checked object by the user, wherein the magnetic field generator is used for transmitting a hemispherical magnetic field so as to position the object in the magnetic field under the magnetic field coordinate system, and the magnetic field covers the checked object, and in a plurality of embodiments, the magnetic field generator is arranged at the upper left side or the upper right side of the checked object.

Step S103, responding to the triggering operation of the magnetic field position selection control, displaying a plurality of alternative magnetic field positions on the organ registration operation interface, and determining a target magnetic field position from the plurality of alternative magnetic field positions.

The alternative magnetic field direction refers to the relative positions of various magnetic field generators and the inspected object, and in the actual application scene, considering that the arrangement space of the instrument does not influence the inspection of the inspected object by medical staff, in order to enable the magnetic field emitted by the magnetic field generator to cover the inspected object, the magnetic field generator is generally arranged at the upper left side of the inspected object or at the upper right side of the inspected object.

The target magnetic field position, i.e. the relative position of the current magnetic field generator and the object under examination, is determined among the candidate magnetic field positions, i.e. the relative position of the current magnetic field generator and the object under examination.

As shown in fig. 2, by triggering the magnetic field orientation selection control, namely "electromagnetic selection", a plurality of alternative magnetic field orientations are displayed in the operation interface, and the medical staff selects the corresponding magnetic field orientations according to the placement situation of the field actual magnetic field generator and the checked object.

And step S104, responding to the triggering operation of the ultrasonic image acquisition control, and displaying an acquisition guide interface for guiding an ultrasonic probe to acquire an ultrasonic image in a second display area, wherein the acquisition guide interface comprises an acquisition guide prompt based on the three-dimensional reconstruction model.

The second display area may be a partial display area in one screen, or may refer to the entire area that can be displayed by one screen, and in one embodiment, the second display area and the first display area appearing above refer to two display screens that are independent of each other.

The acquisition guide interface is used for guiding medical staff to operate the ultrasonic probe to acquire an ultrasonic image of an object to be inspected, the display interface can be displayed in a second display area in a full screen mode or in a partial area of the second display area, and based on multi-mode fusion of organs, the acquired ultrasonic image contains a complete organ image, so that the acquisition guide interface is based on a prior three-dimensional reconstruction model, carries out acquisition guide prompt on the three-dimensional reconstruction model to guide the medical staff to acquire the ultrasonic image by holding the ultrasonic probe according to a given mode, the acquisition guide prompt can be generally arrow guide on the three-dimensional model, can be the motion track of the ultrasonic probe model on the model, or the complete flow demonstration of the ultrasonic image acquired by holding the ultrasonic probe containing the medical staff model, and the like, and the medical staff can accurately and rapidly acquire the ultrasonic image containing the target organ of the object to be inspected, which meets the fusion condition and requirement based on the related prompt.

Step 105, the ultrasonic probe performs ultrasonic sampling on the checked object based on the acquisition guiding prompt, and acquires an ultrasonic image of the checked object.

In one embodiment, the ultrasonic probe is provided with an electromagnetic sensor, and the electromagnetic sensor is used for sensing coordinate information of the electromagnetic sensor in a magnetic field, and further acquiring coordinate information of the ultrasonic probe in the magnetic field, namely a magnetic field coordinate system, and when the ultrasonic probe acquires ultrasonic images, the electromagnetic sensor acquires spatial pose information of each ultrasonic image slice in a three-dimensional magnetic field in real time, so that the ultrasonic image slices finally acquired and output to a client contain the three-dimensional coordinate information of each ultrasonic image slice in the magnetic field coordinate system.

In the ultrasonic sampling process of the checked object, the ultrasonic probe can display the ultrasonic image returned by the current ultrasonic probe in real time in the partial area in the second display area, and the ultrasonic image is used for medical staff to judge whether the ultrasonic image acquired by the ultrasonic probe at the current position meets the requirement of multi-mode fusion on the content of ultrasonic image acquisition.

The ultrasonic image of the checked object can be the ultrasonic image which is acquired by the ultrasonic probe in real time and returned to the second display area for display, or the ultrasonic image which is acquired in real time and screened by medical staff is not limited, and the ultrasonic image participates in the subsequent ultrasonic image registration after the ultrasonic image of the checked object is determined in any acquisition mode.

And step S106, based on the target magnetic field orientation, the target organ image sequence and the ultrasonic image, completing registration of the target image sequence and the ultrasonic image.

According to the target magnetic field orientation, the relative position relation between the current magnetic field generator and the inspected object can be obtained, further, as the three-axis xyz orientation of the magnetic field coordinate system of the magnetic field generator is constant, and the coordinate system of the target organ image sequence is a DICOM coordinate system, the DICOM coordinate system takes the inspected object as the center to be used as the origin, and in certain specific inspections such as prostate inspections, the inspected object is in a sitting posture with the body facing upwards, therefore, after the relative position relation between the magnetic field generator and the inspected object is determined, the respective orientations of the three-axis xyz of the magnetic field coordinate system and the three-axis xyz of the DICOM coordinate system are determined, and the coordinate rotation coefficients enabling the three-axis orientations of the two coordinate systems to be consistent can be determined based on the three-axis orientations of the two coordinate systems.

Identifying a target organ and the outline thereof according to the acquired ultrasonic images by means of algorithm model identification or manual sketching and the like, screening and sorting ultrasonic image slices containing the outline of the target organ, based on the spatial pose information of each ultrasonic image slice on a three-dimensional magnetic field space, namely three-dimensional coordinate information, combining the arrangement positions of pixel points used for representing the outline of the target organ in each ultrasonic image slice in a two-dimensional ultrasonic image, obtaining the three-dimensional coordinate information of the outline of the target organ through spatial mapping, carrying out three-dimensional reconstruction on the outline of the organ on each ultrasonic image slice to obtain a rough organ three-dimensional body, carrying out the processing on the image sequence of the target organ in the same way to obtain rough organ three-dimensional bodies, respectively calculating the geometric centers of two three-dimensional organs, then moving the coordinate system under one organ three-dimensional body to enable the geometric centers of the two three-dimensional organs to coincide, and rotating one coordinate system according to the coordinate rotation coefficient obtained above to enable the xyz axes under the two coordinate systems to face the two coordinate systems to be consistent, and finally enabling the two three-dimensional bodies to be roughly coincident.

According to the two approximately coincident organ three-dimensional volumes, the coordinate transformation matrix of the two coordinate systems can be obtained by performing ICP algorithm registration through coordinates of three-dimensional volume contour points, registration based on an energy field, registration calculation based on at least one algorithm such as image similarity and the like, and any coordinate in the two coordinate systems can be mapped to the other coordinate system through the matrix.

Under a specific application scene, after the registration step is completed, in the following operation mode, such as prostate puncture, kidney puncture and the like, medical staff only needs to collect ultrasonic images of corresponding organs according to a normal operation flow, determine mapping coordinates under a corresponding DICOM coordinate system through a coordinate transformation matrix according to three-dimensional coordinate information under a magnetic field coordinate system of returned ultrasonic images, determine corresponding CT/MRI/PET slices based on the mapping coordinates, and overlap or display the slices and the corresponding ultrasonic images in the same display area so as to realize multi-mode ultrasonic fusion.

Correspondingly, one of the coordinate systems can be arbitrarily selected as a reference coordinate system in the process of organ three-dimensional body superposition and three-dimensional body contour point registration, for example, a magnetic field coordinate system is used as the reference coordinate system, the coordinate system of DICOM is translated and rotated to ensure that the geometric center of the organ three-dimensional body under the coordinate system coincides with the geometric center of the organ three-dimensional body under the magnetic field coordinate system, the three-axis orientation of the DICOM coordinate system coincides with the three-axis orientation under the magnetic field coordinate system, ICP algorithm registration is carried out through the coordinates of the three-dimensional body contour points on the basis of the rotation translation relation, registration based on the energy field, and registration calculation based on at least one or more algorithms such as image similarity are carried out, the obtained coordinates of the DICOM coordinate system are converted into a coordinate transformation matrix of the magnetic field coordinate system, and in the subsequent fusion process, therefore, the coordinate transformation of the magnetic field coordinate under the obtained ultrasonic image is required to be mapped in the DICOM coordinate system, and the coordinate transformation matrix of the DICOM coordinate system is required to be converted into the coordinate transformation matrix of the magnetic field coordinate system after the coordinate transformation is required to be reversely transformed, and the coordinate transformation matrix of the coordinate transformation under the coordinate system is required to be converted into the coordinate transformation matrix of the coordinate system under the coordinate system of the coordinate system.

According to the multi-mode registration method and device based on the organ, after the target image sequence is acquired, the corresponding organ image and the three-dimensional model are obtained through organ identification and three-dimensional reconstruction, the relative positions of the currently inspected object and the magnetic field generator are confirmed, the acquisition guide based on the three-dimensional reconstruction model is displayed in the display area, the user controls the ultrasonic probe to acquire the ultrasonic image of the inspected object according to the acquisition guide, the conversion relation between the image coordinate system of the target image sequence and the magnetic field coordinate system of the magnetic field is obtained through algorithm calculation based on the relative positions of the inspected object and the magnetic field generator, any coordinate can be mutually converted in the two coordinate systems based on the conversion relation, the corresponding image frame can be determined in the target image sequence based on the coordinate information of any ultrasonic image, and the ultrasonic image and the image frame are simultaneously displayed in the same display area, so that the multi-mode fusion display effect is achieved.

In one embodiment, in response to a triggering operation of the magnetic field orientation selection control, displaying a plurality of alternative magnetic field orientations at the organ registration operation interface, and determining a target magnetic field orientation from the plurality of alternative magnetic field orientations, further comprising: and responding to triggering operation for a posture selection control, displaying a plurality of alternative postures on the organ registration operation interface, and determining a target posture from the plurality of alternative postures, wherein the posture selection control is used for selecting the posture of the checked object, and the posture is used for representing the current posture of the checked object.

As shown in fig. 2, in addition to setting an electromagnetic selection, that is, a magnetic field orientation selection control, a body position selection, that is, a body position selection control is further set in the organ registration operation interface, when a user triggers the body position selection control through clicking, touching or other operations, the organ registration operation interface displays a plurality of alternative possible body positions of the related checked object, such as a prone position, a lateral position, a recumbent position and the like, as shown in fig. 3, the medical staff selects a corresponding target body position in the alternative body positions based on the lying position of the current checked object, and the corresponding of the body position determined by the system and the actual body position of the checked object is completed.

Combining the previously determined target magnetic field orientation and the current position of the checked object, determining the lying position of the checked object under the condition that the relative position relation between the magnetic field generator and the checked object is determined, and further determining the coordinate rotation coefficients of the magnetic field coordinate system and the DICOM coordinate system.

In clinical environment, because the target organs aimed by different surgical formulas are different, the requirements on the lying posture of a patient are also different, for example, the surgical formulas of the prostate require the patient to lie on the back, the surgical formulas of the kidney require the patient to expose the back part, so that the required lying posture is different from the front part, different clinical environments can be properly matched by multimode fusion through setting different body position selections, the compatibility of a registration method is improved, and meanwhile, after the body positions are specifically refined, more accurate coordinate rotation coefficients can be provided, the registration calculation flow is optimized, the calculation amount is simplified, and the registration efficiency and precision are improved.

In one embodiment, acquiring guidance cues includes displaying arrow-like trajectory guidance for guiding movement of the ultrasound probe in a target organ region of the three-dimensional reconstruction model.

As shown in fig. 4, the acquisition guidance prompt is a guidance prompt established on a three-dimensional reconstruction model reconstructed from a target image sequence as source data, and in one embodiment, the acquisition guidance prompt is displayed in a partial display area in the second display area, as shown in the lower right corner of fig. 4, which shows the three-dimensional reconstruction model reconstructed from the target image sequence of the object to be examined, and in the corresponding operation, such as in a prostate biopsy puncture, the medical staff is guided by an arrow to operate a timeout probe into the perineum or rectum of the object to be examined according to the arrow direction so as to ultrasonically sample the prostate organ.

The three-dimensional reconstruction model comprises relevant bones of corresponding organs such as pubis, muscle tissues and the like, so that medical staff can conveniently position the acquisition direction through bones of other tissues, and the visualization degree of an inspected object and the coverage of multidimensional information in the acquisition process are improved.

The acquisition guide prompt can also be a motion track of an ultrasonic probe model on a three-dimensional reconstruction model, or a complete flow demonstration of ultrasonic image acquisition of a handheld ultrasonic probe comprising a medical staff model, and is not limited to the embodiment.

After the target image sequence is acquired, the sequence is displayed frame by frame in an organ registration operation interface, as shown in fig. 5, 60 frames are total in the acquired target image sequence, a user can switch the upper frame and the lower frame through an up-down arrow in the operation interface, or drag a progress bar under each frame of picture to switch the currently displayed image frame, because the picture sequences of each frame are sequentially arranged according to the acquisition sequence, the inspected object is usually scanned from the foot to the head in the scanning process of CT/MR, the first frames and the last frames in the sequence do not contain target organs, the medical staff sequentially inspects the image frames, takes the image frame of the first determined target organ as the first frame of the target image sequence, takes the image frame of the last frame of the target organ as the last frame of the target image sequence, namely, the image frame of the target organ can be determined through the first frame and the last frame, the image frame containing the target organ is determined to be the target organ image sequence after the confirmation operation of the medical staff is responded, the target organ image sequence contains relevant information such as two-dimensional image information, coordinate information and the like, and the target organ image sequence is taken as the subsequent image sequence of the registration.

Through the operation, the imported target image sequence can be further simplified and screened, the calculated amount of equipment terminals is reduced, the auditing workload of medical staff on the identification result is reduced, the registration process is further simplified, and the registration efficiency is improved

In one embodiment, the organ registration operation interface further comprises a relative position display area for displaying a relative positional relationship of the magnetic field generator and the object under examination.

As shown in fig. 6, after the acquisition and confirmation of the ultrasound image are completed, the organ contour confirmation interface of the ultrasound image can be further confirmed, after the confirmation of the organ contour recognition result of the ultrasound image is completed, the system can call the related algorithm, and the registration calculation is performed based on the image sequence, the coordinate rotation coefficient and other parameters imported in the previous step, meanwhile, the magnetic field orientation selected previously is displayed in the right upper corner area of the interface, namely, the relative position relationship between the magnetic field generator and the checked object is represented, so that the system is used for assisting the user in performing error correction adjustment at this stage, avoiding the repeated execution of the steps of setting, acquisition and the like caused by the previous selection error, and further improving the registration efficiency and fault tolerance.

Further, the relative position display area may further display the body position of the previously selected inspected object, and the medical care further corrects the target magnetic field position and the target body position in the alternative magnetic field position and the alternative body position shown in fig. 3 by triggering the body position/electromagnetic selection control.

The ultrasonic image acquisition control comprises an automatic acquisition control and a manual acquisition control, and different contents are displayed in the first subarea according to triggering operations of different acquisition controls.

In this embodiment, the second display area refers to a second display screen, as shown in fig. 4, the first sub-area refers to an upper half part in the second display screen, where the first sub-area is further subdivided into different display parts according to different probe surfaces of the ultrasound probe, and is used for respectively displaying ultrasound images acquired by each probe surface of the ultrasound probe in real time, and the first display area refers to a first display screen, and is used for displaying an organ registration operation interface in full screen or in part, and after detecting a triggering operation of a user on an ultrasound image acquisition control, the organ registration operation interface displays corresponding acquisition content.

The ultrasonic image acquisition control comprises an automatic acquisition control and a manual acquisition control, different contents are displayed on the first subarea aiming at triggering operations of different acquisition controls and used for assisting medical staff in judging acquisition conditions under different acquisition modes, such as the number of ultrasonic images needing to be focused and acquired under automatic acquisition and the image content of specific ultrasonic images needing to be focused and acquired under manual acquisition.

In one embodiment, when the triggering operation is an automatic acquisition triggering operation, displaying the acquisition content on the organ registration operation interface, wherein the acquisition content comprises acquisition duration and ultrasonic image frame number acquired by the ultrasonic probe.

As shown in fig. 7, the medical staff triggers the "automatic acquisition" control, and the acquisition content associated with the automatic acquisition mode, such as the number of frames and the acquisition time length of the current acquisition, is displayed in real time in the organ registration operation interface, so that the medical staff can intuitively grasp the number of acquired images and the occupied time length, and can evaluate the calculation amount of the subsequent registration, and whether to adjust the acquisition speed of the ultrasonic probe in time, so as to provide more intervention options, such as the early selection of ending the acquisition after the medical staff judges that the current acquired ultrasonic image is enough to support the subsequent registration calculation.

In one application scenario, medical staff triggers an automatic acquisition control, a second display area displays an ultrasonic image acquired by an ultrasonic probe in real time and simultaneously displays the residual acquisition time, a user controls the ultrasonic probe to move at a certain speed and in a certain direction according to the returned ultrasonic image so as to acquire complete ultrasonic images of organs, synchronously displays acquisition countdown, provides visual acquisition residual time for the medical staff, indicates that the ultrasonic probe may have poor contact and other problems when the numerical value is always 0, can remind the medical staff to adjust in time,

The upper limit of the data of the acquired ultrasonic image is further specified, excessive and repeated redundant data acquired by a user is avoided, and then medical staff can choose whether to finish acquisition in advance according to the continuously increasing numerical value, so that the calculation amount is reduced, and the registration efficiency is improved.

In one embodiment, the trigger operation is a manual acquisition trigger operation: displaying an acquisition control on the organ registration operation interface; responding to the triggering operation of the acquisition control, and acquiring an ultrasonic image currently acquired by the ultrasonic probe; displaying the acquisition content on the organ registration operation interface, wherein the acquisition content comprises ultrasonic images acquired by the ultrasonic probe, and the ultrasonic images are sequentially displayed on the organ registration operation interface according to an acquisition sequence.

As shown in fig. 8, the medical staff triggers the "manual acquisition" control, displays the acquired ultrasonic images in real time in the organ registration operation interface, and each frame of ultrasonic images is sequentially arranged and displayed in the operation interface according to the sequence of acquisition, and simultaneously displays the number of frames currently acquired, and the medical staff judges whether the acquired organ ultrasonic images are complete or not by browsing the acquired ultrasonic images.

In one application scenario, a medical staff triggers a manual acquisition control, a second display area displays an ultrasonic image acquired by an ultrasonic probe in real time, a user controls the ultrasonic probe to slowly move, when the current returned ultrasonic image contains a target organ, the sampling control is triggered to acquire the current ultrasonic image, then the ultrasonic probe is continuously moved, and when the returned ultrasonic image meets the registration requirement, the sampling control is triggered again to acquire the current ultrasonic image, so that the operation is performed for a plurality of times until the complete ultrasonic image of the target organ is acquired.

Through the acquisition mode, the two-dimensional content of the currently acquired ultrasonic image can be determined in real time, and meanwhile, as each frame of ultrasonic image is ordered according to the acquisition sequence, medical staff can intuitively judge whether a series of acquired ultrasonic images can form a complete organ outline or not, further judge whether the acquired ultrasonic images meet the registration requirement or not, increase the visualization degree in the whole acquisition process, provide an error correction function and further improve the fault tolerance.

In one embodiment, completing registration of the target image sequence with the ultrasound image based on the target magnetic field position, the target organ image sequence, and the ultrasound image comprises: determining a relative position relation between the magnetic field generator and the inspected object based on the target magnetic field azimuth, and determining an adjustment coefficient of a magnetic field coordinate system of the magnetic field generator and an image coordinate system of a target organ image sequence based on the relative position relation; identifying an ultrasonic image of a target organ contained in the ultrasonic image of the checked object as an ultrasonic sequence of the target organ; constructing an organ three-dimensional body corresponding to each sequence based on the target organ image sequence and the target organ ultrasonic sequence, and calculating the geometric center of the organ three-dimensional body corresponding to each sequence; rotating at least one of the magnetic field coordinate system and the image coordinate system based on the geometric center and the adjustment coefficient to enable the geometric center of the organ three-dimensional body corresponding to each sequence to coincide, and enabling the three axes of the magnetic field coordinate system and the image coordinate system to be the same in orientation; according to the organ three-dimensional body corresponding to each sequence, calculating to obtain a coordinate transformation matrix according to at least one of ICP algorithm registration, energy field algorithm registration and image similarity algorithm registration; and mapping any coordinate in the magnetic field coordinate system and the image coordinate system to another coordinate system based on the coordinate transformation matrix, and completing registration of the magnetic field coordinate system and the image coordinate system.

ICP algorithm, ICP algorithm registration is a classical data registration algorithm, and is mainly used for solving the registration problem based on free form curved surfaces. The basic principle is that a rotation translation matrix is constructed based on corresponding point pairs by solving the corresponding point pairs between the source point cloud and the target point cloud, the source point cloud is transformed to the coordinate system of the target point cloud by utilizing the solved matrix, and the error function of the transformed source point cloud and the transformed target point cloud is estimated.

The image similarity algorithm registration is that two-dimensional tangent planes are carried out on three-dimensional bodies of different organs under different coordinate systems, similarity calculation is carried out on images based on the tangent planes, corresponding slices with similarity meeting preset standards are obtained, the conversion relation between the two slices is calculated on the basis of three-dimensional coordinate information of the slices, the two-dimensional tangent planes are repeated for a plurality of times, the identification calculation is completed on different slices under a plurality of different coordinate systems, the conversion relation of a plurality of coordinates is further obtained, and the coordinate conversion coefficients of two coordinate systems are obtained after comprehensive calculation.

Identifying a target organ and the outline thereof by means of algorithm model identification or manual sketching and the like according to the acquired ultrasonic images, screening and sorting ultrasonic image slices containing the outline of the target organ, carrying out three-dimensional reconstruction on the basis of the outline of the organ on each slice to obtain a rough organ three-dimensional body, carrying out the processing on the image sequence of the target organ by the same method to obtain rough organ three-dimensional bodies, respectively calculating the geometric centers of the two three-dimensional organs, then moving a coordinate system under one organ three-dimensional body to enable the geometric centers of the two three-dimensional organs to coincide, and rotating one coordinate system according to the coordinate rotation coefficient obtained by the coordinate system so as to enable the xyz axes under the two coordinate systems to be oriented consistently, and finally enabling the three-dimensional bodies of the two organs to be approximately coincident.

In accordance with one or more embodiments of the present disclosure, as shown in fig. 9, there is provided an organ-based multi-modality registration apparatus 200, comprising:

The image acquisition module 201 is configured to acquire a target image sequence of an object to be inspected, and perform target organ identification and three-dimensional reconstruction based on the target image sequence, and acquire a target organ image sequence and a three-dimensional reconstruction model of the object to be inspected;

A first display module 202, configured to display an organ registration operation interface in a first display area, where the organ registration operation interface includes a magnetic field orientation selection control and an ultrasound image acquisition control, where the magnetic field orientation selection control is used to select a magnetic field orientation, and the magnetic field orientation is used to characterize a relative positional relationship between a magnetic field generator and the inspected object;

a magnetic field control response module 203, configured to display a plurality of alternative magnetic field orientations on the organ registration operation interface in response to a triggering operation for the magnetic field orientation selection control, and determine a target magnetic field orientation from the plurality of alternative magnetic field orientations;

A second display module 204, configured to display, in response to a trigger operation for the ultrasound image acquisition control, an acquisition guidance interface for guiding an ultrasound probe to acquire an ultrasound image in a second display area, where the acquisition guidance interface includes an acquisition guidance prompt based on the three-dimensional reconstruction model;

The ultrasonic acquisition module 205 is configured to perform ultrasonic sampling on the object to be inspected based on the acquisition guidance prompt through an ultrasonic probe, and acquire an ultrasonic image of the object to be inspected;

A registration module 206, configured to complete registration of the target image sequence and the ultrasound image based on the target magnetic field orientation, the target organ image sequence, and the ultrasound image.

According to the multi-mode registration device based on the organ, after the target image sequence is acquired, the corresponding organ image and the three-dimensional model are obtained through organ identification and three-dimensional reconstruction, the relative positions of the currently inspected object and the magnetic field generator are confirmed, the acquisition guide based on the three-dimensional reconstruction model is displayed in the display area, the user controls the ultrasonic probe to acquire the ultrasonic image of the inspected object according to the acquisition guide, the conversion relation between the image coordinate system of the target image sequence and the magnetic field coordinate system of the magnetic field is obtained through algorithm calculation based on the relative positions of the inspected object and the magnetic field generator, any coordinate can be mutually converted in the two coordinate systems based on the conversion relation, the corresponding image frame can be determined in the target image sequence based on the coordinate information of any ultrasonic image, and the ultrasonic image and the image frame are simultaneously displayed in the same display area, so that the multi-mode fusion display effect is achieved.

For specific limitations on the organ-based multimodal registration apparatus, reference may be made to the limitations of the organ-based multimodal registration method hereinabove, and will not be described in detail herein. The various modules in the organ-based multimodal registration apparatus described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

Referring to fig. 10, in a specific application scenario, a patient performs a prostate puncture guided surgery, at this time, the patient lies on an operating table in a fixed posture, a multi-modal fusion device is placed beside the operating table, the device integrates a magnetic field generator, an electromagnetic sensor, an ultrasonic probe, a first display screen, a second display screen and other devices, the magnetic field generator is arranged at the upper left or upper right of the patient, and is used for emitting a hemispherical magnetic field to cover the whole body of the patient, a plurality of electromagnetic sensors are detachably and fixedly arranged on the ultrasonic probe, the other end of the electromagnetic sensor is connected with the multi-modal fusion device, when the ultrasonic probe moves within the magnetic field range, corresponding coordinate information and orientation information can be sensed through the electromagnetic sensor, and the corresponding coordinate information and orientation information are returned to the multi-modal fusion device, an MRI image of a prostate organ of the patient, which was previously introduced in the multi-modal fusion device, is displayed on the first display screen of the multi-modal fusion device, namely, the first display area displays an organ registration operation interface, after confirming the leading-in MRI image in a head-tail frame, further selecting the relative position relation between the current posture position of the checked object and the magnetic field generator, then operating the ultrasonic probe by medical staff to collect the ultrasonic image of the checked object in real time, and observing the ultrasonic image returned by the ultrasonic probe in the second display area in real time on a second display screen, meanwhile, the second display screen also displays a guiding prompt based on a three-dimensional reconstruction model in real time, the medical staff controls the ultrasonic probe to move according to the guiding track or direction according to the guiding prompt, after completing the collection of the ultrasonic image by means of automatic collection or manual collection, the medical staff confirms the collection content in the organ registration operation interface, and the multi-mode fusion equipment sets such as the body position, the magnetic field orientation and the like according to the prior parameters, and calling a built-in algorithm by combining the head and tail frames of the organ screened by the MRI image which is introduced in the front and the tail frames and the collected ultrasonic image, and completing registration of two magnetic field coordinate systems and a DICOM coordinate system under the MRI image.

In one embodiment, a computer device is provided, which may be a user terminal, and the internal structure of which may be as shown in fig. 11. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor is configured to provide computing and control capabilities. The memory includes a non-volatile storage medium storing an operating system and a computer program, and an internal memory providing an environment for the operation of the operating system and the computer program in the non-volatile storage medium, which when executed by the processor, implements a medical image display method. The communication interface is used for communicating with an external terminal through a network connection. The display screen may be a liquid crystal display screen or an electronic ink display screen. The input device of the computer equipment can be a touch layer covered on a display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by those skilled in the art that the structure shown in FIG. 11 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In one embodiment, the organ-based multimodal registration apparatus provided by the present application may be implemented in the form of a computer program executable on a computer device as shown in fig. 11. The memory of the computer device may store the various program modules comprising the organ-based multimodal registration apparatus, such as the image acquisition module, the first display module, the magnetic field control response module, the second display module, the ultrasound acquisition module, and the registration module shown in fig. 9. The computer program of each program module causes the processor to carry out the steps in the organ-based multimodal registration method of each embodiment of the application described in the present specification.

Accordingly, in one embodiment, a computer device is provided that includes a memory having a computer program stored therein and a processor that when executed implements the organ-based multimodal registration method provided by any of the embodiments of the application.

Those skilled in the art will appreciate that all or part of the processes in the methods of the above embodiments may be implemented by a computer program for instructing relevant hardware, where the program may be stored in a non-volatile computer readable storage medium, and where the program, when executed, may include processes in the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in embodiments provided herein may include non-volatile and/or volatile memory. The nonvolatile memory can include Read Only Memory (ROM), programmable ROM (PROM), electrically Programmable ROM (EPROM), electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (SYNCHLINK) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), among others.

Accordingly, in one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor implements the medical organ-based multimodal registration method provided by any of the embodiments of the application.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The above examples only represent some embodiments of the application, which are described in more detail and are not to be construed as limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.

Claims

1. A method of organ-based multi-modality registration, comprising:

Acquiring a target image sequence of an inspected object, and carrying out target organ identification and three-dimensional reconstruction based on the target image sequence to acquire a target organ image sequence and a three-dimensional reconstruction model of the inspected object;

Displaying an organ registration operation interface in a first display area, wherein the organ registration operation interface comprises a magnetic field orientation selection control and an ultrasonic image acquisition control, the magnetic field orientation selection control is used for selecting a magnetic field orientation, and the magnetic field orientation is used for representing the relative position relation between a magnetic field generator and the inspected object;

responsive to a triggering operation for the magnetic field orientation selection control, displaying a plurality of alternative magnetic field orientations at the organ registration operation interface, and determining a target magnetic field orientation from the plurality of alternative magnetic field orientations;

In response to triggering operation of the ultrasonic image acquisition control, displaying an acquisition guide interface for guiding an ultrasonic probe to acquire an ultrasonic image in a second display area, wherein the acquisition guide interface comprises an acquisition guide prompt based on the three-dimensional reconstruction model, and the acquisition guide prompt comprises guiding the ultrasonic probe to move through arrow direction, movement track or flow demonstration;

The ultrasonic probe carries out ultrasonic sampling on the checked object based on the acquisition guide prompt, and an ultrasonic image of the checked object is obtained, wherein the ultrasonic image comprises an ultrasonic image of a target organ;

Based on the target magnetic field orientation, the target organ image sequence and the ultrasonic image, completing registration of the target image sequence and the ultrasonic image; the registering of the target image sequence and the ultrasonic image based on the target magnetic field orientation, the target organ image sequence and the ultrasonic image is completed, and the registering comprises the following steps:

determining a relative position relation between the magnetic field generator and the inspected object based on the target magnetic field azimuth, and determining an adjustment coefficient of a magnetic field coordinate system of the magnetic field generator and an image coordinate system of a target organ image sequence based on the relative position relation;

identifying an ultrasonic image of a target organ contained in the ultrasonic image of the checked object as an ultrasonic sequence of the target organ;

constructing an organ three-dimensional body corresponding to each sequence based on the target organ image sequence and the target organ ultrasonic sequence, and calculating the geometric center of the organ three-dimensional body corresponding to each sequence;

Rotating at least one of the magnetic field coordinate system and the image coordinate system based on the geometric center and the adjustment coefficient to enable the geometric center of the organ three-dimensional body corresponding to each sequence to coincide, and enabling the three axes of the magnetic field coordinate system and the image coordinate system to be the same in orientation;

According to the organ three-dimensional body corresponding to each sequence, calculating to obtain a coordinate transformation matrix according to at least one of ICP algorithm registration, energy field algorithm registration and image similarity algorithm registration;

And mapping any coordinate in the magnetic field coordinate system and the image coordinate system to another coordinate system based on the coordinate transformation matrix, and completing registration of the magnetic field coordinate system and the image coordinate system.

2. The registration method of claim 1, wherein the displaying a plurality of alternative magnetic field orientations at the organ registration operation interface in response to a triggering operation for the magnetic field orientation selection control, and determining a target magnetic field orientation from the plurality of alternative magnetic field orientations, further comprises:

And responding to triggering operation for a posture selection control, displaying a plurality of alternative postures on the organ registration operation interface, and determining a target posture from the plurality of alternative postures, wherein the posture selection control is used for selecting the posture of the checked object, and the posture is used for representing the current posture of the checked object.

3. The registration method of claim 1, wherein the acquisition guidance prompt includes displaying arrow-like trajectory guidance for guiding movement of the ultrasound probe in a target organ region of the three-dimensional reconstruction model.

4. The registration method according to claim 1, wherein the acquiring the target organ image sequence includes:

displaying the target image sequence frame by frame on the organ registration operation interface;

Determining that a first frame of the target organ appears in the target image sequence in response to a first frame confirmation operation;

Determining a last frame of the target organ in the target image sequence in response to a tail frame confirmation operation;

and acquiring all the target image sequences from the first frame to the last frame, and determining the target image sequences as target organ image sequences.

5. The registration method according to claim 1 or 2, wherein the organ registration operation interface further includes a relative position display area for displaying a relative positional relationship of the magnetic field generator and the object under examination.

6. The registration method of claim 1, wherein in response to a trigger operation for the ultrasound image acquisition control, displaying an acquisition guidance interface for guiding an ultrasound probe to acquire ultrasound images in a second display area, the acquisition guidance interface including an acquisition guidance prompt based on the three-dimensional reconstruction model, comprising:

Responding to triggering operation for the ultrasonic image acquisition control, wherein the triggering operation comprises automatic acquisition triggering operation and manual acquisition triggering operation;

Displaying an ultrasonic image acquired by the ultrasonic probe in real time in a first subarea of the second display area, and displaying an acquisition guide interface for guiding the ultrasonic probe to acquire the ultrasonic image in a second subarea of the second display area, wherein the acquisition guide interface comprises an acquisition guide prompt based on the three-dimensional reconstruction model;

and displaying acquisition contents on the organ registration operation interface, wherein the acquisition contents are determined by the automatic acquisition triggering operation and the manual acquisition triggering operation.

7. The registration method of claim 6, wherein the acquisition content is displayed on the organ registration operation interface when the triggering operation is an automatic acquisition triggering operation, the acquisition content including an acquisition duration and a number of frames of ultrasound images acquired by the ultrasound probe.

8. The registration method of claim 6, wherein when the trigger operation is a manual acquisition trigger operation:

displaying an acquisition control on the organ registration operation interface;

Responding to the triggering operation of the acquisition control, and acquiring an ultrasonic image currently acquired by the ultrasonic probe;

displaying the acquisition content on the organ registration operation interface, wherein the acquisition content comprises ultrasonic images acquired by the ultrasonic probe, and the ultrasonic images are sequentially displayed on the organ registration operation interface according to an acquisition sequence.

9. An organ-based multi-modality registration device, comprising:

The image acquisition module is used for acquiring a target image sequence of the checked object, and carrying out target organ identification and three-dimensional reconstruction based on the target image sequence to acquire a target organ image sequence and a three-dimensional reconstruction model of the checked object;

The first display module is used for displaying an organ registration operation interface in a first display area, wherein the organ registration operation interface comprises a magnetic field orientation selection control and an ultrasonic image acquisition control, the magnetic field orientation selection control is used for selecting a magnetic field orientation, and the magnetic field orientation is used for representing the relative position relation between a magnetic field generator and the inspected object;

The magnetic field control response module is used for responding to the triggering operation of the magnetic field orientation selection control, displaying a plurality of alternative magnetic field orientations on the organ registration operation interface and determining a target magnetic field orientation from the plurality of alternative magnetic field orientations;

the second display module is used for responding to the triggering operation of the ultrasonic image acquisition control, displaying an acquisition guide interface for guiding an ultrasonic probe to acquire ultrasonic images in a second display area, wherein the acquisition guide interface comprises an acquisition guide prompt based on the three-dimensional reconstruction model, and the acquisition guide prompt comprises a prompt for guiding the ultrasonic probe to move through arrow directions, movement tracks or flow demonstrations;

the ultrasonic acquisition module is used for carrying out ultrasonic sampling on the checked object based on the acquisition guide prompt through an ultrasonic probe to acquire an ultrasonic image of the checked object, wherein the ultrasonic image comprises an ultrasonic image of a target organ;

A registration module for completing registration of the target image sequence with the ultrasound image based on the target magnetic field orientation, the target organ image sequence and the ultrasound image, the registration module further for performing the steps of: