CN118902603B

CN118902603B - Surgery assistance system and surgery assistance method

Info

Publication number: CN118902603B
Application number: CN202410956140.XA
Authority: CN
Inventors: 刘荣; 张修平; 请求不公布姓名
Original assignee: Shanghai Microport Medbot Group Co Ltd
Current assignee: Shanghai Microport Medbot Group Co Ltd
Priority date: 2024-07-16
Filing date: 2024-07-16
Publication date: 2025-02-25
Anticipated expiration: 2044-07-16
Also published as: CN118902603A

Abstract

The invention provides an operation assisting system and an operation assisting method, wherein the operation assisting system comprises an image large model module, a visual large model module, a risk detection large model module and an operator dialogue large model module, the image large model module is used for dividing and reconstructing preoperative images based on the image large model and outputting division result information of the preoperative images, the visual large model module is used for analyzing endoscope scene content in an operation and outputting endoscope content information based on the visual large model, the risk detection large model module is used for analyzing and judging based on the division result information and the endoscope content information and outputting risk prompt information, and the operator dialogue large model module is used for outputting the proposal information according to the division result information, the endoscope content information and the risk prompt information. So configured, the assessment can be integrated and advice can be given to the operator to more intuitively assist the operator in reducing the risk in the operation.

Description

Surgical assistance system and surgical assistance method

Technical Field

The invention relates to the technical field of medical instruments, in particular to a surgery auxiliary system and a surgery auxiliary method.

Background

When performing an endoscopic procedure using a surgical robotic system, the slave end robot is typically manipulated by the operator at the physician console by observing real-time images of the endoscopic feedback. The surgical robot system generally only gives a warning based on the limitation of the technological parameters and gives an alarm feedback when the motor load is excessive, and has no other further prompt and interaction functions. This brings about the following two disadvantages:

1. Highly dependent operators, namely, the operators are completely dependent on the mastering degree of current scenes and information (such as patient medical history conditions and the like), abnormal conditions and dangers in the operation can not be reminded by an operation robot system, the operation process is completely dependent on the operation capability and human eye recognition capability of the operators, and once the operators are poor in hand-eye coordination or tired due to long-time operation, the operation risk can be improved.

2. The cost of manpower is high, an operator needs to manually operate the operation in the whole process at a doctor console, a plurality of medical staff are needed for assisting other works, the whole operation flow completely depends on manpower, a great deal of manpower is needed for an operation robot system, and the manpower burden is increased. On the other hand, the video data after operation is huge, and a great deal of manpower and energy are also required for the analysis after operation.

Disclosure of Invention

The invention aims to provide a surgical auxiliary system and a surgical auxiliary method, which are used for solving the problem that the existing surgical robot system depends on experience of a surgeon.

In order to solve the technical problems, the invention provides an operation assisting system which comprises an image large model module, a vision large model module, a risk detection large model module and an operator dialogue large model module;

The image large model module is used for dividing and reconstructing preoperative images based on the image large model and outputting division result information of the preoperative images;

the visual large model module analyzes the content of the endoscope scene in operation based on the visual large model and outputs the content information of the endoscope;

the risk detection large model module analyzes and judges based on the segmentation result information and the endoscope content information and outputs risk prompt information;

and the operator dialogue language big model module outputs suggestion information according to the segmentation result information, the endoscope content information and the risk prompt information.

Optionally, the large model module of the operator dialogue language comprises an interaction unit, the large model module of the operator dialogue language is further configured to acquire input information through the interaction unit, and at least one of the large model module of the vision, the large model module of the risk detection and the large model module of the operator dialogue language corrects the information output by the user based on the input information.

Optionally, the large model module of the operator dialogue language further comprises a semantic analysis unit, wherein the semantic analysis unit is used for analyzing the input information acquired by the interaction unit and correcting the input information.

Optionally, the surgical assistance system further comprises a data storage module configured to store the segmentation result information, the advice information, and the endoscope content information.

Optionally, the segmentation result information is image data, the suggestion information and the endoscope content information are text data, and the data storage module is configured to combine the segmentation result information, the suggestion information and the endoscope content information with a timestamp to form a multi-mode data structure body for storage.

Optionally, the operation assisting system further comprises a postoperative evaluation module, wherein the postoperative evaluation module evaluates operations based on the multi-mode data structure body stored by the data storage module and outputs evaluation results.

Optionally, the postoperative evaluation module comprises a splitting unit and an intraoperative task analysis unit, wherein the splitting unit is used for splitting the multi-mode data structure body acquired from the data storage module and sending the data obtained by splitting to the intraoperative task analysis unit, and the intraoperative task analysis unit evaluates the operation based on the data obtained by splitting by the splitting unit and outputs an evaluation result.

Optionally, the image large model module is configured to reconstruct the pre-operation image through a first convolution neural network, wherein the input information of the first convolution neural network comprises the pre-operation image and medical record information processed by a word segmentation device;

The visual large model module is configured to analyze the content of the endoscope scene in operation through a second convolution neural network, wherein the input information of the second convolution neural network comprises the endoscope image in operation and the input information processed by the word segmentation device;

the risk detection large model module is configured to analyze and judge through a third convolutional neural network, wherein the input information of the third convolutional neural network comprises the segmentation result information and the endoscope content information processed by the word segmentation device;

the operator dialogue language big model module is configured to analyze and judge through a fourth convolution neural network, and input information of the fourth convolution neural network comprises the segmentation result information, the endoscope content information processed by the word segmentation device and the risk prompt information processed by the word segmentation device.

Optionally, the surgical assistance system further comprises an endoscope calibration module configured to perform the following steps for calibrating an endoscope prior to surgery:

shooting a three-dimensional calibration object to obtain a first image;

Coarse calibrating the endoscope according to the first image and the known size information of the three-dimensional calibration object;

Calculating a calibration error based on the coarse calibration result and outputting the position and angle of the region to be optimized;

arranging a two-dimensional calibration object according to the position and the angle of the area to be optimized, shooting the two-dimensional calibration object, and obtaining a second image;

And according to the second image and the known size information of the two-dimensional calibration object, combining the characteristic information of the three-dimensional calibration object in the coarse calibration step to carry out fine calibration on the endoscope.

Optionally, the step of coarsely calibrating the endoscope includes:

extracting first characteristic points in the first image, and acquiring two-dimensional image coordinate information corresponding to each first characteristic point in the first image;

acquiring world coordinate information of each first feature point according to the size information of the three-dimensional calibration object;

And matching the two-dimensional image coordinate information and the world coordinate information of each first characteristic point, constructing an equation set containing endoscope parameters to be solved, and solving the equation set to obtain a coarse calibration result.

Optionally, the characteristic information of the three-dimensional calibration object in the coarse calibration step comprises two-dimensional image coordinate information and world coordinate information of the first characteristic point, and the step of precisely calibrating the endoscope comprises the following steps:

extracting second characteristic points in the second image, identifying a central characteristic point and a two-dimensional calibration object coordinate system, and obtaining world coordinate information and two-dimensional image coordinate information of each second characteristic point according to the size information of the two-dimensional calibration object;

and combining the two-dimensional image coordinate information and the world coordinate information of the first feature points and the world coordinate information and the two-dimensional image coordinate information of the second feature points to jointly construct an equation set containing endoscope parameters to be solved, and solving the equation set to obtain a precise calibration result.

Optionally, the step of calculating the calibration error based on the coarse calibration result includes:

Mapping the world coordinate information of the first characteristic point calculated in the coarse calibration step to a two-dimensional image coordinate system based on the endoscope parameters after the coarse calibration to obtain virtual two-dimensional image coordinate information of the first characteristic point;

And calculating the Euclidean distance between the coordinate information of the virtual two-dimensional image and the coordinate information of the two-dimensional image of the first characteristic point in the first image, and determining the Euclidean distance as a calibration error.

Optionally, the step of outputting the position and the angle of the region to be optimized includes:

And respectively calculating calibration errors of different segmentation areas in the first image corresponding to planes of the three-dimensional calibration object in different inclination directions, selecting segmentation areas with the calibration errors exceeding the threshold value as areas to be optimized according to a preset threshold value, and determining the inclination angle of the corresponding plane of the three-dimensional calibration object as the angle of the areas to be optimized.

Optionally, the operation auxiliary system further comprises an endoscope fixing device, a calibration object fixing device, an object distance adjusting device, a three-dimensional calibration object and a two-dimensional calibration object;

The endoscope fixing device is used for installing the endoscope, the relative positions of the endoscope fixing device and the marker fixing device are adjustably set through the object distance adjusting device, and the three-dimensional marker and the two-dimensional marker are configured to be detachably arranged on the marker fixing device.

Optionally, the three-dimensional calibration object comprises a plurality of repeated structures, each repeated structure comprises a plurality of planes with different inclination directions, and/or the two-dimensional calibration object comprises a plane, and the plane comprises a plurality of first characteristic symbols and a plurality of second characteristic symbols.

Optionally, the endoscope calibration system further comprises a display device, wherein the display device is used for displaying:

At least one of an intra-operative endoscope image, the advice information, the risk advice information, an image captured by the endoscope, a divided region of a plane corresponding to different oblique directions of the three-dimensional calibration object, and a region to be optimized.

In order to solve the technical problem, the invention also provides an operation assisting method, which comprises the following steps:

based on the image large model, segmenting and reconstructing preoperative images, and outputting segmentation result information of the preoperative images;

Analyzing the content of the endoscope scene in the operation based on the visual large model, and outputting the content information of the endoscope;

based on the segmentation result information and the endoscope content information, analyzing and judging, and outputting risk prompt information;

And outputting advice information according to the segmentation result information, the endoscope content information and the risk prompt information.

To solve the above technical problems, the present invention also provides a surgical robot system including the surgical assistance system as described above.

In summary, in the surgical assistance system and the surgical assistance method, the surgical assistance system comprises an image large model module, a visual large model module, a risk detection large model module and a worker dialogue large model module, wherein the image large model module is used for dividing and reconstructing preoperative images based on the image large model and outputting division result information of the preoperative images, the visual large model module is used for analyzing endoscope scene content in an operation and outputting endoscope content information based on the visual large model, the risk detection large model module is used for analyzing and judging based on the division result information and the endoscope content information and outputting risk prompt information, and the worker dialogue large model module is used for outputting the proposal information according to the division result information, the endoscope content information and the risk prompt information.

So configured, based on the settings of the image large model module, preoperative images can be loaded and segmented for reconstruction, and potential surgical risks can be analyzed in advance. Based on the setting of the vision large model module, real-time images of the endoscope can be analyzed. Based on the setting of the risk detection large model module, the segmentation result of the preoperative image and the real-time image of the endoscope can be fused, analysis and judgment are carried out, and risk prompt information is output. Based on the setting of the large model module of the operator dialogue language, the method can comprehensively evaluate and give suggestions to the operator based on the segmentation result information, the endoscope content information and the risk prompt information so as to more intuitively assist the operator in reducing the risk in the operation.

Drawings

Those of ordinary skill in the art will appreciate that the figures are provided for a better understanding of the present invention and do not constitute any limitation on the scope of the present invention.

Fig. 1 is a functional block diagram of a surgical assistance system according to an embodiment of the present invention.

Fig. 2 is a functional block diagram of an image large model module according to an embodiment of the invention.

FIG. 3 is a functional block diagram of a large visual model module according to an embodiment of the present invention.

Fig. 4 is a schematic diagram of a functional module of a risk detection large model module according to an embodiment of the present invention.

FIG. 5 is a functional block diagram of a large model module of an operator dialog language in accordance with an embodiment of the present invention.

Fig. 6 is a functional schematic of a word segmentation apparatus according to an embodiment of the present invention.

Fig. 7 is a schematic diagram of content displayed on a display screen according to an embodiment of the present invention.

FIG. 8 is a functional schematic of a data storage module according to an embodiment of the invention.

FIG. 9 is a schematic diagram of two storage forms of a data storage module according to an embodiment of the present invention.

FIG. 10 is a functional block diagram of a data storage module according to an embodiment of the present invention.

FIG. 11 is a functional schematic of a post-operative evaluation module according to an embodiment of the present invention.

Fig. 12 is a functional block diagram of a post-operation evaluation module according to an embodiment of the present invention.

FIG. 13 is a schematic diagram of two evaluation forms of a post-operative evaluation module according to an embodiment of the present invention.

Fig. 14 is a schematic view of a display module according to an embodiment of the invention.

FIG. 15 is a schematic illustration of a calibration flow of an endoscope calibration module in accordance with an embodiment of the present invention.

FIG. 16 is a perspective view of a three-dimensional calibration object according to an embodiment of the present invention.

FIG. 17 is a top view of a three-dimensional calibration object according to an embodiment of the invention.

FIG. 18 is a side view of a three-dimensional calibration object according to an embodiment of the invention.

FIG. 19 is a schematic flow chart of a coarse calibration step according to an embodiment of the present invention.

FIG. 20 is a schematic diagram of a two-dimensional calibration object according to an embodiment of the present invention.

Fig. 21 is a schematic diagram of feature point extraction according to an embodiment of the present invention.

FIG. 22 is a schematic flow chart of the fine calibration step according to an embodiment of the present invention.

Fig. 23a and 23b are schematic diagrams of an endoscopic parameter solving according to an embodiment of the present invention.

FIG. 24 is a schematic diagram of calibration accuracy calculation according to an embodiment of the present invention.

Fig. 25 is a schematic diagram of a display and contents displayed by the display according to an embodiment of the present invention.

FIG. 26 is a schematic view of an endoscope calibration assembly in accordance with an embodiment of the present invention.

Fig. 27a and 27b are schematic views of an AR display device and contents displayed by the AR display device according to an embodiment of the present invention.

In the drawing, a 10-image large model module, a 20-visual large model module, a 30-risk detection large model module, a 40-operator dialogue language large model module, a 41-phonetic character recognition column, a 50-data storage module, a 60-postoperative evaluation module, a 61-splitting unit, a 62-intraoperative task analysis unit, a 70-display module, a 71-display screen, a 72-green warning lamp, a 73-red warning lamp, a 74-display, a 75-AR display device, a 80-substrate, a 81-repetitive structure, a 82-slope, a 83-top surface, a 84-plane, a 85-first characteristic symbol, a 86-second characteristic symbol, a 91-endoscope fixing device, a 92-calibration object fixing device, a 93-object distance adjusting device, a 94-three-dimensional calibration object, a 96-calibration unit, a 901-endoscope, a 902-image, a 903-segmentation network, a 904-circular, a 905-triangle, a 906-segmentation area, a 907-area to be optimized, and a 908-position to be placed.

Detailed Description

The invention will be described in further detail with reference to the drawings and the specific embodiments thereof in order to make the objects, advantages and features of the invention more apparent. It should be noted that the drawings are in a very simplified form and are not drawn to scale, merely for convenience and clarity in aiding in the description of embodiments of the invention. Furthermore, the structures shown in the drawings are often part of actual structures. In particular, the drawings are shown with different emphasis instead being placed upon illustrating the various embodiments.

The invention aims to provide a surgical auxiliary system and a surgical auxiliary method, which are used for solving the problem that the existing surgical robot system depends on experience of a surgeon. The following description refers to the accompanying drawings.

Referring to fig. 1, an embodiment of the invention provides a surgery assistance system, which comprises an image big model module 10, a visual big model module 20, a risk detection big model module 30 and a surgery dialogue big model module 40, wherein the image big model module 10 is used for dividing and reconstructing a pre-surgery image based on the image big model and outputting dividing result information of the pre-surgery image, the visual big model module 20 is used for analyzing the content of an endoscope in surgery and outputting endoscope content information based on the visual big model, the risk detection big model module 30 is used for analyzing and judging based on the dividing result information and the endoscope content information and outputting risk prompt information, and the surgery dialogue big model module 40 is used for outputting the proposal information according to the dividing result information, the endoscope content information and the risk prompt information.

The image large model module 10 (IMAGINGLM) may perform segmentation processing on the acquired preoperative images (including CT images or MRI images, etc.) based on the image large model. The segmentation result information can comprise image data and also feature information, such as some identified risks (such as anatomical structure variation, pipeline variation, lymph nodes with metastasis, and the like). Referring to FIG. 2, a functional block diagram of the image large model module 10 (IMAGINGLM) is exemplarily shown. Optionally, the image large model module 10 is configured to reconstruct the pre-operation image through a first convolutional neural network (CNN 1), wherein the input information of the first convolutional neural network (CNN 1) includes the pre-operation image and the medical record information processed by the word segmentation device (tokenizer).

The visual large model module 20 (VisualLM) may analyze acquired intra-operative endoscopic scene content (including real-time images of the intra-operative endoscope, etc.) based on the visual large model. The output endoscope content information may include, for example, textual description information or the like analyzed with respect to the current endoscope image. Referring to FIG. 3, a functional block diagram of the vision large model module 20 (VisualLM) is exemplarily shown. The vision large model module 20 is configured to analyze the content of the intraoperative endoscopic scene through a second convolutional neural network (CNN 2), wherein the input information of the second convolutional neural network (CNN 2) comprises the intraoperative endoscopic image and the input information processed by the word splitter (tokenizer), and the input information is available based on the interaction unit of the intraoperative dialogue large model module 40 (described in detail below).

The risk detection large model module 30 (RiskDetectorLM) may obtain the segmentation result information of the preoperative image output by the image large model module 10 and the endoscope content information obtained by analysis by the visual large model module 20, so as to perform fusion analysis and judgment based on the segmentation result information and the endoscope content information, and the output risk prompt information may be literal description information. Referring to FIG. 4, a functional block diagram of the risk detection large model module 30 (RiskDetectorLM) is exemplarily shown. The risk detection large model module 30 is configured to perform analysis and judgment through a third convolutional neural network (CNN 3), wherein the input information of the third convolutional neural network (CNN 3) comprises the segmentation result information and the endoscope content information processed by the word segmentation device (tokenizer). Here, the input information of the third convolutional neural network (CNN 3) includes a set of image data from the segmentation result information output by the image large model module 10 and a set of text data from the endoscope content information output by the visual large model module 20.

The operator dialogue language big model module 40 (SurgeonChatLLM) may analyze and summarize the information output by the image big model module 10, the vision big model module 20, and the risk detection big model module 30 based on the language big model, preferably analyze and comprehensively evaluate based on the prompt of the current operation plan, and give corresponding advice information (such as prompt for dangerous operation, vascular variation prompt, accidental bleeding prompt, etc.). The output advice information may include descriptive information, image information, voice prompt information, etc., and may include resistance of a control arm applied to a doctor console, etc. The operator may proceed based on the suggested information presented by the operator dialog language big model module 40. It can be appreciated that the advice information is obtained based on comprehensive evaluation of information such as preoperative images, current endoscope images and the like, so that the operator can be more intuitively assisted in reducing the risk in the operation. Referring to FIG. 5, a functional block diagram of the operator dialog language big model module 40 (SurgeonChatLLM) is exemplarily shown. The large operator dialogue language model module 40 is configured to perform analysis and judgment through a fourth convolutional neural network (CNN 4), wherein the input information of the fourth convolutional neural network (CNN 4) comprises the segmentation result information, the endoscope content information processed by the word segmentation device (tokenizer) and the risk prompt information processed by the word segmentation device (tokenizer). Here, the input information of the fourth convolutional neural network (CNN 4) includes one set of image data and two sets of text data, the image data is from the segmentation result information output by the image large model module 10, the text data is from the endoscope content information output by the vision large model module 20 and the risk prompt information output by the risk detection large model module 30.

Referring to fig. 6, it should be noted that the word segmentation unit (tokenizer) is used to vector the input text so that the convolutional neural networks can be input as features. In addition, the large models such as the image large model, the visual large model and the language large model, and the convolutional neural networks can be trained in advance, and the training process can refer to the prior art, and is not described herein.

The operation auxiliary system provided by the embodiment can give operation advice (advice information) in real time in operation by combining preoperative images with large models (including large models of images, large models of vision, large models of language and the like), does not need to mark image data, does not need to use target detection and predefined operation instructions, and has stronger generalization capability in non-preset scenes and unknown scenes.

Optionally, in some embodiments, the large model module 40 comprises an interaction unit, the large model module 40 is further configured to obtain input information through the interaction unit, and at least one of the large model module 20, the large model module 30 for risk detection and the large model module 40 for human dialog corrects the information output by the user based on the input information.

The interactive unit is configured to accept input from an operator. It should be noted that, the input information herein includes, but is not limited to, text input information, voice input information, touch click information, and other input information common in the art. Referring to FIG. 7, a display 71 (described in detail below) is shown, which includes an example of an interactive element, such as a speech and text recognition column 41. The operator can input information by means of text or voice.

Furthermore, the large model module of the operator dialogue language further comprises a semantic analysis unit, wherein the semantic analysis unit is used for analyzing the input information acquired by the interaction unit and correcting the input information. The semantic analysis unit can analyze the input information from the vocabulary level and the sentence level, and correct the input information by combining the output information of the image large model module 10, the vision large model module 20, the risk detection large model module 30 and other modules, thereby further improving the accuracy and the reliability of the input information.

On the one hand, if the operator finds that the recommended information given by the operator dialogue language big model module 40 is wrong, the operator can feed back the recommended information to the operator dialogue language big model module 40 through the interaction unit, and at least one of the visual big model module 20, the risk detection big model module 30 and the operator dialogue language big model module 40 can correct the respective output information according to the feedback of the operator so as to correct the error. In an alternative example, the operator dialog language macrocode module 40 may analyze specific tasks based on input information and communicate to the vision macrocode module 20 for execution (e.g., requiring the vision macrocode module 20 to focus on analyzing the organ being cut in preparation for the next proposal). Thereby dynamically highlighting soft tissue separation regions, highlighting suturing regions, highlighting dangerous vessel regions, etc. at different stages of the procedure.

On the other hand, referring to fig. 3 in combination, the content of the intra-operative endoscope scene acquired by the large visual model module 20 is not limited to include only real-time endoscope images, but may include feedback and judgment information made by the operator based on the current endoscope images, and the feedback and judgment information may be used as input information to feedback to the large visual model module 40 through the interaction unit, and then transmitted to the large visual model module 20 by the large visual model module 40. So configured, the accuracy of the endoscope content information analyzed by the vision large model module 20 can be further improved.

With continued reference to fig. 1 and 8, optionally, the surgical assistance system further comprises a data storage module 50 (LMStorage), the data storage module 50 being configured to store the segmentation result information, the recommendation information, and the endoscope content information. It will be appreciated that the data stored by the data storage module 50 is actually a multi-modal data in that the segmentation result information, the advice information and the endoscope content information contain both images and text. Referring to fig. 9, the data storage module 50 may be stored in various forms, and fig. 9 shows two storage forms of the data storage module 50, one is a local NAS server mode, and the other is a cloud server storage mode based on an encrypted channel. Of course, other storage forms may be configured for the data storage module 50 by those skilled in the art in accordance with the prior art.

Further, referring to fig. 10, the segmentation result information is image data, the suggestion information and the endoscope content information are text data, and the data storage module 50 is configured to combine the segmentation result information, the suggestion information and the endoscope content information with a timestamp to form a multi-modal data structure for storage. The data storage module 50 integrates data of multiple modalities into a multi-modal data structure for storage in a storage medium.

Optionally, referring to fig. 1, the surgical assistance system further includes a post-operation evaluation module 60 (SurgeryEVA), and the post-operation evaluation module 60 evaluates the operation based on the multi-modal data structure stored by the data storage module 50 and outputs an evaluation result. Based on the configuration of the data storage module 50, the surgical procedure may be analyzed and evaluated after surgery based on the multi-modal data structure stored in the data storage module 50, for example, to give the analysis result of surgery (e.g., instrument motion distribution, time-consuming of specific excision motion, etc.).

Referring to fig. 11, post-operative evaluation module 60 may perform a predefined evaluation based on the multimodal data structure. Predefined evaluations include, but are not limited to, procedure length, action proficiency, procedure sequence, and the like.

Referring to fig. 12, the postoperative evaluation module 60 includes a splitting unit 61 and an intraoperative task analysis unit 62, wherein the splitting unit 61 is configured to split the multi-modal data structure acquired from the data storage module 50 and send the split data to the intraoperative task analysis unit 62, and the intraoperative task analysis unit 62 evaluates the operation based on the split data of the splitting unit 61 and outputs the evaluation result. Since the multi-modal data structure is acquired from the data storage module 50, it is required to split it by the splitting unit 61 to obtain three categories of specific image data, text data and time stamp data. The three kinds of data obtained by splitting are input to the intraoperative task analysis unit 62, and analysis and evaluation can be performed.

The post-operative evaluation module 60 may present the evaluation results (e.g., contrast charts, evaluation scores, etc.) in a variety of forms. Referring to fig. 13, the post-operation evaluation module 60 is implemented in various ways, and fig. 13 shows two evaluation modes of the post-operation evaluation module 60, one of which is manual evaluation, for example, providing an access to a web page and providing a function of scoring time periods on the page, so as to facilitate evaluation by the same party. And secondly, automatically evaluating, for example, automatically giving a comparison analysis result with a sample operation formula (such as an expert operation formula). Of course, one skilled in the art may also configure the post-operative evaluation module 60 with other forms of evaluation according to the prior art.

In this embodiment, the visual large model module 20 (VisualLM) needs to analyze the content of the intraoperative endoscope scene, which is based on the endoscopic shots. To improve accuracy, the endoscope generally needs to be calibrated prior to surgery. In the field of medical endoscopes, a currently widely applied endoscopic internal reference calibration method is a Zhang Zhengyou checkerboard calibration method, the method is to shoot a plurality of checkerboard pictures with different angles at different positions of an endoscope visual field, obtain angular point image coordinates of the checkerboard from the pictures, set a certain angular point of the checkerboard as an origin of a world coordinate system, calculate the world coordinate system of the angular point due to the fact that the size of the checkerboard is known, and further construct an equation based on an endoscope imaging model, so that internal references of the endoscope are solved. However, on the one hand, the Zhang Zhengyou checkerboard calibration method has complex calibration flow and low speed, at least needs to shoot 15-20 images, has certain requirements on the shooting angle of the endoscope and the placement of the calibration plate, and ensures that the checkerboard covers the whole image plane as much as possible and comprises various angles and distances during shooting. On the other hand, the Zhang Zhengyou checkerboard calibration method has the problems of unstable calibration precision and strong subjectivity, and because photos of different positions and angles need to be taken, the calibration precision is influenced by the proficiency of operators, and the calibration precision of different operators can be quite different.

In order to solve the problems of complex flow, slow speed and unstable calibration precision of the existing endoscope, it is preferable to refer to fig. 15, in the embodiment of the present invention, the surgical auxiliary system further includes an endoscope calibration module, where the endoscope calibration module is configured to perform the following steps for calibrating the endoscope before the operation:

s1, shooting a three-dimensional calibration object to obtain a first image;

S2, performing coarse calibration on the endoscope according to the first image and the known size information of the three-dimensional calibration object;

step S3, calculating a calibration error based on a coarse calibration result and outputting the position and the angle of the region to be optimized;

S4, arranging a two-dimensional calibration object according to the position and the angle of the area to be optimized, shooting the two-dimensional calibration object, and obtaining a second image;

And S5, carrying out fine calibration on the endoscope according to the second image and the known size information of the two-dimensional calibration object and combining the characteristic information of the three-dimensional calibration object in the coarse calibration step.

So configured, on one hand, only a small amount of images need to be shot, the internal parameters of the endoscope can be solved, the flow is simple, and the calibration speed is high. On the other hand, firstly, the endoscope is roughly calibrated by using a first image of the shot three-dimensional calibration object, and then, the region to be optimized at a specific position and angle is finely calibrated by using the shot two-dimensional calibration object, so that the overall calibration precision and the robustness of the calibration result are effectively improved.

The three-dimensional object preferably has a plurality of planes of different oblique directions, each of which preferably has a plurality of easily extractable signatures thereon. Thus, the endoscope can calibrate by shooting the three-dimensional calibration object according to a plurality of planes with known gradient on the three-dimensional calibration object and combining the characteristic marks on the planes.

Referring to fig. 16 to 18, the three-dimensional calibration object preferably includes a plurality of repeating structures 81, and each repeating structure 81 includes a plurality of planes with different inclination directions. In the three-dimensional object shown in fig. 16 to 18, one substrate 80 and four repeating structures 81 are included. Each repeating structure 81 has a quadrangular frustum-shaped configuration, i.e., each repeating structure 81 includes four slopes 82 of different oblique directions and a top surface 83 parallel to the substrate 80. Further, each plane (including four slopes 82 and a top surface 83) has a plurality of easily extracted feature identifiers, such as graphic symbols, e.g. dots, marked on each plane. Further, dimensional information of the three-dimensional calibration object is known, that is, the size of each plane of each repeating structure 81, the inclination direction and inclination angle of each slope 82, and the like are known.

In step S1, a suitable distance between the endoscope and the three-dimensional calibration object is adjusted according to a focal length, a depth of field and a visual field of the endoscope to be calibrated, so that the three-dimensional calibration object can be shot by the endoscope to obtain a first image.

In step S2, since the size information of the three-dimensional calibration object is known, the endoscope can be roughly calibrated according to the first image and the known size information of the three-dimensional calibration object.

Referring to fig. 19, an exemplary process for coarsely calibrating an endoscope is shown. The steps of coarsely calibrating the endoscope include:

S21, extracting first characteristic points in the first image, and acquiring two-dimensional image coordinate information corresponding to each first characteristic point in the first image;

step S22, world coordinate information of each first feature point is obtained according to the size information of the three-dimensional calibration object;

And S23, matching the two-dimensional image coordinate information and the world coordinate information of each first characteristic point, constructing an equation set containing the endoscope parameters to be solved, and solving the equation set to obtain a coarse calibration result.

Because each plane of the three-dimensional calibration object is provided with a plurality of feature identifiers which are easy to extract, in step S21, for example, a first feature point in the first image, such as the center of a dot, can be extracted by a feature extraction algorithm, so that two-dimensional image coordinate information corresponding to the first feature point, that is, pixel coordinate information of the first feature point in the first image, can be obtained.

In step S22, the plurality of first feature points extracted in step S21 may be numbered in order according to their positional relationship in the first picture, and three-dimensional world coordinate information of each first feature point may be obtained according to the known size information of the three-dimensional calibration object.

After the coarse calibration is completed, step S3 may calculate calibration errors of different regions of the first image, screen out a region to be optimized that needs further fine calibration, and output the position and angle thereof. And further performing fine calibration through the step S4.

Referring to fig. 20, the two-dimensional object preferably includes a plane 84, and the plane 84 includes a plurality of first signatures 85 and a plurality of second signatures 86. The plurality of first feature symbols 85 may be arranged in an array in a certain order, and the plurality of second feature symbols 86 may be arranged in a specific position, where the arrangement pattern formed by the second feature symbols 86 and the first feature symbols 85 does not coincide. In the example shown in fig. 20, the first signature 85 is circular and is arranged in a five-row five-column array, but with several of the arrangement positions replaced by the second signature 86. The second signature 86 is triangular (the triangle shown in fig. 20 also includes inscribed circles to facilitate feature extraction), and the second signature 86 is preferably disposed at a particular location in the corners, center, etc. of a five-row five-column array. Further, the size information of the two-dimensional calibration object is known, that is, the size, number, center position, and arrangement order of each first signature 85, and the size, number, center position, and arrangement order of each second signature 86 are known. It will be appreciated that circles and triangles are merely exemplary of the first and second signatures 85 and 86 and are not limiting, and that other shapes may be selected as signatures in other embodiments. The number, size, arrangement order, etc. of the feature symbols may be selected and arranged according to the actual implementation, and are not limited to those shown in fig. 20.

Referring to fig. 21, one embodiment of a feature extraction algorithm is exemplarily shown. In this embodiment, a neural network algorithm may be used to train a segmentation model, an image 902 captured by an endoscope is input during reasoning, a segmentation network 903 outputs mask images of a plurality of feature points, such as a circle 904 and a triangle 905, and then accurate two-dimensional image coordinates of the feature points are obtained by calculating the center of the circle and the center of the triangle. Of course, the example shown in fig. 21 is only one example of a feature extraction algorithm and is not limiting of the feature extraction algorithm, and other feature extraction algorithms may be employed by those skilled in the art in light of the prior art.

In step S4, arranging the two-dimensional calibration object according to the position and the angle of the area to be optimized means that the plane 84 of the two-dimensional calibration object is arranged according to the position and the angle of the area to be optimized output in step S3, so that the plane 84 of the two-dimensional calibration object is parallel or coincident with the position and the angle of the area to be optimized. Since the position and angle of the area to be optimized can be understood as being selected according to a certain plane of the three-dimensional calibration object, it can be understood that the two-dimensional calibration object is a certain plane for replacing the original three-dimensional calibration object.

In step S5, since the size information of the two-dimensional calibration object is known, the endoscope can be precisely calibrated according to the second image and the known size information of the two-dimensional calibration object in combination with the characteristic information of the three-dimensional calibration object in the coarse calibration step. The characteristic information of the three-dimensional calibration object in the coarse calibration step comprises two-dimensional image coordinate information and world coordinate information of the first characteristic point.

Referring to fig. 22, an exemplary process for fine calibration of an endoscope is shown. The steps of precisely calibrating the endoscope comprise:

step S51, extracting second characteristic points in the second image, identifying a central characteristic point and a two-dimensional calibration object coordinate system, and obtaining world coordinate information and two-dimensional image coordinate information of each second characteristic point according to the size information of the two-dimensional calibration object;

And S52, combining the two-dimensional image coordinate information and the world coordinate information of the first feature points and the world coordinate information and the two-dimensional image coordinate information of the second feature points to jointly construct an equation set containing the endoscope parameters to be solved, and solving the equation set to obtain a precise calibration result.

Since the plane 84 of the two-dimensional calibration object has a plurality of first feature symbols 85 and a plurality of second feature symbols 86 that are easy to be extracted, in step S51, for example, the second feature points in the second image, such as the center of the circle of the first feature symbol 85 and the center of the triangle of the second feature symbol 86, can be extracted by the feature extraction algorithm, so that the center of the second image and the center feature point located at the center of the second image (the center of the triangle located at the center in fig. 20) can be identified, and the two-dimensional calibration object coordinate system can be obtained by identification. And then according to the known size information of the two-dimensional calibration object, the world coordinate information and the two-dimensional image coordinate information of each second characteristic point can be obtained.

In step S52, the two-dimensional image coordinate information and the world coordinate information of the first feature point obtained in step S21 and step S22, and the world coordinate information and the two-dimensional image coordinate information of the second feature point obtained in step S51 may be combined together to form an equation set, and the precise calibration result is obtained by solving.

As will be appreciated by those skilled in the art, an endoscope acts as a camera, which generally has external and internal parameters. The extrinsic parameters include, for example, a rotation matrix R and a translation vector T. The external parameters may change at different camera positions or shooting moments. The internal parameters are parameters describing the internal properties of the camera, including an internal matrix M and a distortion model F _disp, etc. Calibration of the endoscope can be understood as solving for internal parameters of the endoscope.

The steps of achieving calibration by constructing a system of equations and solving in step S23 and step S52 will be exemplarily described with reference to fig. 23a and 23 b.

Fig. 23a shows a two-dimensional image coordinate system of a certain image captured by an endoscope, where o is an image origin, U and V denote two coordinate axes of the image, and a two-dimensional image coordinate of a certain feature point may be expressed as (ui, vi). Fig. 23b shows a state of a certain calibration object in the world coordinate system, and in one embodiment, the central feature point (not limited to the central feature point, but also some other feature point) of the calibration object may be set as the origin O of the world coordinate system, and the world coordinate of a certain feature point may be expressed as (Xi, yi, zi). By extracting a plurality of groups of matched characteristic points and obtaining corresponding two-dimensional image coordinates and world coordinates, the following equation set can be constructed:

Wherein F _disp is a distortion model of the endoscope, M is an internal reference matrix of the endoscope, R is a rotation matrix, T is a translation vector, and F _disp, M, R and T can be obtained by solving the above equation set.

Optionally, in step S3, the step of calculating the calibration error based on the coarse calibration result includes:

step S31, mapping the world coordinate information of the first characteristic point calculated in the coarse calibration step to a two-dimensional image coordinate system based on the endoscope parameters after the coarse calibration to obtain virtual two-dimensional image coordinate information of the first characteristic point;

And S32, calculating the Euclidean distance between the coordinate information of the virtual two-dimensional image and the coordinate information of the two-dimensional image of the first characteristic point in the first image, and determining the Euclidean distance as a calibration error.

Referring to fig. 24, in an alternative exemplary embodiment, a re-projection error may be used as a quantization of the calibration accuracy. The re-projection error is to map the feature points to a two-dimensional image coordinate system based on the calibrated endoscope parameters, and further calculate the distance between the virtual two-dimensional image coordinates of the mapped feature points and the real two-dimensional image coordinates. The reprojection error can be expressed by the following formula:

Err=Dist(P,T(P))

Where P represents the real two-dimensional image coordinates of the feature points, T () represents the conversion function of the world coordinate system to the two-dimensional image coordinate system, and Dist () represents the euclidean distance. The arrow in fig. 24 indicates the error direction and the error magnitude of each feature point.

Further, in step S3, the step of outputting the position and the angle of the region to be optimized includes:

And S33, respectively calculating calibration errors of different segmentation areas in the first image corresponding to planes of the three-dimensional calibration object in different inclination directions, selecting segmentation areas with the calibration errors exceeding a threshold value as areas to be optimized according to a preset threshold value, and determining the inclination angle of the corresponding planes of the three-dimensional calibration object as the angle of the areas to be optimized.

Referring to fig. 25, in an example, taking a three-dimensional calibration object including four repeating structures 81, each repeating structure 81 including 5 planes as an example, a first image obtained by photographing the first image by the endoscope may be divided into 41 divided areas 906, and fig. 25 is schematically divided by dotted lines and solid lines. After the coarse calibration, the calibration error of each of the divided regions may be calculated based on step S31 and step S32, respectively, wherein the divided region 906 in which the calibration error exceeds the threshold value may be used as the region 907 to be optimized. It will be appreciated that the number of regions to be optimized is not limited, and may be one, plural or 0.

Optionally, referring to FIG. 26, the surgical assistance system further comprises an endoscope calibration assembly comprising an endoscope fixture 91, a calibration object fixture 92, an object distance adjustment device 93, a three-dimensional calibration object 94, a two-dimensional calibration object 95, and a calibration calculation unit 96;

The relative positions of the endoscope fixing device 91 and the calibration object fixing device 92 are adjustably set by the object distance adjusting device 93, the three-dimensional calibration object 94 and the two-dimensional calibration object 95 are configured to be detachably mounted on the calibration object fixing device 92, and the calibration calculating unit 96 is configured to calibrate the endoscope 901 mounted on the endoscope fixing device 91 according to the endoscope calibration method as described above.

In one embodiment, an endoscope fixture 91 is used for mounting an endoscope 901 to be calibrated. The marker securing device 92 is configured for removably mounting a three-dimensional marker 94 or a two-dimensional marker 95. Preferably, the marker securing device 92 has a freely angularly adjustable articulation so that the three-dimensional marker 94 or the two-dimensional marker 95 mounted thereon can be freely angularly adjusted relative to the endoscope 901.

The object distance adjusting device 93 includes, for example, an adjusting rail and a driving motor, on which the endoscope fixing device 91 and the marker fixing device 92 are movably mounted, respectively, and the relative distance between the endoscope fixing device 91 and the marker fixing device 92 can be adjusted by the driving motor.

The three-dimensional calibration object 94 and the two-dimensional calibration object 95 are referred to in the foregoing description and will not be repeated here.

The calibration calculation unit 96 is in communication with the endoscope, and can acquire images captured by the endoscope 901 and be used to perform the endoscope calibration method as described above, calculate parameters of the endoscope 901, and thereby calibrate the endoscope 901.

Referring to fig. 14, 25 to 27b, optionally, the surgical assistance system further includes a display module 70, where the display module 70 is configured to display at least one of an intra-operative endoscope image, the advice information and the risk cue information, an image captured by the endoscope, a segmented region corresponding to a plane of different inclination directions of the three-dimensional calibration object, and a region to be optimized. The display module 70 may include a display screen 71, a display 74, or an AR display 75, for example, and may also include a warning light, for example. The display module 70 is provided to more intuitively assist the operator.

In one example, the display module 70 may include a display 71, a green warning light 72, and a red warning light 73. When the operator dialogue large model module 40 outputs advice information, the green warning lamp 72 is turned on while the display screen 71 displays the advice information. While the display 71 displays the risk indicator, the red warning light 73 is illuminated. Of course, fig. 14 illustrates only an exemplary embodiment of the display module 70 and is not a limitation of the display module 70, and those skilled in the art may configure the display module 70 according to actual situations. Optionally, the display module 70 may also display the position 908 where the two-dimensional calibration object 95 is to be placed, as shown in fig. 25. In another embodiment, the display module 70 includes a display 74, such as a liquid crystal display, the display range of the display 74 may be adapted to the full field of view of an endoscope, for example, which displays an exemplary image as shown in FIG. 25. Referring to fig. 27a, in another embodiment, the display module 70 includes an AR display device 75, which is shown in fig. 27b, and can map the position and angle of the region to be optimized into the AR display device 75 to guide the operator to place the two-dimensional calibration object, wherein the dashed line represents the position 908 where the two-dimensional calibration object 95 is to be placed.

In some embodiments, the endoscope fixing device 91 and the calibration object fixing device 92 can be automatically moved by a driving motor according to the input position and angle, so that the placement adjustment of the calibration objects is automated, the calibration speed is further improved, and the method is suitable for batch automatic calibration scenes.

Based on the surgical assistance system as described above, an embodiment of the present invention further provides a surgical assistance method, including:

Step SA, based on the image large model, segmenting and reconstructing preoperative images, and outputting segmentation result information of the preoperative images;

Step SB, analyzing the content of the endoscope scene in operation based on the visual large model, and outputting the content information of the endoscope;

Step SC, based on the segmentation result information and the endoscope content information, analyzing and judging, and outputting risk prompt information;

And step SD, according to the segmentation result information, the endoscope content information and the risk prompt information, outputting suggestion information.

Preferably, the surgical assistance method provided by the embodiment of the present invention may also perform a combination of one or more of the other steps of the surgical assistance system as described above. Reference is made in particular to the foregoing and is not repeated here. Alternatively, the surgical assistance method provided in this embodiment may be assembled into a program and integrated into a readable storage medium, which may be attached to a surgical robot system, for example, a control device of the surgical robot system, or may be provided separately. Similarly, the modules of the surgical assistance system provided in this embodiment may be attached to the surgical robot system, for example, integrated in a control device of the surgical robot system, or may be independently provided, which is not limited in this regard. Further, the embodiment of the invention also provides a surgical robot system, which comprises the surgical auxiliary system. The construction and principles of the other components of the surgical robotic system may be referenced to the prior art and the present invention will not be described further.

In summary, in the surgical assistance system and the surgical assistance method, the surgical assistance system comprises an image large model module, a visual large model module, a risk detection large model module and a worker dialogue large model module, wherein the image large model module is used for dividing and reconstructing preoperative images based on the image large model and outputting division result information of the preoperative images, the visual large model module is used for analyzing endoscope scene content in an operation and outputting endoscope content information based on the visual large model, the risk detection large model module is used for analyzing and judging based on the division result information and the endoscope content information and outputting risk prompt information, and the worker dialogue large model module is used for outputting the proposal information according to the division result information, the endoscope content information and the risk prompt information. So configured, based on the settings of the image large model module, preoperative images can be loaded and segmented for reconstruction, and potential surgical risks can be analyzed in advance. Based on the setting of the vision large model module, real-time images of the endoscope can be analyzed. Based on the setting of the risk detection large model module, the segmentation result of the preoperative image and the real-time image of the endoscope can be fused, analysis and judgment are carried out, and risk prompt information is output. Based on the setting of the large model module of the operator dialogue language, the method can comprehensively evaluate and give suggestions to the operator based on the segmentation result information, the endoscope content information and the risk prompt information so as to more intuitively assist the operator in reducing the risk in the operation.

It should be noted that the above embodiments may be combined with each other. The above description is only illustrative of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention, and any alterations and modifications made by those skilled in the art based on the above disclosure shall fall within the scope of the present invention.

Claims

1. A surgical assistance system, characterized by comprising: an image big model module, a visual big model module, a risk detection big model module and a surgeon dialogue language big model module;

The image large model module segments and reconstructs the preoperative image based on the image large model, and outputs the segmentation result information of the preoperative image;

The visual big model module analyzes the endoscope scene content during the operation based on the visual big model and outputs the endoscope content information;

The risk detection large model module performs analysis and judgment based on the segmentation result information and the endoscope content information, and outputs risk warning information;

The operator dialogue language large model module outputs suggestion information according to the segmentation result information, the endoscope content information and the risk warning information;

The image large model module is configured to segment and reconstruct the preoperative image through a first convolutional neural network; the input information of the first convolutional neural network includes the preoperative image and the medical record information processed by the word segmenter;

The visual large model module is configured to analyze the intraoperative endoscopic scene content through a second convolutional neural network; the input information of the second convolutional neural network includes the intraoperative endoscopic image and the input information processed by the word segmenter;

The risk detection large model module is configured to perform analysis and judgment through a third convolutional neural network; the input information of the third convolutional neural network includes the segmentation result information and the endoscope content information processed by the word segmenter;

The operator dialogue language large model module is configured to perform analysis and judgment through a fourth convolutional neural network; the input information of the fourth convolutional neural network includes the segmentation result information, the endoscope content information processed by the word segmenter, and the risk warning information processed by the word segmenter.

2. The surgical assistance system according to claim 1 is characterized in that the operator dialogue language large model module includes an interaction unit, and the operator dialogue language large model module is also configured to obtain input information through the interaction unit; at least one of the visual large model module, the risk detection large model module and the operator dialogue language large model module corrects its own output information based on the input information.

3. The surgical assistance system according to claim 2 is characterized in that the surgeon dialogue language large model module also includes a semantic analysis unit; the semantic analysis unit is used to analyze the input information obtained by the interaction unit and correct the input information.

4. The surgical assistance system according to claim 1 is characterized in that the surgical assistance system also includes a data storage module; the data storage module is configured to store the segmentation result information, the recommendation information and the endoscope content information.

5. The surgical assistance system according to claim 4 is characterized in that the segmentation result information is image data, the recommendation information and the endoscopic content information are text data, and the data storage module is configured to combine the segmentation result information, the recommendation information and the endoscopic content information with a timestamp to form a multimodal data structure for storage.

6. The surgical assistance system according to claim 5 is characterized in that the surgical assistance system also includes a postoperative evaluation module; the postoperative evaluation module evaluates the surgery based on the multimodal data structure stored in the data storage module and outputs the evaluation results.

7. The surgical assistance system according to claim 6 is characterized in that the postoperative evaluation module includes a splitting unit and an intraoperative task analysis unit, the splitting unit is used to split the multimodal data structure obtained from the data storage module, and send the split data to the intraoperative task analysis unit; the intraoperative task analysis unit evaluates the operation based on the data split by the splitting unit, and outputs the evaluation result.

8. The surgical assistance system according to claim 1, characterized in that the surgical assistance system further comprises an endoscope calibration module; the endoscope calibration module is configured to perform the following steps to calibrate the endoscope before surgery:

photographing a three-dimensional calibration object to obtain a first image;

Performing rough calibration on the endoscope according to the first image and known size information of the three-dimensional calibration object;

Based on the rough calibration results, the calibration error is calculated and the position and angle of the area to be optimized are output;

Arranging a two-dimensional calibration object according to the position and angle of the area to be optimized, and photographing the two-dimensional calibration object to obtain a second image;

The endoscope is precisely calibrated according to the second image and the known size information of the two-dimensional calibration object in combination with the feature information of the three-dimensional calibration object in the rough calibration step.

9. The surgical assistance system according to claim 8, wherein the step of roughly calibrating the endoscope comprises:

Extracting first feature points in the first image, and obtaining two-dimensional image coordinate information corresponding to each of the first feature points in the first image;

Acquire world coordinate information of each of the first feature points according to the size information of the three-dimensional calibration object;

The two-dimensional image coordinate information and the world coordinate information of each of the first feature points are matched, a set of equations including the endoscope parameters to be solved is constructed, and the set of equations is solved to obtain a rough calibration result.

10. The surgical assistance system according to claim 9, wherein the feature information of the three-dimensional calibration object in the rough calibration step includes the two-dimensional image coordinate information and world coordinate information of the first feature point; and the step of finely calibrating the endoscope comprises:

Extracting second feature points in the second image, identifying a central feature point and a two-dimensional calibration object coordinate system, and obtaining world coordinate information and two-dimensional image coordinate information of each second feature point based on size information of the two-dimensional calibration object;

Combine the two-dimensional image coordinate information and world coordinate information of the first feature point, and the world coordinate information and two-dimensional image coordinate information of the second feature point, and jointly construct a group of equations containing the endoscope parameters to be solved, solve the group of equations, and obtain a precise calibration result.

11. The surgical assistance system according to claim 9, wherein the step of calculating the calibration error based on the rough calibration result comprises:

Based on the roughly calibrated endoscope parameters, mapping the world coordinate information of the first feature point calculated in the rough calibration step to a two-dimensional image coordinate system to obtain virtual two-dimensional image coordinate information of the first feature point;

The Euclidean distance between the virtual two-dimensional image coordinate information and the two-dimensional image coordinate information of the first feature point in the first image is calculated, and the Euclidean distance is determined as a calibration error.

12. The surgical assistance system according to claim 11, wherein the step of outputting the position and angle of the area to be optimized comprises:

Corresponding to the planes of multiple different tilt directions of the three-dimensional calibration object, the calibration errors of different segmented areas in the first image are calculated respectively, and according to a preset threshold, the segmented area whose calibration error exceeds the threshold is selected as the area to be optimized, and the tilt angle of the corresponding plane of the three-dimensional calibration object is determined as the angle of the area to be optimized.

13. The surgical assistance system according to claim 8, characterized in that the surgical assistance system further comprises: an endoscope fixing device, a calibration object fixing device, an object distance adjustment device, a three-dimensional calibration object and a two-dimensional calibration object;

The endoscope fixture is used for installing the endoscope; the relative positions of the endoscope fixture and the calibration object fixture are adjustably set by the object distance adjustment device; the three-dimensional calibration object and the two-dimensional calibration object are configured to be detachably mounted on the calibration object fixture.

14. The surgical assistance system according to claim 13 is characterized in that the three-dimensional calibration object includes multiple repeated structures, each of which includes multiple planes with different inclination directions; and/or the two-dimensional calibration object includes a plane, which includes multiple first characteristic symbols and multiple second characteristic symbols.

15. A surgical robot system, comprising: a surgical assistance system according to any one of claims 1 to 14.