CN118236166B - Automatic tracking system and method for surgical instruments - Google Patents

Abstract

The present invention relates to an automatic tracking system and method for surgical instruments. The system comprises: a robotic arm; a visual sensing unit, arranged at the end of the robotic arm, for acquiring image data; a host computer, connected to the visual sensing unit, for analyzing the image data based on a neural network model, obtaining the position information of the surgical instrument and the position information of the biological cavity tissue, and obtaining the movement information of the robotic arm according to the position information of the surgical instrument and the position information of the biological cavity tissue; a control unit, connected to the host computer and the robotic arm, for controlling the movement direction of the robotic arm according to the movement information of the robotic arm. The present invention can automatically track surgical instruments, replacing manual operation of the laparoscope, with more precise operation and lower cost than a surgical robot.

Description

Automatic tracking system and method for surgical instruments

Technical Field

The present invention relates to the field of intelligent control technology, and in particular to an automatic tracking system and method for surgical instruments.

Background Art

Laparoscopic surgery is a type of minimally invasive surgery. Through a keyhole-sized incision, surgeons can insert a laparoscope and surgical instruments. Surgeons use images transmitted to the monitor to guide the operation of surgical instruments to perform the operation. Compared with traditional surgery, laparoscopic surgery has smaller incisions, less damage, less postoperative pain, and faster recovery. It has become a commonly used surgical method in clinical practice.

During the operation, surgeons need to guide the laparoscope for a long time to take pictures of the biological tissues in the cavity and the position of surgical instruments, and transmit them to the monitor. Due to the instability of human hand movements and long-term uninterrupted operation, the movement of the laparoscope may cause collision with the biological tissues in the cavity, resulting in contamination of the laparoscope and damage to the biological tissues in the cavity.

In recent years, with the cross-development of robotics and medical science, surgical robots have been increasingly widely used. Compared with traditional manual operations, the movement and observation of surgical robots are highly accurate and reliable. However, the cost of existing surgical robots is much higher than that of traditional surgery.

Summary of the invention

Based on this, an object of the present invention is to provide an automatic tracking system and method for surgical instruments.

In a first aspect, the present invention provides an automatic tracking system for a surgical instrument, comprising:

Robotic arm;

A visual sensing unit, disposed at the end of the robotic arm, for acquiring image data;

A host computer connected to the visual sensing unit, configured to analyze the image data based on a neural network model to obtain position information of the surgical instrument and position information of the biological cavity tissue, and obtain movement information of the robotic arm according to the position information of the surgical instrument and the position information of the biological cavity tissue;

A control unit is connected to the host computer and the robotic arm, and is used to control the movement direction of the robotic arm according to the movement information of the robotic arm.

Compared with the existing technology, the present invention analyzes image data through a neural network model to obtain the position information of surgical instruments and biological cavity tissue information, and then obtains the movement information of the robotic arm, so that it can automatically track surgical instruments and replace manual operation of the laparoscope. The operation is more precise and the cost is lower than that of a surgical robot.

In one of the embodiments, the visual sensing unit includes an image sensor, and the image sensor is a monocular or binocular camera;

The host computer is used to perform target detection on the image data based on a neural network model to identify the surgical instrument and the biological cavity tissue; obtain first distance information between the surgical instrument and the image sensor, and second distance information between the biological cavity tissue and the image sensor through a video ranging algorithm based on pixel information of the surgical instrument and the biological cavity tissue; and obtain movement information of the robotic arm based on the first distance information and the second distance information.

In one embodiment, the visual sensing unit comprises:

An image sensor, used for acquiring image data;

A fill light element is used to provide fill light to the image sensor.

In one of the embodiments, the host computer is used to divide the image data into a fill light image and a position information image, and control the fill light element to perform fill light according to the fill light image, and obtain the movement information of the robot arm according to the position information image.

In one of the embodiments, the host computer is used to determine whether the brightness of the fill light image is lower than a preset threshold; if so, control the fill light element to perform fill light.

In a second aspect, the present invention provides an automatic tracking method for a surgical instrument, comprising the steps of:

S10: Acquire image data transmitted by the image sensor;

S20: Analyze the image data based on a neural network model to obtain position information of the surgical instrument and position information of the biological cavity tissue;

S30: obtaining movement information of the robotic arm according to the position information of the surgical instrument and the position information of the biological cavity tissue;

S40: Sending the movement information of the robot arm to a control unit.

In one embodiment, step S20 includes:

S21: dividing the image data into a fill light image and a position information image;

S22A: Determine whether the brightness of the fill-light image is lower than a preset threshold; if so, control the fill-light element of the visual sensing unit to perform fill-light;

S22B: performing target retrieval on the position information image based on a neural network model to identify surgical instruments and biological cavity tissues;

S23: According to the pixel information of the surgical instrument and the biological cavity tissue, obtain the first distance information between the surgical instrument and the image sensor, and the second distance information between the biological cavity tissue and the image sensor through a video ranging algorithm.

In a third aspect, the present invention provides an automatic tracking device for a surgical instrument, comprising:

An image data acquisition module, used to acquire image data transmitted by an image sensor;

A position information analysis module, used to analyze the image data based on a neural network model to obtain position information of surgical instruments and position information of biological cavity tissue;

A movement information acquisition module, used to obtain movement information of the robot arm according to the position information of the surgical instrument and the position information of the biological cavity tissue;

The movement information sending module is used to send the movement information of the mechanical arm to the control unit.

In a fourth aspect, the present invention provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the aforementioned method when executing the computer program.

In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the conventional technology, the drawings required for use in the embodiments or the conventional technology descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a schematic diagram of the structure of an automatic tracking system for surgical instruments according to an embodiment of the present invention;

FIG2 is a flow chart of an automatic tracking method for a surgical instrument according to an embodiment of the present invention;

FIG. 3 is a schematic structural diagram of an automatic tracking device for a surgical instrument according to an embodiment of the present invention.

DETAILED DESCRIPTION

In order to facilitate understanding of the present invention, the present application will be described more fully below with reference to the relevant drawings. Embodiments of the present invention are given in the drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

It should be noted that when an element is considered to be "connected" to another element, it can be directly connected to the other element, or connected to the other element through an intermediate element. In addition, the "connection" in the following embodiments should be understood as "electrical connection", "communication connection", etc. if there is transmission of electrical signals or data between the connected objects.

In view of the instability of existing manually operated laparoscopes and the high cost of surgical robots, the present invention provides an automatic tracking system and method for surgical instruments. The host computer analyzes image data through a neural network model to obtain the position information of the surgical instrument and the biological cavity tissue information, and then obtains the movement information of the robotic arm, so as to automatically track the surgical instrument and replace the manual operation of the laparoscope.

1 , the automatic tracking system of the surgical instrument of the present invention comprises a robotic arm 10, a stepping motor 20, a visual sensor unit 30, a host computer 40, a control unit 50 and a display unit 60. The stepping motor 20 and the visual sensor unit 30 are arranged on the robotic arm 10, and the visual sensor unit 30 is located at the end of the robotic arm 10.

Specifically, the robot arm 10 is a six-axis robot arm with six degrees of freedom, and can make almost any trajectory movement. The stepper motor 20 can provide six degrees of freedom, and the rotation accuracy reaches 0.007°, which can ensure the stability of the tracking shooting picture and speed.

The visual sensing unit 30 includes an image sensor 31 and a fill light element 32. The image sensor 31 is used to acquire image data, and the fill light element 32 is used to fill light for the image sensor.

Specifically, the image sensor 31 adopts an imaging lens with an aspherical design and a pixel of 64 million, so that the image resolution of the object is high, the image is complete and clear, and the field of view is wide. In the embodiment of the present invention, the image sensor 31 is a monocular or binocular camera, and the distance information of the object can be obtained through the image information. The fill light element 32 includes six high-lumen LED lights, which have good adaptability to different environments and can restore the image color to reality.

The host computer 40 is connected to the visual sensing unit 30, and is used to analyze the image data based on the neural network model to obtain the position information of the surgical instrument and the position information of the biological cavity tissue, and obtain the movement information of the robot arm 10 according to the position information of the surgical instrument and the position information of the biological cavity tissue. Specifically, the host computer 40 divides the image data into a fill light image and a position information image, determines whether the brightness of the fill light image is lower than a threshold, and obtains the position information of the surgical instrument and the position information of the biological cavity tissue according to the position information image.

In the embodiment of the present invention, the image sensor 31 acquires a video image of one second, and the host computer 40 uses the image data of the first half second as the fill light image, and the image data of the other half second as the position information image. For the fill light image, the host computer 40 uses the grayscale histogram for quantitative analysis to obtain the brightness value of the fill light image. When the brightness value of the fill light image is lower than the preset threshold, the host computer 40 sends a variable command to control the fill light element 32 to perform fill light.

For the position information image, after performing noise reduction processing, the host computer 40 performs target detection through a neural network model to identify the surgical instrument and the biological cavity tissue; based on the pixel information of the surgical instrument and the biological cavity tissue, the first distance information between the surgical instrument and the image sensor 31 and the second distance information between the biological cavity tissue and the image sensor 31 are obtained through a video ranging algorithm; based on the first distance information and the second distance information, the movement information of the robotic arm 10 is obtained.

Specifically, a coordinate system is established based on the image sensor 31 as the origin to obtain the position point of the surgical instrument and the position point of the biological cavity tissue. In order to prevent the robot arm 10 from colliding with the biological cavity tissue, the present invention presets a distance threshold. The host computer 40 obtains the moving position point of the robot arm 10 based on this distance threshold, the position point of the surgical instrument and the position point of the biological cavity tissue, and sends this moving position point to the controller. At the same time, the host computer 40 can also send the image data to the display unit 60 for the doctor to view.

In an embodiment of the present invention, the neural network model adopts an improved DenseNet structure, which is composed of 124 layers of neural networks. In addition, the neural network model adopts the LReLy activation function, which is a variant of LReLU, which allows negative input values to have a small non-zero output, thereby solving the problem that LReLU will completely shut down the gradient under negative input values, so that the neural network model has better performance on different input data, especially when there is a slight negative input. It can still maintain a good activation state and avoid the problem of neuron death. Furthermore, in order to improve the accuracy of surgical instrument recognition, the present invention uses a training set to optimize the training of the neural network model. Among them, the training set is a collection of images including biological cavity tissues and surgical instruments during various laparoscopic surgeries.

The control unit 50 is connected to the host computer 40 and the stepper motor 20 , and is used to control the movement direction of the robot arm 10 according to the movement information of the robot arm 10 .

Specifically, the control unit 50 analyzes the moving position point of the robot arm 10, and calculates the angle that each stepper motor 20 needs to rotate through the kinematics of the six-axis robot arm, and then sends the corresponding angle to each stepper motor 20, and uses vector closed-loop control to accurately and stably move the robot arm 10 to the moving position point sent by the host computer 10.

Compared with the prior art, the present invention analyzes the image data through a neural network model to obtain the position information of the surgical instrument and the biological cavity tissue information, and then obtains the movement information of the robotic arm, so that the surgical instrument can be automatically tracked, replacing the manual operation of the laparoscope, and the operation is more precise, and the cost is lower than that of the surgical robot. In addition, the present invention is also provided with a fill light element, which controls the fill light element to perform fill light according to the image data, thereby improving the recognition ability of the neural network model. Moreover, the present invention simultaneously performs brightness analysis and target detection on the image data, and adopts dual threads to dynamically adapt to the environment.

Based on the same inventive concept, an embodiment of the present invention also provides an automatic tracking method for surgical instruments. The implementation solution provided by the method is similar to the implementation solution recorded in the above system, so the specific definition of the automatic tracking method for surgical instruments provided below can refer to the definition of the automatic tracking system for surgical instruments above, and will not be repeated here.

Please refer to FIG. 2 , the automatic tracking method of the surgical instrument of the present invention is applied to the host computer 40 in FIG. 1 , and includes the following steps:

S10: Acquire image data transmitted by the image sensor 31 .

S21: dividing the image data into a fill-in light image and a position information image.

S22A: Determine whether the brightness of the fill-light image is lower than a preset threshold; if so, control the fill-light element of the visual sensing unit to perform fill-light.

S22B: Perform target detection on the position information image based on a neural network model to identify surgical instruments and biological cavity tissues.

S23: According to the pixel information of the surgical instrument and the biological cavity tissue, the first distance information between the surgical instrument and the image sensor 31 and the second distance information between the biological cavity tissue and the image sensor 31 are obtained by a video ranging algorithm.

S30: Obtain movement information of the robot arm 10 according to the first distance information and the second distance information.

S40 : Sending the movement information of the robot arm 10 to the control unit 50 .

Based on the same inventive concept, an embodiment of the present invention further provides an automatic tracking device for a surgical instrument for implementing the automatic tracking method of a surgical instrument mentioned above. Referring to FIG3 , the automatic tracking device for a surgical instrument of the present invention comprises an image data acquisition module, a position information analysis module, a movement information acquisition module and a movement information sending module, wherein:

An image data acquisition module, used to acquire image data transmitted by an image sensor;

A position information analysis module, used to analyze the image data based on a neural network model to obtain position information of surgical instruments and position information of biological cavity tissue;

A movement information acquisition module, used to obtain movement information of the robot arm according to the position information of the surgical instrument and the position information of the biological cavity tissue;

The movement information sending module is used to send the movement information of the robot arm to the control unit.

Based on the same inventive concept, an embodiment of the present invention also provides a computer device, which may be a terminal device such as a server, a desktop computing device or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet computer, a netbook, etc.). The device includes one or more processors and a memory, wherein the processor is used to execute the automatic tracking method of the surgical instrument of the program implementation method embodiment; and the memory is used to store a computer program executable by the processor.

Based on the same inventive concept, an embodiment of the present invention also provides a computer storage medium, corresponding to the embodiment of the aforementioned automatic tracking method of surgical instruments, wherein the computer storage medium stores a computer program thereon, and when the program is executed by a processor, the steps of the automatic tracking method of surgical instruments recorded in any of the aforementioned embodiments are implemented.

The present application may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims (4)

1. An automatic tracking system for surgical instruments, comprising: Robotic arm; A visual sensing unit is arranged at the end of the robotic arm; the visual sensing unit comprises an image sensor and a fill light element, the image sensor is a monocular or binocular camera for acquiring image data, and the fill light element is used to fill light for the image sensor; A host computer connected to the visual sensing unit, for performing target detection on the image data based on a neural network model, and identifying surgical instruments and biological cavity tissues; obtaining first distance information between the surgical instrument and the image sensor, and second distance information between the biological cavity tissue and the image sensor through a video ranging algorithm according to pixel information of the surgical instrument and the biological cavity tissue; obtaining movement information of the robotic arm according to the first distance information, the second distance information, and a preset distance threshold between the robotic arm and the biological cavity tissue; A control unit is connected to the host computer and the robotic arm, and is used to control the movement direction of the robotic arm according to the movement information of the robotic arm. 2. The automatic tracking system for surgical instruments according to claim 1 is characterized in that the host computer is used to divide the image data into a fill light image and a position information image, and control the fill light element to perform fill light according to the fill light image, and obtain the movement information of the robotic arm according to the position information image. 3. The automatic tracking system for surgical instruments according to claim 2 is characterized in that the host computer is used to determine whether the brightness of the fill light image is lower than a preset threshold; if so, control the fill light element to perform fill light. 4. An automatic tracking device for a surgical instrument, comprising: An image data acquisition module, used to acquire image data transmitted by an image sensor; a position information analysis module, configured to perform target detection on the image data based on a neural network model, and identify surgical instruments and biological cavity tissues; and obtain first distance information between the surgical instrument and the image sensor, and second distance information between the biological cavity tissue and the image sensor through a video ranging algorithm according to pixel information of the surgical instrument and the biological cavity tissue; A movement information acquisition module, used for obtaining movement information of the robotic arm according to the first distance information, the second distance information and a preset distance threshold between the robotic arm and the biological cavity tissue; wherein the visual sensing unit is arranged at the end of the robotic arm, the visual sensing unit comprises the image sensor and a fill light element, the image sensor is a monocular or binocular camera, used for obtaining image data, and the fill light element is used for fill light to the image sensor; A movement information sending module, used for sending the movement information of the mechanical arm to a control unit; The position information analysis module is further used to control the fill light element to perform fill light according to the image data.