CN118542635B - Endoscope control system based on eyeball movement recognition - Google Patents
Endoscope control system based on eyeball movement recognition Download PDFInfo
- Publication number
- CN118542635B CN118542635B CN202410995207.0A CN202410995207A CN118542635B CN 118542635 B CN118542635 B CN 118542635B CN 202410995207 A CN202410995207 A CN 202410995207A CN 118542635 B CN118542635 B CN 118542635B
- Authority
- CN
- China
- Prior art keywords
- endoscope
- eyeball
- movement
- mode
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/00002—Operational features of endoscopes
- A61B1/00004—Operational features of endoscopes characterised by electronic signal processing
- A61B1/00006—Operational features of endoscopes characterised by electronic signal processing of control signals
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/04—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor combined with photographic or television appliances
- A61B1/045—Control thereof
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B1/00—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor
- A61B1/06—Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes; Illuminating arrangements therefor with illuminating arrangements
- A61B1/0655—Control therefor
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/013—Eye tracking input arrangements
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/18—Eye characteristics, e.g. of the iris
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H40/00—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices
- G16H40/60—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices
- G16H40/63—ICT specially adapted for the management or administration of healthcare resources or facilities; ICT specially adapted for the management or operation of medical equipment or devices for the operation of medical equipment or devices for local operation
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Surgery (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Medical Informatics (AREA)
- Theoretical Computer Science (AREA)
- Public Health (AREA)
- Heart & Thoracic Surgery (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Veterinary Medicine (AREA)
- Animal Behavior & Ethology (AREA)
- Pathology (AREA)
- Human Computer Interaction (AREA)
- Radiology & Medical Imaging (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Optics & Photonics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Ophthalmology & Optometry (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Primary Health Care (AREA)
- Epidemiology (AREA)
- General Business, Economics & Management (AREA)
- Business, Economics & Management (AREA)
- Databases & Information Systems (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Artificial Intelligence (AREA)
- Signal Processing (AREA)
- Endoscopes (AREA)
Abstract
The invention discloses an endoscope control system based on eyeball movement recognition, wherein a multidimensional eyeball movement recognition unit is arranged in a head-mounted display device, and the head-mounted display device is provided with an infrared camera and is used for capturing multidimensional movement data of eyeballs of a user in real time, and the multidimensional eyeball movement data comprises up-and-down movement, left-and-right movement and forward-and-backward rolling; the endoscope space positioning and control unit is electrically connected with the multi-dimensional eyeball motion recognition unit and receives eyeball motion data from the recognition unit; the direct eye movement control of the endoscope function is realized by integrating the double infrared camera system, the image processing and the LED infrared light source with the multipoint layout in the head-mounted display device, and the position, the angle and the focal length of the endoscope are adjusted by continuously monitoring and analyzing the eyeball movement of an operator, so that the control method adapts to the instant requirement in the operation, and not only improves the accuracy and the response speed of the operation, but also obviously improves the convenience of the operation and the overall efficiency of the operation.
Description
Technical Field
The invention relates to the technical field of endoscope control, in particular to an endoscope control system based on eyeball movement identification.
Background
In modern medical practice, endoscopes are widely used in a variety of minimally invasive surgical and diagnostic procedures. The manual operation of the endoscope requires a doctor to adjust the position and focal length of the endoscope while performing the operation, and the control method is complex in operation, and is easy to cause fatigue of an operator in the long-time operation process, and due to the fatigue and limitation of the manual operation, the stability and accuracy of the endoscope are difficult to maintain by using the conventional control method.
Based on this, there is a need for an endoscope control system based on eye movement recognition.
Disclosure of Invention
The invention aims to provide an endoscope control system based on eyeball movement identification, which is used for realizing direct eye movement control of an endoscope function by integrating a double-infrared camera system, image processing and a multi-point layout LED infrared light source in a head-mounted display device, and adjusting the position, angle and focal length of an endoscope by continuously monitoring and analyzing the eyeball movement of an operator so as to adapt to the instant requirement in an operation.
In order to achieve the above object, the present invention provides the following solutions:
an endoscope control system based on eyeball motion recognition comprises a multidimensional eyeball motion recognition unit, an endoscope space positioning and control unit and an operation mode intelligent selection unit;
The multidimensional eyeball movement identification unit, the endoscope space positioning and control unit and the operation mode intelligent selection unit are interconnected through an internal data processing network to realize the instant transmission and processing of data and commands;
The multi-dimensional eyeball movement recognition unit is arranged in the head-mounted display device, and the head-mounted display device is provided with an infrared camera and is used for capturing multi-dimensional movement data of eyeballs of a user in real time, wherein the multi-dimensional movement data comprise up-down movement, left-right movement and front-back scrolling;
the infrared camera and the image processing module work in a combined mode and are used for identifying a specific eyeball movement direction from the captured eyeball image;
The endoscope space positioning and controlling unit is electrically connected with the multidimensional eyeball motion recognition unit, receives eyeball motion data from the recognition unit, and adjusts the space position and focal length of the endoscope according to the received data so as to match the sight and the operation intention of a user;
The control unit controls the multi-dimensional position adjustment and focal length change of the endoscope;
The intelligent operation mode selection unit is provided with a mode identification module which is used for identifying the operation intention sent by a user through a specific eyeball movement mode and switching the observation mode, the treatment mode and the working mode of the illumination adjustment mode of the endoscope according to the identification result;
In this embodiment, the multidimensional eye movement recognition unit further includes an LED infrared light source for enhancing the accuracy of eye movement capturing, the LED infrared light source is configured at the inner side of the head-mounted display device and is aligned to the eye region of the user, the LED infrared light source adopts a multi-point layout manner, and surrounds the eye region of the user, so as to uniformly irradiate the eyeball and eliminate the influence caused by the change of ambient light, so as to improve the stability and accuracy of eye tracking, the wavelength of the LED infrared light source is set to 850 nm, and the wavelength is safe to the eye and has minimal interference under most ambient light conditions;
The method comprises the steps of providing two infrared cameras, wherein the high-precision infrared cameras have a frame rate of 1000 FPS and a resolution of at least 1920x1080 pixels, and are arranged at a specific angle to realize cross capture of eyeball movement, and the capture precision of the movement is enhanced through stereoscopic vision, so that each tiny movement of the eyeball can be accurately recorded;
the captured eye movement image is analyzed and parsed in real time by an image processing module that uses a Convolutional Neural Network (CNN) for deep learning to accurately identify and track eye movement from a complex background.
In this embodiment, the endoscope spatial positioning and control unit includes a linear driver for realizing linear displacement of the endoscope, the driving precision of which reaches ±0.01 mm, and a rotation mechanism for adjusting the rotation angle of the endoscope according to the received eyeball movement data, the rotation precision being ±0.1 degrees;
Wherein the linear driver and the rotating mechanism adopt a PID (proportion-integral-derivative) control algorithm for fine adjustment response, and the parameters of the PID controller are dynamically adjusted according to the speed and the acceleration of eyeball movement so as to adapt to rapid changes in operation, in particular: by calculation formula WhereinIs the coefficient of the rotational response and,Is the compensation coefficient of the compensation coefficient,Is the rotation angle data of the eyeball,The change rate of the eyeball rotation angle is the change rate, so that the ultra-precise control of the endoscope is realized;
further comprising automatic error correction, automatically adjusting parameters of the PID controller by continuously monitoring the difference between the actual and the expected position of the endoscope, to minimize operational errors, ensure surgical accuracy, and in particular to achieve accurate control of the endoscope by:
continuously monitoring the difference between the actual position and the expected position of the endoscope to obtain a position error value;
Judging whether the position error value exceeds a first preset error threshold (+ -0.02 mm) to obtain a first judgment result;
If the first judgment result is yes, increasing the adjustment amplitude of the PID controller so as to quickly reduce the position error;
if the first judgment result is negative, further judging whether the position error value is smaller than or equal to a second preset error threshold (+ -0.01 mm);
If yes, maintaining the current parameter setting of the PID controller, and maintaining a fine tuning state to gradually optimize the position accuracy;
if not, reducing the adjustment amplitude of the PID controller;
In this embodiment, the operation mode intelligent selection unit is provided with an acoustic feedback module, and when it is recognized that the user switches operation modes through eye movements, an audio signal is sent out to remind the operator that the modes have been switched.
In this embodiment, the head-mounted display device displays the current view image of the endoscope in real time; the display unit employs an OLED screen with a resolution of at least 2K, providing a refresh rate of at least 120Hz to reduce visual delay and blur.
In this embodiment, the system further includes an ambient light sensing unit for detecting the light intensity of the operating environment and automatically adjusting the brightness of the head-mounted display device and the LED infrared light source, so as to ensure that the accuracy of eye tracking can be maintained under any illumination condition.
In this embodiment, the image processing module includes a function of performing color enhancement processing on the captured eyeball image to improve the recognition rate of the eyeball characteristics, especially in a low-light environment.
In this embodiment, the system is configured with a wireless data transmission unit for implementing data communication between the module units; the wireless transmission unit supports at least Wi-Fi and bluetooth technologies.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
The invention provides an endoscope control system based on eyeball movement identification, which is characterized in that a double-infrared camera system, image processing and LED infrared light sources with multi-point layout are integrated in a head-mounted display device to realize direct eye movement control of an endoscope function, and the position, angle and focal length of the endoscope are adjusted by continuously monitoring and analyzing the eyeball movement of an operator to adapt to the instant requirement in operation.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flow chart of a control method of adaptive infrared imaging detection according to embodiment 1 of the present invention;
FIG. 2 is a schematic diagram showing the light intensity and automatic adjustment determination of the environment according to embodiment 1 of the present invention;
Fig. 3 is a schematic diagram of intelligent selection and judgment of an operation mode according to embodiment 1 of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to provide an endoscope control system based on eyeball movement identification, which is used for realizing direct eye movement control of an endoscope function by integrating a double-infrared camera system, image processing and a multi-point layout LED infrared light source in a head-mounted display device, and adjusting the position, angle and focal length of an endoscope by continuously monitoring and analyzing the eyeball movement of an operator so as to adapt to the instant requirement in an operation.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
Examples
As shown in fig. 1, an endoscope control system based on eye movement recognition in the present embodiment includes a multidimensional eye movement recognition unit, an endoscope space positioning and control unit, and an operation mode intelligent selection unit;
The multidimensional eyeball motion recognition unit, the endoscope space positioning and control unit and the operation mode intelligent selection unit are interconnected through an internal data processing network to realize the instant transmission and processing of data and commands, the network adopts an Ethernet-based real-time communication protocol (EtherCAT or PROFINET) which is specially designed for high-speed and low-delay data exchange, the protocol supports high-speed data transmission, the delay is as low as a few microseconds, and the instant reaction and synchronization among the units of the system are ensured;
The multi-dimensional eyeball movement recognition unit is arranged in the head-mounted display device, and the head-mounted display device is provided with an infrared camera and is used for capturing multi-dimensional movement data of eyeballs of a user in real time, wherein the multi-dimensional movement data comprise up-down movement, left-right movement and front-back scrolling; specifically, the vertical movement of eyeballs is captured through double infrared cameras, each infrared camera is provided with image processing, the movement speed and angle of the eyeballs can be calculated in real time, data are converted into control signals, the tilting motor of the endoscope is accurately regulated, a micro stepping technology is used for motor control, extremely fine tilting angle adjustment is allowed, the tilting speed is dynamically regulated according to the movement speed of the eyeballs, smoothness and accuracy of visual field adjustment are guaranteed, the movement of the eyeballs from side to side is captured in real time by the high-precision infrared cameras, specific movement parameters are obtained through image processing software analysis, the parameters are transmitted to a rotation control unit through a data link, the rotation control unit controls one or more high-precision rotation servo motors, the orientation of the endoscope is regulated according to specific instructions of the movement of the eyeballs, the response time of the rotation commands is not more than millisecond level, the real-time of operation is guaranteed, the specific movement pattern of the endoscope is identified through rapid pupil fluctuation detection of the eyeballs, the movement of the endoscope is analyzed to advance and retreat control commands of the endoscope, the control system adjusts the endoscope by using an accurate linear driver, the adjustment of the endoscope is based on the high-precision track system, stable movement path setting is guaranteed, and physical interference in surgery is reduced;
The multi-dimensional eye movement recognition unit further includes an LED infrared light source for enhancing eye movement capturing accuracy, the LED infrared light source being disposed at an inner side of the head-mounted display device to be directed to an eye region of a user, the LED infrared light source being disposed in a multi-point layout manner to surround the eye region of the user, to uniformly illuminate the eye and eliminate influence due to variation of ambient light, to enhance stability and accuracy of eye tracking, the wavelength of the LED infrared light source being set to 850 nm, the LED infrared light source having a wavelength of 850 nm being selected based on consideration of safety to eyes and minimization of interference of ambient light, light having the wavelength being hardly visible to human eyes, thus not causing discomfort of vision or interference with normal vision of the user, meanwhile, under various ambient light conditions, the interference of 850 and nm wavelength infrared light on the existing illumination is extremely small, the stability of a system is enhanced, the LED infrared light sources on the inner side of the head-mounted display device are distributed in multiple points and encircle the whole eye area, the distribution mode can uniformly irradiate the eyeballs of a user from all angles, the sufficient infrared light irradiation can be obtained no matter how the eyes of the user move, dead angles and blind areas are reduced, each LED infrared light source is provided with a micro lens, light rays can be precisely oriented to specific parts of the eyeballs, the orientation is beneficial to improving the service efficiency of the light sources, light scattering and loss are reduced, and the light rays are ensured to directly act on key areas required by eyeball tracking;
As shown in fig. 2, the system further comprises an ambient light sensing unit for detecting the light intensity of the operation environment and automatically adjusting the brightness of the head-mounted display device and the LED infrared light source, so as to ensure that the accuracy of eye tracking can be maintained under any illumination condition; ambient light sensing unit use WhereinIs the intensity of the ambient light,Is an adjustment coefficient;
automatic adjustment logic for automatically adjusting brightness of the head-mounted display device and the LED infrared light source by the ambient light sensing unit:
initial judgment: judging whether the current ambient light intensity is lower than a first preset intensity (200 lux);
If the intensity is lower than the first preset intensity: the backlight brightness of a display screen in the head-mounted display device and the intensity of an LED infrared light source are increased, so that the accuracy of eyeball tracking and the visibility of the display screen in a dark environment are ensured;
if the intensity is not lower than the first preset intensity: then enter the second stage to judge;
And (3) second-stage judgment: continuing to judge whether the ambient light intensity is lower than a second preset intensity (500 lux);
If yes: keeping the current brightness settings of the head-mounted display device and the LED infrared light source unchanged, because the current brightness is sufficient to provide a good eye tracking and visual experience;
if no: the brightness of the LED infrared light source is reduced, so that discomfort of eyes or influence on the performance of the eyeball tracking sensor caused by the generation of excessively strong infrared light in a brighter environment is avoided;
Real-time data from the light sensor is continuously received, and the brightness of the LED infrared light source and the brightness of the display screen are adjusted, so that the eye tracking effect and the visual comfort of a user are optimized.
The method comprises the steps of providing two infrared cameras, wherein the high-precision infrared cameras have a frame rate of 1000 FPS and a resolution of at least 1920x1080 pixels, and are arranged at a specific angle to realize cross capture of eyeball movement, and the capture precision of the movement is enhanced through stereoscopic vision, so that each tiny movement of the eyeball can be accurately recorded;
the captured eye movement image is analyzed and parsed in real time by an image processing module that uses a Convolutional Neural Network (CNN) for deep learning to accurately identify and track eye movement from a complex background.
The infrared camera and the image processing module work in a combined mode and are used for identifying a specific eyeball movement direction from the captured eyeball image;
as shown in fig. 3, the operation mode intelligent selection unit is configured with a mode recognition module for recognizing an operation intention of a user through a specific eye movement mode and switching an observation mode, a treatment mode and a working mode of an illumination adjustment mode of the endoscope according to a recognition result;
The mode recognition module integrates advanced image processing technology and a deep learning framework, can recognize and distinguish different eye movement modes through training a large amount of eyeball movement data, quickly blinks, continuously gazes and quickly moves eyeballs, each action is associated with a specific operation mode, and the combination of a Convolutional Neural Network (CNN) and a cyclic neural network (RNN) is adopted to improve the processing effect on time series data, so that the quick continuous eye movement modes and the intention thereof can be recognized more accurately;
Fast blink is defined as closing the eyelid twice or more in a short time (1-2 seconds) and in the present system is configured as a switch trigger signal for activating or switching to a treatment mode of a specific surgical mode, capturing the fast closing and opening actions of the eyelid by the infrared camera, the frequency and number of these actions being analyzed by the image processing algorithm, and upon identifying a fast blink mode meeting preset conditions, initiating a mode switching protocol by the internal logic controller.
Continuous fixation refers to fixation of an eyeball in one direction or object for a long time, in a surgical mode, continuous fixation is used for activating an observation mode of an enlarged or specific view, and by analyzing the position stability of the eyeball and the duration of fixation in real time, the observation intention of a user can be judged, and for continuous fixation, the system can adjust the focal length or illumination intensity of an endoscope so as to optimize the view quality.
The fast movement of the eye, used in surgery to rapidly switch the viewing area, the action is used in the system to rapidly adjust the direction of the endoscope to follow the user's gaze movement, which tracks the dynamic path and speed of the eye in real time through the infrared camera, and when a rapid and obvious eye jump action is detected, instantly adjusts the position and angle of the endoscope to match the user's gaze direction.
Judging conditions of intelligent switching:
Initial detection stage
Detecting an eyeball movement pattern: continuously monitoring the dynamic state of the eyeball at a high frame rate through an infrared camera, and judging whether the eyeball meets the preset operation mode switching condition or not;
judging whether the eyeball movement is continuous twice rapid blinking or not to obtain a first judgment result;
if the first judgment result is yes, entering the next judgment stage;
if the first judgment result is negative, continuing monitoring.
Deep judging stage
Further confirming the handover requirement: based on the preliminary determination, the system further evaluates whether a mode switch is required.
Judging whether the current operation mode is a target mode or not;
if the target mode is already the target mode, the switching is not performed;
if not, the system is ready to switch modes.
Requesting the user to confirm: to prevent mishandling, the system requests the user to confirm the mode switch by displaying or voice prompts within the head mounted display device.
Sending a sound prompt to a user, inquiring whether to confirm to switch to the treatment mode;
if the user confirms through a specific eye movement (rapid blink upwards), performing mode switching;
if the user cancels by a specific eye movement (rapid blink down), no switching is performed, and returns to the normal monitoring state.
Performing mode switching
Once the user confirms the switch, the system immediately adjusts the endoscope settings, switching to the corresponding surgical mode;
according to the user confirmation, the system adjusts the focal length, illumination and position of the endoscope, ensures compliance with the requirements of the treatment mode, and displays the current mode as "treatment mode".
In this embodiment, the endoscope spatial positioning and control unit includes a linear driver for realizing linear displacement of the endoscope, the driving precision of which reaches ±0.01 mm, and a rotation mechanism for adjusting the rotation angle of the endoscope according to the received eyeball movement data, the rotation precision being ±0.1 degrees;
Wherein the linear driver and the rotating mechanism adopt a PID (proportion-integral-derivative) control algorithm for fine adjustment response, and the parameters of the PID controller are dynamically adjusted according to the speed and the acceleration of eyeball movement so as to adapt to rapid changes in operation, in particular: by calculation formula WhereinIs the coefficient of the rotational response and,Is the compensation coefficient of the compensation coefficient,Is the rotation angle data of the eyeball,The change rate of the eyeball rotation angle is the change rate, so that the ultra-precise control of the endoscope is realized;
further comprising automatic error correction, automatically adjusting parameters of the PID controller by continuously monitoring the difference between the actual and the expected position of the endoscope, to minimize operational errors, ensure surgical accuracy, and in particular to achieve accurate control of the endoscope by:
continuously monitoring the difference between the actual position and the expected position of the endoscope to obtain a position error value;
Judging whether the position error value exceeds a first preset error threshold (+ -0.02 mm) to obtain a first judgment result;
If the first judgment result is yes, increasing the adjustment amplitude of the PID controller so as to quickly reduce the position error;
if the first judgment result is negative, further judging whether the position error value is smaller than or equal to a second preset error threshold (+ -0.01 mm);
If yes, maintaining the current parameter setting of the PID controller, and maintaining a fine tuning state to gradually optimize the position accuracy;
if not, reducing the adjustment amplitude of the PID controller;
In this embodiment, the operation mode intelligent selection unit is provided with an acoustic feedback module, and when it is recognized that the user switches operation modes through eye movements, an audio signal is sent out to remind the operator that the modes have been switched.
In this embodiment, the head-mounted display device displays the current view image of the endoscope in real time; the display unit employs an OLED screen with a resolution of at least 2K, providing a refresh rate of at least 120Hz to reduce visual delay and blur.
In this embodiment, the image processing module includes a function of performing color enhancement processing on the captured eyeball image to improve the recognition rate of the eyeball characteristics, especially in a low-light environment.
In this embodiment, the system is configured with a wireless data transmission unit for implementing data communication between the module units; the wireless transmission unit supports at least Wi-Fi and bluetooth technologies.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.
Claims (1)
1. An endoscope control system based on eyeball movement identification is characterized in that: comprising
The device comprises a multidimensional eyeball motion recognition unit, an endoscope space positioning and control unit and an operation mode intelligent selection unit;
the multidimensional eyeball motion recognition unit, the endoscope space positioning and control unit and the operation mode intelligent selection unit are interconnected through an internal data processing network to realize the instant transmission and processing of data and commands;
The multi-dimensional eyeball motion recognition unit is arranged in a head-mounted display device, and the head-mounted display device is provided with an infrared camera and is used for capturing multi-dimensional motion data of the eyeballs of a user in real time, wherein the multi-dimensional motion data comprises up-down motion, left-right motion and front-back scrolling;
the infrared camera and the image processing module work in a combined mode and are used for identifying a specific eyeball movement direction from the captured eyeball image;
The endoscope space positioning and controlling unit is electrically connected with the multi-dimensional eyeball motion recognition unit, receives eyeball motion data from the recognition unit, and adjusts the space position and focal length of the endoscope according to the received data so as to match the sight and the operation intention of a user;
The control unit controls the multi-dimensional position adjustment and focal length change of the endoscope;
The intelligent operation mode selection unit is provided with a mode identification module, and is used for identifying the operation intention sent by a user through a specific eyeball movement mode and switching the observation mode, the treatment mode and the working mode of the illumination adjustment mode of the endoscope according to the identification result;
The multi-dimensional eye movement recognition unit further comprises an LED infrared light source for enhancing eye movement capturing accuracy, wherein the LED infrared light source is arranged on the inner side of the head-mounted display device and is aligned to the eye area of a user, the LED infrared light source surrounds the eye area of the user in a multi-point layout mode so as to uniformly irradiate the eyeballs and eliminate the influence caused by the change of ambient light, the stability and accuracy of eye tracking are improved, the wavelength of the LED infrared light source is set to 850 nm, and the wavelength is safe to the eyes and has the least interference under most ambient light conditions;
Two infrared cameras are provided, the infrared cameras have a frame rate of 1000 FPS and a resolution of at least 1920x1080 pixels, the infrared cameras are arranged at a specific angle so as to realize cross capture of eyeball movement, and the capture precision of the movement is enhanced through stereoscopic vision, so that each tiny movement of the eyeball can be accurately recorded;
the captured eyeball motion image is analyzed in real time through the image processing module, the image processing module uses a convolutional neural network CNN to perform deep learning, and eyeball motion is accurately identified and tracked from a complex background;
the endoscope space positioning and controlling unit comprises a linear driver, a rotating mechanism and a rotating mechanism, wherein the linear driver is used for realizing linear displacement of the endoscope, the driving precision of the linear driver reaches +/-0.01 millimeter, and the rotating mechanism is used for adjusting the rotating angle of the endoscope according to received eyeball motion data, and the rotating precision is +/-0.1 degree;
Wherein the linear driver and the rotating mechanism adopt a proportional-integral-derivative control algorithm based on PID for fine adjustment of response, and the parameters of the PID controller are dynamically adjusted according to the speed and acceleration of eyeball movement so as to adapt to rapid change in operation;
further comprising automatic error correction, automatically adjusting parameters of the PID controller by continuously monitoring the difference between the actual and the expected position of the endoscope, to minimize operational errors, ensure surgical accuracy, and in particular to achieve accurate control of the endoscope by:
A position sensor is arranged on the endoscope or a supporting structure thereof, the movement distance and the angle of the endoscope are measured, the position feedback is provided for a control unit by converting mechanical displacement into an electric signal, the control unit receives the electric signal, and the difference between the actual position and the expected reference position of the endoscope is continuously monitored to obtain a position error value;
judging whether the position error value exceeds a first preset error threshold value plus or minus 0.02 mm or not to obtain a first judgment result;
if the first judgment result is yes, increasing the adjustment amplitude of the PID controller so as to quickly reduce the position error;
if the first judgment result is negative, further judging whether the position error value is smaller than or equal to a second preset error threshold value plus or minus 0.01 mm;
If yes, maintaining the current parameter setting of the PID controller, and maintaining a fine tuning state to gradually optimize the position accuracy;
if not, reducing the adjustment amplitude of the PID controller;
the intelligent surgical mode selection unit is provided with an acoustic feedback module, and sends out an audio signal when the user is identified to switch the surgical mode through eyeball movement so as to remind the operator that the mode is switched;
the head-mounted display device displays the current view image of the endoscope in real time; the display unit employs an OLED screen with a resolution of at least 2K, providing a refresh rate of at least 120Hz to reduce visual delay and blur;
the system further comprises an ambient light sensing unit, a head-mounted display device and an LED infrared light source, wherein the ambient light sensing unit is used for detecting the light intensity of an operation environment and automatically adjusting the brightness of the head-mounted display device and the LED infrared light source, so that the accuracy of eyeball tracking can be ensured under any illumination condition;
In a low-light environment, the image processing module comprises a function of performing color enhancement processing on the captured eyeball image so as to improve the recognition rate of eyeball characteristics;
The system is configured with a wireless data transmission unit for realizing data communication between the module units; the wireless transmission unit supports at least Wi-Fi and bluetooth technologies.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410995207.0A CN118542635B (en) | 2024-07-24 | 2024-07-24 | Endoscope control system based on eyeball movement recognition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410995207.0A CN118542635B (en) | 2024-07-24 | 2024-07-24 | Endoscope control system based on eyeball movement recognition |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN118542635A CN118542635A (en) | 2024-08-27 |
| CN118542635B true CN118542635B (en) | 2024-10-22 |
Family
ID=92451361
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410995207.0A Active CN118542635B (en) | 2024-07-24 | 2024-07-24 | Endoscope control system based on eyeball movement recognition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118542635B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN120032851A (en) * | 2025-01-15 | 2025-05-23 | 河南嘉宇医疗科技有限责任公司 | A remote evaluation method and system based on smart mirror device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106456148A (en) * | 2014-03-19 | 2017-02-22 | 直观外科手术操作公司 | Medical devices, systems, and methods using eye gaze tracking |
| CN114041103A (en) * | 2019-05-29 | 2022-02-11 | 直观外科手术操作公司 | Operating Mode Control System and Method for Computer Assisted Surgery System |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6714085B2 (en) * | 2015-12-29 | 2020-06-24 | コーニンクレッカ フィリップス エヌ ヴェＫｏｎｉｎｋｌｉｊｋｅ Ｐｈｉｌｉｐｓ Ｎ．Ｖ． | System, controller, and method for using virtual reality devices for robotic surgery |
| CN108065904A (en) * | 2018-01-02 | 2018-05-25 | 京东方科技集团股份有限公司 | Endoscopic system and its control method |
| KR102770936B1 (en) * | 2022-09-05 | 2025-02-21 | 서울대학교병원 | headset-based vision system for providing multiple medical view and surgery assistance |
-
2024
- 2024-07-24 CN CN202410995207.0A patent/CN118542635B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106456148A (en) * | 2014-03-19 | 2017-02-22 | 直观外科手术操作公司 | Medical devices, systems, and methods using eye gaze tracking |
| CN114041103A (en) * | 2019-05-29 | 2022-02-11 | 直观外科手术操作公司 | Operating Mode Control System and Method for Computer Assisted Surgery System |
Also Published As
| Publication number | Publication date |
|---|---|
| CN118542635A (en) | 2024-08-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110502100B (en) | Virtual reality interaction method and device based on eye movement tracking | |
| US11844573B2 (en) | Active visual alignment stimuli in fundus photography | |
| US9319662B2 (en) | Systems and methods for differentiating between dominant and weak eyes in 3D display technology | |
| RU2661027C2 (en) | Method and system for treatment of visual impairment | |
| US9486386B2 (en) | Using a 3D display to train a weak eye | |
| CN118542635B (en) | Endoscope control system based on eyeball movement recognition | |
| JPH02164335A (en) | Eyeball motion analyzing device | |
| CN105431765A (en) | Focal length adjustment | |
| JPWO2016072272A1 (en) | Visual inspection device and head-mounted display device | |
| EP4359894A1 (en) | Eye-tracking using embedded electrodes in a wearable device | |
| CN110568627A (en) | Control method of zoom glasses | |
| EP3919963A1 (en) | Gaze tracking apparatus and systems | |
| EP4083682B1 (en) | Head-mountable display apparatus and methods | |
| JP2021129993A (en) | Ophthalmologic apparatus | |
| JP2011050479A (en) | Visual function measuring device and visual function training device | |
| CN113908025A (en) | Multifunctional medical training system and method for eyes | |
| US20230309819A1 (en) | Ophthalmologic apparatus | |
| GB2613084A (en) | Gaze tracking apparatus and systems | |
| JP2021137130A (en) | Medical image processing device and medical observation system | |
| US20250302295A1 (en) | Ophthalmologic apparatus | |
| CN120029466B (en) | Dynamic virtual image plane adjusting device and method based on multi-image plane eye tracking | |
| CN115875652A (en) | VR equipment and adjusting method thereof, doctor operating table and surgical robot system | |
| CN120168299A (en) | A head-mounted integrated intelligent system for myopia prevention and control | |
| JP2002010980A (en) | Ophthalmic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |