CN111803013A - An endoscopic imaging method and an endoscopic imaging system - Google Patents

Abstract

The invention discloses an endoscope imaging method and an endoscope imaging system, wherein the endoscope imaging method comprises the following steps: illumination, visible light and fluorescence collection, image signal collection and image processing. Collecting visible light reflected by a human body object and excited fluorescence by two sensors positioned at the front end of the endoscope tube to generate corresponding image signals, and transmitting the image signals to a rear end for processing to obtain a final endoscope image; compared with the prior art that visible light and fluorescence are transmitted to the rear end through the light path for collection and processing, the endoscope solves the problem of accuracy reduction caused by light path transmission, improves imaging accuracy, and saves light path cost and light path occupation space; in addition, due to the design of independent light paths of visible light and fluorescence, the visible light and the fluorescence can be respectively focused and collected, so that the definition consistency of a visible light image and a fluorescence image is ensured; the endoscope imaging system is binocular stereoscopic vision, and can obtain three-dimensional information to compensate the fluorescence brightness of different depth areas.

Description

An endoscopic imaging method and an endoscopic imaging system

technical field

The invention relates to the technical field of medical devices, in particular to an endoscopic imaging method and an endoscopic imaging system.

Background technique

In minimally invasive surgery, the application of excitation light endoscopy (or fluorescence endoscopy) can accurately locate the location of cancer, and then more accurately remove cancerous tissue. The main principle is: after the excitation photodrug is dispersed or injected into the target part of the living body, the subject is irradiated with excitation light from the light source device, and the fluorescence from the cancer is captured to diagnose the existence of cancer and malignancy. Qualitative diagnosis of degree, etc.

The prior art is mainly realized by single-camera time-division imaging and dual-camera spectroscopic imaging. Spectroscopic technology is mainly used at the handle end to separate white light, excitation light, and excitation light, and two or more sensors (CCD or CMOS) are used to sense white light and fluorescent monochromatic images respectively. One is realized as one color sensor and one excitation light monochromatic sensor. Another implementation is three-way R, G, B monochrome sensors, plus one-way excitation light monochrome sensor.

The prior art requires complex spectroscopic and filtering techniques to avoid contamination of fluorescent monochromatic images with white light or contamination of white light color images with excitation light. Whether it is time-sharing or light-splitting, in the prior art, visible light, return excitation light, and excitation light all share a set of front-end optical links. Due to the different wavelengths of visible light and excitation light, the focal lengths are different, resulting in visible light images and fluorescent monochrome. Image clarity is inconsistent.

SUMMARY OF THE INVENTION

The invention provides an endoscopic imaging method and an endoscopic imaging system with simple optical path and clear imaging.

According to a first aspect, an embodiment provides an endoscopic imaging method, comprising the following steps:

Controlling the first light source and the second light source to emit white light and excitation light respectively, and the white light and the excitation light are combined to form mixed light and irradiate on the human body;

The visible light reflected by the human body is obtained through the first sensor located at the front end of the endoscope, and the first sensor converts the visible light into a first image signal; at the same time, the fluorescence generated by the human body excited by the second sensor is obtained through the second sensor located at the front end of the endoscope. The sensor converts the fluorescence into a second image signal;

The first image signal and the second image signal are acquired, the first image signal is converted into a white light color image, the second image signal is converted into a fluorescent monochrome image, and the white light color image and the fluorescent image are processed to output an endoscopic image.

Further, the image processing includes the following steps:

Convert white light color image to RGB image;

Convert RGB image to grayscale image;

Normalize the grayscale information of the fluorescent monochromatic image and the grayscale image, and then use the stereo matching algorithm to obtain the stereo disparity map of the fluorescent monochromatic image matched to the white light color image;

According to the stereo disparity map, reconstruct the fluorescent monochromatic image into the coordinate system of the white light color image to obtain the reconstructed fluorescent monochromatic image;

The reconstructed fluorescent monochromatic image is superimposed on the RGB image to form an endoscopic image.

Further, in the step of image processing, after the stereo disparity map is calculated, the distance between the object in the human body and the end face of the camera is calculated through the stereo disparity map, and then the depth information of different fluorescent areas is obtained, and then the distance between the fluorescent areas with far distances is calculated. Fluorescence brightness is compensated.

Further, the compensation of fluorescence brightness includes the following steps:

Calculate the average value of fluorescence light for each fluorescence area, and find the fluorescence area with the highest average value;

The other fluorescence regions except the fluorescence region with the highest average value are respectively multiplied by a corresponding gain coefficient, so that the fluorescence brightness of the other fluorescence regions and the fluorescence region with the highest average value are consistent.

Further, the compensation of fluorescence brightness includes the following steps:

Pre-design and calculate a fluorescence compensation gain curve according to the distance between the human subject and the end face of the camera;

After obtaining the depth information of different fluorescence regions, each region performs fluorescence compensation according to the compensation gain curve.

According to a second aspect, an embodiment provides an endoscopic imaging system, comprising:

The handle, the handle has a threading channel;

Mirror tube, the mirror tube is connected to the front end of the handle;

Light source assembly, the light source assembly includes a first light source, a second light source and a light guide, the first light source is used for emitting white light, the second light source is used for emitting excitation light, one end of the light guide is penetrated in the mirror tube and extends to the front end of the mirror tube , the other end of the light guide is connected with the first light source and the second light source;

The first front-end assembly, the first front-end assembly is installed at the front end of the mirror tube, the first front-end assembly includes a first mirror group and a first sensor arranged in sequence, the first mirror group is used to obtain the visible light reflected by the human body subject, and the first A sensor is used for acquiring the visible light filtered by the first lens group and generating the first image signal;

The second front-end assembly, the second front-end assembly and the first front-end assembly are installed side by side at the front end of the mirror tube, the second front-end assembly includes a second mirror group and a second sensor arranged in sequence, and the second mirror group is used to capture the human body. the excited fluorescence of the object, and the second sensor is used to acquire the fluorescence filtered by the second lens group and generate a second image signal;

and a control device, the control device is connected with the first light source, the second light source, the first sensor and the second sensor; the control device is used for controlling the first light source to emit white light and the second light source to emit excitation light respectively, for collecting the first sensor to generate The first image information and the second image information generated by the second sensor are used to convert the first image signal into a white light color image, the second image signal into a fluorescent monochrome image, and then convert the white light color image and the fluorescent monochrome image. The image is processed to output an endoscopic image.

Further, the control device includes a controller, an image acquisition module and an image processing module, the controller is used to control the first light source to emit white light and the second light source to respectively emit excitation light, and the image acquisition module is used to collect the first sensor generated by the first sensor. The image information and the second image information generated by the second sensor, the image processing module is used to convert the first image signal into a white light color image, the second image signal into a fluorescent monochrome image, and then convert the white light color image and the fluorescent monochrome image. The endoscopic image is output after image processing.

Further, the control device further includes a sensor drive module, the input end of the sensor drive module is connected to the controller, the output end of the sensor drive module is connected to the first sensor and the second sensor respectively, and the sensor drive module is used to calculate the controller The sensor setting information is output to the first sensor and the second sensor.

Further, the light source assembly further includes a light source control module, the input end of the light source control module is connected with the controller, the output end of the light source control module is connected with the first light source and the second light source, and the light source control module is used for receiving the control signal of the controller and generating the control signal. The light intensity of the light emitted by the first light source and the second light source is controlled.

Further, the first mirror group includes a fluorescence cut filter, and the second mirror group includes a visible light cut filter.

According to the endoscopic imaging method and endoscopic imaging system of the above-mentioned embodiments, the visible light reflected by the human body object and the excited fluorescence can be collected respectively by the two sensors located at the front end of the mirror tube, and corresponding image signals can be generated, and then the The image signal is transmitted to the back end for processing to obtain the final endoscopic image; compared with the prior art, the visible light and fluorescence are transmitted from the front end of the mirror tube to the back end through the optical path and then collected and processed, the endoscope solves the accuracy caused by the propagation of the optical path. In addition, the visible light and fluorescence are independent of the optical path design, and the visible light and fluorescence can be collected separately, so as to ensure the same clarity of the visible light image and the fluorescence image; The speculum imaging system is binocular stereo vision, which can obtain three-dimensional information to compensate the fluorescence brightness in different depth regions.

Description of drawings

1 is a schematic structural diagram of an endoscopic imaging system in an embodiment;

2 is a flowchart of an endoscopic imaging method in an embodiment;

FIG. 3 is a flowchart of image processing of an endoscopic imaging method in an embodiment.

Detailed ways

The present invention will be further described in detail below through specific embodiments in conjunction with the accompanying drawings. Wherein similar elements in different embodiments have used associated similar element numbers. In the following embodiments, many details are described so that the present application can be better understood. However, those skilled in the art will readily recognize that some of the features may be omitted under different circumstances, or may be replaced by other elements, materials, and methods. In some cases, some operations related to the present application are not shown or described in the specification, in order to avoid the core part of the present application from being overwhelmed by excessive description, and for those skilled in the art, these are described in detail. The relevant operations are not necessary, and they can fully understand the relevant operations according to the descriptions in the specification and general technical knowledge in the field.

Additionally, the features, acts, or characteristics described in the specification may be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can also be exchanged or adjusted in order in a manner obvious to those skilled in the art. Therefore, the various sequences in the specification and drawings are only for the purpose of clearly describing a certain embodiment and are not meant to be a necessary order unless otherwise stated, a certain order must be followed.

The serial numbers themselves, such as "first", "second", etc., for the components herein are only used to distinguish the described objects, and do not have any order or technical meaning. The "connection" and "connection" mentioned in this application, unless otherwise specified, include both direct and indirect connections (connections). In the text, the front end is the end close to the human subject, and the rear end is the end away from the human subject.

Example 1:

This embodiment provides an endoscope imaging system. The endoscope imaging system is a binocular endoscope, which is mainly used for detecting human canceration.

As shown in FIG. 1 , the endoscopic imaging system mainly includes a handle 10 , a mirror tube 20 , a light source assembly 30 , a first front end assembly 40 , a second front end assembly 50 and a control device 60 , and other parts of the endoscopic imaging system are not It is not involved in this application and will not be described in detail.

The mirror tube 20 is a hard mirror tube or a flexible mirror tube, the rear end of the mirror tube 20 is connected to the front end of the handle 10, and the handle 10 is provided with a threading channel that communicates with the mirror tube 20, and the handle 10 is used for the doctor to operate the mirror tube 20. The front end protrudes into the human body.

The light source assembly 30 includes a first light source 31, a second light source 32 and a light guide 33. The first light source 31 and the second light source 32 are located at the rear of the handle 10, such as installed in the control device 60, or independently installed in a separate device , the two ends of the light guide 33 are the input end and the output end respectively, the input end and the output end of the light guide 33 both have two branch beams, and the two branch beams at the input end of the light guide 33 are respectively connected with the first light source 31 and the second light source. The light source 32 is connected, and the two branch beams at the output end of the light guide 33 extend through the handle 10 to the front end position in the mirror tube 20 , and the two output branch beams are symmetrically arranged. The first light source 31 emits white light (visible light), the second light source 32 emits excitation light, and the white light emitted by the first light source 31 and the excitation light emitted by the second light source 32 are merged into mixed light in the light guide 33 . A fusion light path 34 can also be installed at the front ends of the first light source 31 and the second light source 32. The fusion light path 34 is composed of two input ends and an output end, and is provided with a refractor and other structures inside, so that the two paths of light are combined into one light. Output, the output end of the fusion light path 34 is connected to the input end of the light guide 33, the two input ends of the fusion light path 34 are aligned with the outgoing light paths of the first light source 31 and the second light source 32, and the fusion light path 34 emits the first light source 31. The white light emitted from the second light source 32 and the excitation light emitted by the second light source 32 are merged into mixed light and then irradiated onto the human subject through the light guide 33 . The output end of the light guide 33 is provided with two branched light beams, so that the mixed light can be uniformly irradiated on the human subject.

The first front end assembly 40 and the second front end assembly 50 are installed side by side at the front end position in the mirror tube 20, and the first front end assembly 40 and the second front end assembly 50 are located in the middle of the mirror tube 20, and the output end of the light guide 33 is located in the mirror tube 20. at the edge of the mirror tube 20 . The first front-end assembly 40 is used to collect visible light reflected by the human body subject, and the second front-end assembly 50 is used to collect the excited fluorescence of the human body subject.

The first front-end assembly 40 includes a first mirror group 41 and a first sensor 42. The first mirror group 41 and the first sensor 42 are sequentially arranged in the front-to-rear direction and aligned on the optical axis. The rear end of the first mirror group 41 has a fluorescence cut filter 411 , the fluorescence cut filter 411 is used to filter the fluorescence in the mixed light, and the fluorescence cut filter 411 irradiates the filtered visible light (white light) to the first sensor 42 superior. The first sensor 42 is a color sensor, and the first sensor 42 is used to acquire visible light (white light) and generate a corresponding first image signal.

The second front-end assembly 50 includes a second mirror group 51 and a second sensor 52. The second mirror group 51 and the second sensor 52 are sequentially arranged in the front-to-rear direction and aligned on the optical axis. The rear end of the second mirror group 51 has a visible light cut filter 511 , the visible light cut filter 511 is used to filter visible light in the mixed light, and the visible light cut filter 511 irradiates the filtered fluorescence to the second sensor 52 . The second sensor 52 is a monochrome sensor, and the second sensor 52 is used to acquire fluorescence and generate a corresponding second image signal. The first lens group 41 and the second lens group 51 are focusing lens groups, respectively, which are set to focus in visible light and fluorescence modes, respectively. The lenses of the first lens group 41 and the second lens group 51 except for the filters can be the same. . In other embodiments, the fluorescence cut filter 411 and the visible light cut filter 511 are filter films, and the filter films are attached to the front or rear lens of the second lens group 51 .

In this embodiment, the control device 60 is installed in the equipment behind the handle 10 , the control device 60 is the host part of the endoscopic imaging system, and the control device 60 has control and processing functions. The control device 60 includes a controller 61 , an image acquisition module 62 and an image processing module 63 . The controller 61 is respectively connected with the image acquisition module 62 , the image processing module 63 , the first light source 31 , the second light source 32 , the first sensor 42 and the second light source 32 . Sensor 52 is connected. The controller 61 is a control center for controlling the entire endoscope imaging work.

The light source assembly 30 further includes a light source control module 35, the input end of the light source control module 35 is connected to the controller 61, and the two output ends of the light source control module 35 are respectively connected to the first light source 31 and the second light source 32. The light source control module 35 is used for acquiring the control signal of the receiving controller 61 and controlling the light intensity of the light emitted by the first light source 31 and the second light source 32 .

The control device 60 further includes a sensor driving module 64, the input end of the sensor driving module 64 is connected to the controller 61, and an output end of the sensor driving module 64 is connected to the first sensor 42 in the lens tube 20 through the handle 10 through the cable 43 , the other output end of the sensor driving module 64 is connected to the second sensor 52 in the mirror tube 20 through the handle 10 through the cable 53 . The sensor driving module 64 is configured to output the sensor setting information calculated by the controller 61 to the first sensor 42 and the second sensor 52 .

The control device 60 also includes an input device 65 connected, the input device 65 is a keyboard, a touch screen and other devices, the input device 65 is connected to the controller 61, and the input device 65 is used for inputting operation instructions and parameters to the controller 61. The input commands respond to control other parts of the workpiece.

The input end of the image acquisition module 62 is connected to the first sensor 42 in the mirror tube 20 through the cable 44 through the handle 10 , and the image acquisition module 62 acquires the first image signal generated by the first sensor 42 through the cable 44 . The input end of the image acquisition module 62 is also connected to the second sensor 52 in the mirror tube 20 through the cable 54 through the handle 10 , and the image acquisition module 62 acquires the second image signal generated by the second sensor 52 through the cable 54 . The output end of the image acquisition module 62 is connected to the image processing module 63, the image processing module 63 is used to acquire the first image signal and the second image signal collected by the image acquisition module 62, and the image processing module 63 is also used to convert the first image signal. For a white light color image, the second image signal is converted into a fluorescent monochromatic image, and then the white light color image and the fluorescent monochromatic image are processed to output an endoscopic image.

The image processing module 63 is connected to the display 70 , and the image processing module 63 outputs the endoscopic image generated by the calculation to the display 70 for display.

In the endoscopic imaging system of this embodiment, since the first front end assembly 40 and the second front end assembly 50 are installed at the front ends of the mirror tube 20, the first front end assembly 40 and the second front end assembly 50 collect visible light and fluorescence respectively, and The white light and fluorescence are converted into corresponding image signals and sent to the image processing module 63 at the rear end of the handle 10 for calculation and processing. Compared with the prior art, the visible light and fluorescence are transmitted from the front end of the mirror tube to the back end through the optical path and then collected and processed, the endoscope solves the problem of accuracy degradation caused by the propagation of the optical path, improves the imaging accuracy, and saves the cost and cost of the optical path. Optical paths take up space. In addition, the visible light and fluorescence are independent optical path design, visible light and fluorescence can be focused and collected separately, so as to ensure the same clarity of visible light image and fluorescence image; this endoscope imaging system is binocular stereo vision, which can obtain three-dimensional information to compensate for different depths Fluorescence brightness of the area.

Embodiment 2:

This embodiment provides an endoscopic imaging method, and the imaging method is implemented based on the endoscopic imaging system in the foregoing embodiment.

As shown in FIG. 2 , the endoscopic imaging method of this embodiment includes the following steps:

S10, lighting;

The light source control module 35 controls the first light source 31 to emit white light, and at the same time controls the second light source 32 to emit excitation light, the white light and the excitation light are merged into mixed light through the optical path, and the mixed light is irradiated to the human body tissue (subject) through the light guide 33 ; The tissue (subject) in the human body will reflect white light (visible light), and the cancerous area in the human body will be excited to generate fluorescence after being irradiated by excitation light;

S20, collecting visible light and fluorescence;

Visible light is collected by the first front end assembly 40 located at the front end of the mirror tube 20 , and fluorescence is collected by the second front end assembly 50 located at the front end of the mirror tube 20 .

The first lens group 41 transmits the mixed light formed by visible light and fluorescence reflected and excited by the human body to the fluorescence cut filter 411 , and the fluorescence cut filter 411 filters the fluorescence and irradiates the visible light to the first sensor 42 , the first sensor 42 converts visible light into a first image signal.

At the same time, the second mirror group 51 transmits the mixed light formed by visible light and fluorescence reflected and excited by the human subject to the visible light cut filter 511 , and the visible light cut filter 511 filters the visible light to irradiate the fluorescence to the second sensor 52 , the second sensor 52 converts the fluorescence into a second image signal.

S30, collecting image signals;

The first image signal and the second image signal are collected by the image collection module 62 , and the first image signal and the second image signal are transmitted to the image processing module 63 .

S40, image processing.

After acquiring the first image signal and the second image signal, the image processing module 63 converts the first image signal into a white light color image, converts the second image signal into a fluorescent monochrome image, and then converts the white light color image and the fluorescent monochrome image into The endoscopic image is output after image processing.

As shown in Figure 3, the image processing of the white light color image and the fluorescent monochrome image specifically includes the following sub-steps:

S41. Convert the white light color image to an RGB image;

Since the white light color image is a Bayer mode image, first adjust the white balance of the white light color image, adjust the white balance and convert it to an RGB image, and then perform color correction on the RGB image.

Among them, the adjustment of the white balance of the white light color image is to adjust the ratio of the three color channels of R, G, and B, so that the values of the R, G, and B components in the white or gray image are equal.

After adjusting the white balance of the white light color image, use the difference algorithm to complete the R, G, and B component values of each pixel in the white light color image. That is, the G and B components are filled on the R pixels in the Bayer mode, the R and B components are filled on the G pixels, and the R and G components are filled on the B pixels. The final output is an RGB three-channel color image.

During the color correction process, the color reproduction degree of the image is corrected by the following color correction matrix:

为输入，

为矫正后的输出。Among them, M is the color correction matrix,

for input,

is the corrected output.

S42. Convert the RGB image to a grayscale image;

Convert a color-corrected RGB image to a grayscale image, where the grayscale values can be taken as follows:

Method 1: Take the average value of R, G, and B, namely (R+G+B)/3;

Mode 2: adopt the coding mode specified in ITU-R BT.709, Y=0.2125R+0.7153G+0.0721B;

Method 3: Convert to L channel in L*a*b* color space.

S43, image normalization;

The grayscale information of the fluorescent monochrome image and the grayscale image is normalized.

Fluorescent monochrome images are preprocessed first and then image normalized. The preprocessing of fluorescent monochrome images includes operations such as image de-noising, smoothing, region segmentation and morphology. The key point is to segment the fluorescent monochrome image and extract the fluorescent area from the background area.

In the process of image normalization, the fluorescence region extracted by segmentation and the grayscale image are used for image normalization.

Image normalization can be calculated in the following two ways:

method one:

荧光图像平均值，其中

为第i个荧光区域的区域平均值。设

为灰度图像的平均值。设

归一化的荧光图像为Ut＝{W×U1,W×U2,…,W×Un}。Suppose U={U1,U2,...,Un} is the fluorescence area segmented on the fluorescence image, and n is the number of fluorescence areas.

Fluorescence image mean, where

is the area average of the i-th fluorescent area. Assume

is the average value of grayscale images. Assume

The normalized fluorescence image is Ut={W×U1,W×U2,…,W×Un}.

Method two:

Similar to the first method, the difference is the calculation method of the average value of the grayscale image. Let X _U be the set of coordinate points corresponding to U, and N _U be the area sum of all fluorescent regions (that is, the total number of pixel points corresponding to U). The grayscale image average is calculated as

S44. Calculate a stereo disparity map;

Stereoscopic disparity maps of fluorescent monochrome images matched to white light color images were obtained using a stereo matching algorithm.

Before using the algorithm below, the binocular camera module needs to perform stereorectification. For the object to be photographed, the left and right images after correction will have the same horizontal position, that is, the same y-coordinate value. When performing the matching algorithm, only a lateral search is required, thereby reducing the amount of computation. The binocular correction algorithm is a mature algorithm and will not be described again. When performing binocular calibration, the visible light cut filter in the fluorescence camera can be temporarily replaced with a fluorescence cut filter, which is convenient for calibration using white light images.

Let the first lens (white light camera) be the left camera, and the second lens (fluorescence camera) be the right camera. The stereo matching algorithm has the following two ways:

Method 1: Simple block matching algorithm

Do the following for all pixels in the fluorescent area:

Let the coordinate of a pixel in the current fluorescence image be X _i , take its adjacent 3x3 or 5x5 area, and denote it as Y _i . On the grayscale image, starting from the coordinate X _i , look to the right for the grayscale block most similar to _Yi (block matching), and note the coordinate _Xm of the matching block on the grayscale image. d _Xi =X _m -X _i is the parallax corresponding to the fluorescent image pixel Xi _.

Method 2: SGM (Semi-Global Matching)

Method 1 Although the algorithm is simple and runs fast, it will cause more noise and inaccurate disparity calculation. In practical applications, the SGM algorithm will be used more.

Ref:H.Hirschmuller,"Stereo Processing by Semiglobal Matching andMutual Information,"in IEEE Transactions on Pattern Analysis and MachineIntelligence,vol.30,no.2,pp.328-341,Feb.2008,doi:10.1109/TPAMI.2007.1166 .

The purpose of the above two methods is to obtain the disparity value of each fluorescent pixel point corresponding to the grayscale image.

S45. Reconstruct a fluorescent monochrome image;

According to the stereo disparity map, the fluorescent monochromatic image is reconstructed into the coordinate system of the white light color image to obtain the reconstructed fluorescent monochromatic image.

Reconstructing a fluorescent monochromatic image can be achieved in two ways:

Method 1: pixel based

For any fluorescent pixel X _i , its corresponding d _Xi is obtained, and its corresponding coordinate on the grayscale image (white light image) is X _m =X _i +d _Xi .

Method 2: Region based

将荧光区域U_i中所有像素向右平移

即可得将U_i重建到在白光图坐标系中。For any fluorescent region U _i , calculate the average value of the corresponding disparity of all pixels in it

Shift all pixels in the fluorescent region U _i to the right

That is, U _i can be reconstructed into the white light map coordinate system.

S46, image overlay;

The reconstructed fluorescent monochromatic image is superimposed on the color-corrected RGB image to form an endoscopic image.

The RGB images are also processed for image addition and image parameter adjustment in the superposition. The image addition includes common processes such as sharpening, denoising, and edge enhancement, and the image parameter adjustment includes adjusting the brightness, contrast, saturation and other parameters of the image. The RGB image after image addition and image parameter adjustment processing has higher image quality, so that a clearer endoscopic image can be obtained by subsequent superposition processing.

In the endoscope imaging method provided in this embodiment, visible light and fluorescence are collected through binoculars at the front end, and visible light and fluorescence can be collected by focusing separately, so as to ensure the same clarity of visible light image and fluorescence image; Calculate and reconstruct the fluorescence image into the white light coordinate system to obtain a clear and accurate image.

In this another embodiment, in order to further improve the imaging effect, fluorescence intensity compensation is also performed after step S44. Since the fluorescence ability of the lens to capture the fluorescence area at a longer distance is weaker, the intensity of the imaging area far away from the fluorescence area is weaker. Therefore, by compensating the fluorescence intensity, the fluorescence area with different distances from the lens can exhibit the same fluorescence intensity.

The principle steps of fluorescence intensity compensation are: calculate the distance between the object in the human body and the end face of the camera through the stereo disparity map, and then obtain the depth information of different fluorescence regions, and then compensate the fluorescence brightness of the far-distance fluorescence regions.

The depth information is calculated as follows:

Let a pixel coordinate in the current fluorescence image be _Xi , and its corresponding parallax be d _Xi . The depth value formula is:

Among them, f is the focal length of the camera, and T is the center distance of the left and right cameras, both of which can be obtained in the binocular correction result.

Fluorescence intensity compensation includes the following two methods:

method one:

Calculate the average value of fluorescence light for each fluorescence area, and find the fluorescence area with the highest average value;

The other fluorescence regions except the fluorescence region with the highest average value are respectively multiplied by a corresponding gain coefficient, so that the fluorescence brightness of the other fluorescence regions and the fluorescence region with the highest average value are consistent.

Method two:

Pre-design and calculate a fluorescence compensation gain curve according to the distance between the human subject and the end face of the camera;

After obtaining the depth information of different fluorescence regions, each region performs fluorescence compensation according to the compensation gain curve.

The compensation curve is calculated as follows:

Using a fluorescent target, a fluorescent camera is used to collect fluorescent images from 1cm, 2cm,..., to 20cm away from the target. Take the central 32x32 area image and calculate the average. Taking the distance as the abscissa and the fluorescence value as the ordinate, draw the attenuation curve of the intensity of the fluorescence value changing with the distance. Assuming that 5cm (or any distance deemed appropriate) is the optimal fluorescence distance, the fluorescence intensity corresponding to 5cm is used to normalize the decay curve to obtain the compensation value of the fluorescence intensity that changes with the distance. An implementation method is to set y _b as the optimal fluorescence value, the compensation curve b _i =y _b /y _i , i={1,2,...,20}, and y _i is measured by dividing the distance i cm fluorescence value.

The fluorescent target is a solid-color uniform target. In order to reduce noise interference, when collecting images at various distances, multiple images can be continuously collected and averaged to obtain an average image.

The compensation curve is a discrete curve. In the application, the interpolation algorithm can be used to obtain the compensation coefficient under the current distance.

The specific method of applying the compensation curve to the above compensation is as follows:

1) In any fluorescent area in the fluorescent image, calculate the draw depth value of its pixels;

2) Calculate the corresponding compensation coefficient α in the fluorescence depth compensation curve using an interpolation algorithm;

3) Multiply the compensation coefficient α for each fluorescent pixel in the current area.

The above specific examples are used to illustrate the present invention, which are only used to help understand the present invention, and are not intended to limit the present invention. For those skilled in the art to which the present invention pertains, according to the idea of the present invention, several simple deductions, modifications or substitutions can also be made.

Claims (10)

1. an endoscope imaging method, is characterized in that, comprises the steps: Controlling the first light source and the second light source to emit white light and excitation light respectively, the white light and the excitation light are combined to form mixed light and irradiate on the human body; The visible light reflected by the human body is obtained through the first sensor located at the front end of the endoscope, and the first sensor converts the visible light into a first image signal; at the same time, the fluorescence generated by the excited human body is obtained through the second sensor located at the front end of the endoscope, the second sensor converts the fluorescence into a second image signal; Obtain a first image signal and a second image signal, convert the first image signal into a white light color image, convert the second image signal into a fluorescent monochrome image, and then perform image processing on the white light color image and the fluorescent image Output endoscopic images. 2. The endoscopic imaging method of claim 1, wherein the image processing comprises the steps of: Convert white light color image to RGB image; converting the RGB image to a grayscale image; Perform image normalization processing on the grayscale information of the fluorescent monochrome image and the grayscale image, and then use a stereo matching algorithm to obtain a stereo disparity map of the fluorescent monochrome image matched to the white light color image; According to the stereo disparity map, reconstruct the fluorescent monochromatic image into the coordinate system of the white light color image to obtain a reconstructed fluorescent monochromatic image; The reconstructed fluorescent monochromatic image is superimposed on the RGB image to form an endoscopic image. 3. The endoscopic imaging method according to claim 2, wherein in the step of image processing, after the stereo disparity map is calculated, the distance between the object in the human body and the end face of the camera is calculated by the stereo disparity map, and then the distance between the object and the end face of the camera is calculated. The depth information of different fluorescent areas is obtained, and then the fluorescent brightness of the fluorescent areas far away is compensated. 4. The endoscopic imaging method according to claim 3, wherein the compensation of the fluorescence brightness comprises the following steps: Calculate the average value of fluorescence light for each fluorescence area, and find the fluorescence area with the highest average value; The other fluorescence regions except the fluorescence region with the highest average value are respectively multiplied by a corresponding gain coefficient, so that the fluorescence brightness of the other fluorescence regions and the fluorescence region with the highest average value are consistent. 5. The endoscopic imaging method according to claim 3, wherein the compensation of the fluorescence brightness comprises the following steps: Pre-design and calculate a fluorescence compensation gain curve according to the distance between the human subject and the end face of the camera; After obtaining the depth information of different fluorescence regions, each region performs fluorescence compensation according to the compensation gain curve. 6. An endoscopic imaging system, comprising: a handle, the handle has a threading channel; a mirror tube, the mirror tube is connected to the front end of the handle; A light source assembly, the light source assembly includes a first light source, a second light source and a light guide, the first light source is used for emitting white light, the second light source is used for emitting excitation light, and one end of the light guide is passed through the inside the mirror tube and extending to the front end of the mirror tube, and the other end of the light guide is connected with the first light source and the second light source; a first front-end assembly, the first front-end assembly is mounted on the front end of the mirror tube, the first front-end assembly includes a first mirror group and a first sensor arranged in sequence in front and back, the first mirror group is used to acquire the human body Visible light reflected by the subject, the first sensor is used to acquire the visible light filtered by the first lens group and generate a first image signal; A second front-end assembly, the second front-end assembly is installed side by side with the first front-end assembly at the front end of the mirror tube, the second front-end assembly includes a second mirror group and a second sensor that are arranged in sequence, the second The lens group is used to acquire the excited fluorescence of the human body subject, and the second sensor is used to acquire the fluorescence filtered by the second lens group and generate a second image signal; and a control device, the control device is connected with the first light source, the second light source, the first sensor and the second sensor; the control device is used for controlling the first light source to emit white light and the second light source to emit white light respectively Excitation light, used to collect the first image information generated by the first sensor and the second image information generated by the second sensor, and used to convert the first image signal into a white light color image, and the second image signal into a fluorescent monochrome image , and then the white light color image and the fluorescent monochrome image are processed to output the endoscopic image. 7. The endoscopic imaging system according to claim 6, wherein the control device comprises a controller, an image acquisition module and an image processing module, the controller is configured to control the first light source to emit white light and the second light source respectively emit excitation light, the image acquisition module is used to collect the first image information generated by the first sensor and the second image information generated by the second sensor, and the image processing module is used to convert the first image signal into a white light color image , the second image signal is converted into a fluorescent monochromatic image, and then the white light color image and the fluorescent monochromatic image are processed to output an endoscopic image. 8 . The endoscopic imaging system according to claim 7 , wherein the control device further comprises a sensor driving module, an input end of the sensor driving module is connected to the controller, and an output end of the sensor driving module is connected to the controller. 9 . respectively connected with the first sensor and the second sensor, and the sensor driving module is used for outputting the sensor setting information calculated by the controller to the first sensor and the second sensor. 9 . The endoscopic imaging system according to claim 7 , wherein the light source assembly further comprises a light source control module, the input end of the light source control module is connected with the controller, and the output end of the light source control module is connected with the light source control module. 10 . The first light source and the second light source are connected, and the light source control module is used for receiving a control signal from the controller and controlling the light intensity of the light emitted by the first light source and the second light source. 10. The endoscopic imaging system of claim 6, wherein the first mirror group comprises a fluorescence cut filter, and the second mirror group comprises a visible light cut filter.

Images