JP6684005B1 - Endoscopic device capable of high-definition imaging and spectrum analysis - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a small endoscope device capable of observing a subject with high definition, obtaining qualitative information of the subject by analyzing a reflection spectrum, and realizing miniaturization.
An optical fiber (1) that emits illumination light supplied from a light source (209) and receives reflected light that returns from a subject (3), and illumination light emitted from the optical fiber (1) is generated on a scanning line (4) in a one-dimensional direction, An endoscope apparatus comprising an optical system 2 for irradiating 3 and a scanning mechanism 8 for scanning a scanning line 4 in a direction Y intersecting a one-dimensional direction.
[Selection diagram] Figure 1

Description

  The present invention relates to an endoscope apparatus capable of high-definition imaging and analysis of a reflection spectrum regardless of its small size and small diameter.

-There is a 2mmφ small-diameter endoscope that uses an image fiber bundle, but due to the wave-optic limit of the image fiber diameter, the number of pixels remains at about 70,000.
・ Although there is a 2mmφ small-diameter endoscope that incorporates an image sensor manufactured according to the most advanced design rules, the number of pixels is only about 140,000.
-Neither of the above endoscope devices has a function of analyzing a reflection spectrum.

-There is a strong need for magnifying observation of diseases such as cancers of 3 mm or less with high-definition images such as HD, 4K, and 8K. In addition, there is a strong need to obtain qualitative information of the subject at the pixel level by analyzing the reflection spectrum at the same time as such high-definition observation. However, there is no small-diameter endoscope apparatus that can provide these functions.

・ The reason is that if an HD image sensor is installed inside the tip of a 2mmφ small-diameter endoscope, the diagonal dimension of the image pickup surface of the image sensor must be kept below 1.5mm. Since the pitch is required to be 0.65 μm or less, the current semiconductor design rules, image sensor sensitivity, and optical system resolution will be exceeded. The difficulty increases even further when it becomes 4K or 8K.

・ In addition, if an image sensor is to be equipped with a function to detect a spectrum, it is necessary to separately prepare pixels equipped with an optical filter for spectrum detection in addition to RGB, which makes it much more difficult to downsize the image sensor. .

  The present invention has been made in view of such circumstances. That is, the present invention replaces the configuration of the light guide and the image sensor of the conventional endoscope with one optical fiber and a one-dimensional scanning mechanism, and the reflected light received by the optical fiber is resolved and spectrum outside the imaging unit. By performing the optical processing required for analysis, we will solve the problems related to image sensors and realize an endoscopic device that can be made smaller and thinner in the part inserted into the human body.

  In order to solve the above-mentioned problems and achieve the object, a first aspect of an endoscope apparatus of the present invention is an optical fiber that emits illumination light supplied from a light source and receives reflected light returning from a subject, An optical system that generates illumination light emitted from the optical fiber on a scanning line in a one-dimensional direction and illuminates a subject, and a scanning mechanism that scans the scanning line in a direction intersecting the one-dimensional direction.

  A second aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to the first aspect, wherein the illumination light emitted from the optical fiber is applied obliquely to the surface of the subject. .

A third aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to the first or second aspect, wherein the intersecting directions are orthogonal to the one-dimensional direction.
A fourth aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to any one of the first to third aspects, wherein the light source generates broadband light or broadband wavelength swept light.

  A fifth aspect of the endoscope device of the present invention is the endoscope device according to the fourth aspect, wherein the light source generates excitation light for causing the subject to fluoresce.

  A sixth aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to any one of the first to fifth aspects, wherein the optical system has a mechanism for focusing or zooming the subject. Characterized by

  A seventh aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to any one of the first to sixth aspects, wherein the light source is excitation light for generating Raman scattered light from the subject. To generate.

  An eighth aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to any one of the first to seventh aspects, wherein the light source generates stimulated emission light for causing the subject to fluoresce. .

  A ninth aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to any one of the first to eighth aspects, wherein interference for interfering reference light with the reflected light received by the optical fiber is provided. A coupler and a reflector for generating the reference light are provided.

  A tenth aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to any one of the first to ninth aspects, wherein the reflection plate generates reference light by Raman scattered light.

  An eleventh aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to any one of the first to tenth aspects, wherein the reference light is made to interfere with the reflected light received by the optical fiber. Then, a predetermined wavelength band component is taken out, Fourier transform is performed, and an image in the predetermined wavelength band component is generated.

  A twelfth aspect of the endoscopic device of the present invention is the endoscopic device according to the eleventh aspect, wherein the reference light is interfered to form three primary colors with respect to the reflected light received by the optical fiber. Means for taking out corresponding wavelength band components, performing Fourier transform, generating image signals of three primary colors, and generating an RGB image signal based on the image signals of the three primary colors.

  A thirteenth aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to the twelfth aspect, wherein the luminance signal components of the image signals of the three primary colors are Fourier-transformed in the near infrared region. Contains the signal obtained.

  A fourteenth aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to any one of the first to thirteenth aspects, wherein a plurality of clusters are arranged in descending order of Fisher ratio from a reflection spectrum of a known subject. An identification unit is provided that calculates a spectral component and uses the spectral component to identify from a reflection spectrum of an object whose cluster is unknown.

  A fifteenth aspect of the endoscopic device of the present invention is the endoscopic device according to the fourteenth aspect, wherein the identifying means uses an AI that executes deep learning.

  A sixteenth aspect of the endoscopic device of the present invention is the endoscopic device according to any one of the first to fifteenth aspects, in which the correlation between the image of the previous frame and the image of the current frame The blur amount of the image of the current frame is detected, and the image of the current frame is corrected.

  A seventeenth aspect of the endoscope apparatus of the present invention is the endoscope apparatus according to any one of the first to sixteenth aspects, wherein a difference between signals of a plurality of scanning lines in the one-dimensional direction adjacent to each other is calculated. By acquiring and accumulating, the moving image of the subject is rendered.

The endoscope device of the present invention has a simple structure in which the image pickup unit includes an optical system, an optical fiber, and a one-dimensional scanning mechanism, and thus can be downsized and reduced in diameter.

The endoscopic device of the present invention is capable of high-definition imaging in HD, 4K, and 8K even though it is small and has a small diameter, and enables microscopic magnifying observation with such a pixel number.

Since the endoscope apparatus of the present invention is capable of high-speed repetitive scanning, it is possible to extract a fast-moving subject, and it is possible to depict blood vessels of the superficial mucosa without a contrast agent.

The endoscopic device of the present invention has a small size and a small diameter, but can specify a substance by spectral analysis at the pixel level.

    Since the endoscopic device of the present invention does not use an image sensor which is weak against heat, autoclave sterilization is possible.

It is a figure for demonstrating the principle of the imaging system of the endoscope apparatus of this invention. It is a figure for demonstrating an observation image. It is a block diagram which shows the structure of an imaging part. It is a block diagram which shows the structure of the imaging part of a MEMS mirror drive. It is a block diagram which shows the structure which incorporated the imaging part in the endoscope. It is a figure for demonstrating the resolution of a scanning direction. It is explanatory drawing for demonstrating the resolution and depth of focus in a scanning direction. It is a block diagram which shows the OCI process of a broadband light source. It is a block diagram which shows the OCI process of a wavelength swept light source. It is a block diagram which shows the usage example-1 of an endoscope. It is a block diagram which shows the usage example-2 of an endoscope. It is a block diagram which shows the usage example-3 of an endoscope. It is a block diagram which shows the usage example-4 of an endoscope. It is a block diagram which shows blurring correction. It is a block diagram which shows the structure of the imaging part of a contact-type small diameter endoscope. It is a block diagram which shows the structure of the imaging part of a contact-type small diameter endoscope. It is a block diagram which shows the structure of the usage example of a contact-type small diameter endoscope. It is a figure explaining the usage method of a contact type small diameter endoscope. It is a block diagram which shows detection of RGB and a spectrum image. It is a block diagram which shows a broadband light receiving line detector. It is a figure explaining the range of Fourier transformation. It is a figure explaining FS (Foley Sammon) conversion. It is a figure explaining a cluster identification flow. It is a figure explaining the Stokes shift of a Raman spectrum. It is a block diagram which shows the OCI process-1 for Raman spectra. It is a block diagram which shows detection of RGB and a Raman spectrum image. It is a block diagram which shows Raman spectrum detection by a variable filter. FIG. 6 is a block diagram showing an OCI process for stimulated emission of fluorescence. It is a block diagram which shows the structure of the intravascular scope. It is a block diagram which shows the structure of the intravascular scope. It is a block diagram which shows the structure of the intravascular scope. It is a block diagram which shows the structure of the scope for puncture needles.

Hereinafter, an endoscope apparatus according to an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same parts are designated by the same reference numerals.
The principle of the image pickup method of the present invention will be described below.
1-1. Principle of Imaging Method The principle of the imaging method of the endoscope apparatus of the present invention is as shown in FIG. 1, in which the illumination light from the optical fiber 1 is passed through the optical system 2 from an oblique direction to the surface 3 of the subject. OCI (Optical Coherence Imaging: hereinafter, referred to as OCI in the present specification) installed outside the imaging unit by receiving light reflected from the line 4 by converging it in the shape of a line 4 and illuminating it. In the processing unit, processing that applies the principle of Michelson interferometry is performed, and resolution in four line directions is performed to obtain one-dimensional image data. Then, the scanning mechanism 8 repeats the OCI process while scanning the line 4 in a direction (arrow Y) perpendicular to the line 4 in FIG. 1 to obtain two-dimensional image data. In FIG. 1, reference numeral 201 is a tip imaging unit, 203 is an optical circulator, 205 is a display unit, 207 is an OCI processing unit, and 209 is a light source.

-The OCI processing is based on the processing of OCT (Optical Coherence Tomography) as described in the embodiment in Section 2. With Michelson interferometry and Fourier transform processing, a micron-order short light pulse is emitted obliquely, as if it were a radar, and the light pulses that are sequentially reflected from the subject surface are received to obtain a one-dimensional image. It is a process that produces the same effect as.

As shown in FIG. 2, the image obtained by the imaging method of the present invention is such that the subject is illuminated obliquely, and the reflection point 6 at which the line 4 on the subject intersects the wavefront 5 of the illumination light is tangential to the wavefront 5. It is the same as the image observed from the direction 7. In addition, when the subject is semi-transparent, as shown in the enlarged view 2A in the lower left of FIG. 2, the reflection signals from the same wavefront inside the subject are superimposed, and the same image as when the transmission image is observed from the tangential direction 7 is obtained. To be If near-infrared light (0.68 μm to 1.5 μm), which is highly transparent to the living body, is used for illumination, a transmitted image can be obtained at a depth of several mm from the surface of the living body. Further, since the illumination is performed obliquely, the specular reflection of the mucous membrane surface escapes to the front, and the shadow of the tissue T such as a blood vessel is likely to appear, so that a high-quality image having a stereoscopic effect can be obtained. For example, when observing a blood vessel in a shallow portion of the surface mucosa of an organ by applying it to an endoscope, the blood vessel is emphasized by a shadow without being disturbed by specular reflection, and the structure of the blood vessel can be easily grasped. Incidentally, the malignancy and progress of cancer have been diagnosed by observing the vascular structure.

-OCI is not a process that detects reflected light that is greatly attenuated from inside the living body like OCT, but is a process that detects reflected light on the surface of the subject and acquires an RGB image and analyzes the reflection spectrum. Compared with, there is a sufficient margin of sensitivity, and by adjusting the aperture of the optical fiber 1 with a slit or pinhole, the depth of focus of the optical system 2 can be set shallow, and the resolution (NA) in the scanning direction is similar to that of a microscope. Can be set high.

  Also, with respect to the resolution in the direction of the line 4, the resolution can be set to the wave-optical limit by setting the wavelength bandwidth of the illumination wide and adjusting the aperture of the optical fiber 1 with the slit or the pinhole. Therefore, it becomes possible to perform magnified observation with high-definition images such as HD, 4K, and 8K. The numerical verification is described in Section 3.

-As described in the background art, the critical element that limits the miniaturization and diameter reduction of the conventional endoscope is the size of the image sensor and the light guide fiber, but the endoscope apparatus of the present invention is A single optical fiber that serves both as illumination and light reception and a simple one-dimensional scanning mechanism replaces them, so the imaging unit can be made smaller and smaller in diameter. One-dimensional scanning mechanism, advances in microfabrication technology, and micro motor for OCT of 0.9Mmfai, galvano motor, a voice coil, a small scanning mechanism, such as MEMS mirrors are commercially available, concerns about the size of a scanning mechanism There is no. Rather, the critical factor is the lens aperture size, which is determined by the resolution and work distance. The numerical verification is described in Section 3.

1-2. Application of the present invention-The image pickup method of the endoscope apparatus of the present invention enables high-definition magnified image pickup, and the line scanning repetition time becomes the exposure time. Compared with, the exposure time is shorter and the image blur does not occur between lines. By utilizing this, the OCI processing in the line 4 direction is repeated twice, and the difference between the images is obtained (that is, the difference between the signals of the scanning lines in the plurality of one-dimensional directions adjacent to each other is acquired and accumulated), Images of only moving blood can be extracted. By superimposing the images thus obtained over a plurality of times in a superimposed manner, a blood vessel image with high contrast can be visualized in detail without using a contrast agent. As mentioned above, there is a sufficient margin of sensitivity compared to OCT, so it is possible to speed up the iterative processing in the line 4 direction, and even if the number of iterative processing is increased to perform accumulation, the frame rate is reduced. It is possible to draw a high-definition blood vessel image without having to do so.

  If this detection method is used together with NBI (Narrow Band Imaging) processing that increases the difference in reflectance between the blood vessel and the superficial mucosa, the blood vessel image rendering effect can be further enhanced. Also, by using near-infrared rays, which has a high biological transmittance, for illumination, it becomes possible to visualize the transparent image of the deep vascular structure as described in Section 1-1.

The endoscopic device of the present invention is capable of extracting the received light signal to the outside of the imaging unit by an optical fiber and performing optical processing such as interference and spectroscopy, so as described in Section 4-5. At the same time as high-definition imaging, spectral analysis for identifying a substance can be performed at the pixel level. Specifically, a large amount of multispectral data can be acquired in vivo for multivariate analysis. Then, an off-line computer performs multivariate analysis of the large amount of multispectral data to extract the optimum spectral component axis for each substance to be specified, and then the matrix conversion coefficient for projecting to the spectral component axis is calculated. By storing in the endoscope apparatus of the present invention and switching the coefficient for each specified substance to be used, it is possible to perform accurate real-time substance identification at the pixel level.

2. Example of Imaging Unit • FIG. 3-1 shows an example of the tip imaging unit of the endoscope apparatus of the present invention.
The illumination light emitted from the end face 51 of the optical fiber 11 is converged into a line 15 on the subject by the optical system 12 and the mirror 13, and the reflected light from the line 15 is passed through the same mirror 13 and the optical system 12 to form the optical fiber 11. An image is formed on the end surface of the image pickup device and is taken out as an optical signal to the outside of the image pickup unit. At that time, in order to obtain the resolution in the vertical direction (scanning direction) with respect to the paper surface of FIG. 3A, the optical resolution is adjusted so that the resolution (NA) in the vertical direction with respect to the paper surface is constant at any position on the line 15. The aperture of the system 12 in the direction perpendicular to the plane of the paper is made to gradually increase as the convergent position on the line 15 becomes farther from the end face 51 of the optical fiber 11. With respect to the direction of the line 15, an appropriate diffusion degree is set by the optical system 12 and the mirror 13 so that the line 15 is illuminated obliquely and uniformly. Therefore, the optical system 12 is a cylindrical free-form surface lens. In FIG. 3A, reference numeral 18 is a rotary scanning axis, 525 is a bearing, 527 is a lens frame, 529 is a magnet, 531 is an electric coil, and 533 is a coiled spring.

  In FIG. 3A, the mirror 13 and the optical system 12 are described separately in order to explain the principle of illumination, but it is preferable to integrally mold them as a free-form surface prism. Thereby, the tip structure can be downsized, the manufacturing process can be reduced, and the optical accuracy can be improved. Further, a free-form surface prism and a GRIN lens (Graded-Index Lenses) may be combined. Further, the optical system 12 may be combined with an auto focus or a zoom mechanism.

Also, by changing the distance between the optical system 12 and the end face 51 of the optical fiber 11 from the outside, the focal length and aperture angle of the optical system 2 can be changed, so that it is possible to change from normal imaging to microscope observation. Can be At this time, the resolution in the direction of the line 15 is adjusted close to the optical limit (limit of resolution) by adjusting the opening of the fiber end face with a pinhole.

The scanning mechanism 16 in the embodiment of FIG. 3-1 includes a microgalvano motor (a microgalvano scanner or a micromotor rotating and scanning at 60 Hz) that reciprocally scans the line 15 at 30 Hz in a direction perpendicular to the paper surface, and an optical system 12. It is composed of a mirror 13. In the case of a galvano motor, a non-resonant type is used because it makes a reciprocating scan at a low frequency of 30 Hz. At the time of the above-described angiography, it is necessary to change the scanning speed in order to perform cumulative addition of line images, and for that reason, the non-resonance type is better.

FIG. 3-2 (a) shows an example in which the MEMS mirror 131 is used for the scanning mechanism 16 and the tip portion is miniaturized. In the case of this embodiment, since the rotary scanning axis 18 is inclined with respect to the line 15, the scanning area SA has a fan shape as shown in FIG. 3-2 (b). Using multiple line memory and interpolation circuit, TV scan is scan converted and displayed. Reference numeral 121 in FIG. 3-2 is an optical system, r1 and r2 are rotational scanning radii, S is an object, reference numeral 124 in FIG. 3-3 is an optical system, 18 is a rotational scanning axis, and F is F. The focus variable direction, I indicates the imaging range variable direction.

-FIG. 3-3 shows an embodiment in which the image pickup apparatus of the present invention is incorporated in the endoscope apparatus. By providing the optical system with a focusing and zooming mechanism, it is possible to change the imaging position and range. The image on the fan is displayed by performing scan conversion on TV scanning using a multi-line memory and an interpolation circuit.

The resolution of the line 15 in the direction perpendicular to the paper surface (see (3) in FIG. 4) is projected on the PSF (Point Spread Function) of the optical system 12 (see (1) in FIG. 4) and the line 15. Since a convolution integral with the PSF of the end face 51 (aperture) of the optical fiber 11 (the light intensity distribution of the projected image: see (2) in FIG. 4) is performed, a slit or pinhole is installed on the end face 51 of the optical fiber 11. By increasing the point light source property, the resolution can be brought close to the PSF of the optical system 12. As shown in FIG. 4, since the width W becomes wider, the overall resolution is reduced. However, since there is a trade-off between sensitivity and partial coherence, the aperture shapes of the optical system 12 and the end face 51 of the optical fiber 11 are appropriately set according to the purpose.

・ In addition, the total resolution (or depth of focus) of illumination and light reception is, as shown in FIG. 5, PSF of illumination (or depth of focus of illumination) (1) and PSF of light reception in the direction perpendicular to the paper surface. (Or the depth of focus of light reception) (2). Thus, it is the square of the PSF (or depth of focus) of the optical system 12. Therefore, the width W5 in FIG. 5C is 1 / 1.4, the resolution is about 1.4 times that of the camera image, and the depth of focus is 1 / 1.4. At this time, since the peripheral side lobes (SL) are greatly suppressed, the sense of resolution is improved. The resolution δ can be approximated by the equation δ = 0.37 × λ / NA (= sin α), although there is a difference between the rectangular aperture and the circular aperture. The depth of focus γ also becomes a square because it also becomes a square, and can be approximated by the formula γ = 0.565 × λc / sin ² (α / 2).

-The resolution in the direction of line 15 is obtained by OCI processing that applies the principle of Michelson interferometry. As shown in FIG. 6A, the illumination light of the broadband light source 21 is input to the branch coupler 22 of the optical fiber. The split coupler 22 divides the light into two, and one of the illumination lights is obtained as a reference light 25 by a reflector 24 through an optical circulator 23. The other illuminating light is guided by the optical circulator 26 to the optical fiber 11 of the imaging unit shown in FIG. 3, and illuminates the subject surface obliquely. Since the light is illuminated from an oblique direction, the reflected light from the line 15 shown in Fig. 3-1 is proportional to the difference in the round trip distance from the end face of the optical fiber 11 to each reflection point on the line 15 (the reflection on the line 15). The phases are shifted (as the points are separated), and they are received by the optical fiber 11 as a superimposed signal. The reflected light thus superimposed passes through the circulator 26 and is interfered with the reference light 25 by the interference coupler 27. When interference occurs, interference fringes are generated at a frequency proportional to the phase shift, and they are transformed into a superimposed signal. The frequency of the interference fringe becomes higher at a reflection point whose optical path length is farther from the reflection plate 24.

  Next, the output light of the interference coupler 27 is separated into wavelength components by the grating spectroscope 28, and converted into a time-series electric signal by the light receiving line detector 29. By being converted into an electric signal, carriers of the wavelength (frequency) of the light disappear, and the electric signal is changed to an electric signal on which interference fringes which are the envelope of the reflected light are superimposed. The FFT 30 then performs a Fourier transform on this electrical signal. As described above, since the interference fringes of the frequency proportional to the position of each reflection point on line 15 are superimposed, when the frequency component is converted by FFT30, the frequency component axis becomes the position of each reflection point on line 15. Correspondingly, the amplitude axis of each frequency component corresponds to the reflectance at each reflection point on the line 15. As a result, the image on the line 15 is detected as shown in the lower right diagram of FIG. 6-1.

Next, FIG. 6-2 shows the OCI processing when the broadband wavelength swept light source 31 is used instead of the broadband light source 21. As shown in the upper right diagram of FIG. 6-2, the wavelength of the illumination light is linearly modulated by the wavelength swept light source 31 within the repetitive scanning time of one line, and is input to the branching coupler 32. Further, the output light from the branch coupler 33 is converted into an electric signal by the light receiving detector 34. Other processes are the same as the broadband light source system OCI process shown in FIG. 6-1. When the wavelength swept light source is used, the grating spectroscope 28 and the light receiving line detector 29 shown in FIG. 6-1 are not necessary, and the light receiving detector 34 that is most suitable for receiving light in the wavelength band can be used. However, in order to prepare a broadband wavelength swept light source such as visible light, it is assumed that the structure of the wavelength swept light source becomes complicated, such as combining using a plurality of wavelength swept light sources. Therefore, since the method to be adopted has advantages and disadvantages, the light source method and the OCI process are appropriately selected and combined according to the purpose, including applications such as multispectral analysis described later.

When the wavelength bandwidth of the broadband light source 21 (see FIG. 6-1) and the wavelength swept light source 31 (see FIG. 6-2), which are the above-mentioned illumination light sources, is widened, the frequency band of the interference fringes is proportionally increased. As the frequency component band after the Fourier transform by the FFT 30 increases (the number of frequency components corresponding to the number of pixels in the line 15 direction increases), the resolution in the line 15 direction increases. The resolution σ in the direction of the line 15 at the reflection point 17 in FIG. 3-1 is σ = [0.44 · (λc) ² / Δλ when the end face 51 of the optical fiber 11 is regarded as a point light source and the wavelength band has a Gaussian shape. ] / Sin θ Δλ indicates the wavelength bandwidth, and λc indicates the center wavelength. θ is the angle formed by the line 15 and the traveling direction of the wavefront of the illumination light at the reflection point 17. The actual resolution is σ and the convolution integral of the PSF of the end face 51 of the optical fiber 11.

・ Furthermore, the sensitivity of OCI processing is increased by the FFT Fourier because the wavelength number of wavelength component per unit time increases as the wavelength sweep time (= scan repetition time) increases (or the output of the photodetector increases). The amplitude of the frequency component after conversion is increased, and the sensitivity is increased.

-The OCI process is not a process of detecting the reflected light inside the living body that is exponentially greatly attenuated to obtain a tomographic image inside the living body like OCT, but is for detecting the reflected light from the subject surface that is not attenuated. , The sensitivity margin is much larger than that of OCT. Therefore, slits or pinholes are installed on the end face 51 of the optical fiber 11 (see FIG. 3-1) to increase the resolution in the direction perpendicular to the paper surface of FIG. 1, or the repetition frequency is increased to increase the number of lines in the scanning direction. It is easier to do than OCT.

3. Numerical verification of miniaturization and high definition of the imaging unit ・ The numerical verification of the possibility of miniaturization and high definition of the imaging unit is described below.
As one of the factors that determine the diameter of a small-diameter endoscope, there is concern about the size of the scanning mechanism, but with advances in fine processing in recent years, 0.9 mmφ micromotors (for OCT), galvanomotors, Looking at the situation where voice coils, MEMS mirrors, etc. are generally present and further miniaturization is progressing, the critical factors in reducing the diameter are the resolution and work distance (focusing point) depending on the application of the thin endoscope. The aperture size of the optical system is determined by the distance). The aperture size of the optical system according to the application will be described below.

-Now, in the endoscope device 1 according to the embodiment of the present invention, a resolution of 20 μm in the direction perpendicular to the line 15 is obtained, and the resolution in the direction of the line 15 by the OCI process is average (resolution of the line center at θ = 45 degrees). ) When 20 μm is obtained, according to the sampling theorem, HD imaging with 2000 × 1000 pixels is possible in a subject area of 20 × 10 mm. In addition, for a subject area of 40 × 20 mm, 4K imaging with 4000 × 2000 pixels becomes possible. Obtain the aperture angle of the optical system required for this 20 μm resolution, and calculate the focal length (work distance) and depth of focus at that aperture angle from the aperture size of the optical system limited by the diameter of the thin endoscope. Verifies whether or not it can be applied to an endoscopic device.

-Now, if the aperture of the optical fiber 11 in the direction perpendicular to the paper surface of FIG. 3A is narrowed to 6 μm using a slit and the resolution of the optical system 12 is set to 14 μm, the direction perpendicular to the paper surface of FIG. Since the resolution of is the convolution integral of each PSF, about 20 μm can be obtained. The total resolution δ of illumination (illumination light) and light reception at this time can be approximated by the equation δ = 0.37 × λ / NA (= sin α) (the difference between circular and rectangular apertures is approximated). In order to realize the resolution of the system 12 of 14 μm, when the center wavelength λ is 550 nm, the aperture angle α needs to be 0.83 degrees. The following verification will be performed based on this opening angle of 0.83 degrees.

3-1. Verification of 1.5 mmφ HD endoscope ・ Now, in the endoscope 301 having a small diameter of 1.5 mmφ, the size of the portion of the free curved surface optical system 12 having the largest aperture is 1 mm. Therefore, the value of the part with the longest focal length (a + b in Fig. 3-1) is 35 mm when the aperture angle is 0.83 degrees. The depth of focus γ is 5.9 mm from the formula of γ = 0.565 · λc / sin ² (α / 2), but when the position and direction of the image are determined so that the focus will be on the center of the screen, and rotational scanning is performed. The amount by which the edge of the screen in the scanning direction shifts from the focus position is 0.35 mm, so it falls within the depth of field range.

As shown in FIGS. 7 to 10, the focal length of 35 mm is a work distance (positioning or treatment device when using the small-diameter endoscope 301 through the forceps port 303 (2 to 3 mmφ) of the parent endoscope 307. There is no problem in space when using. While monitoring the normal observation of the endoscope 301, the small-diameter endoscope 301 of the present embodiment is directed to a portion to be magnified and observed through the forceps opening 303, as shown in FIGS. It is possible to magnify an area with an HD image. Reference numeral 305 in the figure indicates a flexible portion.

・ When the work distance is doubled to 70 mm, if the aperture angle of the optical system is set to 0.4 degrees and the resolution on the subject is set to 40 μ, it is possible to magnify and observe a 40 × 20 mm area with an HD image.

3-2.2 Verification of 4 mm endoscope of 2 mmφ ・ Next, an example of applying a 4K image to a small diameter endoscope of 2 mmφ will be described. When the aperture of the optical system is 1.5 mm and the aperture angle is 0.83 degrees, the focal length f is 52 mm, and 4K imaging is possible in a subject area of 40 × 20 mm. The depth of focus is 5.9 mm because the aperture angle is the same. When rotating and scanning with the focus on the center of the image, the edge of the screen in the scanning direction deviates from the focus position by 1.0 mm, so the depth of field is within 5.9 mm.

3-3. Verification of 4 mmφ 8K endoscope-Next, an example in the case of being applied to an endoscope having a margin in diameter such as a laparoscope will be described. Now, in a 4 mmφ laparoscope, when the aperture of the optical system is 3 mm and the aperture angle is also 0.83 degrees, the focal length f is 104 mm, and 8K imaging is possible in a subject area of 80 × 40 mm. The depth of focus is 5.9 mm because the aperture angle is the same. When rotating and scanning with the focus on the center of the image, the edge of the screen in the scanning direction deviates from the focus position by 2.0 mm, so it falls within the range of 5.9 mm depth of field.

  This focal length of 104 mm is not a problem in work distance (space for positioning and using treatment equipment) when using a small-diameter endoscope through the forceps port (2 to 3 mmφ) of the parent endoscope. Don't get up

  As mentioned above, numerical verification was shown by giving some application examples regarding miniaturization and high definition of the imaging unit of the endoscope, but there are many other application examples depending on the use of the endoscope. Needless to say. In addition, the application examples shown in 3-1 to 3-3 above and 3-7 described later can be realized with one endoscope by combining with an optical mechanism such as autofocus, diaphragm, and zoom. It is possible to variably set the resolution, the focal length, and the imaging range according to the application.

3-4. Repetition frequency of OCI process The number of pixels in the scanning direction of the imaging method of the present embodiment is not determined only by the resolution of the optical system, but is limited by the repetition frequency of the OCI process performed during scanning. The repetition frequency is determined by the trade-off with sensitivity. The current OCT repetition frequency has an upper limit of about 100 kHz, but when the present invention is applied to an HD image, the horizontal scanning frequency of the HD image is 67.5 kHz, and therefore the repetition frequency does not limit the frequency. In the case of a 4K image, the horizontal scanning frequency is 135 kHz, but as described above, since the OCI processing has a sufficient margin of sensitivity as compared with OCT, it is possible to raise the repetition frequency to 135 kHz.

3-5. Focusing of small-diameter endoscope Since the embodiment described above is for high-definition magnified imaging, when trying to focus on the entire screen, the three-dimensional position (x, y, z) of the imaging unit and the imaging direction (Xθ). , Yθ) becomes difficult to align. The alignment method will be described below.

The focal length can be changed by reciprocally scanning the end face 51 of the optical fiber 11 in FIG. 3-2 in the direction of the rotary scanning axis 18 or changing the distance between the image pickup unit and the subject S by reciprocal scanning. The peak value of the high-frequency component is detected by the high-pass filter for the line blocks in the scanning direction at three or more places in the screen while changing the. Since the positions of the peaks are the in-focus positions at the three positions, the optimum three-dimensional position (x, y, z) of the imaging unit with respect to the object plane and the imaging direction (xθ, yθ) are obtained from the data of the peak positions at the three positions. ) Can be calculated. From this x, y, z, xθ, yθ data, it is possible to display the navigation information of the operation for focusing on the monitor. Alternatively, if the flexible part of the small-diameter endoscope has a structure similar to that of a robot manipulator, it becomes possible to automatically perform alignment of the imaging unit with respect to the subject S, and with respect to loose movement of the subject S. Focus tracking becomes possible.

3-6. Blurring correction of a small-diameter endoscope When high-resolution magnified imaging is performed, the deterioration of resolution (blur) due to minute vibrations becomes a problem in observation. Since the image pickup method of the endoscope apparatus of the present invention is not the exposure at the frame repeat time as in the image sensor, but the exposure at the line repeat scan time, a minute and fast motion such as vibration is detected. Can be corrected. The means for correcting blur will be described below.

As shown in FIG. 11, the image is divided into blocks 41 of about 8 × 8 pixels, and the two-dimensional correlator 43 performs the two-dimensional correlation with the previous frame stored in the frame memory 551 for each block. The block 42 in the range where the two-dimensional correlation of the previous frame is performed is set in advance assuming the range of blur. The block 41 of 8 × 8 pixels in the current frame has the number of pixels and the number of pixels required in consideration of the situation where the blurring in the screen 351 is not uniform (uneven blurring due to partial movement of the subject or movement of the imaging unit). The number is set in the frame.

  The motion vector of the blur of the block 41 is detected by the two-dimensional correlator 43 and stored in the plural line memory 44 prepared for the number of vertical blocks in the screen 351. Next, in order to calculate the motion vector for each pixel of the previous frame, the motion vector around the pixel of interest is read from the multi-line memory 44, interpolation is performed by the interpolation circuit 45, and the motion vector for each pixel is calculated. , A predetermined read address is given (49) and stored in the multi-line memory 44. From this motion vector, the luminance value corresponding to the position where the pixel has moved due to blurring is calculated by the interpolation circuit 46 from the peripheral pixels of the current frame, and the pixel is returned to the position of the pixel of the previous frame to be displayed. A missing image is generated. This also enables tracking of the subject.

3-7. Microscopic Observation Using a Small-Diameter Endoscope Next, an example of microscopic magnifying observation using the endoscope apparatus of the present invention will be described.

An example of capturing a 3 × 2 mm small area with an HD image will be described with reference to FIGS. 12-1, 12-2, 13, and 14. FIG. 12 shows the configuration of a 2 mmφ small-diameter endoscope capable of microscopic observation. By using the free-form surface optical system 52 and the mirror 53, the optical paths of illumination and light reception are folded back in the imaging unit 56, and the convergence line 55 is set to be close to the outer diameter of the thin endoscope. In FIG. 12-1, the mirror 53 and the optical system 52 are described separately to show the principle of illumination, but they can be integrally molded as a free-form surface prism. Thereby, the tip structure can be further downsized. Further, as shown in FIG. 12-2, a free-form surface prism 521 and a GRIN lens 523 may be combined.

With this structure, as shown in FIGS. 13 and 14, it is possible to bring the subject into contact with the subject for imaging, and it is possible to focus on the entire screen. In addition, it becomes possible to perform observation without blur even by microscopic magnifying imaging. Even if blurring occurs, it can be removed by the blurring correction shown in Section 3-6. When the scanning angle when pressed against the subject is 120 degrees, the circumferential length in contact with the subject is 2 mm, so that it is possible to detect an imaging area of 3 × 2 mm. Reference numeral 301 in FIG. 13 is a 2 mmφ small-diameter endoscope, 305 a is a spring-shaped flexible portion, and 305 b is a wire-controlled flexible portion. Reference numeral 301 in FIG. 14 is a 2 mmφ small-diameter endoscope, 341 is a superficial mucosa, 343 is a contact portion of 2 mm, and β is a scanning angle of 120 degrees.

  As mentioned above, since the OCI process has a margin of sensitivity, if a 2 μm slit is installed on the end face of the optical fiber and the average resolution in the direction perpendicular to the paper surface of the optical system 52 (see Figure 12-1) is set to 2 μm. The resolution in the direction perpendicular to the paper surface of FIG. 14 is about 2 μm by convolution integration. When the illumination light is in the visible light band (λc = 550 nm, Δλ = 250 nm), the resolution of the optical system is 2 μm, but the resolution in the direction of line 55 (see Figure 12-1) is the same as the resolution of the optical system and the pinhole. Since the total resolution is 4 μm by convolution integration with the aperture of, it is possible to image a small area of 3 × 2 mm with the number of pixels of 1500 × 1000. In order to set the resolution of the optical system 52 at this time to 2 μm, it is necessary to set the aperture angle of the optical system 52 in the direction perpendicular to the paper surface (see FIG. 12-1) to 5.8 degrees. If the size of the part of the optical system 52 in Fig. 12-1 that has the largest opening in the direction perpendicular to the paper surface is 1.5 mm, then the focal length will be 7.4 mm, and the focal length a + b (7.4 mm in the configuration shown in Fig. 12-1. ) Can be matched.

4. Spectral Analysis by the Endoscope Device of the Present Invention Next, a spectrum analysis utilizing the endoscopic device of the present invention will be described. First, (4-1.) Describes the significance of multispectral analysis based on the ability of the human eye to recognize the spectrum of light, and then (4-2.) Describes the nature of the reflection spectrum of living organisms and the possibility of application. Then, (4-3.) Issues in applying the multispectral analysis to diagnosis will be described, and finally (4-4.) Examples of the multispectral analysis by the endoscope apparatus of the present invention will be described.

4-1. Significance of multispectral analysis-70% of the cerebral cortex is involved in visual function, and the human visual function exerts a high ability by moving the eyes and the brain together. For example, the high-definition visual field (fovea centralis) of the human eye has a range of only 2 degrees, and the resolution of the other visual fields decreases rapidly toward the outside. By swiftly scanning the part where the eye gazes and synthesizing the obtained images in the brain, it is possible to capture the entire visual field as if observing it in high definition. At that time, the pupil is adjusted at high speed according to the average brightness of the high-definition visual field of 2 degrees to obtain an image with a high dynamic range over the entire visual field. The human visual function has a much higher resolution and dynamic range than existing imaging devices, and has a high ability to recognize the shape and texture of a subject and the movement of the subject. Indeed, the visual function of human beings is a function that can be seen when viewed.

・ However, regarding the spectrum of light, human vision has only the resolution of the three primary colors of X, Y, and Z, and its ability to identify a substance by recognizing the spectrum component of light is by no means high. When the principal component analysis of the spectral characteristics of the subject is performed, the cumulative contribution rate of the XYZ axes is high, and the spectral characteristics can be reproduced to some extent by Wiener estimation, but to identify various compositions and substances, the spectrum that changes rapidly A component (for example, a bright line spectrum of infrared spectroscopy or Raman spectroscopy) must be detected. The evolution process of the human eye may have been good with the resolution of the XYZ axes, but in modern times, the need for remote sensing to identify the composition of the subject at the same time as imaging is high, especially in medical treatment, and as one of the means therefor. Multispectral analysis that exceeds human color perception is expected.

  By using multi-spectral analysis to extract the optimal spectral component axis for each specified substance, the detected information is converted into color and texture information that can be visually recognized by the human eye and provided to assist diagnosis. be able to. Multispectral analysis at that time is not limited to the visible light band, and analysis that includes infrared and terahertz regions invisible to the human eye is required.

・ As mentioned above, the human visual function is that the brain and eyes are integrated and high performance is demonstrated, but there are parts where AI (artificial intelligence) is better than human brain, By effectively utilizing it for multispectral analysis, it is possible to enhance the discrimination power of substances. AI is better at learning a large amount of data in a short time and making judgments in a short time using a large amount of data. The main reason for this is that AI is able to learn in a short time at any time of day or night at the speed of electricity without fatigue, and because it does not forget the learning results (although forgetting is a human brain). It is said to be an important means for debugging, but if the field and scope of application are narrowed, there are many cases in which learning, analysis, and judgment using a large amount of various data are superior to humans.

  As described in the example of multispectral analysis in Section 4-5., In multispectral analysis that requires learning and identification of a large amount of data, effective use of AI can improve the ability to identify substances. . The imaging device aims for high definition and high dynamic range close to human eyes, and at the same time realizes spectrum analysis that exceeds human eyes in combination with AI, opening new usefulness.

4-2. Characteristics of Reflection Spectrum of Living Body and Possibility of Application Next, characteristics of reflection spectrum of living body and possibility of application will be described.
-Spectral components of reflected light are formed by reflection and absorption. Since the refractive index of living tissue is unevenly distributed due to granular, layered, or fibrous tissue, light entering the living body is repeatedly reflected and diffracted, and backscattering at that time is reflected. Become light. Rayleigh scattering is the main one.

  When the particle size is sufficiently small with respect to the wavelength, Rayleigh scattering becomes strong in inverse proportion to the fourth power of the wavelength. Although the spectral components of Rayleigh scattering change due to these factors, the refractive index distribution of biological tissue is not highly correlated with the type of tissue (for example, cancer or normal tissue), and even if the direction of illumination changes. Since the spectral component changes, the spectral component of Rayleigh scattering has poor stability (poor quantification) in identifying the subject substance. Rayleigh scattering is utilized for imaging the shape and texture of a subject by detecting the backscattering. The human visual brain covers this unstable information with the learning knowledge cultivated over many years, and is capable of highly recognizing the shape and texture of the subject without being aware of it.

・ However, as visible light band light is repeatedly scattered in the living body, the electron distribution in living organic molecules is resonantly oscillated by the vibration of the electric field of the light to excite energy, so that a part of the spectrum is absorbed. It In addition, a part of the energy is changed to molecular vibration, and the remaining energy is emitted as wavelength-shifted fluorescence (spontaneous fluorescence). This changes the component ratio of the reflection spectrum. These changes strongly reflect the structure of organic molecules and have high stability and SN in identifying substances. However, the reflected light from a living body adds unstable Rayleigh scattering as background light to the absorption spectrum, and as a result, stability and SN do not increase in identifying a substance, and it has been actively used in the past. I haven't come.

・ When the bandwidth is wide like the XYZ characteristics of the human eye, the unstable spectrum of Rayleigh scattering that changes depending on the illumination and observation conditions is made uniform to some extent, and the stability as a color and the SN ratio are improved. It is considered that this is one of the reasons why the spectral resolution of the human eye is low.
As will be described later, if the detection band is optimized for each substance specified by multivariate analysis, and if it is combined with AI identification, the background light of Rayleigh scattering is suppressed, and stability and accuracy in specifying the substance can be reduced. Can be improved.

-Biological substances that absorb visible light and change spectral components include collagen (absorbs and emits autofluorescence), blood hemoglobin with different oxygen saturation concentrations, melanin pigments, and components contained in blood, such as glucose. Although there are many, there is a possibility that the properties of the tissue can be specified by performing multispectral analysis of these absorption spectrum components.

  It is said that cancer has a different oxygen saturation concentration due to vigorous growth, because the tissue is disordered and there is little absorption and fluorescence due to collagen. In addition, a number of dyes (probes) that attach to tissues such as cancer and strongly show absorption and fluorescence in the visible light band have been developed, and the spectral changes due to the dyes are efficiently expressed by multispectral analysis, resulting in high accuracy. Tissue identification can be performed.

  For example, ICG (Indocyanine Green) dye that stains tissue has stronger absorption in the near infrared region than in visible light, so by detecting the optimal band by multivariate analysis and displaying in color, The amount of weakly toxic ICG used can be reduced to enable highly accurate substance identification and staining.

・ Mid-infrared to far-infrared light has a longer wavelength than visible light and is easily transmitted because of less scattering, but strong absorption occurs by exciting molecular vibrations and spins with a low resonance frequency during transmission. , The actual transmittance drops. In particular, the absorption of water and hemoglobin, which are abundant in the living body, is large, so the transmittance is reduced. However, since the absorption to the organic molecule is strong, the stability of the spectrum is high in specifying the organic substance. There is a band called the fingerprint area, and infrared spectroscopy has been established.

  However, infrared spectroscopy is suitable for detection with microscopes that transmit thin analytes, and when the reflection method used for in vivo detection is used, Rayleigh scattering is mixed as in the visible light band to identify substances. Therefore, stability and SN are not so high.

・ In the near-infrared region, the situation is the same, but in hemoglobin and water, there are few valley wavelength regions (biologically transparent windows) where absorption due to energy excitation by scattering or electrons or molecules occurs, and tomographic image detection by OCT or Various applications have been made, such as detection of oxygen saturation concentration in blood and blood vessel detection using ICG (indocyanine green) fluorescence.

-As described above, there are staining and fluorescence using a property (probe) that a specific dye strongly binds to a specific biomolecule, and staining and fluorescence using a substrate that develops color by reacting with a specific enzyme. . Staining is due to the absorption spectrum. Rayleigh scattered light is mixed in the reflected light, but because of strong absorption, stability and SN are high. In fluorescence, the energy of molecules is excited by excitation light, part of the excitation light is absorbed as intermolecular vibration, and the remaining energy is emitted as light. A wavelength shift corresponding to the absorbed energy occurs, and this can be used to remove the Rayleigh scattering of the excitation light by the spectral filter.

  Although the fluorescence is weak, it is much larger than the Raman scattered light described later, and has high stability and SN for identifying a substance. This is a spectroscopic method suitable for reflection type detection. There are many stains and fluorescent dyes such as ICG (Indocyanine Green), 5-ALA, BBG, triamcinolone acetonide, and fluorescein mentioned above.In addition, the fluorescent probe that attaches to cancer and the attached cancer die when near infrared rays are applied. Various new reagents are being developed, including those that add photodynamic therapy to fluorescent probes and those that add photoimmunotherapy that kills cancer that has translocated due to the immune effect on the remains of cancer that has been killed by photodynamic therapy. There is.

・ In addition, although the signal energy is extremely small compared to the excitation light (10 ⁻⁶ ), absorption and emission of light by the Raman effect occur. This is because when light is scattered by a molecule, part of the energy of that photon is lost to the vibration energy of the molecule (may be rotation / electron energy), or the vibration energy of the molecule is added to the energy of the light wave. It is caused by being done. Since the amount of energy to be added and subtracted is constant depending on the energy level, a Raman spectrum that is a constant wave number shift (Stokes shift) is generated from the pump light regardless of the wavelength of the pump light. Raman spectra are used in microscopes and the like as Raman spectroscopy for identifying molecular structures and the like. Since the Raman spectrum has different wavelengths by the amount shifted from the excitation light, Rayleigh scattering of the excitation light can be removed by a spectral filter, and this is a spectroscopic method suitable for reflection type detection. However, since the signal energy is extremely small, the complete image time is sacrificed.

4-3. Means to identify biological material from multi-spectrum ・ As described in Section 4-2., The spectrum reflected from living body is visible fluorescence or Raman light that can remove Rayleigh scattering. Except for, Rayleigh scattering is mixed as background light (noise) in the change of spectral components due to absorption of electrons and molecules, so that stability and SN are not generally high in identifying substances. Therefore, statistical analysis and ambiguity determination methods are required for spectrum analysis.

  Obtaining a large amount of spectral component data and performing multivariate analysis in a space with the spectral components as multidimensional orthogonal axes shows that, for example, there are many cases in which clusters of normal cells and cancer cells can be separated and recognized. ing. Therefore, by using multivariate analysis with spectral components as independent variables, the optimal spectrum component axis for identification is extracted, and the cluster formed on the spectrum component axis is identified, and further non-linear carving and ambiguity determination are good. By identifying with AI (Deep Learning), it is possible to improve the accuracy of substance identification.

4-4. Procedure for Performing Multispectral Analysis and Required Functions Based on the above background, the procedure for performing multispectral analysis by the endoscope apparatus of the present invention will be described below.

First of all, in order to perform multivariate analysis, multi-spectral data necessary for specifying the target substance is acquired as much as possible by the endoscopic device of the present invention, and a tag or regularity that is consistent with definitive diagnosis (pathological examination etc.) is acquired. Information necessary for pre-processing (to reduce cluster variance) for optimization is added and accumulated.

-Next, a large amount of accumulated multi-spectral data is pre-processed by an off-line computer, and then multivariate analysis is performed to calculate and narrow down the spectrum component axis that is most suitable for specifying the target substance. Since the axes are calculated in descending order of contribution rate, the axis with high contribution rate is narrowed down as in data compression.

-Next, projective transformation of multispectral data is performed on the narrowed spectrum component axis, and using that data, AI is trained (supervised) on the computer to identify two clusters. The AI has the same structure as that incorporated in the endoscope apparatus of the present invention.

Next, the component values of the spectrum component axis (component values of the basis vector) narrowed down by the computer and the knowledge data learned by the AI (coefficients of the neurons of the hierarchical neural network forming the AI) are used as an internal view of the present invention. Send it to a mirror device for storage.

  Since the number of spectral component axes narrowed down by the multivariate analysis is at most 5 to 6, the circuit scale of the endoscope apparatus of the present invention can be simplified and the scale of AI can be reduced. The speed of AI judgment will be improved, and it will be possible to identify substances in vivo in real time.

From the above, the endoscopic device of the present invention has a function capable of acquiring and accumulating a large number of multispectral data in vivo, and a spectral component determined by multivariate analysis in vivo to identify a substance in real time. It is necessary to have two functions that can be performed together. An example of these two functions will be described in Section 4-5.

4-5. Examples of RGB image generation and multispectral analysis An example of generating an RGB image by the endoscope apparatus of the present invention and an example of the procedure of the above-described multispectral analysis are shown in FIGS. 15-1 and 15-2. , 16, 17 will be described.

First, an example of generating an RGB image will be described. The output of the light receiving line detector 67 shown in FIG. 15-1 is input to the FFT 61, and the Fourier transform is performed for the visible light band 80 shown in FIG. 16, and the W signal 81 shown in FIG. 16 is generated. When a part of the surface of the living body is drawn to some extent, Fourier transform is performed on the range including the near-infrared region 85 with good biological permeability shown in FIG. 16, and a W signal 81 is generated.

In parallel, the FFT 62 first performs the Fourier transform of the R band shown in FIG. 16 to generate the R signal 82. The R signal 82 is subjected to pixel interpolation by the interpolation memory unit 63 shown in FIG. 15A, and the W signal 81 and the time axis (pixel position) are synchronized. Then, the FFT 62 shown in FIG. 15-1 performs the Fourier transform of the B band shown in FIG. 16 to generate the B signal 83. Similarly, the B signal 83 is also subjected to pixel interpolation by the interpolation memory unit 64 shown in FIG. 15-1, and the W signal 81 and the pixel position are synchronized.

Next, these W, R and B signals are matrix-converted by the matrix converter 65 to generate RGB video signals.

The resolution of the R signal 82 and the B signal 83 is about 1/3 of that of the B signal 83 and the R signal 82 as compared with the W signal 81 because the wavelength bandwidth and the center wavelength change and the number of samples changes. Since the resolution of human eyes is also 1/3, there is no problem.

In this way, as shown in FIG. 16, the wavelength band is divided into ⅓, and the Fourier transform of the range corresponding to the divided R band and B band is performed to obtain the R signal 82 and the B signal. 83 can be generated. The principle can be considered that a broadband light source is composed of a linear sum of multiple light sources that divide the wavelength band, including R, G, B, and infrared light, and interferes with illumination, reflection, and reference light. , All of the signal processing of the spectrum and Fourier transform are linear processing, the time series signal of the part corresponding to the R or B band of the output of the light receiving line detector 67 is calculated from the principle of superposition in the linear system. By extracting and performing the Fourier transform, the same image as that obtained by performing the OCI process using the R and B independent light sources can be obtained. With the same idea, the multi-spectrum MS1 to MSn and the eigen spectra EU1 to EUn shown by reference numeral 84 in FIG. 16 are extracted by performing the Fourier transform by extracting the range of each band, and thereby the image on each spectrum axis is obtained. Obtainable.

  Next, a procedure and an example of multispectral analysis will be described. An example of identifying two tissues such as cancer and normal tissue or blood vessels and superficial mucosa will be described.

・ First, in order to find the optimum spectral component axis for distinguishing two tissues from the wide band from the visible light band to the far infrared region by multivariate analysis, the widebands in Figs. 6-1 and 6-2 are used. As the light source 21 and the wavelength swept light source 31, a broadband light source capable of covering the visible light band to the far infrared region is prepared. A wide band may be realized by dividing a wide band, preparing a plurality of light sources corresponding to it, or preparing a plurality of wavelength swept light sources, and combining their outputs by an optical coupler or the like. Alternatively, illumination may be performed by switching the above light source for each line scanning.

At this time, as the optical fiber 11 in FIGS. 3-1 to 3-3, a broadband optical fiber such as a polycrystalline fiber is used in order to pass light in a wide band from visible light to far infrared. Alternatively, the wavelength band may be divided and a plurality of fibers corresponding to the respective bands may be stacked and used. In that case, the positional deviation of the image caused by the rotational scanning is corrected by a plurality of line memories and pixel interpolation.

Also, in order to make the light-receiving line detector 67 of FIG. 15-1 correspond to a wide wavelength band from visible light to far infrared, as shown in FIG. 15-2 (a), the wavelength band is a multi-plate spectral prism. Divide by 91 and split the divided band by spectroscopes 92, 93, 94, and then use the appropriate coefficients k1, k2, k3 for the output of the photo detectors 95, 96, 97 that have the optimum sensitivity for each band. You may multiply by and synthesize | combine. If a wavelength swept light source corresponding to the divided bands can be prepared, the spectroscopes 92, 93, 94 and line detectors 95, 96, 97 shown in Fig. 15-2 (a) are unnecessary, and sensitivity is set for each divided wavelength band. It is also possible to prepare a plurality of light receiving detectors that have the same and multiply their outputs by appropriate coefficients k1, k2 and k3 to synthesize them.

  Magnification adjustment and sensitivity correction to combine the divided bands are performed in accordance with the outputs of the photodetection line detectors 95, 96, and 97, as shown in Fig. 15-2 (b). , K3 are read from the memory in the control unit 71 of FIG. 15-1, multiplied by multipliers 98, 99 and 100, and added by an adder 301 to perform synthesis and correction. The values of the coefficients k1, k2, and k3 include correction values for the spectral characteristic of illumination and the light receiving characteristic of the line detector. These values are constantly measured or measured in advance and reflected in the values of k1, k2, and k3. In FIG. 15-2 (b), reference numeral 601 is a CMOS line detector, 603 is an InGaAs line detector, 605 is a bolometer line detector, 607 is a synthesized band, 609 is a visible to far infrared region, and 611 is a mid infrared ray. A region 613 indicates a far infrared region.

In order to acquire the multispectral data MS1 to MSn (reference numeral 84 in FIG. 16) for multivariate analysis, first, the site to be acquired on the RGB image displayed on the monitor is designated by an input means such as a mouse. A gate signal 76 (see FIG. 15-1) for each line corresponding to the designated area is generated by a personal computer, sent to the control unit 71, and stored therein. Next, the multi-spectrum MS1 to MSn84 output from the light receiving line detector 67 is converted into a line image by the FFT 68 of FIG. 15-1 and stored in the memory unit 69. Then, according to the gate signal 76 sent from the control unit 71 from the line image, images of the multi-spectrum MS1 to MSn84 of the target portion are extracted.

Next, in the data format creation unit 70, a tag indicating cancer and normal tissue, which is consistent with a definitive diagnosis (by pathological examination, etc.), a tag indicating the type of cancer, malignancy, progression, etc., and data on the spectral characteristics of illumination. Information necessary for preprocessing is attached to the images of multispectral MS1 to MSn84 and stored. Follow the same procedure for normal tissue. In this way, as many data of multispectral MS1 to MSn84 of cancer and normal part as possible are acquired and accumulated for each case.

-In multispectral analysis, there is a trade-off between the spectral resolution and the spectral image resolution. Not only the image pickup method of the present invention but also any detection method, a trade-off relationship including the complete image time is established. If you apply multispectral analysis to texture enhancement or contour identification, put emphasis on the resolution of the spectral image, and if you place emphasis on the accuracy of substance identification, the spectral resolution. Will increase the number of. In this way, the number of multispectrals MS1 to MSn, the center wavelength, and each bandwidth are appropriately set according to the application purpose.

  The multispectral MS1 to MSn used for the multivariate analysis should have the bandwidth divided into as many pieces as possible, so the resolution of the spectrum should be emphasized, but in order to specify and extract the target site from the image, Since some resolution of the spectrum image is required, balance the settings according to the application purpose.

・ Next, send a large amount of multispectral data MS1 to MSn84 of cancer and normal cells detected and accumulated by the above to an off-line external computer (not shown in Fig. 15-1) to measure the spectral characteristics of the illumination, the sensitivity of the device, and the spectrum. Perform necessary normalization such as the average brightness of. The preprocessing of normalization is important to suppress the cluster variance and improve the identification accuracy.

-Next, multivariate analysis is performed using the normalized data. When the spectra of cancer and normal cells are displayed in a space having multispectral components as multidimensional orthogonal axes, for example, as shown in FIG. 17, the coordinates of the spectrum components of cancer and normal cells are represented by two clusters. . If the spectral characteristics of the illumination, the sensitivity of the device, etc. are normalized by the preprocessing, the cluster dispersion will be caused mainly by the instability of the superimposed Rayleigh scattering. In order to distinguish the two clusters from each other, the projection space having the smallest variance and the largest distance between the clusters (the largest Fisher ratio) is obtained by orthogonal transformation. In FIG. 17, reference numeral 361 denotes a cancer cluster, 362 denotes a normal tissue cluster, 363 denotes a cancer cluster projection, 364 denotes a normal tissue cluster group projection, and Z denotes a non-linear division by AI.

  For example, when the projective space is obtained by the FS (Foley-Sammon) transformation, the projective axes (eigenvectors) are calculated in descending order of Fisher ratio. By looking at the projection on the upper axis and the projection result on the plane combining the upper axes, it is grasped whether or not the two clusters can be separated and recognized, and the upper axes EU1 to EUn having a large contribution rate are selected. From experience, the number of EU1 to EUn can be limited to 5 to 6 or less at most, regardless of the application purpose.

-Next, using a large amount of data that re-projects two clusters on the axes EU1 to EUn, the AI on the computer (same configuration as AI75 in Figure 15-1) has a teacher to identify the clusters. Let them learn. AI is composed of a multi-layered neural network capable of learning by the Deep Learning method. Then, the knowledge data (neuron coefficient) obtained there is sent to the memory of the AI75 in FIG. 15-1 and stored therein. Then, to improve the accuracy of identification (elimination of local minimum) and reduce the scale (number of layers) of AI75, learning is performed for each identification purpose, and the knowledge data (neuron coefficient) is switched from the control unit 71. The signal 77 is used by being switched for each identification purpose.

At the same time, the base vector components of EU1 to EUn are sent from the external computer to the control unit 71 and stored in the memory of the control unit 71 as the coefficients 72-1 to 72-n of the multipliers 73-1 to 73-n. Let Then, through the same procedure, the coefficients 72-1 to 72-n corresponding to the identification purpose are stored in the memory in the control unit 71. The values of the coefficients 72-1 to 72-n are constantly monitored for the spectral characteristics of the illumination light and the correction values thereof are included.

Then, the coefficients 72-1 to 72-n are read from the memory of the control unit 71 in accordance with the time-series spectral components that are the output of the light receiving line detector 67, and are multiplied by the multipliers 73-1 to 73-n. As a result, it is possible to generate a spectrum component in which the reflection spectrum is projected on the axes EU1 to EUn in time series. Image signals for EU1 to EUn can be obtained by performing Fourier transform on the time-series signals using FFTs 74-1 to 74-n. Diagnosis support may be provided by assigning a pseudo color that is easy to distinguish between cancer and normal cells to the image of EU1 to EUn and displaying the image.

-Or, through the procedure described above, it becomes possible to input the image signals of EU1 to EUn to the learned AI75 and perform the discrimination between cancer and normal cells in real time at the pixel level. Depending on the degree of firing of the output signal identified by AI75, the hue of the RGB image of the cancer part is changed to express the identification result, the contour of the cancer is displayed, and the like, and the visual identification ability is high. Efficient diagnosis support may be provided by converting the expression. Even if the two clusters partially overlap with each other even though they are projected in the optimal spectral space for discrimination, for example, if the hue is changed according to the firing degree of AI75, the hue change density is high. The observer can judge at a glance that the part is likely to be the central part of the cancer.

-As described above, the spectrum analysis is described for the identification of two clusters such as cancer and normal tissue, but it is also possible to collectively identify (classify) a plurality of clusters by AI. However, narrowing down the feature axis by multivariate analysis becomes complicated, and the scale of hardware including AI becomes large.

  Therefore, as shown in 1 of Fig. 18, first, using the detected spectral components, two clusters of normal and cancer are discriminated (screening) for each region of the image, and then the cancer is detected. For the region where the determination is made, as shown in 2. of FIG. 18, the type of cancer and the grade of malignancy are determined from the combination of identification of two clusters, and then, as shown in 3. of FIG. It is possible to identify a plurality of clusters by sequentially performing the identification by the combination of the two clusters, such as determining the progress degree from the combination of the identifications of the two clusters. When the identifications are sequentially performed, the coefficients 66-1 to 66-n corresponding to EU1 to EUn, which are roughly stored, are read from the control unit 71 and used. The simple system shown in the example allows multiple clusters to be identified with little time penalty.

  Next, numerical verification of the resolution of a spectral image when the imaging method of the present invention is applied to an example of spectral diagnosis that is actually used in medical care will be described.

-For example, in the case of NBI (Narrow Band Imaging) diagnosis of an endoscope, a narrow band spectrum of 415 nm and 540 nm is used as an effective spectrum for separating clusters of superficial mucosa and blood vessels. By the FS conversion, it is assumed that these two narrow band spectra are extracted as a band representing the proper eigen axis for discrimination, and the resolution when the spectrum image is detected by the imaging method of the present invention, the central wavelength λc is 415 nm. The wavelength band width Δλ is 520 nm, the wavelength bandwidth Δλ is 40 nm, and the Fourier transform is performed to obtain 2.4 μm, which is sufficient in resolution as compared with 10 μm set in the verification of the above-described embodiment.

-Also, substances such as collagen, which are often present in normal mucous membranes, absorb light of 450 to 500 nm and emit autofluorescence of 500 to 550 nm, but absorption and fluorescence do not occur in cancer tissue with disordered tissues. There is a method of diagnosing cancer by detecting this difference in absorption spectrum. Now, according to the FS conversion, a spectral component having a central wavelength λc of 475 nm and a wavelength bandwidth Δλ of 50 nm is optimal for identification, and when the spectral image is detected by the imaging method of the present invention, the resolution in the line direction becomes 2.8 μm. It can be seen that there is no problem in resolution even in this spectrum diagnosis.

-For a spectral component of dyeing with high stability and SN, a more vivid dyeing effect can be obtained by extracting the effective spectral component by multivariate analysis and displaying it in pseudo color. As a result, the amount of a weakly toxic reagent such as ICG used can be reduced. In the case of ICG, there is a large absorption band at 700 to 780 nm.According to FS conversion, the spectrum component with the center wavelength λc of 740 nm and the wavelength band width Δλ of 80 nm is optimal for identification, so the line direction at that time is When the resolution is calculated, it becomes 4.3 μm, which shows that there is no problem in resolution.

5. Example of Raman spectrum analysis Next, an example of Raman spectrum analysis to which the endoscope apparatus of the present invention is applied will be described.

・ As shown in Fig. 19 (a), the Raman spectrum shows a spectrum corresponding to the molecular structure of the analyte when the wavelength of the excitation light is shifted by a certain wave number (Stokes shift) regardless of the wavelength of the excitation light. Appear. As described in Section 5-2., Raman spectroscopy is stable in identifying substances in parallel with infrared spectroscopy, because the scattering of the excitation light that becomes the background light can be removed by using the wave number shift with an optical filter. This spectroscopic method is suitable for in vivo reflection method.

・ The problem of Raman spectroscopy is that the intensity of Raman spectroscopy is as small as 10 ^-6 with respect to the intensity of excitation light. However, when the subject is brought into contact with a metal thin film such as gold and irradiated with excitation light, the emission of Raman spectrum is greatly amplified by the SERS (Surface Enhanced Raman Scattering) effect. The SERS effect is that when light is applied to a metal thin film in which metal particles are distributed with a size and roughness of nanometer level, the electric field of light is concentrated at the closest tips between the metal particles, and a very strong vibration of the electric field occurs. Wake up and localized surface plasmons (LSP) occur locally. When the subject contacts the metal particles, the electrons resonate with the electric field of light to cause mutual electron transfer with the subject, and at that time, strong amplified emission of Raman spectrum occurs from the subject. Depending on the shape of the particles in the thin film and the wavelength of the excitation light, amplification of about 10 ⁴ to 10 ⁶ is possible for spontaneous emission. For the metal thin film and the particles, gold or silver, which has low electric resistance (energy loss), is used because it easily causes resonance vibration.

• An example of applying SERS to the contact-type small-diameter endoscope described in Section 3-7 will be described below. As shown in FIG. 12-1, a gold thin film is coated on the outside of the transparent cover 57 of the small-diameter endoscope. The surface of the gold thin film in contact with the subject is coated with gold fine particles with an appropriate size and arrangement so that the SERS effect is strongly exerted. The transparent cover on which the gold thin film is formed can be replaced for durability and infection prevention. The gold thin film has a grain arrangement that makes it transparent to visible light.

As shown in FIG. 20-1, as an illumination light source, an RGB light source 101 in the visible light band for obtaining an RGB image and a narrow band light source (Raman excitation light source) 102 for Raman light excitation are used. The narrow-band light source 102 uses near-infrared light which emits less spontaneous fluorescence (becomes background light) from a living body and can be processed separately from an RGB image.

・ The reason for using a narrow band light source instead of a laser light source for Raman excitation is described below. When the excitation light is close to monochromatic light like a laser, the Raman spectrum approaches the emission line spectrum, and the accuracy of identifying the substance is improved. When the pumping light has a bandwidth, the Raman spectrum line width becomes wide because it is convolved with the wavenumber band of the pumping light. As a result, the resolution of the spectrum, that is, the accuracy of identifying the substance is reduced, but the band of the Raman spectrum itself is increased, so that the resolution of the spectrum image is increased. If the emission line spectra of the two substances to be distinguished are separated from each other, the resolution of the Raman spectrum image can be increased by appropriately increasing the bandwidth of the excitation light. When the Raman spectrum analysis results are used for texture enhancement or contour extraction, the resolution of the spectral image is emphasized, and when the accuracy of substance identification is important, the resolution of the spectrum is emphasized. The bandwidth of the excitation light is set appropriately according to the purpose.

・ Now, when Raman spectrum excitation is performed using a narrow-band light source with a central wavelength of 780 nm and a bandwidth of 4 nm, the resolution of the spectrum is about 1/4 of the resolution when using a laser light source. The average bandwidth of the spectrum is about 16 nm, and the image resolution of Raman light is about 17 μm on average. When applied to the example in the section 3-7., The Raman spectrum image has 880,000 pixels, which is higher than the resolution of XGA (Extended Graphics Array) for the RGB image of HD.

-Next, the OCI process for Raman spectroscopy will be described. As shown in FIG. 20-1, the illumination light of the RGB light source 101 is split by the split coupler 103 and input to the circulator 104 and the multiplexing coupler 105. The multiplexing coupler 105 combines the illumination light for the RGB image and the excitation light for the Raman spectrum, and supplies it to the optical fiber 11 shown in FIG. 3 via the circulator 106.

The reflected light from the subject passes through the optical fiber 11 and the circulator 106, and is split into visible light and Raman spectrum by the demultiplexing coupler 107, the visible light signal is input to the interference coupler 108, and the Raman light signal is input to the interference coupler 109. Entered in.

The visible light input to the interference coupler 108 interferes with the RGB reference light 110, the Raman light excitation light is removed by the RGB pass filter 111, and then the visible light is input to the grating spectrometer 120 of FIG. 20-2. Then, the same processing as described in Section 4-5. Is performed and the RGB image is detected.

On the other hand, the Raman light input to the interference coupler 109 interferes with the Raman reference light 113 from the reflection plate 112, the excitation light is removed by the Raman light passing filter 114, and then the grating spectrum shown in FIG. 20-2. Input to the container 122. Here, since the Raman spectrum is shifted from the excitation light by a constant wave number, the excitation light cannot be used as the reference light in the OCI process for obtaining the Raman spectrum image. Therefore, the reflector 112 is coated with the same gold thin film 115 as the gold thin film of the transparent cover, and is further coated with the substance 116 that emits the same Raman spectrum as the subject. By applying excitation light to this substance 116, Raman reference light 113 is generated by the SERS (surface enhanced Raman scattering) effect. The reflecting plate 112 can be set by exchanging it according to a specified substance.

・ As shown in Fig. 20-2, after being converted into an electric signal by the photodetector line detector 123, it is adjusted to the time-series Raman spectrum signal as in the multispectral analysis described in Section 4-5. , Coefficients 124-1 to 124-n for the Raman spectrum are read out in time series from the memory in the control unit 125 and are multiplied by the multipliers 125-1 to 125-n to project on the EU1 to EUn axes for the Raman spectrum. The generated time series signal is generated. By Fourier transforming the time series signals by FFT 126-1 to 126-n, Raman spectrum images when projected onto EU1 to EUn can be obtained.

-By the way, Raman spectra have a database of spectra, and it is relatively easy to calculate EU1 to EUn for distinguishing two substances by multivariate analysis.

By inputting the image signals of EU1 to EUn to the already learned AI 127 through the procedure described above, it becomes possible to identify two substances in real time at the pixel level.
Other processes are the same as the multispectral analysis process in section 4-5.

-To ensure the stability of the SERS effect, the size and arrangement of the particles on the metal thin film must be uniform. Although improvements have been made due to advances in nanotechnology, they are not yet perfect. Such manufacturing fluctuations (larger cluster variance) can be suppressed by combining multivariate analysis and AI discrimination, as in the above-described multispectral analysis.

Next, an example will be described in which both the substance identification accuracy of the Raman spectrum and the image resolution are obtained. A wavelength swept light source is used as the Raman pumping light source 102 in FIG. 20-1. Now, when the wavelength sweep 69 of 780 to 820 nm, which has a high biological transmittance, is performed, the Raman spectrum of the biological material having the smallest wavenumber shift is swept 68 of 863 to 910 nm as shown in FIG. 19 (b). Is released. The Raman spectrum image resolution in this wavelength band is 11 μm on average. Since the Raman spectra of other biological substances shift to the longer wavelength side where the wave number is larger than this, the excitation light can be removed by the Raman light passing filter 114.

The Raman light that has interfered with the Raman reference light 113 is composed of an acousto-optic tunable filter, a liquid crystal tunable filter, etc. after the excitation light is removed by the Raman light passing filter 114, as shown in Fig. 20-3. The swept Raman spectra are separated by the band variable band pass filters (variable BPF) 131-1 to 131-n.

  After that, the photodetectors 132-1 to 132-n convert the signals into electric signals, the FFTs 133-1 to 133-n convert the signals into image signals, and then the matrix conversion unit 134 projects the image signals projected onto the axes EU1 to EUn. Is converted to. As the coefficient (n × n) 136 for matrix conversion, the one corresponding to the specified substance is read from the memory in the control unit 125 of FIG. 20-2 and set in the memory in the matrix conversion unit 134 to be used. It Then, the matrix-converted EU1 to EUn images are synchronized with the RGB image and the pixels are interpolated by a plurality of interpolation memories 135-1 to 135-n. By inputting the image signals of EU1 to EUn to the learned AI 137 through the procedure described above, it is possible to identify two substances in real time at the pixel level. Other processes are the same as the multispectral analysis process in section 4-5.

6. Multispectral Analysis by Stimulated Fluorescence Emission ・ Next, we will explain an example in which stimulated emission of fluorescent dye is used by using stimulated emission light, and the strong absorption spectrum that occurs at that time is utilized to identify the substance. . When excitation light is applied to the fluorescent dye or probe injected into the living body, spontaneous emission of fluorescence occurs, but when strong induced light is added to this, fluorescence with a phase and wavelength that matches the induced light is strongly emitted, and that much excitation occurs. Strong absorption of light occurs. The excitation light includes Rayleigh scattering, but the spectral component due to absorption is large, and therefore stability and SN are high in identifying a substance.

As shown in FIG. 21, the excitation lights of the RGB light source 141 and the fluorescence excitation light source 142 are combined by the multiplexing coupler 143 and then branched by the branching coupler 144, one of which passes through the circulator 145 and the reference light by the reflecting plate 146. 147 is obtained. The other side is combined with the fluorescence stimulated emission light source 157 by the multiplexing coupler 148, and is supplied to the optical fiber 11 of FIG. 3 via the circulator 149. The RGB light source 141 and the fluorescence excitation light source 142 are separated for the purpose of adjusting the intensity and band of light for fluorescence excitation and fluorescence stimulated emission separately from the illumination light for RGB.

The reflected light from the living body received by the optical fiber 11 (see FIG. 3-1) interferes with the reference light 147 by the interference coupler 150 (see FIG. 21), and the fluorescence is removed by the fluorescence removal filter 151. It is input to the grating spectroscope 66 of 15-1.

-The process for detecting the absorption spectrum amplified by the stimulated emission of fluorescence is the same as the multispectral analysis process in section 4-5.

7. Other Application Examples of the Present Invention Above, the application to the digestive system endoscope and the laparoscope was described, but the endoscope apparatus of the present invention is a high-definition small-diameter endoscope of 1 mm or less. Since it can be realized, it can be applied to various scopes. An example will be shown below, and a part thereof will be described.

(A) Intravascular scope Figures 22-1 and 22-2 show examples of simultaneously observing the inner wall of a blood vessel and observing a tomographic image.
The observation 175 of the inner wall of the blood vessel by OCI uses the visible light band, and the observation 177 of the tomographic image by OCT uses the near infrared band. The mirror 161 is a dichroic mirror that separates wavelength bands to be used. When performing multispectral analysis including the near infrared band, a liquid crystal mirror or the like is used as the mirror 161, and the illumination light is switched to perform multispectral analysis. 22-1, 22-2, and 22-3, reference numeral 171 indicates a blood vessel, and 173 indicates a plaque.

When forward vision is required, the wavefront 162 of the illumination light shown in FIG. 22-3 is obtained by forming the optical fiber 211 in two stages and detecting the phase difference of the received light signal as shown in FIG. 22-3. By detecting the upper position 163, a front-view image can be generated by a method similar to CG image composition. 3D display of OCI images is also possible. The phase difference may be detected by causing the output light of the two fibers to interfere with each other and detecting it from the frequency of the interference fringes, or by performing block correlation of the image generated from the light reception signals of the two fibers. . When the correlation is performed, the accuracy can be improved by the shadow generated by the oblique illumination. The signals received by the two optical fibers 211 are taken out of the scope through the two optical rotary joints.

(A) Scope for puncture needle Fig. 23 shows an example of a scope for cytodiagnosis. Since the inner diameter of the puncture needle (the transparent puncture needle 401 in this embodiment) is about 0.5 to 1 mm, the diameter of the scope to be inserted therein has no problem in manufacturing. The rotary scanning mechanism such as the cytology scope 403 is installed in the hub near the puncture needle. The strength of the puncture needle may be maintained by using a part of the puncture needle as a transparent member, but a transparent puncture needle made of plastic is commercially available, and a transparent puncture needle is used when the entire circumference is set as the imaging range. To do. The imaging surface of the puncture needle, which is in contact with the subject, may be coated with fine gold particles having an appropriate size and arrangement so that the SERS effect can be obtained. Further, as shown in FIG. 22, OCT may be used together to obtain a tomographic image. In FIG. 23, reference numeral 405 indicates a detection line and 407 indicates an affected area.

The thin scope and the puncture needle scope described above can be applied to a plurality of thin scopes such as a percutaneous blood vessel scope and a joint scope, as illustrated below.

(C) Bronchial scope, (D) Joint scope, (E) Indirect jaw scope,
(F) Spinal cord scope, (K) Ventricle scope,
(H) olfactory cleft, nasopharynx, sinus entrance scope, (K) middle ear scope,
(Ko) ear canal scope, (sa) intraocular scope, (shi) lacrimal duct scope,
(Su) Bladder / urethra scope, (S) Renal pelvic scope (percutaneous), (So) Breast duct scope,
(T) Kurd scope, (H) hysteroscope, (T) chest cavity scope,
(TE) Mediastinal scope, (TO) Biliary / pancreatic duct child scope (via upper gastrointestinal scope),
(D) Compact, high-definition advanced camera for surgical robots

1 optical fiber; 4 lines; 8 scanning mechanism;
11 optical fiber; 12 optical system; 13 mirror;
15 lines; 16 scanning mechanism; 17 reflection points; 18 rotation scanning axis;
21 Broadband light source; 22 Branch coupler; 23, 26 Circulator;
24 Reflector; 27 Interference coupler; 28 Grating spectrometer;
29 Light receiving line detector; 31 Wavelength swept light source; 32, 33 Branch coupler;
34 Photodetector; 41, 42 block; 43-dimensional correlator;
44 line memory; 45,46 interpolation circuit; 51 end face; 52 optical system;
53 mirrors; 55 lines; 56 imaging unit; 57 transparent cover;
63,64 Interpolation memory block; 65 Matrix converter;
66 Grating spectrometer; 67 Light receiving line detector; 69 Memory section;
70 Data format creation unit; 71 Control unit 73 Multiplier;
76 gate signal; 77 switching signal; 80 visible light band; 81 W signal;
82 R signal; 83 B signal; 85 near infrared region; 91 multi-plate spectral prism;
92, 93, 94 spectrometer
100 Multiplier; 101 Light Source; 301 Adder; 102 Raman Excitation Light Source;
103 Branch coupler; 104 Circulator; 105 Multiplex coupler;
106 Circular; 107 Demultiplexing coupler; 108,109 Interfering coupler;
111 Pass filter; 112 Reflector; 113 Raman reference beam 113;
114 Raman light pass filter; 115 Gold thin film; 116 Reference sample
120,122 Grating spectrometer
123 Light receiving line detector; 125 Control section;
125-1 to 125-n Multiplier; 131 Mirror; 132 Photodetector;
134 matrix converter; 135 interpolation memory; 141 light source;
142 Fluorescence excitation light source; 143 Multiplexing coupler; 144 Branching coupler;
145,149 Circulator; 146 Reflector; 147 Reference light;
148 Multiplexing coupler; 150 Interference coupler; 151 Fluorescence elimination filter;
157 Fluorescence stimulated emission light source; 161 Mirror; 162 Wavefront;
211 optical fiber;
401 transparent puncture needle; 403 cell test scope;
551 Frame memory; 521 Free-form surface prism; 523 Lens

Claims (21)

An optical fiber that emits the light supplied from the light source and receives the reflected light that returns from the subject,
An optical system having a cylindrical surface for generating, from the light emitted from the optical fiber, illumination light having a condensing line that is linearly condensed on the subject, and the illumination light generated by the cylindrical surface is , Has the same wavelength band at any condensing position in the longitudinal direction of the condensing line, and the optical path length from the one end face of the optical fiber to the condensing position is far from one end in the longitudinal direction of the condensing line. As described above, the optical system gradually becomes longer ,
An endoscope apparatus comprising: a scanning mechanism that scans the illumination light in a scanning direction that intersects the condensing line. The endoscope apparatus according to claim 1, wherein the scanning direction is orthogonal to the condensing line . The endoscope apparatus according to claim 1 or 2 , wherein the light source generates broadband light or broadband wavelength swept light. The endoscopic device according to claim 3 , wherein the light source generates excitation light for causing the subject to fluoresce. The optical system is the cylindrical surface where the focal length becomes longer as the optical path length gradually becomes longer, so that the illumination light converges constantly on the converging line, as the focal length becomes longer, opening endoscope apparatus according to any one of claims 1-4, characterized in that it has a larger said cylindrical surface. Said light source, the endoscope apparatus according to any one of claim 1 to 5, characterized in that to generate the excitation light for generating a Raman scattered light from the object. It said light source, the endoscope apparatus according to any one of claim 1 to 6, characterized in that to produce a stimulated emission light for fluorescence the subject. And interference coupler to interfere with reference light to said reflected light received by the optical fiber, to any one of claim 1 to 7, characterized in that it comprises a reflecting plate for generating the reference light The endoscopic device described. The endoscope apparatus according to claim 8 , wherein the reflector plate generates reference light by Raman scattered light. Relates to the aforementioned reflected light received by the optical fiber, by interfering the reference light, taking out a predetermined wavelength band component, performs Fourier transform, claim 8, characterized in that to generate an image at a predetermined wavelength band component Or the endoscope apparatus according to 9 . With respect to the reflected light received by the optical fiber, the reference light is interfered to extract a wavelength band component corresponding to the three primary colors, Fourier transform is performed, and an image signal of the three primary colors is generated. the endoscope apparatus according to basis, claim 1 0, characterized in that it comprises means for generating a RGB image signal, to the. Luminance signal component of the image signal of the three primary colors, the endoscope apparatus according to claim 1 1, characterized in that it is obtained by Fourier transform of the signal in the wavelength band including the near-infrared region. A plurality of spectral components are calculated from the reflection spectrum of a subject whose cluster is known in descending order of Fisher ratio, and the cluster is provided with a discriminating means for discriminating from the reflection spectrum of an unknown subject. The endoscope apparatus according to any one of claims 1 to 12 . It said identification means, the endoscope apparatus according to claim 1, wherein the use of AI to perform a deep learning. By correlating the image of the image and the current frame the previous frame, the detected blur amount of the image of the current frame, according to claim 1 to 1 4 in which the image of the current frame is characterized in that it is corrected The endoscope apparatus according to any one of claims. With respect to an image signal obtained from the reflected light from the subject due to the illumination light, a difference between the image signals adjacent in the scanning direction is acquired and accumulated, whereby an image of a moving part of the subject is drawn. The endoscope apparatus according to any one of claims 1 to 15 , characterized in that. The endoscope apparatus according to any one of claims 1 to 16 , wherein the reflected light is received by the optical fiber via the optical system. Wherein the optical system, the illumination light endoscope apparatus according to any one of claims 1 to 1 7, characterized in that it is focused linearly. From light emitted from the optical fiber, according to any one of claim 1 to 18, further comprising a second optical system for producing a tomographic image of the subject to spectroscopic near-infrared light Endoscopic device. An optical fiber that emits light supplied from a light source that generates broadband light and receives reflected light that returns from the subject,
An optical system having a free-form curved cylindrical surface for generating illumination light having a condensing line linearly condensed on the object from the light emitted from the optical fiber, the optical system being generated by the cylindrical surface. The illumination light converges in the same wavelength band at any condensing position in the longitudinal direction of the condensing line, and the optical path length from one end face of the optical fiber to the condensing position is one end in the longitudinal direction of the condensing line. The optical system that gradually becomes longer as it gets farther from the section ,
An interference coupler for interfering a reference light with the reflected light of the condensing line,
A spectroscope that disperses the output light of the interference coupler into wavelength components, and generates an interference fringe having a frequency proportional to the difference in the optical path lengths of the reference light and the reflected light over the dispersed wavelength band.
A light receiving line detector for converting the interference fringes into an electric signal,
Fourier transforming the electric signal of the interference fringes converted by the light receiving line detector, calculating the amplitude of each frequency component of the interference fringes, and detecting the one-dimensional image of the subject Fourier transform circuit,
A scanning mechanism that scans the illumination light in a scanning direction intersecting the condensing line while repeatedly detecting the image in the one-dimensional direction by the Fourier transform circuit to generate a two-dimensional image. Endoscope device. An optical fiber that emits light supplied from a light source that generates a broadband wavelength swept light and receives reflected light that returns from the subject,
An optical system having a free-form curved cylindrical surface for generating illumination light having a condensing line linearly condensed on the object from the light emitted from the optical fiber, the optical system being generated by the cylindrical surface. The illumination light converges in the same wavelength band at any condensing position in the longitudinal direction of the condensing line, and the optical path length from one end face of the optical fiber to the condensing position is one end in the longitudinal direction of the condensing line. The optical system that gradually becomes longer as it gets farther from the section ,
By interfering reference light with the reflected light of the condensing line, an interference coupler that generates an interference fringe having a frequency proportional to the difference in optical path length between the reference light and the reflected light,
A photodetector for converting the interference fringes into an electric signal,
Fourier transforming the electric signal of the interference fringes converted by the light receiving detector, calculating the amplitude of each frequency component of the interference fringes, and detecting the one-dimensional image of the subject Fourier transform circuit,
A scanning mechanism that scans the illumination light in a scanning direction intersecting the condensing line while repeatedly detecting the image in the one-dimensional direction by the Fourier transform circuit to generate a two-dimensional image. Endoscope device.

Images