JP2020049231A - Information processing apparatus and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To facilitate correspondence between a motion contrast image obtained by OCTA and structural information obtained by OCT or the like. SOLUTION: A subject image acquisition step of acquiring a subject image in a first predetermined range of a subject and a second predetermined range included in the first predetermined range are intentionally acquired with the same cross section. Between each frame in the signal acquisition step of acquiring multiple frames of interference signal sets for forming a three-dimensional tomographic image for a plurality of different cross sections, and in the multiple frames of interference signal sets constituting the same acquired cross section. In an image processing method having an OCTA image generation step of generating an OCTA image using the corresponding pixel data in, a display step of displaying the subject image and the generated OCTA image side by side, and the subject image and the OCTA image. When the information of one image is selected, a superimposed display step of superimposing the selected information on the area where the information is displayed in the other image is arranged. [Selection diagram] FIG. 8

Description

  The present invention relates to an image forming method and apparatus using optical coherence tomography.

  Optical coherence tomography (hereinafter referred to as OCT) has been put into practical use as a method for non-destructively and non-invasively acquiring a tomographic image of a measurement target such as a living body. In OCT, a tomographic image of the retina at the fundus of the eye to be inspected is obtained, particularly in an ophthalmic region, and is widely used in ophthalmic diagnosis of the retina.

  In OCT, a tomographic image is obtained by causing light reflected from a measurement target and light reflected from a reference mirror to interfere with each other and analyzing the time dependence or wave number dependence of the intensity of the interfered light. As such an optical coherence tomographic image acquisition device, a time domain OCT, a spectral domain OCT, or a wavelength sweeping OCT is known. In the time domain OCT, depth information of a measurement target is obtained by changing the position of a reference mirror. In a spectral domain optical coherence tomography (SD-OCT) using a broadband light source, interference light is separated for each wavelength by a spectroscope to obtain depth information of a measurement target. In a wavelength-swept optical coherence tomography (SS-OCT) apparatus, a variable wavelength light source apparatus capable of changing an oscillation wavelength is used. Note that SD-OCT and SS-OCT are collectively called (FD-OCT: Fourier Domain Optical Coherence Tomography).

  In recent years, a pseudo angiography using this FD-OCT has been proposed, and is called OCT angiography (OCTA) (Non-Patent Document 1). Fluorescence imaging, a common angiography method in modern clinical medicine, requires injection of a fluorescent dye (for example, fluorescein or indocyanine green) into the body, and two-dimensionally displays blood vessels through which the fluorescent dye passes. On the other hand, OCTA enables non-invasive pseudo angiography, and enables a three-dimensional display of a network of blood flow sites. Furthermore, OCTA has attracted attention because it has a higher resolution than fluorescence imaging and can render microvessels or blood flow in the fundus.

  A plurality of OCTA methods have been proposed depending on the difference in the blood flow detection method. For example, Patent Document 2 proposes a method of extracting an OCT signal from a blood flow by extracting only a signal in which time modulation has occurred from the OCT signal. Non-Patent Document 3 proposes a method using variation in phase due to blood flow, and Non-Patent Document 4 and Patent Document 1 proposes a method using variation in intensity due to blood flow.

USP Pub. No. 2014/221827 A1

Makita et al., “Optical Coherence Angiography,” Optics Express, 14 (17), 7821-7840 (2006) An et al. “Optical microangiography provides correlation between microstructure and microvasculature of optic nerve head in human subjects,” J. Biomed. Opt. 17, 116018 (2012) Fingler et al. “Mobility and transverse flow visualization using phase variance contrast with spectral domain optical coherence tomography” Optics Express. Vol. 15, No. 20.pp 12637-12653 (2007) Mariampillai et al., “Optimized speckle variance OCT imaging of microvasculature,” Optics Letters 35, 1257-1259 (2010)

  However, in the above-mentioned OCTA, since a detailed blood flow site can be obtained, the connection or travel of a specific blood vessel may be difficult for the examiner to understand. In addition, the image related to the blood flow region obtained by OCTA and the image related to the structural information obtained by OCT are obtained as independent images. Therefore, at the time of actual diagnosis, the examiner needs to alternately compare these two images. However, since the detailed blood flow site obtained by OCTA is detailed, it is difficult to understand the correspondence with the structural information of the subject's eye obtained from the OCT intensity image or the fundus photograph.

  The present invention has been made in view of such a situation, and has as its object to facilitate correspondence between an image relating to a blood flow site obtained by OCTA and structural information obtained by OCT or the like.

In order to solve the above-described problem, an image processing method according to the present invention includes:
A subject image obtaining step of obtaining a subject image in a first predetermined range of the subject,
In a second predetermined range included in the first predetermined range, an interference signal set for a plurality of frames obtained with the intention of the same cross section is obtained for a plurality of different cross sections for forming a three-dimensional tomographic image. A signal acquisition step;
A motion contrast image generation step of generating a motion contrast image using pixel data corresponding to each frame in an interference signal set for a plurality of frames acquired with the intention of the same cross section,
A display step of displaying the subject image and the generated motion contrast image side by side,
When selecting information of one of the subject image and the motion contrast image, a superimposed display step of superimposing and displaying the selected information in a region where the information is displayed in the other image. Features.

  According to the present invention, it is possible to easily obtain a correspondence between an image relating to a blood flow site obtained by OCTA and structural information obtained by OCT or the like.

It is a figure showing the outline of the whole composition of the device in a first embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a scan pattern according to the first embodiment. 5 is a flowchart illustrating an overall processing procedure of image generation according to the first embodiment. 5 is a flowchart illustrating an interference signal acquisition procedure according to the first embodiment. 5 is a flowchart illustrating a signal processing procedure according to the first embodiment. 5 is a flowchart illustrating a procedure for acquiring and displaying three-dimensional blood flow site information according to the first embodiment. FIG. 9 is an explanatory diagram of a segmentation result in the first embodiment. FIG. 3 is a diagram illustrating an example of a display screen according to the first embodiment. FIG. 5 is a diagram illustrating an example of a display method according to the first embodiment. FIG. 11 is a diagram illustrating an example of a depth selection method according to the second embodiment. FIG. 10 is a diagram illustrating an example of a method for generating a two-dimensional intensity image according to the second embodiment. FIG. 14 is a diagram illustrating an example of a method for displaying a selected part according to the second embodiment. 9 is a flowchart illustrating an example of a region division processing procedure. It is a figure showing an example of the display method which displays two images. It is a flowchart showing an example of the comparison procedure performed when estimating a blood vessel. It is a figure showing an example of the display method of the analysis result of blood vessel estimation. It is a figure showing an example of a display screen in a third embodiment. It is a figure showing an example of the display screen in a 4th embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention related to the claims, and all combinations of features described in the following embodiments are essential to the solution of the present invention. Not exclusively.

  Further, in this specification, an OCT signal displayed as an image is referred to as an intensity image. Furthermore, a signal in which a time-modulated signal is displayed from the OCT signal as an image is called a motion contrast image, its pixel value is called a motion contrast, and its data set is called a motion contrast data.

(First embodiment)
In the present embodiment, an example will be described in which a tomographic image is generated from a captured three-dimensional optical interference signal, motion contrast is calculated, and three-dimensional blood flow site information is obtained.

[Configuration of entire image forming apparatus]
FIG. 1 is a diagram illustrating a configuration example of an image forming method and apparatus using optical coherence tomography according to an embodiment of the present invention. The image forming method and apparatus include an OCT apparatus 100 that is an optical coherence tomography acquisition unit that acquires an optical coherence tomography signal and a control unit 143. As the OCT apparatus, for example, the above-described SD-OCT or SS-OCT is applicable to the present invention. The embodiment described below shows a configuration in a case where the OCT apparatus is SS-OCT.

<Configuration of OCT device 100>
The configuration of the OCT apparatus 100 will be described below with reference to FIG.
The light source 101 is a wavelength sweep type (Swept Source: SS) light source, and emits light while sweeping at a sweep center wavelength of 1050 nm and a sweep width of 100 nm, for example.

  Light emitted from the light source 101 is guided to the beam splitter 110 via the optical fiber 102, and is split into measurement light (also referred to as OCT measurement light) and reference light (also referred to as reference light corresponding to the OCT measurement light). . The splitting ratio of the beam splitter 110 is 90 (reference light): 10 (measurement light). The branched measurement light is emitted to the measurement optical path via the optical fiber 111. The measurement optical path includes a collimator lens 112, a galvano scanner 114, a scan lens 115, and a focus lens in order from the optical fiber 111 to the subject's eye 118. 116 is arranged.

  The measurement light emitted to the measurement optical path is made into parallel light by the collimator lens 112. The parallel measuring light enters the eye 118 via a galvano scanner 114, a scanning lens 115, and a focus lens 116 that scans the measuring light at the fundus Er of the eye 118. Here, the galvano scanner 114 is described as a single mirror, but is actually constituted by two galvano scanners (not shown) such as an X-axis scanner and a Y-axis scanner so as to raster-scan the fundus Er of the eye 118 to be inspected. ing.

  The focus lens 116 is fixed on the stage 117, and the focus of the focus lens 116 can be adjusted by moving the stage 117 in the optical axis direction. The galvano scanner 114 and the stage 117 are controlled by a signal acquisition control unit 145, which will be described later, and within a desired range of the fundus Er of the subject's eye 118 (also referred to as a tomographic image acquisition range, tomographic image acquisition position, and measurement light irradiation position). The measuring light can be scanned.

  Although not described in detail in the present embodiment, it is preferable that a tracking function for detecting the movement of the fundus Er and scanning the mirror of the galvano scanner 114 by following the movement of the fundus Er is provided. The tracking method can be performed using a general technique, and can be performed in real time or by post-processing.

  As a method of obtaining a fundus image for detecting the movement of the fundus Er, for example, there is a method using a scanning laser ophthalmoscope (SLO). In this method, an image in the plane perpendicular to the optical axis (fundus surface image) is acquired with time using an SLO with respect to an image of the fundus Er, and characteristic portions such as blood vessel bifurcations in the image are extracted. It is possible to perform real-time tracking by calculating how the characteristic portion in the acquired two-dimensional image has moved as a movement amount of the fundus Er, and feeding back the calculated movement amount to the galvano scanner 114.

  As described above, the measurement light is incident on the subject's eye 118 via the focus lens 116 on the stage 117, and is focused on the fundus Er by the focus lens 116. The measurement light irradiated to the fundus Er is reflected and scattered by each retinal layer, and returns to the beam splitter 110 via the above-described measurement optical path. The return light of the measurement light that has entered the beam splitter 110 enters the beam splitter 128 via the optical fiber 126.

  On the other hand, the reference light split by the beam splitter 110 is output to the reference light path via the optical fiber 119a, the polarization controller 150, and the optical fiber 119b. 120, a dispersion compensation glass 122, an ND filter 123, and a collimator lens 124 are arranged.

  The reference light emitted from the optical fiber 119b is collimated by the collimator lens 120. The polarization controller 150 can change the polarization of the reference light to a desired polarization state. The reference light enters the optical fiber 127 via the dispersion compensating glass 122, the ND filter 123, and the collimator lens 124. One end of the collimator lens 124 and one end of the optical fiber 127 are fixed on the coherence gate stage 125 so that the collimator lens 124 and the like are driven in the optical axis direction according to the difference in the axial length of the subject. , And is controlled by a signal acquisition control unit 145 described later. In the present embodiment, the optical path length of the reference light is changed. However, it is sufficient that the optical path length difference between the optical path of the measurement light and the optical path of the reference light can be changed.

  The reference light that has passed through the optical fiber 127 enters the beam splitter 128. In the beam splitter 128, the return light of the measurement light and the reference light described above are multiplexed to form interference light, and then split into two. The split interference light is interference light having phases inverted to each other (hereinafter, referred to as a positive component and a negative component). The positive component of the split interference light enters one input port of the detector 141 via the optical fiber 129. On the other hand, the negative component of the interference light enters the other input port of the detector 141 via the optical fiber 130. The detector 141 is a differential detector. When two interference signals having phases inverted by 180 ° are input, the detector 141 removes a DC component and outputs only an interference component.

  The interference signal detected by the detector 141 is output as an electric signal corresponding to the light intensity, and is input to a signal processing unit 144 which is an example of a tomographic image generation unit.

<Configuration of control unit 143>
The control unit 143 for controlling the entire apparatus will be described.
The control unit 143 includes a signal processing unit 144, a signal acquisition control unit 145, a display unit 146, and a display control unit 149. Further, the signal processing unit 144 further includes an image generation unit 147 and a map generation unit 148. The image generator 147 has a function of generating a luminance image and a motion contrast image from the transmitted electric signal, and the map generator 148 has a function of generating layer information (retinal segmentation) from the luminance image.

  The signal acquisition control unit 145 controls each unit such as the stage 117 and the coherence gate stage 125 as described above. The signal processing unit 144 generates an image, analyzes the generated image, and generates visible information of the analysis result based on the signal output from the detector 141.

  The image and the analysis result generated by the signal processing unit 144 are sent to the display control unit 149, and the display control unit 149 causes the display screen of the display unit 146 to display the image and the analysis result. Here, the display unit 146 is a display such as a liquid crystal display, for example. Note that the image data generated by the signal processing unit 144 may be sent to the display control unit 149 and then sent to the display unit 146 by wire or wirelessly. In the present embodiment, the display unit 146 and the like are included in the control unit 143. However, the present invention is not limited to this, and may be provided separately from the control unit 143. For example, an example of a device that can be carried by a user Tablet. In this case, it is preferable that a touch panel function be mounted on the display unit so that the display position of the image can be moved, enlarged / reduced, and the displayed image can be changed on the touch panel.

[Scan pattern]
<Scanning mode of measuring light in OCT device>
Here, a manner of scanning the measuring light in the OCT apparatus will be described. As described above, interference light is obtained from the return light of the measurement light applied to an arbitrary point on the fundus of the subject's eye 118 and the corresponding reference light. The signal processing unit 144 processes an electric signal according to the intensity of the interference light detected by the detector 141 to obtain image data in the depth direction at the arbitrary point. The above is the description of the process of acquiring information relating to a tomogram at one point of the subject's eye 118. Acquiring information about a tomographic slice in the depth direction of the subject's eye 118 in this manner is called A-scan.

  Further, acquiring information about a tomographic image of the subject's eye 118 in a direction orthogonal to A-scan, that is, acquiring information about a plurality of tomographic images in the depth direction described above in the scanning direction for acquiring a two-dimensional image is referred to as B-scan. Call. Further, acquiring information on a plurality of tomograms in the depth direction in a scanning direction orthogonal to both the A-scan and B-scan scanning directions is referred to as C-scan. When two-dimensional raster scanning is performed on the fundus oculi when acquiring a three-dimensional tomographic image, the high-speed scanning direction is the B-scan direction, and the low-speed scanning direction in which the B-scans are arranged in the orthogonal direction is the C-scan direction. Call. By performing A-scan and B-scan, a two-dimensional tomographic image can be obtained. By performing A-scan, B-scan, and C-scan, a three-dimensional tomographic image can be obtained. These B-scan and C-scan are performed by the galvano scanner 114 described above.

  As described above, the galvano scanner 114 is constituted by an X-axis scanner and a Y-axis scanner (not shown), each of which is constituted by a deflecting mirror whose rotation axis is orthogonal to each other. The X-axis scanner scans the measurement light in the X-axis direction on the fundus Er, for example, and the Y-axis scanner scans in the Y-axis direction. Each of the X-axis direction and the Y-axis direction is a direction perpendicular to the direction of the eye axis of the eyeball, and is a direction perpendicular to each other.

  Further, the line scanning direction such as B-scan and C-scan does not need to coincide with the X-axis direction or the Y-axis direction. For this reason, the B-scan and C-scan line scanning directions can be appropriately determined according to the two-dimensional tomographic image or the three-dimensional tomographic image to be captured.

<Scanning mode of measurement light in the present embodiment>
In OCTA, a time change of an OCT interference signal due to a blood flow is measured, so that multiple measurements are required at the same place. In the present embodiment, the OCT apparatus repeats the B scan (scan in the X-axis direction) at the same place m times, and moves the scan position to n y positions (positions where the measurement light irradiation position is moved in the Y-axis direction). Do the scan to move. FIG. 2 shows a specific scan pattern. As shown in the figure, in this embodiment, the B scan is repeatedly performed m times for each of n positions y1 to yn on the fundus plane.

  Here, if the value of m is large, the number of times of measurement at the same place increases, so that the accuracy of blood flow detection is improved. On the other hand, the scan time becomes longer, and there arises a problem that motion artifacts occur in the image due to the movement of the eyes (fine fixation) during the scan and a problem that the burden on the subject increases. In this embodiment, m = 4 in consideration of the balance between the two. The control unit 143 may automatically change m in accordance with the A-scan speed of the OCT apparatus and the motion analysis of the fundus oculi surface image of the subject's eye 118.

  In FIG. 2, p indicates the sampling number of the A scan in one B scan. That is, in one B scan, the A scan is performed at each of the positions x1 to xp, and the plane image size is determined by p × n. If the value of p × n is large, a wide range can be scanned with the same measurement pitch, but the scan time becomes long, and the above-described problems of motion artifact and patient burden occur. In this embodiment, n = p = 300 in consideration of the balance between the two. The values of n and p described above can be freely changed as needed.

  Further, Δx in FIG. 2 is an interval (x pitch) between adjacent x positions, and Δy is an interval (y pitch) between adjacent y positions. In the present embodiment, the x pitch and the y pitch are determined as の of the beam spot diameter of the irradiation light on the fundus, and are set to 10 μm. By setting the x pitch and the y pitch to ビ ー ム of the beam spot diameter on the fundus, a generated image can be formed with high definition. Note that even if the x pitch and the y pitch are smaller than 1/2 of the fundus beam spot diameter, the effect of further increasing the definition of the generated image is small.

Conversely, when the x pitch and the y pitch are larger than 1/2 of the fundus beam spot diameter, the definition deteriorates, but a wide range of images can be acquired with a small data capacity. However, the x pitch and the y pitch may be freely changed according to clinical requirements.
The scan range of this embodiment is p × Δx = 3 mm in the x direction and n × Δy = 3 mm in the y direction.

<Acquisition style of OCT interference signal and OCTA signal>
The acquisition of the OCT interference signal and the acquisition of the OCTA signal may be the same step or separate steps. Here, a case where acquisition of an OCT interference signal and acquisition of an OCTA signal are acquired in the same step will be described. When acquiring in the same process, the signal acquisition control unit 145 acquires an interference signal set for a plurality of frames used when forming a three-dimensional tomographic image by measuring the same position a plurality of times in scanning with the measurement light. I do. Here, the signal acquisition control unit 145 functions as a signal acquisition unit. The interference signal set is a set of interference signals obtained by the above-described one B scan, and means a set of interference signals capable of generating one frame of a tomographic image by the B scan. In addition, since the eye to be examined always performs fine fixation, even if the same position in the eye to be examined is scanned a plurality of times to obtain an interference signal set for a plurality of frames to form an image of the same cross section of the eye to be examined, actually the same It is difficult to obtain an accurate cross section. Therefore, the interference signal set for a plurality of frames described here is grasped as an interference signal set for a plurality of frames acquired with the intention of the same cross section.

  The signal processing unit 144 calculates the OCT interference signal and the OCTA signal by processing these interference signal sets. By using the interference signal set acquired in the same step, each pixel of the intensity image by OCT and the motion contrast image by OCTA can be at the same position. That is, in the present embodiment, the interference signal set used for generating the OCT intensity image is included in the interference signal set for a plurality of frames acquired to generate the motion contrast image by OCTA. Further, the OCT intensity image displayed as a two-dimensional image is averaged (superimposed) using at least two or more interference signal sets of the interference signal sets acquired for a plurality of frames, that is, two frames. May be obtained. With such a configuration, random noise included in the interference signal set is averaged, and noise in the intensity image can be reduced. Here, the signal processing unit 144 functions as a motion contrast image generation unit that generates a motion contrast image using pixel data corresponding to each frame in an interference signal set for a plurality of frames forming the same cross section.

  Next, a case where acquisition of an OCT interference signal and acquisition of an OCTA signal are acquired in separate steps will be described. In the case of separate steps, the signal acquisition control unit 145 causes the OCTA signal acquisition to be executed, for example, after the acquisition of the OCT interference signal. In the scan pattern at the time of acquisition of the OCT interference signal, values such as the number of repetitions and the pitch corresponding to the scan pattern are used. For example, the acquisition of the OCT interference signal does not necessarily need to repeat the B scan m times. Further, the scan range does not need to be the same between the acquisition of the OCT interference signal and the acquisition of the OCTA signal. Therefore, if the scan time is the same, the OCT interference signal can be obtained in a wider range as the first predetermined range. In the case of separate steps, OCTA signals are acquired in detail only in a predetermined region of interest as a second predetermined range included in the first predetermined range while observing a wide range with a three-dimensional tomographic image by OCT. Can be used.

  Next, a specific processing procedure of the image forming method according to the present embodiment will be described with reference to a flowchart illustrated in FIG. A detailed description of each process shown in the flow will be described later. FIG. 3 illustrates an example in which acquisition of an OCT interference signal and acquisition of an OCTA signal are performed in separate steps.

  In step S101, which is a first signal acquisition step, that is, an object image acquisition step displayed as a two-dimensional image, the signal acquisition control unit 145 controls the OCT apparatus 100 to acquire an optical coherence tomographic signal.

  In step S102, which is a second signal acquisition step, that is, a signal acquisition step, the signal acquisition control unit 145 controls the OCT apparatus 100 to acquire an optical coherence tomographic signal for a plurality of frames. More specifically, an interference signal set for a plurality of frames based on the measurement light controlled to scan the same position in the sub-scanning direction a plurality of times is acquired. Further, this operation is performed a plurality of times in order to obtain interference signal sets from a plurality of different cross sections while shifting the scanning position in the sub-scanning direction, thereby acquiring an interference signal set that can form a three-dimensional tomographic image.

  In step S103, which is an intensity image generation step, the control unit 143 calculates three-dimensional tomographic image data of the subject's eye 118 based on the optical coherence tomographic signal acquired in the first signal acquisition step, and generates an intensity image. . Further, the control unit 143 calculates three-dimensional motion contrast data from the optical coherence tomographic signals of a plurality of frames acquired in the second signal acquiring step, and generates a three-dimensional motion contrast image.

In step S104, which is a display step, the control unit 143 generates and displays display information from the intensity image and the three-dimensional motion contrast image. The control unit 143 reconstructs and redisplays the display information according to the operation of the examiner. The detailed description of the reconstruction process will be described later.
After performing the above steps, the processing procedure of the image forming method of the present embodiment ends.

[Interference signal acquisition procedure]
Next, a specific procedure of the first and second signal acquisition steps S101 and S102 of the above-described embodiment will be described with reference to FIG.

  In step S109, the signal acquisition control unit 145 sets the index i of the position yi in FIG. In step S110, the OCT apparatus 100 moves the scan position to yi. In step S119, the signal acquisition control unit 145 sets the index j of the repetitive B scan to 1. In step S120, the OCT apparatus 100 performs a B scan.

  In step S130, the detector 141 detects an interference signal for each A scan, and the above-described interference signal is stored in the signal processing unit 144 via an A / D converter (not shown). The signal processing unit 144 obtains p samples of the interference signal of the A scan, thereby obtaining an interference signal for 1B scan.

  In step S139, the signal acquisition control unit 145 repeatedly increments the index j of the B scan.

  In step S140, the signal acquisition control unit 145 determines whether j is larger than a predetermined number (m). That is, it is determined whether the B scan at the position yi has been repeated m times. If not, the process returns to S120 and repeats the B-scan measurement at the same position. If it has been repeated a predetermined number of times, the process proceeds to S149. In step S149, the signal acquisition control unit 145 increments the index i of the position yi. In step S150, the signal acquisition control unit 145 determines whether i is larger than the predetermined number of measurements (n) at the Y position, that is, whether the B scan has been performed at all n y positions. If the number of measurements at the predetermined Y position is less than (no), the process returns to S110, and the measurement at the next measurement position is repeated. If the number of measurements at the predetermined Y position has been completed (yes), the process proceeds to the next step S160. In the first and second signal acquisition steps S101 and S102 described above, i = 1 and m = 4, respectively.

  In step S160, the OCT apparatus 100 acquires background data. The OCT apparatus 100 executes 100 A-scans with the shutter (not shown) closed, and the signal acquisition control unit 145 averages and stores the signals obtained in the 100 A-scans. Note that the number of background measurements is not limited to 100 as exemplified herein.

  By performing each of the above processes, the interference signal obtaining procedure of the present invention is completed.

  In the acquisition of the OCTA interference signal, the OCTA interference signal can be acquired in the same procedure as the OCTA signal acquisition by setting the number of measurements and the position to values corresponding to the acquisition of the OCTA interference signal.

  In the present embodiment, an OCT signal is obtained, and a two-dimensional object image is generated based on data obtained from an interference signal set based on the OCT signal to form a three-dimensional tomographic image. I have. As described above, the range in which the OCT signal is obtained does not need to be the same as the range in which the OCTA signal is obtained. Therefore, as described above, this subject image includes the second range in which the OCTA signal is obtained in the subject's eye 118. Generated as a first range image.

[Signal processing procedure]
Next, a specific process of generating the display information including the three-dimensional OCT image information and the three-dimensional blood flow region information of the OCTA in step S103 illustrated in FIG. 3 will be described with reference to FIG. In the present invention, a motion contrast of OCTA is calculated in order to generate three-dimensional blood flow region information from OCTA information.

  In step S210, the signal processing unit 144 sets the index i of the position yi to 1. In step S220, the signal processing unit 144 extracts a repeated B-scan interference signal (m times) at the position yi (here, i = 1). In step S230, the signal processing unit 144 sets the index j of the repetitive B scan to 1. In step S240, the signal processing unit 144 extracts the j-th (here, j = 1) B-scan data.

  In step S250, the signal processing unit 144 generates a luminance image of a tomographic image by performing a reconstruction process on the interference signal of the B scan data in step S240. First, the image generation unit 147 removes fixed pattern noise composed of background data from the interference signal. The fixed pattern noise removal is performed by averaging the plurality of detected A-scan signals of the background data to extract fixed pattern noise, and subtracting this from the input interference signal. Next, the image generation unit 147 performs a desired window function process in order to optimize the depth resolution and the dynamic range, which have a trade-off relationship when the Fourier transform is performed in a finite section. Thereafter, a luminance image of the tomographic image is generated by performing the FFT processing.

  In step S260, the signal processing unit 144 increments the index j of the repeated B scan. In step S270, the signal processing unit 144 determines whether or not the incremented j is greater than m. That is, it is determined whether the brightness calculation of the B scan at the position yi has been repeated m times. In the case of no, the process returns to S240, and the brightness calculation of the repeated B scan at the same Y position is repeated. If yes, the flow proceeds to the next step.

  In step S280, the signal processing unit 144 positions m frames of the repeated B scan at a certain yi position. Specifically, first, the signal processing unit 144 selects an arbitrary one of the m frames as a template. For the frames to be selected as templates, the correlation may be calculated for all the combinations, the sum of the correlation coefficients may be obtained for each frame, and the frame having the maximum sum may be selected. Next, matching is performed for each frame using the template, and the amount of displacement (δX, δY, δθ) is determined for each frame. Specifically, Normalized Cross-Correlation (NCC), which is an index representing the similarity while changing the position and angle of the template image, is calculated, and the difference between the image positions when this value is the maximum is obtained as the positional deviation amount .

  In the present invention, the index indicating the similarity can be variously changed as long as it is a scale indicating the similarity between the features of the template and the image in the frame. For example, Sum of Abusolute Difference (SAD), Sum of Squared Difference (SSD), Zero-means Normalized Cross-Correlation (ZNCC), Phase Only Correlation (POC), and Rotation Invariant Phase Only Correlation (RIPOC) may be used.

  Next, the signal processing unit 144 applies position correction to the (m-1) frames other than the template in accordance with the amount of positional deviation (δX, δY, δθ), and aligns the m frames. Further, when the three-dimensional image information of the OCT and the three-dimensional blood flow site information of the OCTA are obtained in separate steps as in the present embodiment, the three-dimensional image information and the three-dimensional blood flow site information are also obtained. Perform positioning. That is, the three-dimensional tomographic image data calculated based on the interference signal set acquired for the purpose of generating the OCT intensity image and the interference signal set acquired for the purpose of generating the motion contrast image are calculated. Alignment with the three-dimensional tomographic image data is performed. By performing these alignments, it becomes easy to make correspondence between the three-dimensional image information and the three-dimensional blood flow site information. The positioning can be performed in the same manner as the positioning between frames.

  In step S290, the signal processing unit 144 averages the aligned luminance images calculated in step S280 to generate a luminance averaged image.

  In step S300, the map generator 148 performs retinal segmentation (part information acquisition) from the luminance averaged image generated by the signal processor 144 in step S290. This step is not used in the first embodiment described here, so that this step is skipped. The description will be given in the second embodiment.

  In step S310, the image generation unit 147 calculates a motion contrast. In the present embodiment, a variance value is calculated for each pixel at the same position from a luminance image of each tomographic image of m frames output by the signal processing unit 144 in step S250, and the variance value is used as a motion contrast.

  Note that there are various methods for obtaining the motion contrast. For example, the type of the feature amount used as the motion contrast in the present invention is a change in the luminance value of each pixel of the plurality of B scan images at the same Y position. Therefore, any index indicating the change in the luminance value can be applied to obtain the motion contrast. As the motion contrast, another method may be used instead of the variance value for each pixel at the same position from the luminance image of the m-frame tomographic image. For example, it is also possible to use a variation coefficient normalized by an average value for each pixel in each frame.

  In step S320, the signal processing unit 144 performs the first threshold processing of the motion contrast output by the signal processing unit 144. The value of the first threshold value is obtained from the luminance averaged image output by the signal processing unit 144 in step S290. Specifically, an area where only random noise is displayed on the noise floor is extracted from the luminance averaged image, the standard deviation σ is calculated, and the average luminance of the noise floor + 2σ is set as the first threshold. The signal processing unit 144 sets a motion contrast value corresponding to an area where each luminance is equal to or less than the first threshold to 0.

  By the first threshold processing in S320, noise can be reduced by removing motion contrast resulting from luminance change due to random noise.

  The smaller the value of the first threshold is, the higher the sensitivity of detecting the motion contrast is, while the noise component is also increased. The larger the noise, the lower the noise but the lower the sensitivity of motion contrast detection. In the present embodiment, the threshold is set as the average luminance of the noise floor + 2σ, but the threshold is not limited to this.

  In step S330, the signal processing unit 144 increments the index i of the position yi.

  In step S340, the signal processing unit 144 determines whether i is greater than n. That is, it is determined whether positioning has been performed at all of the n y-positions, luminance image averaging calculation, motion contrast calculation, and threshold processing have been performed.

  If no, the process returns to S220. If yes, the flow proceeds to the next step S350.

  When S340 ends, three-dimensional data (three-dimensional data) of the motion contrast of each pixel of the B scan image (Z depth vs. X direction data) at all Y positions is obtained. In step S350, the signal processing unit 144 performs a display information generation process from the three-dimensional image information.

  Hereinafter, the display information generation processing of the present embodiment will be described. FIG. 6 is a flowchart illustrating details of the process performed in step S350.

  When generating the display information, in step S351, the signal processing unit 144 acquires the three-dimensional data of the motion contrast determined earlier.

  In step S352, the signal processing unit 144 performs smoothing processing on the motion contrast three-dimensional data in order to remove noise while leaving blood flow site information. Although the optimum smoothing processing differs depending on the nature of the motion contrast, for example, the following can be considered.

  A smoothing method for outputting the maximum value of the motion contrast from nx × ny × nz voxels near a target pixel.

  Alternatively, a smoothing method for outputting the average value of the motion contrast of nx × ny × nz voxels in the vicinity of the target pixel.

  Alternatively, a smoothing method for outputting the median value of the motion contrast of nx × ny × nz voxels near the pixel of interest.

  Alternatively, a smoothing method that weights the motion contrast of nx × ny × nz voxels in the vicinity of the pixel of interest by a distance.

  Alternatively, a smoothing method in which the motion contrast of nx × ny × nz voxels in the vicinity of the target pixel is weighted according to the difference between the weight based on the distance and the pixel value of the target pixel.

  Alternatively, a smoothing method that outputs a value using a weight according to the similarity between the motion contrast pattern of the small area around the pixel of interest and the motion contrast pattern of the small area around the peripheral pixel.

  A method of performing smoothing while leaving other blood flow site information may be used.

  After the above-described smoothing processing, in step S353, the signal processing unit 144 obtains, from the display control unit 149, a threshold value used for determining a pixel to be displayed in step S354 and an initial value of a range in the depth direction to be displayed. . The initial value of the display range is usually set to about 1/4 of the depth direction, and is set to a position substantially including the range of the retinal surface layer. The reason why the initial value of the display range is not set to the entire range in the depth direction is that the main blood vessel / capillary network in the surface layer portion is desired to be displayed in an easy-to-read manner. That is, if the surface layer including the main blood vessel / capillary network and the RPE layer having no blood vessel and having a large noise are displayed at the same time, it is difficult to determine the main blood vessel / capillary network in the surface layer. Note that the initial value of the display threshold may be set in advance by obtaining a typical value for a healthy eye or the like. Alternatively, when repetitive photographing is performed, the previous photographing data may be used. Further, the display threshold may be adjusted in a step described later.

  Next, in step 354, display threshold processing for displaying pixels exceeding the smoothed three-dimensional data using the initial value of the display threshold is performed. FIG. 7 shows an example of conversion from the motion contrast to the display pixel value by this processing. FIG. 7A shows an example in which pixel values below the display threshold are assigned to zero, and display pixel values proportional to pixels from the pixel value above the threshold to the maximum intensity are assigned. FIG. 7B shows pixel values below the display threshold. Is an example in which display values are multiplied by zero, and pixel values higher than the multiplied values are multiplied by one. In any case, if the motion contrast is equal to or less than the display threshold, the motion contrast is invalidated, and the region having the motion contrast having the displayed connection is displayed in a separated form.

  Step 355 executed by the display control unit 149 shown in FIG. 6 is a step of displaying the motion contrast image subjected to the display threshold processing shown in FIG.

  Next, step 356 is a step of changing display conditions. Display conditions that can be changed will be described later. When the display condition is changed, the process returns to step 354, and the updated image is displayed by changing the display condition. When the display condition is not changed, the display information generation processing ends.

  Next, FIG. 8 shows a display example of an image generated by the display information generation processing of the present embodiment. On the display screen, an intensity image 400 and a motion contrast image 401 each displaying an OCT signal as an image are displayed side by side. In this display example, a slider 402 for operating a display threshold of the motion contrast is provided beside the motion contrast image 401.

  A tomographic image 403 of a part of the intensity image 400 is displayed beside the intensity image 400. The tomographic image is displayed on the xz section 403a and the yz section 403b. Further, a line 404 indicating the display position of the tomographic image may be displayed on the intensity image 400 or the motion contrast image 401. The line 404 indicating the display position may be automatically determined under specific conditions from the image information, or may be designated by the examiner. For example, as an example of the specific condition, a tomographic image 403 centering on the macula may be specified. Alternatively, the examiner may specify an arbitrary position and length from the intensity image 400 or the motion contrast image 401 and display a tomographic image so as to correspond thereto.

  Further, a slider 405 operated to indicate the display range in the depth direction of the intensity image 400 and the motion contrast image 401 from the tomographic image may be provided. In the slider 405, a depth or a display range in the depth direction at which the intensity image 400 and the motion contrast image 401 are displayed is designated by a slide portion 405a in a bar 405b indicating an adjustable range in the depth direction. Further, a GUI operation unit 406 for operating the display position may be provided. The details of the GUI operation unit 406 will be described later. Further, a cursor 407 indicating the selected position may be displayed. In this case, it is preferable that the cursor 407 is displayed so as to simultaneously indicate the corresponding positions of the intensity image 400 and the motion contrast image 401. The display is instructed by the display control unit 149. Note that, in the example of FIG. 8, an example is shown in which the image is displayed as a two-dimensional plane image. However, a three-dimensional image, an arbitrary tomographic image, or a combination thereof may be used.

  In the present embodiment, the image displayed on the two-dimensional plane shown in FIG. 8 is generated from an OCT intensity signal without using a fundus camera or a laser scanning ophthalmoscope. Such an image is called an Enface image. Here, a method of generating an Enface image will be briefly described.

  The information obtained by A-scan is arranged from the near side in the optical axis direction, and from the position where there is no signal, the position where the first strong signal can be obtained is the information of the fundus surface layer (retina surface). Therefore, by connecting the reflection intensities of the strong signals first for each of the A-scan signals at each measurement position (imaging position) on the fundus when the measurement light is scanned in two axes, the fundus surface layer is two-dimensionally connected. An Enface image about the fundus oculi surface layer represented by the following equation can be obtained. In addition, by connecting the reaction intensities of the signals having the specific order from the signals, an Enface image at a specific layer or depth in the retina can be obtained.

  Further, the tomographic image acquired by the B-scan is constructed by continuously acquiring information by A-scan in one axis direction. Therefore, by determining the position where the image signal of a part of the Enface image and the image signal of the fundus oculi surface layer part of the acquired tomographic image are collated, the A-scan is the tomographic image at any position on the Enface image. Can be accurately detected.

  Next, the display operation will be described. FIG. 9 shows the display operation. FIG. 9 shows the intensity image 400 and the motion contrast image 401 of FIG. As described above, in the present embodiment, the acquisition of the OCT interference signal and the acquisition of the OCTA signal are performed in the same process, and the alignment of these signals on the same scanning line is completed when a display image is generated. ing. Therefore, the correspondence between each pixel in the intensity image 400 and the motion contrast image 401 is grasped in advance.

In the display operation of the present embodiment, when a part of one image (for example, a blood vessel portion of the intensity image) is selected, a corresponding region of the other image (a corresponding blood vessel portion of the motion contrast image) is selected as the original image (the intensity image). ). In the example of FIG. 9, when the blood vessel portion 450 of the intensity image 400 is selected, the corresponding blood vessel and the connected blood vessel 450a in the motion contrast image 401 are superimposed and displayed on the corresponding position of the intensity image 400. More specifically, first, a blood vessel is selected in one image. Here, the position information of each pixel or blood vessel is already associated with the corresponding other image by positioning. In this state, the position information of the blood vessel selected in one image is obtained. Subsequently, based on the obtained position information, corresponding position information in the other image is selected. A blood vessel in the other image is designated as a corresponding blood vessel based on the selected position information. Through the above procedure, the blood vessel of one image is superimposed and displayed on the other blood vessel image based on the positional information of the corresponding blood vessel in each image.
It is assumed that the connection state of blood vessels (a group of thick blood vessels and a group of thin blood vessels branched from the thick blood vessels) in each image is grasped in advance by the region division processing described later. Therefore, the selection and superimposition of the blood vessels are executed in consideration of this connection state (in a manner of associating the blood vessel groups with each other). As described later, this superimposed display displays information (information on blood vessels) of one of the intensity image and the motion contrast image selected by an input from the cursor 407 or the like in an area where the information is displayed in the other image. It is performed to overlap. This superimposed display is executed by the display control unit 149 which is a display control unit for displaying the image thus superimposed on the display unit 146 which is a display unit.

  As a specific selection method, a specific blood vessel may be selected as shown in FIG. 9, or an area may be specified. The designated area may be a rectangle or a circle at a certain distance from the center of the cursor 407. If the OCTA measurement area is smaller than the OCT measurement area, the entire motion contrast image may be overlapped. Furthermore, the display color of the blood vessel that is the source of the superimposition in the other image (the motion contrast image) may be changed so that it is possible to know which blood vessel is superimposed and displayed on the original image (the intensity image).

  Although the example in which the motion contrast image is superimposed on the intensity image has been described, the reverse may be applied. When the intensity image is superimposed on the motion contrast image, when a specific blood vessel portion of the motion contrast image 401 is selected, the corresponding blood vessel and the connected blood vessel in the intensity image 400 are displayed on the corresponding position of the motion contrast image 401 in an overlapping manner. Is done. In this case, it is preferable to express a portion corresponding to a blood vessel portion displayed in a superimposed manner with a thick line and a dotted line in order to enhance visibility. Alternatively, the display color of the blood vessel to be superimposed may be changed. In general, blood vessels that can be distinguished from an intensity image are relatively thick blood vessels, and it is easy to establish positional correspondence with structural information of the intensity image. Therefore, by superimposing the intensity image on the motion contrast image in this manner, a thick blood vessel can be used as a mark, and comparison with structural information other than the blood vessel becomes easy. Further, by using the intensity image 400 as a base, it is possible to display not only blood vessels but also structural information such as a macula or a lesion.

  As described above, by adopting a configuration in which one image information of the OCT intensity image and the OCTA motion contrast image can be superimposed and displayed on the other, the correspondence between the intensity image and the motion contrast image becomes easy. Therefore, it is possible to provide an image that is easy to understand intuitively.

  A cursor display will be described as an example of the display according to the present embodiment. As shown in FIGS. 8 and 9, a cursor 407 may be displayed on both the intensity image 400 and the motion contrast image 401. The shape of the cursor 407 may be, for example, an arrow as illustrated, a circle, or a square. Further, the two images may have different shapes. Alternatively, it may be a crosshair crossing in the X and Y directions. The cursor points simultaneously to the corresponding positions in both images. It is desirable that the cursor can be freely moved by operating a mouse (not shown) attached to the image forming apparatus. Further, the cutting position of the tomographic image may be changed with the movement of the cursor. By displaying the corresponding positions at the same time, the two images can be easily compared.

  Next, as an example of the change of the display condition of the present embodiment, the change of the display range and the viewpoint will be described. The display magnification and display position of both images, or the viewpoint position may be changed at the same time. For example, the display conditions are changed on the GUI operation unit 406 on the display screen in FIG. The enlargement / reduction button 406a simultaneously enlarges (+) and reduces (-) both images. When the image is enlarged, the translation of the display range of both images is simultaneously performed by the translation button 406b. Similarly, the rotation of the display viewpoint of both images is simultaneously performed by the rotation button 406c. The rotational movement can be applied when both images are displayed three-dimensionally. That is, with these configurations, it is possible to simultaneously change at least one of the magnification, the display position, and the viewpoint position with respect to the intensity image and the motion contrast image in both images.

  Here, the GUI operation unit 406 has been described as an example of three types of buttons, but the number, shape, and type of the buttons may be different. Alternatively, another method may be used. For example, it may be performed by operating a mouse attached to the image forming apparatus. Move the mouse cursor over one of the two images and use the wheel associated with the mouse to zoom in or out, or move the mouse while pressing the mouse button to move (translate or rotate) the image. Or you may. In an example other than a mouse, a touch panel may be used to operate an image displayed on the panel with a finger. In any case, the display range and viewpoint of the intensity image and the motion contrast image displayed side by side may be changed at the same time. According to this configuration, since the image is changed at the same time for both the intensity image and the motion contrast image, it is possible to easily understand which position the user is looking at even if the display method of the image is changed.

  The display threshold of the motion contrast pixel value will be described. FIG. 8 shows a slider 402 for adjusting the threshold value of the pixel to be displayed. The initial position of the display threshold may be a representative value set in advance. When the examiner drags this slider with the mouse, the signal processing unit 144 determines in step 356 in FIG. 6 that the display condition has been changed (YES). The signal processing unit 144 changes the display threshold, returns to step S354, and updates the three-dimensional motion contrast image to be displayed. At this time, if the adjustment of the threshold value is set so that it can be changed by a relative value with respect to the initial value, an equivalent effect can be obtained even for different data such as different eyes and parts to be examined.

  Next, a modified example of the display method of the blood vessel information will be described. Here, a step of acquiring blood vessel information is performed on the two-dimensional motion contrast image. The step of acquiring blood vessel information may be performed at the same timing as step S352, which is the smoothing processing step described with reference to FIG. 6, or may be performed as a subsequent step. Specifically, a step of applying a smoothing process to the motion contrast data and a step of dividing a region corresponding to a blood vessel are performed. By performing these processes, a smooth blood vessel region can be extracted.

Next, the area division processing will be described. FIG. 13 shows an example of the flow of the area division processing.
First, in step S601 which is a blood vessel candidate extraction step, the signal processing unit 144 performs a process of extracting, as a blood vessel candidate pixel, a pixel value representing a volume data corresponding to the motion contrast of the pixel of interest as a pixel of a predetermined threshold or more. Execute.

  Next, in step S602, which is a blood vessel connection estimation step, the signal processing unit 144 executes a process of estimating, for each of the extracted pixels of the blood vessel candidate, whether or not there is a blood vessel connection relationship that is a connection relationship between these blood vessels. I do. In the blood vessel connection estimation step, for each pixel of the blood vessel candidate, if an adjacent pixel has a pixel value equal to or greater than a predetermined threshold, or if the adjacent pixel is also a blood vessel candidate in advance, these pixels are connected. Is determined to be a blood vessel candidate. As described above, when a specific pixel is determined as a blood vessel candidate to be connected, a range to be evaluated for a further adjacent pixel is expanded. If the adjacent pixel is smaller than the threshold value (or is not a blood vessel candidate), the adjacent pixel is excluded from the blood vessel candidate, and the other adjacent pixels are evaluated. The operation of blood vessel connection estimation may be performed for all or a specified range of blood vessel candidates.

  After the evaluation of all the adjacent pixels or all the pixels of the blood vessel candidate is completed, the signal processing unit 144 performs the process of step S603 which is a blood vessel recognition process based on the size of the connected blood vessel candidate. In the blood vessel recognition processing, a blood vessel candidate having a connection of a predetermined number or more of pixels is identified as the same relatively thick or main blood vessel.

  Finally, in step S604, which is a blood vessel connection relationship data generation step, the signal processing unit 144 executes a process of generating first blood vessel connection relationship data. The first blood vessel connection relationship data has at least three-dimensional distribution information of blood vessels to be connected and blood vessel connection information. For example, first volume connection data may be used as long as the volume data has information on the presence or absence of a blood vessel for each pixel and information for distinguishing each blood vessel.

  The threshold for extracting the blood vessel candidates may be a predetermined value or may be changed. For example, the examiner may measure the OCTA signal for the subject's eye and adjust the threshold value based on how the area is divided into blood vessels. Further, a representative value may be used as the initial value of the threshold. In the region dividing step, when the motion contrast data has been subjected to the binarization processing for the blood vessel area and the other areas, blood vessel candidates may be extracted using the result of the binarization processing. Similarly, the threshold value of the predetermined number of connections determined as a blood vessel may be a predetermined value or may be changed. For example, a blood vessel may be excluded when a plurality of pixels are connected, and may be excluded from a candidate when a single pixel is connected. By performing such processing, isolated regions can be removed from the motion contrast image, and connected blood vessels can be extracted. An example of an isolated region is a local and random change in reflection intensity such as speckle noise.

  In addition, the example in which the size of the intensity image and the size of the motion contrast image are the same has been described. However, as described above, the image can be similarly displayed even in an area where the range for acquiring the motion contrast image is narrow. FIG. 14 shows an example in which the size of the intensity image and the size of the motion contrast image are different. FIG. 14A shows an example in which the size of the motion contrast image is changed to the same scale. FIG. 14B shows an example in which the scale of the image is changed to make the size of the image uniform. It is preferable to provide a frame line 416 indicating the other display area on the side having a wider angle of view. The enlargement / reduction button 406a or the like shown in FIG. 8 may be used to enlarge / reduce the intensity image and the motion contrast image, move the viewpoint, or superimpose one state on the other.

  Next, an example of comparing a blood vessel obtained from the intensity image with a blood vessel obtained from the motion contrast image will be described. The operation performed in this case will be described with reference to the flowchart shown in FIG.

  First, in step S651, which is the first blood vessel connection estimation step, the signal processing unit 144 executes a process of estimating a first blood vessel connection relationship from a two-dimensional motion contrast image. In step S651, the first blood vessel connection relationship can be estimated by a method similar to the method used when estimating the blood vessel connection relationship performed in step S602 described above.

  Next, in step S652, which is the second blood vessel connection estimation step, the signal processing unit 144 executes a process of estimating a second blood vessel connection relationship from data of the three-dimensional tomographic image. The second blood vessel connection relationship may be estimated for a specific layer. For example, if three-dimensional tomographic image data is used to generate a two-dimensional intensity image (for example, intensity image 400) from three-dimensional tomographic image data having a depth corresponding to the retinal layer and further estimate a second blood vessel connection relationship Good. In the intensity image, the intensity of the signal obtained from the blood vessel is relatively weaker than the surroundings due to absorption by blood and the like. In this case, by selecting a retinal layer, it becomes easy to extract a pixel corresponding to a retinal arteriovenous blood vessel as a blood vessel.

  The division of the region corresponding to the blood vessel may be performed in the same manner as the motion contrast data. Further, a threshold value of a pixel when extracting a pixel as a blood vessel may be set to a value corresponding to the intensity image. The blood vessels obtained by estimating the second blood vessel connection relationship have a lower resolution of the blood vessels that can be extracted than the blood vessel connection relationship obtained from the motion contrast data (first blood vessel connection relationship), but are obtained from information derived from the blood vessel structure. is there.

  Next, in step S653, which is a blood vessel connection state comparison step, the signal processing unit 144 performs a process of comparing the connection state of the blood vessels in the second blood vessel connection relation and the first blood vessel connection relation. The comparison of the connection state of the blood vessels may be made by comparing the presence or absence of a blood vessel with respect to the first and second blood vessel connection relations at corresponding positions.

  Next, in step S654, which is a comparison result display step (display step), the control unit 143 displays on the display unit 146 at least a part of the comparison result of the blood vessel connection state as an intensity image and / or a motion contrast image. To be superimposed. For example, on the other hand, a portion not estimated as a blood vessel is displayed. Conversely, a portion that is estimated to be a blood vessel may be displayed on both sides. The comparison may be for the entire image or only for a specific area.

  In the case where the blood vessel image of the motion contrast image is superimposed on the intensity image, it is equivalent to superimposing and displaying capillary information that cannot be obtained from the intensity image on the intensity image. On the contrary, when the blood vessel image of the intensity image is superimposed on the motion contrast image, it corresponds to displaying the region where the blood flow is stagnant. In the area where the blood flow is stagnant, the pixel value of the motion contrast is smaller than the area where the blood flow is present. When the pixel value becomes equal to or less than the threshold value, it is determined that there is no blood vessel. Examples of the region where the blood flow is stagnant include a portion where stagnation has occurred due to a vascular aneurysm, a portion where there is a portion having a small inner diameter on an intermediate route, and a blocked region.

  As described above, by superimposing the information on the connection state of one blood vessel on the other, both images can be more easily compared.

  Further, in displaying blood vessels, a specific blood vessel may be selected and highlighted. In this case, after step S654, which is a comparison result display step, a step of selecting a pixel near a specific blood vessel is further provided. Specifically, for example, an arbitrary pixel is selected from a display image with a mouse or the like. In response to this selection operation, the signal processing unit 144 selects a blood vessel closest to the selected pixel. Here, the selected blood vessel has the information on the first and second blood vessel connection relationships described above. The result of comparing the selected blood vessel with the blood vessel obtained from the intensity image and the motion contrast image is displayed. As a method of displaying the comparison result, for example, a translucent blood vessel image with a different display color may be displayed in a superimposed manner. FIG. 9 shows an example in which a specific blood vessel is selected and a blood vessel image of motion contrast is superimposed on the intensity image.

  It should be noted that a mode having a plurality of modes may be considered as a blood vessel selection mode when superimposing. As a mode in this case, for example, a mode in which a blood vessel designated by the cursor 407 from the nipple to the end of the blood vessel is superimposed on the intensity image 400 can be considered. In addition, the blood vessel specified by the cursor 407 may include a mode of superimposing the intensity image from the specified position to the end. Alternatively, a region may be designated by the cursor 407 and a method of superimposing blood vessels included in the region on the intensity image 400 may be included. Furthermore, a method may be included in which a blood vessel diameter is estimated, which will be described later, and a blood vessel estimated to be smaller than a predetermined blood vessel diameter is not superimposed at the time of superposition. Further, these modes may be combined.

  With such a configuration, a comparison result focusing on a specific blood vessel can be obtained. For example, one of the retinal arteriovenous blood vessels can be selected, and the state of the blood vessels connected to the selected blood vessel can be extracted and displayed. Alternatively, instead of selecting a specific blood vessel, a color may be previously colored for each run from the optic disc. For example, the colors may be classified based on the direction spreading from the center of the optic disc.

  Further, a step of acquiring blood vessel information for a blood vessel extracted from the motion contrast image may be further provided. For example, at least a blood vessel diameter may be detected for a blood vessel region. The blood vessel diameter is obtained by estimating the axial direction and the radial direction when the extracted blood vessel is approximated to a cylinder, and obtaining the blood vessel diameter. The blood vessel diameter data may be further added to, for example, the volume data of the first blood vessel connection relationship data (the three-dimensional distribution information of blood vessels and the connection information of blood vessels). With such a configuration, information on the form of the blood vessel can be obtained from the motion contrast image.

  Furthermore, blood vessels may be classified based on the calculated blood vessel diameter. First, a blood vessel smaller than a predetermined diameter is extracted. The predetermined diameter may be, for example, a diameter corresponding to a capillary. Further, in a predetermined region of the motion contrast image, the existence ratio of a pixel extracted as the blood vessel (for example, a capillary blood vessel) and other pixels is calculated. Further, the existence ratio is displayed as a two-dimensional map. With such a configuration for mapping, the density of blood vessels can be quantitatively visualized. The predetermined area may be divided based on, for example, the macula (or the optic disc). FIG. 16 shows an example of mapping. In the example of FIG. 16, a map 408 divided into a plurality of regions around the macula is superimposed on the two-dimensional motion contrast image. The division into a plurality of regions facilitates the qualitative evaluation of the density of the capillaries. Further, the mapped information may be displayed not only on the motion contrast image but also on the two-dimensional intensity image in a superimposed manner, or may be displayed on another screen.

  With such a configuration, it is possible to visualize the information on the form of the blood vessel from the motion contrast image in association with the intensity image.

(Second embodiment)
Next, as a second embodiment of the present invention, a change in the display range in the depth direction will be described.
By operating the slider 405 in FIG. 8, the display range in the depth direction in which the intensity image and the motion contrast image are displayed is changed. The slider 405 includes the slide portion 405a indicating the depth range to be displayed and the bar 405b indicating the adjustable range in the depth direction of the retina as described above. The slide portion 405a can change the width and position in the depth direction to be displayed as an image, and is used for selecting a display range in the depth direction. The intensity image 400 and the motion contrast image 401 to be displayed are simultaneously updated according to the change of the display range by the operation of the slide unit 405a. According to the present embodiment, since the change in the depth direction is performed at the same time, even if the range of the display depth of the image is changed, it is easy to understand which position is being viewed.

  Here, an example using a slider has been described as a method of changing the depth range, but other methods may be used. For example, the display depth may be changed for each layer structure of the eye to be examined, thereby substantially changing the displayed depth. The present embodiment has a step of detecting the layer structure of the tomographic image of the subject's eye from the data of the three-dimensional tomographic image (corresponding to step 300 in FIG. 5). The step of setting the display depth enables selection based on the layer structure information of the eye to be inspected, instead of the slider. The layer structure of the human eye is known and includes, for example, the following six layers. The six layers are (1) nerve fiber layer (NFL), (2) ganglion cell layer (GCL) + inner plexiform layer (IPL), (3) inner granular layer (INL) + outer plexiform (4) Outer granular layer (ONL) + outer limiting membrane (ELM) combined layer, (5) Ellipsoid Zone (EZ) + Interdigitation Zone (IZ) + retinal pigment epithelium (RPE) And (6) Choroid.

  The segmentation of each of these layers in the retina will be described. The map generator 148 creates an image by applying a median filter and a Sobel filter to a tomographic image to be processed extracted from the intensity image, respectively (hereinafter also referred to as a median image and a Sobel image, respectively). Next, a profile is created for each A-scan from the created median image and Sobel image. The median image has a luminance value profile, and the Sobel image has a luminance gradient profile. Then, a peak in the profile created from the Sobel image is detected. The boundary of each region of the retinal layer is extracted by referring to the profile of the median image corresponding to before and after the detected peak and between the peaks. The correspondence between pixels and layers is performed based on the extracted boundary information.

  FIG. 10 shows an example of a method of selecting a layer to be displayed after the segmentation. In this example, a selection button 410 corresponding to each layer in the tomographic image is provided. By selecting the selection button 410, a display range of an image or a layer to be displayed is selectively designated. The selection buttons 410 may be provided one by one as illustrated in the figure, or may be formed by combining a plurality of layers. For example, the selection buttons may be divided into an upper retina layer and a lower retina layer. Alternatively, a button may be provided so that selection can be made in boundary units. Furthermore, a radio button 411 may be provided to switch between selection in units of layers or selection in units of boundaries as shown in FIG. Further, the tomographic image 412 may be displayed side by side with the selection buttons 410 and the like in order to easily understand the considerable correspondence.

  The intensity image and the motion contrast image are updated according to the selection processing of the layer to be displayed. Note that a slider 413 for adjusting the display threshold of the motion contrast image may be provided for each layer, and the display threshold may be readjusted when the image is updated. With the above configuration, the displayed layers are changed simultaneously in the intensity image and the motion contrast image, so that even when the displayed layers are changed, it is easy to understand which layer is being viewed. Also, by changing the depth range based on the layer structure, information focusing on a specific layer can be displayed.

  When displaying an image based on the layer structure of the eye to be inspected, the intensity image and the motion contrast image may be two-dimensional images. Here, the process of generating a two-dimensional image will be described. In the two-dimensional image generation step, the two-dimensional intensity image is generated by projecting and integrating the three-dimensional tomographic image data in a corresponding range in the depth direction based on the designated depth or the selected layer. An example of the projection method will be described with reference to FIG.

  FIG. 11A shows a tomographic image, which corresponds to a selected range between Z1 and Z2 in the depth direction. FIG. 11B shows three-dimensional volume data 500 and a depth display bar 501. The volume area 502 designated as between the depths Z1 and Z2 on the depth display bar 501 is the selected range. A two-dimensional intensity image can be generated by projecting and integrating voxels (volume data of a three-dimensional tomographic image) in the volume region 502 in the depth direction. Similarly, for the three-dimensional motion contrast, each voxel in the same region is projected and integrated in the depth direction, or projected or integrated to generate a two-dimensional motion contrast image.

  In order to generate a two-dimensional image, in addition to integrating the pixel values of the corresponding pixels, it is also possible to extract and project a representative value such as a maximum value. The generated two-dimensional image is displayed as an intensity image and a motion contrast image. Further, these display processes may be performed not simultaneously, but sequentially in such a manner that a display state is confirmed by forming a two-dimensional image and then the display threshold of the motion contrast image is changed. By forming a two-dimensional image and displaying it in parallel, it is possible to easily grasp the entire information focused on a specific layer at a time.

  Note that, in the example shown in FIG. 11, voxel projection / integration is performed in a specified fixed depth range, but this may be performed in a depth range corresponding to the layer structure. That is, a depth range corresponding to the layer selected for each location may be selected, and voxel projection / integration in this range may be performed. By doing so, a two-dimensional image reflecting the layer structure can be generated.

  Further, when displaying the two-dimensional image described above, a tomographic image may be displayed in parallel as in FIG. In this case, the position of the cut surface of the tomographic image is designated by one of a two-dimensional intensity image and a two-dimensional motion contrast image. As the tomographic image to be displayed, a tomographic image at a designated position or a corresponding position is selected (or generated) from three-dimensional tomographic image data and / or three-dimensional motion contrast. The original image specifying the position of the tomographic image to be displayed and the tomographic image to be displayed can be appropriately combined. Further, a tomographic image of a motion contrast image may be displayed so as to be superimposed on a tomographic image of an intensity image. As shown in FIG. 8, a display line 404 indicating a cross section of a tomographic image may be provided in both images. Since the position of the tomographic image can be designated and displayed from the two-dimensional image of either the intensity image or the motion contrast image, the tomographic image at the desired position can be easily displayed.

  Further, the tomographic image described above may further display information on a layer structure. In this case, the layer structure is superimposed and displayed on the tomographic image based on information obtained by detecting the layer structure of the tomographic image of the eye to be examined from the data of the three-dimensional tomographic image. The display method may be such that the layer structure can be identified. For example, a line may be displayed at each layer boundary, or each layer may be colored. FIG. 12 shows an example in which a tomographic image is displayed for each layer. In the example of FIG. 12, the boundary line 422 (dotted line) based on the layer detected for the tomographic image is displayed in an overlapping manner. The boundary line 422a of the selected layer is a thick dotted line, and the selected layer region 423 is gray. By superimposing the information on the detected layer structure, it is possible to make the display easier to understand.

(Third embodiment)
Next, a third embodiment of the present invention will be described. In the present embodiment, a fundus image is obtained separately from obtaining a motion contrast image by OCT. Examples of a fundus image acquiring unit include a fundus camera, a scanning laser ophthalmoscope (SLO), and a blood vessel image obtained by fluorescence contrast. These configurations constitute a subject image acquisition unit that acquires a subject image in a first predetermined range of the subject's eye 118, similarly to the configuration of generating an intensity image composed of OCT in the first embodiment. Similarly to the intensity image obtained by the OCT, the third blood vessel connection relationship is calculated from the fundus image obtained by these methods. The calculated blood vessel connection relationship and the first blood vessel connection relationship based on the motion contrast image are aligned based on the feature amount of the blood vessel. A known alignment algorithm can be used for the alignment.

  After the alignment, both images are displayed side by side. FIG. 17 shows a display example. In the example of FIG. 17, a motion contrast image (OCTA image) 401 and a fundus image 420 are displayed side by side. The fundus image to be displayed is switched by the radio switch unit 421. For example, the switching operation switches between a two-dimensional intensity image obtained by integrating OCT, a fundus image by a fundus camera, and a fundus image by SLO. As in the first embodiment, both images are provided with a cursor 407 at corresponding positions.

  Further, as in the first embodiment (the display example shown in FIG. 8), both images may be simultaneously enlarged or moved. With such a configuration, it is possible to easily compare an image obtained by another observation device with a motion contrast image. For example, by displaying the images obtained from the color fundus camera side by side, it is possible to easily compare the blood vessel photograph and the motion contrast image viewed in color.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. In the present embodiment, the blood vessel connection relationship based on the measured motion contrast image and the blood vessel connection relationship based on another motion contrast image are displayed side by side. Other motion contrast images may be previously captured images. FIG. 18 shows a display example of the fourth embodiment. In the example of FIG. 18, the left motion contrast image 401 is compared with the right motion contrast image 430 shot in the past. Position alignment is performed between these images based on the connection relationship of the blood vessels as in the third embodiment. In addition, it is sufficient that both images can be enlarged and moved at the same time. In this case, in the above-described embodiment, the operation executed in the step of acquiring the subject image is a step of acquiring another motion contrast image at a different time from the motion contrast image in the signal acquiring step which is a measurement time. Is replaced by

  In the example of FIG. 18A, an image obtained when there is no blood vessel signal in a region 431 surrounded by a broken line of the motion contrast image 401 is schematically shown. The connected state of the two displayed images is compared, and at least a part of the comparison result is displayed so as to be superimposed on one of the images. FIG. 18B shows an example in which the comparison result (blood vessel) of the region 431 is displayed in a superimposed manner by a dotted line. With such a configuration, it is possible to easily grasp a temporal change in the connection relationship of blood vessels.

  The comparison process executed here may be a comparison with a pair of images or a comparison between a plurality of images. When comparing a plurality of images, display colors may be divided. Alternatively, the comparison target may be sequentially switched.

  As described above, according to each of the above-described embodiments, the information obtained from one image is displayed on the other image with respect to the motion contrast image obtained by OCTA and the intensity image obtained by OCT, so that the image can be more intuitively understood. It is possible to provide an easy-to-use image. That is, the correspondence between the motion contrast image obtained by OCTA and the intensity image obtained by OCT can be facilitated. In addition, when displaying both images side by side, displaying a cursor indicating the corresponding pixel can provide an image that is easier to compare.

  Furthermore, by providing a process in which the magnification, display position, and viewpoint position of both images can be changed at the same time, it is possible to provide an image that makes it easy to understand which position the user is looking at even if the display method of the images is changed. .

  As described above, the preferred embodiments of the present invention have been described in detail, but the present invention is not limited to the specific embodiments, and various modifications may be made within the scope of the present invention described in the appended claims.・ Change is possible. For example, the positional relationship of the image display and the shape of the GUI can be changed. Alternatively, the display may be a stereoscopic display using a 3D display.

(Other embodiments)
Note that the present invention is not limited to the above-described embodiment, and can be implemented with various modifications and changes without departing from the spirit of the present invention. For example, although an embodiment has been described in which the subject is an eye, the present invention can be applied to subjects other than eyes, such as skin and organs. In this case, the present invention has an aspect as an image forming method or apparatus for forming an image based on data obtained from a medical device such as an endoscope other than the ophthalmic device. In addition, it is preferable that the above-described OCT apparatus is grasped as an examination apparatus exemplified by an ophthalmologic apparatus, and that the eye to be examined is grasped as one mode of an object to be examined.

  Further, the present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus execute the program. The processing can be implemented by reading and executing. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

100 Optical coherence tomographic acquisition unit 143 Control unit 144 Signal processing unit 145 Signal acquisition control unit 146 Display unit 147 Image generation unit 148 Map generation unit 149 Display control unit

The present invention relates to an information processing device and an information processing method .

In order to solve the above-described problem, an information processing device according to one embodiment of the present invention includes:
Data acquisition means for acquiring three-dimensional motion contrast data of the eye to be inspected;
Information acquisition means for acquiring measurement information on at least one blood vessel obtained using the obtained three-dimensional motion contrast data;
A classification unit that classifies the at least one blood vessel using the acquired measurement information;
Is provided .

Claims (19)

A subject image obtaining step of obtaining a subject image in a first predetermined range of the subject,
In a second predetermined range included in the first predetermined range, an interference signal set for a plurality of frames obtained with the intention of the same cross section is obtained for a plurality of different cross sections to form a three-dimensional tomographic image. A signal acquisition step;
A motion contrast image generation step of generating a motion contrast image using pixel data corresponding to each frame in an interference signal set for a plurality of frames acquired with the intention of the same cross section,
A display step of displaying the subject image and the generated motion contrast image side by side; and selecting information of one of the subject image and the motion contrast image, a region where the information is displayed in the other image. A superimposed display step of superimposing and displaying the selected information;
An image forming method comprising:   2. The subject image acquiring step, wherein the subject image is generated based on an interference signal set acquired from the first predetermined range in order to form a three-dimensional tomographic image. 3. Image forming method.   The image forming method according to claim 2, wherein the interference signal set acquired in the subject image acquiring step is included in the interference signal set for the plurality of frames acquired in the signal acquiring step.   4. The image forming apparatus according to claim 2, wherein the subject image is an image generated by overlapping data of the three-dimensional tomographic image obtained from the interference signal set by at least two frames. 5. Method. Data of a three-dimensional tomographic image of the subject calculated based on the interference signal set acquired in the subject image acquiring step,
5. The method according to claim 2, further comprising a step of performing positioning of the subject with data of a three-dimensional tomographic image of the subject calculated based on the interference signal set acquired in the signal acquiring step. The image forming method according to claim 1. The display step includes:
The image according to claim 5, further comprising a step of instructing a change of a display range in the depth direction in the three-dimensional tomographic image, wherein the displayed motion contrast image is updated when the display range is changed. Forming method. Further comprising a step of detecting a layer structure of the tomographic image of the subject from the data of the three-dimensional tomographic image acquired in the signal acquiring step,
7. The image forming method according to claim 6, wherein in the step of changing the display range, the display range is selectively designated based on the detected layer structure. The object image is obtained by projecting and integrating data of the three-dimensional tomographic image of the object in the display range, and
In the display step, a two-dimensional motion contrast image generated by projecting and integrating volume data of the motion contrast image in a depth direction in the display range and the subject image are displayed. 8. The image forming method according to 6 or 7.   The display means displays at least one of the three-dimensional tomographic image data and the volume data of the motion contrast image at a position corresponding to a position designated by one of the subject image and the two-dimensional motion contrast image. 9. The image forming method according to claim 8, wherein the image is generated and displayed from one side.   The image forming method according to claim 2, wherein the subject image acquiring step is a step of acquiring the motion contrast image as the subject image at a different time from the signal acquiring step.   The image forming method according to any one of claims 1 to 10, wherein the displaying step includes a step of displaying a cursor that simultaneously indicates a corresponding position between the subject image and the motion contrast image. .   12. The method according to claim 1, wherein the display step includes a step of simultaneously changing at least one of a magnification, a display position, and a viewpoint position between the subject image and the motion contrast image. 2. The image forming method according to 1.,   The motion contrast image generating step includes a step of performing a smoothing process on volume data of the motion contrast image, and a step of performing a region dividing process on the volume data on which the smoothing process has been performed. The image forming method according to claim 1, wherein: The subject includes a blood vessel,
The region dividing process, a pixel value representing the volume data is a pixel of a blood vessel candidate or more of a predetermined threshold or more,
A blood vessel connection estimating step of estimating a pixel having a blood vessel connection relationship with respect to the pixel that is the blood vessel candidate,
14. The image forming method according to claim 13, further comprising the step of: recognizing, as a blood vessel, the pixel which is a blood vessel candidate estimated to have a blood vessel connection relationship with a predetermined number or more of pixels. A second blood vessel connection estimation step of estimating a second blood vessel connection relationship from the subject image,
Comparing the connection state of the blood vessels in the second blood vessel connection relationship and the blood vessel connection relationship,
15. The image forming method according to claim 14, wherein the displaying step displays at least a part of a result of comparing the connection states of the blood vessels on at least one of the subject image and the motion contrast image. . The display step includes a step of selecting a pixel near the pixel recognized as the blood vessel,
The display of the pixel estimated to have the blood vessel connection relationship with the selected neighboring pixel is performed in a different manner from the selected pixel and the neighboring pixel. The image forming method according to claim 1. Extracting a blood vessel smaller than the predetermined diameter by comparing the blood vessel diameter calculated based on the pixel recognized as the blood vessel and a predetermined diameter,
17. The method according to claim 14, further comprising a step of mapping and displaying an abundance ratio of the pixels recognized as the extracted thin blood vessels and other pixels in a predetermined area of the motion contrast image. The image forming method according to claim 1.   A non-transitory computer-readable storage medium storing a program that causes a computer to execute each step of the image forming method according to claim 1. A subject image acquiring means for acquiring a subject image in a first predetermined range of the subject,
Signal acquisition for acquiring a plurality of frames of interference signal sets intended for the same cross section in a second predetermined range included in the first predetermined range for different cross sections to form a three-dimensional tomographic image Means,
Motion contrast image generating means for generating a motion contrast image using pixel data corresponding between each frame in the interference signal set for a plurality of frames obtained with the intention of the same cross section,
Display control means for displaying the subject image and the generated motion contrast image side by side on a display means,
Superimposing display means for displaying, when information of one of the subject image and the motion contrast image is selected, the selected information in a region where the information is displayed in the other image, .