JP2012122852A - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To properly specify an area of a predetermined sample component in a sample and improve visibility during observation.SOLUTION: In an image processing device 70, a visible light image acquisition control part 771 controls operation of a microscope 10 and acquires a visible light image by imaging visible light observation image of a sample to which HE dyeing has been applied and dyeing (fluorescent label) has been applied with fluorescent dye binding specifically to a cell nucleus. A fluorescent image acquisition control part 772 controls the operation of the microscope 10 and acquires a fluorescence image by imaging fluorescence observation image of the sample. A nucleus area specifying part 741 specifies a nucleus area in the visible light image based on the fluorescence image. An abnormal nucleus area identification image generating part 749 indicates dyeing condition of the HE dyeing of the sample based on the visible light image, and generates an abnormal nucleus area identification image identifiably indicating an abnormal nucleus area included in the nucleus area of the sample.

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program for processing specimen images of specimens that have been subjected to multiple different staining.

  In pathological diagnosis, preparation of a specimen from a tissue specimen obtained by organ excision or needle biopsy, and magnified observation using an optical microscope in order to obtain various findings are widely performed. Since the specimen is almost colorless and transparent, it is generally stained with a dye prior to observation. Various staining methods have been proposed. Particularly, regarding pathological specimens, hematoxylin (dye H) and eosin (dye E) are used as stains for observing the morphology of the specimen (morphological observation staining). Hematoxylin-eosin staining (hereinafter referred to as “HE staining”) using two dyes is standardly used.

  Here, the observation of the specimen is performed by, for example, multiband imaging of the specimen and displaying it on a screen, in addition to the visual observation of the observer. In this case, after estimating the pigment amount of the pigment staining the specimen from the pixel values of the captured multiband image, correcting the color of the image appropriately based on the estimated pigment amount, and then viewing the RGB image Is displayed on the screen.

  In the pathological diagnosis, for example, HE is used to confirm a predetermined sample component constituting a sample such as a cell component such as a cell nucleus, a cell membrane, or a cytoplasm, or a target molecule (antigen) expressed on the cell component. In addition to staining, other staining such as immunostaining and special staining may be applied.

  Here, for example, Patent Document 1 discloses a sample that is multiple-stained with a plurality of dyes, for example, a sample that has been subjected to HE staining for morphological observation and further stained with a target molecule by immunostaining that allows visible light observation. A technique for displaying a sample image with high visibility is disclosed. In this Patent Document 1, a display target dye is selected from a plurality of dyes staining the specimen, and the dye amount of the display target dye is displayed based on the dye amount of the display target dye in each pixel of the sample image. The generated display image is generated.

JP 2010-134195 A JP 2008-51654 A Japanese Patent No. 3469619

  According to the technique of Patent Document 1 described above, for example, a staining state by HE staining is obtained from a specimen image of a specimen in which HE staining is performed and a target specimen constituent element is specifically stained by immunostaining capable of observing visible light. A display image that represents the amount of dye that is staining the target specimen component, that is, a display image that represents only the region of the target specimen component Is possible. However, since HE staining and immunostaining are performed on the specimen in an overlapping manner, the dyes that are superimposed cannot actually be completely separated due to the influence of overlapping dyes, and the region of the target specimen component is not correctly determined. There was a case that could not be identified. In addition, the colors of a plurality of pigments may be superimposed and displayed, and visibility may be deteriorated.

  The present invention has been made in view of the above-described conventional problems, and can appropriately specify a region of a predetermined sample component in a sample and improve the visibility at the time of observation. An object of the present invention is to provide an image processing method and an image processing program.

  In order to solve the above-described problems and achieve the object, an image processing apparatus according to an aspect of the present invention stains a first staining and a predetermined specimen component, and is different from the first staining. First image acquisition means for acquiring a first sample image by imaging a staining state of the first staining of a sample that has been subjected to multiple second stainings observed by the method; A second image acquisition means for acquiring a second specimen image by imaging a staining state by the second staining; and the predetermined specimen in the first specimen image based on the second specimen image A region specifying means for specifying a region of the component, and a staining state of the sample by the first staining based on the first sample image, and at least of the predetermined sample component in the sample Observation image generation means for generating an observation image showing a part of the image in an identifiable manner , Characterized in that it comprises a.

  Here, the specimen constituent element refers to a constituent such as a cell constituent element such as a cell nucleus, a cell membrane, a cytoplasm, or a target molecule (antigen) expressed on the cell constituent element.

  The image processing method according to another aspect of the present invention includes a first staining and a second staining that stains a predetermined specimen component and is observed by an observation method different from the first staining. A first image acquisition step of acquiring a first specimen image by imaging a staining state of the specimen that has been subjected to the first staining and imaging a staining state of the specimen by the second staining; A second image acquisition step of acquiring a second specimen image; and an area specifying step of specifying an area of the predetermined specimen constituent element in the first specimen image based on the second specimen image; An observation image representing, based on the first specimen image, a staining state of the specimen by the first staining and identifiable at least a part of the predetermined specimen component in the specimen An observation image generation step to be generated.

  According to another aspect of the present invention, there is provided an image processing program in which a first staining and a predetermined specimen component are stained in a computer, and the second staining is observed by an observation method different from the first staining. A first image acquisition step of acquiring a first specimen image by imaging a staining state of the specimen subjected to multiple staining and acquiring the first staining image; and a staining state of the specimen by the second staining. A second image acquisition step of capturing and acquiring a second specimen image; and an area for specifying an area of the predetermined specimen component in the first specimen image based on the second specimen image Based on the specific step and the first specimen image, the staining state of the specimen by the first staining is represented, and at least a part of the predetermined specimen component in the specimen is identifiable Observation image generation step for generating an observation image , Characterized in that for the execution.

  According to the present invention, the first staining image is obtained by imaging the staining state of the sample by the first staining, and the second staining of the sample observed by an observation method different from the first staining is obtained. A stained state, that is, a second specimen image can be obtained by imaging a predetermined specimen component in the specimen, and a predetermined specimen configuration in the first specimen image is obtained based on the second specimen image. The area of the element can be specified. Then, it is possible to generate an observation image representing the staining state of the specimen by the first staining and representing at least a part of predetermined specimen constituent elements in the specimen in an identifiable manner. Therefore, it is possible to appropriately specify a region of a predetermined sample component in the sample and improve visibility during observation.

FIG. 1 is a block diagram illustrating an example of the overall configuration of the microscope system according to the first embodiment. FIG. 2 is a schematic diagram illustrating a schematic configuration of the microscope. FIG. 3 is a diagram showing excitation wavelength characteristics and fluorescence wavelength characteristics of LDS751 / DNA. FIG. 4 is a diagram showing the spectral transmission characteristics of the first bandpass filter in the first embodiment. FIG. 5 is a diagram showing the spectral transmission characteristics of the first dichroic mirror in the first embodiment. FIG. 6 is a diagram showing the spectral transmission characteristics of the second dichroic mirror in the first embodiment. FIG. 7 is a flowchart illustrating a processing procedure performed by the image processing apparatus according to the first embodiment. FIG. 8 is an explanatory diagram illustrating a processing procedure performed by the image processing apparatus according to the first embodiment. FIG. 9 is a block diagram illustrating an example of the overall configuration of the microscope system according to the second embodiment. FIG. 10 is a diagram showing excitation wavelength characteristics and fluorescence wavelength characteristics of indocyanine green. FIG. 11 is a diagram showing the spectral transmission characteristics of the first bandpass filter in the second embodiment. FIG. 12 is a diagram showing the spectral transmission characteristics of the first dichroic mirror in the second embodiment. FIG. 13 is a diagram showing the spectral transmission characteristics of the second dichroic mirror in the second embodiment. FIG. 14 is a flowchart illustrating a processing procedure performed by the image processing apparatus according to the second embodiment. FIG. 15 is an explanatory diagram illustrating a processing procedure performed by the image processing apparatus according to the second embodiment. FIG. 16 is a block diagram illustrating an example of the overall configuration of the microscope system according to the third embodiment. FIG. 17 is a flowchart illustrating a processing procedure performed by the image processing apparatus according to the third embodiment. FIG. 18 is a block diagram illustrating an example of the overall configuration of the microscope system according to the fourth embodiment. FIG. 19 is a flowchart illustrating a processing procedure performed by the image processing apparatus according to the fourth embodiment.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments. Moreover, in description of drawing, the same code | symbol is attached | subjected and shown to the same part.

(Embodiment 1)
FIG. 1 is a block diagram illustrating an example of the overall configuration of a microscope system 1 according to the first embodiment. As shown in FIG. 1, the microscope system 1 is configured by connecting a microscope 10 and an image processing device 70 realized by using a general-purpose computer such as a workstation or a personal computer so that data can be transmitted and received.

  FIG. 2 is a schematic diagram illustrating a schematic configuration of the microscope 10. The microscope 10 performs visible light observation and fluorescence observation of the specimen S to be observed. In the first embodiment, for example, HE staining is performed as the first staining, and staining using a fluorescent dye that specifically binds to a cell nucleus, which is an example of a specimen component, as the second staining (fluorescent label) The biological tissue specimen that has been subjected to () is the specimen S to be observed, and the microscope 10 performs visible light observation and fluorescence observation of the specimen S. More specifically, the specimen S is a biological tissue specimen in which the cell nucleus is stained blue-purple with the dye H and the cytoplasm and connective tissue are stained light red with the dye E. In addition, this biological tissue specimen is obtained by fluorescently labeling the cell nucleus with, for example, LDS751 / DNA (manufactured by Invitrogen), which is an example of a near-infrared fluorescent dye.

  FIG. 3 is a diagram showing the excitation wavelength characteristic and the fluorescence wavelength characteristic of LDS751 / DNA, in which a change curve L11 of the excitation wavelength characteristic is indicated by a one-dot chain line, and a change curve L13 of the fluorescence wavelength characteristic is indicated by a two-dot chain line. As shown in FIG. 3, LDS751 / DNA has an excitation wavelength around 550 nm and a fluorescence wavelength around 710 nm. Therefore, when light (excitation light) near 550 nm is irradiated, light (fluorescence) near 710 nm is emitted.

  As shown in FIG. 2, the microscope 10 that performs visible light observation and fluorescence observation of the specimen S transmits from the stage 2 on which the specimen S is placed and the back side of the specimen S placed on the stage 2. A transmission illumination optical system 3 for irradiating illumination light, an epi-illumination optical system 4 for irradiating epi-illumination light from the surface side of the specimen S placed on the stage 2, and an observation image of the specimen S are visually observed. And an imaging observation system 6 for observing by observing the image of the specimen S by capturing the image and displaying it on the screen.

  The microscope 10 uses transmission illumination optical system 3 or epi-illumination optical system 4 selectively according to the type of specimen S to be observed, the purpose of observation, etc. Observation of the specimen S and observation of the specimen S by epi-illumination using the epi-illumination optical system 4 can be performed. In the following description, it is assumed that the visible light observation of the specimen S is performed using the transmission illumination optical system 3, and the fluorescence observation of the specimen S is performed using the epi-illumination optical system 4.

  The stage 2 is driven in a plane perpendicular to the observation optical axis (the optical axis of the objective lens 51 constituting the naked eye observation system 5) (in the XY plane) and in the direction of the observation optical axis (Z-axis direction) by a driving mechanism (not shown). The region on the specimen S to be observed is adjusted by this, and the specimen S is focused and moved (focused) along the observation optical axis.

  The transmission illumination optical system 3 includes a first lamp 31, a first collector lens 32, a first band pass filter 33, a total reflection mirror 34, and a condenser lens 35.

  The first lamp 31 is realized by, for example, a halogen lamp, and emits light in a wavelength range from the visible range to the near infrared range as transmitted illumination light. In the first embodiment, the first lamp 31 transmits at least a part of light in the visible range because the specimen S is observed with the transmission illumination by using the transmission illumination optical system 3 to perform the transmission illumination. What is necessary is just to inject | emit the light which contains it as transmitted illumination light. The first collector lens 32 collects the transmitted illumination light from the first lamp 31.

  The first band-pass filter 33 is for limiting the wavelength range of the transmitted illumination light applied to the specimen S to a predetermined wavelength range. Of the transmitted illumination light collected by the first collector lens 32, Transmits light in a predetermined wavelength region. FIG. 4 is a diagram showing the spectral transmission characteristics of the first bandpass filter 33 in the first embodiment. In the first embodiment, as shown in FIG. 4, the first bandpass filter 33 has a wavelength of 400 nm to 700 nm including light in the visible range and light (excitation light) at the excitation wavelength (near 550 nm) of LDS751 / DNA. It has the characteristic of transmitting light in the range and blocking light outside this wavelength range.

  The total reflection mirror 34 reflects the transmitted illumination light of a predetermined wavelength range that has passed through the first bandpass filter 33 so as to irradiate from the back side of the sample S. The transmitted illumination light transmitted through the condenser lens 35 subsequent to the total reflection mirror 34 is irradiated to the back side of the sample S on the stage 2 and enters the objective lens 51 as observation light.

  The epi-illumination optical system 4 includes a second lamp 41, a second collector lens 42, a second band pass filter 43, and a first dichroic mirror 44.

  The second lamp 41 can be realized with the same configuration as the first lamp 31. However, in the first embodiment, since the specimen S is subjected to fluorescence observation by epi-illuminating the specimen S using the epi-illumination optical system 4, the second lamp 41 is in the near infrared where the specimen S is labeled. In the first embodiment, any light that includes at least part of the excitation wavelength of the LDS751 / DNA (near 550 nm) may be emitted as incident illumination light. The second collector lens 42 collects the incident illumination light from the second lamp 41. The second band-pass filter 43 is for limiting the wavelength range of the illumination light irradiated to the specimen S to a predetermined wavelength region. Of the epi-illumination light collected by the second collector lens 42, Transmits light in a predetermined wavelength region. The second band pass filter 43 has the same characteristics as the first band pass filter 33 shown in FIG. 4, for example.

  The first dichroic mirror 44 reflects epi-illumination light in a predetermined wavelength range that has passed through the second bandpass filter 43 so as to irradiate from the surface side of the specimen S, and transmits observation light that has passed through the objective lens 51. To do. FIG. 5 is a diagram showing the spectral transmission characteristics of the first dichroic mirror 44 in the first embodiment. In the first embodiment, as shown in FIG. 5, the first dichroic mirror 44 has a characteristic of transmitting about 50% of light in a wavelength range shorter than 700 nm and reflecting about 50%. Further, the first dichroic mirror 44 has a characteristic of transmitting light in a wavelength range longer than 700 nm, that is, a characteristic of transmitting light (fluorescence) having a fluorescence wavelength of LDS751 / DNA (near 710 nm).

  The epi-illumination light reflected by the first dichroic mirror 44 is irradiated on the surface side of the specimen S through the objective lens 51. As described above, in the first embodiment, fluorescence observation of the specimen S is performed using the epi-illumination optical system 4. At this fluorescence observation, light (excitation light) having the excitation wavelength of the LDS751 / DNA is emitted from the specimen S. Irradiated on the surface side of When the sample S is irradiated with the excitation light in this way, the LDS751 / DNA in the sample S is excited to emit fluorescence, and the fluorescence emitted from the LDS751 / DNA enters the objective lens 51 as observation light.

  The naked eye observation system 5 includes an objective lens 51, an optical path conversion mirror 52, a prism 53, and an eyepiece lens 54 to which an observer's eyes are applied in order to visually observe an observation image of the specimen S.

  The optical path conversion mirror 52 is detachably provided on the observation optical axis, and switches the optical path of the observation light to the naked eye observation system 5 or the imaging observation system 6. Specifically, the optical path conversion mirror 52 is inserted into or removed from the observation optical axis, whereby the observation light transmitted through the first dichroic mirror 44 is transmitted to the second dichroic mirror 61 side, and The total reflection state in which the observation light transmitted through the first dichroic mirror 44 is reflected to the eyepiece lens 54 side can be switched.

  When the optical path conversion mirror 52 is inserted on the observation optical axis and is in the total reflection state, the prism 53 reflects the observation light reflected by the optical path conversion mirror 52 in a predetermined direction and guides it to the eyepiece lens 54. The observation light introduced into the eyepiece lens 54 (visible light observation image of specimen S / fluorescence observation image of specimen S) is visually observed by an observer.

  On the other hand, when the optical path conversion mirror 52 is removed from the observation optical axis and is in a totally transmissive state, the observation light transmitted through the first dichroic mirror 44 (the visible light observation image of the sample S / the sample S The fluorescence observation image) enters the second dichroic mirror 61 of the imaging observation system 6 through the prism 53.

  The imaging observation system 6 includes a second dichroic mirror 61, an excitation light cut filter 62, an image intensifier (II) 63, a fluorescence image imaging unit 64, and a visible light image imaging unit 65. .

  The second dichroic mirror 61 reflects a fluorescent (infrared light) component that is light in a wavelength region including at least a part of the fluorescence wavelength of LDS751 / DNA among the observation light transmitted through the prism 53, and visible light. The visible light component which is light in a wavelength region including at least a part of the light is transmitted and spatially separated. FIG. 6 is a diagram showing the spectral transmission characteristics of the second dichroic mirror 61 in the first embodiment. In the first embodiment, as shown in FIG. 6, the second dichroic mirror 61 transmits light in a wavelength range shorter than 700 nm including light in the visible light range (visible light component), while LDS 751 / It has a characteristic of reflecting light in a wavelength range longer than 700 nm including a fluorescent component of DNA. The fluorescent component reflected by the second dichroic mirror 61 enters the excitation light cut filter 62. On the other hand, the visible light component transmitted through the second dichroic mirror 61 is imaged on the imaging surface of the visible light image capturing unit 65 via a necessary optical system.

  The excitation light cut filter 62 is a spectral transmission characteristic that blocks the excitation light component and transmits only the fluorescence component. In the first embodiment, the excitation light cut filter 62 blocks the excitation light component of LDS751 / DNA and transmits the fluorescence component of LDS751 / DNA. Spectral transmission characteristics. The fluorescence component incident on the excitation light cut filter 62 is removed by the excitation light cut filter 62 and is incident on the image intensifier 63.

  The image intensifier 63 is sensitive to wavelengths near 350 nm to 910 nm, for example, and the observation light that has passed through the excitation light cut filter 62 is amplified by the image intensifier 63 to be emitted from the fluorescence image capturing unit 64. An image is formed on the imaging surface.

  The fluorescence image capturing unit 64 is configured by a camera including an image sensor such as a CCD or a CMOS, and captures the fluorescence component reflected by the second dichroic mirror 61 (fluorescence observation image of the sample S) and converts it into an electrical signal. To do. The pixel value of each pixel of the image data obtained by the fluorescence image capturing unit 64 corresponds to the fluorescence intensity at each point (sample point) on the sample S. Here, the sample point is each position on the sample S corresponding to each pixel position of the image sensor that constitutes the camera. The obtained image data of the fluorescence image as the second specimen image of the specimen S is output to the image processing device 70.

  On the other hand, the visible light image capturing unit 65 applies, for example, an imaging method disclosed in Patent Document 3 and switches a predetermined number (for example, 16) of band-pass filters having different wavelength bands (bands) of transmitted light. The sample S is composed of a multiband camera that captures a visible light observation image of the specimen S by a multi-band imaging in a frame sequential manner. When 16 bandpass filters are used, the visible light image is obtained as a 16-band multiband image. The visible light image capturing unit 65 is, for example, a filter in which the predetermined number of bandpass filters are mounted on a filter wheel, and each bandpass filter is selectively inserted on the observation optical axis by rotating the filter wheel. A visible light component (visible light observation image of the specimen S) that is transmitted through the second dichroic mirror 61 and is converted into an electrical signal by a multi-band imaging of the visible light component that is configured by a unit and a camera equipped with an image sensor such as a CCD or CMOS. . The pixel value of each pixel of the image data obtained by the visible light image capturing unit 65 corresponds to the intensity of visible light in each band pass filter, and the pixel value (spectral data) for each band for each sample point. can get. The obtained image data of the visible light image as the first specimen image of the specimen S is output to the image processing device 70.

  On the other hand, the image processing apparatus 70 includes an input unit 71, a display unit 72, a recording unit 73, an image processing unit 74, and a control unit 77 that controls each unit of the apparatus.

  The input unit 71 is realized by various input devices such as a keyboard, a mouse, a touch panel, and various switches, and outputs an input signal corresponding to an operation input to the control unit 77. The display unit 72 is realized by a display device such as an LCD, an EL display, or a CRT display, and displays various screens based on a display signal input from the control unit 77.

  The recording unit 73 is realized by various IC memories such as ROM and RAM such as flash memory that can be updated and stored, a built-in hard disk connected by a data communication terminal, an information recording medium such as a CD-ROM, and a reading device thereof. It is. In the recording unit 73, a program for operating the microscope system 1 and realizing various functions of the microscope system 1, data used during the execution of the program, and the like are recorded in advance or processed. Recorded temporarily each time. In the recording unit 73, image data of a visible light image and image data of a fluorescence image input from the microscope 10, image data of an abnormal nucleus region identification image as an observation image generated by the image processing unit 74, and the like are recorded. The The recording unit 73 records an image processing program 731 for realizing the processing of the first embodiment and generating a visible light image, a fluorescent image, and an abnormal nucleus region identification image.

  The image processing unit 74 includes a nuclear region specifying unit 741 as a region specifying unit, a spectral characteristic estimating unit 743, a dye amount estimating unit 745 as a dye amount estimating unit, and an abnormal nuclear region detecting unit 747 as a target location detecting unit. And an abnormal nucleus region identification image generating unit 749 as an observation image generating unit and an abnormal mark adding unit 753. The nucleus region specifying unit 741 extracts the nucleus region (nucleus region) in the fluorescence image based on the pixel value (fluorescence intensity) of each pixel constituting the fluorescence image, thereby identifying the nucleus region in the visible light image. Identify. The spectral characteristic estimation unit 743 estimates the spectral transmittance at each sample point on the corresponding sample S for each pixel constituting the visible light image. The pigment amount estimation unit 745 estimates the pigment amount of the sample S based on the spectral transmittance estimated for each pixel of the visible light image by the spectral characteristic estimation unit 743. The abnormal nucleus region detection unit 747 is an abnormal region that is an example of a portion to be identified from the nucleus region based on the pigment amount estimated by the pigment amount estimation unit 745 for the pixels of the nucleus region identified by the nucleus region identifying unit 741. The cell nucleus area (abnormal nucleus area) is detected. The abnormal nucleus region identification image generation unit 749 generates an image (abnormal nucleus region identification image) representing the staining state (dye state of dye H and dye E) of the specimen S by HE staining and identifying and displaying the abnormal nucleus region. To do. The abnormal nucleus region identification image generating unit 749 includes an HE stained image combining unit 751 as a stained image combining unit. The HE-stained image synthesizing unit 751 synthesizes a HE-stained image as a sample-stained image representing the staining state of the sample S by HE staining. The abnormal mark adding unit 753 performs a process of adding an abnormal mark indicating that the cell nucleus corresponding to the vicinity of the abnormal nucleus region in the abnormal nucleus region identification image is abnormal and displaying it on the display unit 72.

  The control unit 77 performs an instruction and data transfer to each unit constituting the microscope 10 based on an input signal input from the input unit 71, a program and data recorded in the recording unit 73, or image processing. The operation of the entire microscope system 1 is comprehensively controlled by giving instructions to each unit constituting the apparatus 70 and transferring data. The control unit 77 includes a visible light image acquisition control unit 771 as a first image acquisition unit and a fluorescent image acquisition control unit 772 as a second image acquisition unit. The visible light image acquisition control unit 771 acquires the visible light image of the specimen S by controlling the operation of the microscope 10. The fluorescence image acquisition control unit 772 acquires the fluorescence image of the specimen S by controlling the operation of the microscope 10.

  FIG. 7 is a flowchart illustrating a processing procedure performed by the image processing apparatus 70 according to the first embodiment. FIG. 7 is an explanatory diagram for explaining a processing procedure performed by the image processing apparatus 70. The processing described here is realized by each unit of the image processing apparatus 70 operating according to the image processing program 731 recorded in the recording unit 73.

  As shown in FIG. 7, first, the visible light image acquisition control unit 771 controls the operation of the microscope 10 as a first image acquisition step and a first image acquisition step, and displays a visible light observation image of the specimen S. Band imaging is performed to obtain a visible light image of the specimen S (step a1). The acquired image data of the visible light image is recorded in the recording unit 73. In addition, as a second image acquisition step and a second image acquisition step, the fluorescence image acquisition control unit 772 controls the operation of the microscope 10 to capture the fluorescence observation image of the sample S and acquire the fluorescence image of the sample S. (Step a3). The acquired fluorescence image data is recorded in the recording unit 73. In step a1 and step a3, the visible light image and the fluorescence of the sample S in which the sample points on the sample S corresponding to each pixel coincide with each other by imaging the same sample S range (the visual field range of the objective lens 51). Get an image.

  As a result, for example, a visible light image shown in FIG. 8A is acquired, and a fluorescent image shown in FIG. 8B is acquired. The visible light image shown in FIG. 8A and the fluorescence image shown in FIG. 8B are obtained by imaging a specimen S including four cell nuclei. Here, as described above, in the specimen S to be observed in the first embodiment, the cell nucleus is stained blue-purple with the dye H, the cytoplasm and connective tissue are stained light red with the dye E, and the sample S is stained with the LDS751 / DNA. The cell nucleus is fluorescently labeled. Therefore, in the visible light observation, the staining state of the specimen S by HE staining is imaged, and the visible light image shows cell nuclei C21 to C24 (cell nucleus C21 in the HE-stained specimen S as shown in FIG. 8A). ~ C24 dyeing state with dye H). On the other hand, in the fluorescence observation, since the staining state of the specimen S with LDS751 / DNA, that is, only the fluorescence emitted by the LDS751 / DNA is imaged, the fluorescence image is shown in FIG. The boundary between the cell nuclei C21 to C24 is obtained as a clear image.

  If the visible light image and the fluorescence image are acquired as described above, the nucleus region specifying unit 741 of the image processing unit 74 first performs a nucleus from the fluorescence image acquired in step a3 as the region specifying step and the region specifying step. An area is extracted (step a5). Specifically, the nucleus region specifying unit 741 performs threshold processing on the pixel value (fluorescence intensity) of each pixel of the fluorescence image, for example, and uses pixels whose fluorescence intensity is greater than or equal to a predetermined threshold as a pixel in the nucleus region. Extract. Then, the nucleus region specifying unit 741 specifies the same pixel in the visible light image as the pixel extracted as the nucleus region in the fluorescence image as the pixel of the nucleus region in the visible light image (step a7). At this time, the nucleus region specifying unit 741 divides the pixels of the nucleus region in the visible light image for each connected component, and attaches a unique label to each divided pixel group, so that each pixel group is assigned to one cell nucleus. Specify as an area.

  Subsequently, the spectral characteristic estimation unit 743 estimates (calculates) the spectral transmittance for each wavelength (for each band) at each sample point on the corresponding sample S based on the pixel value of each pixel of the visible light image. (Step a9). As a method for estimating the spectral transmittance from the multiband image, for example, known Wiener estimation can be used. Then, the dye amount estimation unit 745 estimates (calculates) the dye amount of the sample S based on the spectral transmittance of each pixel estimated in step a9 (step a11). As described above, in the visible light observation, since the staining state of the specimen S by HE staining is imaged, there are two pigments to be estimated, that is, pigment H and pigment E. Note that dye components other than dye H and dye E may be included in the estimation target as appropriate to estimate the amount of dye. Specifically, the color of eosin stained with red blood cells or unstained red blood cells may be included in the estimation target. The estimation of the dye amount can be realized by applying a known technique described in Patent Document 2, for example.

Here, the estimation of the pigment amount will be briefly described. In general, in a material that transmits light, a Lambert bale (Lambert) expressed by the following equation (1) is set between the intensity I ₀ (λ) of incident light and the intensity I (λ) of emitted light for each wavelength λ. -Beer) law is known to hold. k (λ) is a material-specific value determined depending on the wavelength, and d represents the thickness of the material. Further, the left side of the formula (1) means the spectral transmittance t (λ).

For example, when the specimen is dyed with n kinds of dyes of dye 1, dye 2,..., Dye n, the following expression (2) is established at each wavelength λ according to the Lambert-Beer law. k ₁ (λ), k ₂ (λ),..., k _n (λ) represent k (λ) corresponding to dye 1, dye 2,..., dye n, respectively. It is a dye spectrum of each dye dye | stained. The d _1, d _2, ..., d _n is the dye 1 in the sample point on the specimen S corresponding to each pixel of the multi-band image, Dye 2, ..., the virtual thickness of the dye n To express. Since the dye is inherently dispersed in the specimen, the concept of thickness is not accurate, but how much dye is compared to the assumption that the specimen is stained with a single dye. It is an indicator of the relative amount of pigment that indicates whether or not the selenium is present. That, d _1, d _2, it can be said, ..., d _n respectively dyes 1, Dye 2, ..., and represents the amount of dye pigments n. K ₁ (λ), k ₂ (λ),..., K _n (λ) are samples individually dyed with each of the dye 1, dye 2,. By preparing in advance and measuring the spectral transmittance with a spectroscope, it can be easily obtained from the Lambert-Beer law.

Taking the logarithm of both sides of equation (2), the following equation (3) is obtained.

When the element corresponding to the wavelength λ of the spectral transmittance estimated for each pixel of the visible light image in step a9 is t ^ (x, λ) and this is substituted into the equation (3), the following equation (4) is obtained. Here, t ^ indicates that a hat "^" representing an estimated value is attached on t.

In Equation (4), there are _n unknown variables d ₁ , d ₂ ,..., D _n , so that Equation (4) can be solved simultaneously for at least n different wavelengths λ. it can. In order to increase the accuracy, the multiple regression analysis may be performed by simultaneously formula (4) for n or more different wavelengths λ.

  The above is a simple procedure for calculating the amount of pigment, but the pigments to be estimated in Embodiment 1 are pigment H and pigment E, and n = 2. The dye amount estimation unit 745 estimates the dye amounts of the dye H and the dye E fixed to each corresponding sample point based on the spectral transmittance of each pixel estimated in step a9.

  Subsequently, the abnormal nucleus region detection unit 747 determines normality / abnormality of each cell nucleus based on the pigment amount of the pigment H estimated in Step a11 for the pixels constituting the nucleus region identified in Step a7. An abnormal nucleus region is detected (step a13). For example, a specimen containing normal cell nuclei is prepared in advance and subjected to HE staining to obtain a visible light image. Then, the amount of dye H fixed to normal cell nuclei is obtained by estimating the amount of dye in the same procedure as in steps a9 to a11, and recorded in the recording unit 73 as a reference H dye amount that is a reference feature value. Keep it. In step a13, the dye amount of the dye H of the pixel in the nucleus area specified in step a7 is used as a feature value, and the dye amount of the dye H is compared with the reference H dye amount. For example, the average value, the maximum value, the minimum value, and the like are calculated from the dye amount of the dye H of the pixels constituting each nucleus region, and compared with the reference H dye amount. Then, a cell nucleus whose calculated value does not satisfy the reference H dye amount is determined to be abnormal. If the amount is higher than the reference H dye amount, the cell nucleus is determined to be normal. Note that the reference H dye amount is not limited to the dye H dye amount fixed to a normal cell nucleus, but may be the dye H dye amount fixed to an abnormal cell nucleus.

  Subsequently, as an observation image generation step and an observation image generation step, first, the HE-stained image synthesis unit 751 of the abnormal nucleus region identification image generation unit 749 calculates each pixel based on the pigment amount estimated for each pixel in step a11. By calculating pixel values (RGB values), a HE-stained image representing the staining state of the sample S by HE staining is synthesized (step a15). Here, the process of converting the pigment amount into a pixel value (RGB value) can be realized by applying a known technique described in Patent Document 2, for example.

To explain briefly procedure First, dye amount d ₁ estimated in step _{a11, d 2, ···, d} n to the correction coefficient α _1, α _2, ···, and the multiplied by alpha _n, respectively formula Substituting into (2), the following equation (5) is obtained. Then, the dye amount is appropriately corrected by giving a predetermined value to the correction coefficient α _n , and the combined spectral transmittance t ^* (x, λ) is calculated from the corrected dye amount. Here, by adjusting the value of the correction coefficient α _n , the dye amount at each sample point can be virtually corrected, and the variation in the staining state can be corrected. The specimen is usually made by a clinical laboratory technician. For this reason, individual differences such as the experience of clinical laboratory technicians affect, and it is difficult to keep the staining state of the specimen uniform. In addition, the staining state may vary depending on the surrounding environment such as individual differences in the specimen itself and room temperature at the time of specimen preparation. By adjusting the correction coefficient α _n , it is possible to correct the variation in the staining state to a desired staining state. If correction of the dye amount is unnecessary, the correction coefficient α _n may be set to 1. In this case, the spectral transmittance calculated in step a9 without calculating the combined spectral transmittance t ^* (x, λ). May be used.

For an arbitrary point (pixel) x of the captured multiband image, between the pixel value g (x, b) in the band b and the spectral transmittance t (x, λ) of the corresponding point on the sample The relationship of the following equation (6) based on the camera response system is established. λ is the wavelength, f (b, λ) is the spectral transmittance of the b-th filter, s (λ) is the spectral sensitivity characteristic of the camera, e (λ) is the spectral radiation characteristic of the illumination, and n (b) is in the band b. Represents each observation noise. b is a serial number for identifying a band. For example, when the number of bands is 16, the integer is an integer that satisfies 1 ≦ b ≦ 16.

Therefore, the pixel value g ^* (x, b) of each pixel can be obtained by substituting the equation (5) into the above equation (6) and obtaining the pixel value according to the following equation (7). In this case, the observation noise n (b) may be calculated as zero.

  When the HE-stained image is synthesized as described above, the abnormal nucleus region identification image generation unit 749 identifies the abnormal nucleus region detected in step a13 in the synthesized HE-stained image. An image is generated (step a17). Specifically, the abnormal nucleus region identification image generation unit 749 replaces the pixel values of the pixels constituting the abnormal nucleus region in the HE-stained image with a predetermined display color (for example, green), thereby causing the abnormal nucleus region in the HE-stained image. An abnormal nucleus region identification image in which the region is identified and displayed is generated. For example, it is assumed that the cell nucleus C22 is determined to be abnormal among the cell nuclei C21 to C24 shown in the visible light image of FIG. In this case, as shown in FIG. 8C, in the HE-stained image representing the staining state of the specimen S by HE staining, the abnormal nucleus region in which the display color of the region of the cell nucleus C22 that is the abnormal nucleus region is replaced. An identification image is generated.

  Note that the display color of the abnormal nucleus region may be fixedly set in advance, or may be determined by receiving a display color selection operation by the user. In addition, the abnormal nuclear region identification display mode is not limited to the illustrated one, and any display mode that can be distinguished from other nuclear regions, for example, blinking the abnormal nuclear region is displayed. It may be a thing.

  Then, as shown in FIG. 7, the abnormal mark adding unit 753 performs a process of adding an abnormal mark near the abnormal nucleus region in the generated abnormal nucleus region identification image and displaying it on the display unit 72 (step a19). . For example, in the example of FIG. 8D, an abnormal mark M2 surrounding the cell nucleus C22 region is added in the vicinity of the cell nucleus C22 region, and the abnormal nucleus region identification image is displayed on the display unit 72 in this state. Presented to the user.

  The abnormality mark is not limited to those illustrated. For example, a predetermined symbol indicating that the nuclear region is abnormal may be added as an abnormal mark in the vicinity of the abnormal nuclear region. Or it is good also as a structure which adds the abnormal mark which showed that the nucleus area | region is abnormal by the display of the text in the vicinity of the abnormal nucleus area | region.

  In step a19, the abnormal nucleus region identification image is displayed on the display unit 72. However, according to the user operation, the HE stained image and the abnormal nucleus region identification image are switched and displayed on the display unit 72, or It is also possible to display the HE-stained image and the abnormal nucleus region identification image side by side on the display unit 72. According to this, it is also possible to switch between observation of the abnormal nucleus region and observation of the staining state by HE staining (morphological observation) or simultaneously.

  As described above, according to the first embodiment, the fluorescence observation image and the visible light of the specimen S that has been stained with HE and stained (fluorescently labeled) with a fluorescent dye that specifically binds to the cell nucleus. By acquiring an image and extracting a cell nucleus region (nucleus region) in the fluorescence image based on the pixel value (fluorescence intensity) of the fluorescence image, the nucleus region in the visible light image can be specified. As described above, since the fluorescence image is obtained as an image in which the boundary of the cell nucleus in the specimen S is clear, the nucleus region is extracted from this fluorescence image, so that the region of the cell nucleus in the visible light image is appropriately specified. be able to.

  Further, by observing the specimen S with fluorescence, the specimen S is stained with LDS751 / DNA, that is, only the fluorescence emitted by the LDS751 / DNA is captured and acquired as a fluorescence image, while the specimen S is observed with visible light. Since the HE-stained image representing the state of staining by HE staining can be synthesized from the visible light image acquired in step 1, the staining for specifying the region of the cell nucleus does not affect the state of staining by HE staining. Therefore, the nucleus region can be identified without impairing the staining state by HE staining.

  In addition, normality / abnormality of individual cell nuclei can be determined using the amount of pigment estimated from the pixel value of each region in the visible light image. That is, the pigment amount of the sample S is estimated based on the pixel value (spectral data for each band) of each pixel of the visible light image, and the pigment amount of the pigment H of the pixel constituting the nucleus region is acquired in advance. The normal / abnormality of the corresponding cell nucleus can be appropriately determined by comparing the amount of the dye H fixed to the normal cell nucleus (reference H dye amount). Here, in HE staining, cell nuclei are stained with dye H. Conventionally, in this HE staining, for example, the dye amount (concentration) of dye H fixed to abnormal cell nuclei such as cancer cell nuclei, It is known that the amount (concentration) of the dye H fixed to a normal cell nucleus is small (abnormal cell nuclei are stained lighter than normal cell nuclei). According to the first embodiment, since the nucleus region in the visible light image can be appropriately specified, the abnormal nucleus region can be accurately detected from the identified nucleus region.

  Then, an HE-stained image representing the staining state of the sample S by HE staining is synthesized, and an abnormal nucleus region identification image in which the abnormal nucleus region in the HE-stained image is identified and displayed with a predetermined display color is generated and displayed on the display unit 72. Can be displayed. Accordingly, the abnormal nucleus region can be displayed with high visibility in the HE-stained image and presented to a user such as a doctor. Therefore, for the user, the abnormal nucleus region can be easily distinguished from other nucleus regions during observation. Can improve the diagnostic accuracy. In addition, with the HE-stained image representing the staining state of the specimen S by HE staining, the observation of the staining state of the specimen S by HE staining (morphological observation) can be performed with good visibility.

  In the first embodiment, the reference H dye amount used for determining normality / abnormality of the cell nucleus is acquired in advance and recorded in the recording unit 73. Here, the staining state of the specimen is the staining environment such as the type of staining solution actually used (manufacturer of the staining solution), the combination of staining solutions to be used, the staining time, the staining location (medical facility), etc. It may vary depending on attributes relating to the process, and the reference H dye amount may also vary depending on the dyeing state. Therefore, the reference H dye amount in different staining states may be acquired in advance and recorded in the recording unit 73 as a database. Then, the reference H dye amount corresponding to the staining state of the specimen S actually to be observed may be read out and acquired from the database and used for determination of normal / abnormal cell nuclei. Alternatively, the reference H dye amount used for determination may be acquired from a database of the external device by communicating with an external device having a reference H dye amount database for each staining state.

  In the first embodiment, LDS751 / DNA is exemplified as the fluorescent dye. However, the exemplified fluorescent dye is an example, and the type thereof is not limited thereto. That is, instead of LDS751 / DNA, another fluorescent dye that specifically binds to the cell nucleus may be used. In this case, as the first band-pass filter 33, the first dichroic mirror 44, and the second dichroic mirror 61 constituting the microscope 10, the spectral characteristics according to the excitation wavelength characteristics and the fluorescence wavelength characteristics of the fluorescent dye to be used are used. A material having transmission characteristics may be used.

  In the above-described embodiment, the spectral transmittance is estimated based on the pixel value of each pixel of the visible light image, and the pigment amount of the sample S is estimated based on the estimated spectral transmittance of each pixel ( Calculation). On the other hand, a weight is set higher for each pixel in the visible light image identified as the nucleus region in step a7 in FIG. 7 than the pixels other than the nucleus region, and the value of the pigment amount is weighted by the set weight. The calculated value (weighted pigment amount) may be calculated.

(Embodiment 2)
FIG. 9 is a block diagram illustrating an example of the overall configuration of the microscope system 1a according to the second embodiment. In FIG. 9, the same components as those in the first embodiment are denoted by the same reference numerals. As shown in FIG. 9, the microscope system 1a is configured by connecting a microscope 10a and an image processing device 70a so that data can be transmitted and received.

  Here, the specimen S to be observed by the microscope 10a according to the second embodiment is obtained by subjecting a biological tissue specimen that has been subjected to HE staining as the first staining and further immunostaining as the second staining to obtain a desired target. An antibody against a molecule (antigen) is allowed to act, and this antibody is stained (fluorescently labeled) with a fluorescent dye that specifically binds to the acted antibody. For example, an actin antibody is allowed to act on a HE-stained biological tissue specimen, and the actin antibody is fluorescently labeled with indocyanine green that specifically binds to the actin antibody. In the second embodiment, the binding site of the actin antibody in the sample S corresponds to the sample component, and the entire region of the sample component is the identification target location.

  FIG. 10 is a diagram showing the excitation wavelength characteristic and the fluorescence wavelength characteristic of indocyanine green. The change curve of the excitation wavelength characteristic is indicated by a one-dot chain line, and the change curve of the fluorescence wavelength characteristic is indicated by a two-dot chain line. As shown in FIG. 10, indocyanine green has an excitation wavelength near 770 nm and a fluorescence wavelength near 810 nm. Therefore, when light near 770 nm (excitation light) is irradiated, light near 810 nm (fluorescence) is emitted.

  The microscope 10a for performing visible light observation and fluorescence observation of the specimen S can be realized with a configuration substantially the same as that of the microscope 10 according to the first embodiment described with reference to FIG. The spectral transmission characteristics of the dichroic mirror 44 and the second dichroic mirror 61 are different from those of the first embodiment.

  FIG. 11 is a diagram showing the spectral transmission characteristics of the first bandpass filter 33 in the second embodiment. In the second embodiment, as shown in FIG. 11, the first band-pass filter 33 has a wavelength of about 400 nm to 800 nm including light in the visible range and light (excitation light) in the excitation wavelength (near 770 nm) of indocyanine green. It has the characteristic of transmitting light in the wavelength range and blocking light outside this wavelength range.

  FIG. 12 is a diagram showing the spectral transmission characteristics of the first dichroic mirror 44 in the second embodiment. In the second embodiment, as shown in FIG. 12, the first dichroic mirror 44 transmits about 50% of light in the wavelength range shorter than 700 nm (visible light range), while reflecting about 50%. It has the characteristic to do. The first dichroic mirror 44 has a characteristic of reflecting light in a wavelength range of 700 nm to 800 nm including light (excitation light) having an excitation wavelength (near 770 nm) of indocyanine green. Further, the first dichroic mirror 44 has a characteristic of transmitting light in a wavelength range longer than 800 nm, that is, a characteristic of transmitting light (fluorescence) having a fluorescence wavelength of indocyanine green (around 810 nm).

  FIG. 13 is a diagram showing the spectral transmission characteristics of the second dichroic mirror 61 in the second embodiment. In the second embodiment, as shown in FIG. 13, the second dichroic mirror 61 transmits light in a wavelength range shorter than 800 nm including light in the visible light region (visible light component), while indocyanine. It has a characteristic of reflecting light in a wavelength range longer than 800 nm including a green fluorescent component.

  As shown in FIG. 9, the image processing device 70a includes an input unit 71, a display unit 72, a recording unit 73a, an image processing unit 74a, and a control unit 77 that controls each unit of the device.

  The recording unit 73a records an image processing program 731a for realizing the processing of the second embodiment and generating an antibody binding region identification image from the visible light image and the fluorescence image.

  The image processing unit 74a includes an antibody binding region specifying unit 755a as a region specifying unit, a spectral characteristic estimating unit 743, a dye amount estimating unit 745 as a dye amount estimating unit, and an antibody binding region identification image as an observation image generating unit. And a generation unit 757a. The antibody binding region specifying unit 755a specifies the antibody binding region in the visible light image by extracting the antibody binding region in the fluorescence image based on the pixel value (fluorescence intensity) of each pixel constituting the fluorescence image. . The antibody binding region identification image generation unit 757a generates, as an observation image, an image (antibody binding region identification image) representing the staining state of the specimen S by HE staining and identifying and displaying the antibody binding region. The antibody binding region identification image generating unit 757a includes an HE stained image combining unit 751 as a stained image combining unit and an H single stained image combining unit 759a as a single stained image combining unit. The H single stained image composition unit 759a synthesizes an H single stained image as a specimen single stained image that represents the staining state of the specimen S with only the dye H.

  FIG. 14 is a flowchart illustrating a processing procedure performed by the image processing apparatus 70a according to the second embodiment. FIG. 14 is an explanatory diagram for explaining a processing procedure performed by the image processing apparatus 70a. The process described here is realized by each part of the image processing apparatus 70a operating according to the image processing program 731a recorded in the recording part 73a.

  As shown in FIG. 14, first, the visible light image acquisition control unit 771 controls the operation of the microscope 10a as the first image acquisition step and the first image acquisition step, as in the first embodiment. A visible light observation image of S is subjected to multiband imaging, and a visible light image of the specimen S is acquired (step b1). Further, as in the second image acquisition step and the second image acquisition step, the fluorescence image acquisition control unit 772 controls the operation of the microscope 10a to capture the fluorescence observation image of the specimen S as in the first embodiment. Then, a fluorescence image of the specimen S is acquired (step b3).

  As a result, for example, a visible light image shown in FIG. 15A is acquired, and a fluorescent image shown in FIG. 15B is acquired. Here, as described above, the specimen S to be observed in the second embodiment is obtained by applying an actin antibody to a HE-stained biological tissue specimen and applying a fluorescent label to the actin antibody with indocyanine green. Therefore, the visible light image is obtained as an image representing the staining state of the specimen S by HE staining, as in the first embodiment (FIG. 15A). On the other hand, in the fluorescence observation, since the staining state of the sample S with indocyanine green, that is, only the fluorescence emitted by indocyanine green is imaged, the fluorescence image is shown in the sample S as shown in FIG. It is obtained as an image representing only the region (antibody binding region) E3 to which the actin antibody is bound.

  If the visible light image and the fluorescence image are acquired as described above, the antibody binding region specifying unit 755a of the image processing unit 74a first uses the fluorescence image acquired in step b3 as the region specifying step and the region specifying step. An antibody binding region is extracted (step b5). Specifically, the antibody binding region specifying unit 755a performs, for example, threshold processing on the pixel value (fluorescence intensity) of each pixel of the fluorescence image, and pixels whose fluorescence intensity is greater than or equal to a predetermined threshold value set in advance in the antibody binding region. Extract as pixels. Then, the antibody binding region specifying unit 755a specifies the same pixel in the visible light image as the pixel extracted as the antibody binding region in the fluorescence image as the pixel of the antibody binding region in the visible light image (step b7).

  Subsequently, in the same manner as in the first embodiment, the spectral characteristic estimation unit 743 performs each wavelength (for each band) at each sample point on the corresponding sample S based on the pixel value of each pixel of the visible light image. Spectral transmittance is estimated (calculated) (step b9). Then, the dye amount estimation unit 745 estimates (calculates) the dye amount of the sample S based on the spectral transmittance of each pixel estimated in step b9 (step b11).

  Subsequently, as the observation image generation step and the observation image generation step, first, the dye stained by the HE-stained image synthesis unit 751 of the antibody binding region identification image generation unit 757a estimated for each pixel in step b11 as in the first embodiment. By calculating the pixel value (RGB value) of each pixel based on the amount, a HE-stained image representing the staining state of the sample S by HE staining is synthesized (step b13).

Subsequently, the H single-stained image synthesis unit 759a calculates only the pigment H of the sample S by calculating the pixel value (RGB value) of each pixel based on the pigment amount of the pigment H estimated for each pixel in step b11. A single H-stained image representing the dyeing state is synthesized (step b15). For example, in the above equation (5), the correction coefficient α _n multiplied by the dye H is set to 1, and the correction coefficient α _n multiplied by the dye E is set to 0, whereby the spectral transmittance t ^* for only the dye amount of the dye H is targeted ^. (X, λ) is calculated, and the pixel value g ^* (x, b) may be obtained according to equation (7). In this way, in step b15, an image representing the dye amount of the dye H, that is, an image representing the staining state of only the dye H of the specimen S is synthesized as an H single-stained image.

  Then, the antibody binding region identification image generation unit 757a generates a combined HE stained image and an antibody binding region identification image in which the antibody binding region identified in step b7 is identified and displayed in the synthesized H stained image (step b17). Specifically, the antibody binding region identification image generation unit 757a identifies and displays the antibody binding region in the HE-stained image by replacing pixel values of pixels constituting the antibody-binding region in the HE-stained image with a predetermined display color. An antibody binding region identification image (antibody identification HE stained image) is generated. For example, as shown in FIG. 15C, an antibody binding region identification image in which the display color of the antibody binding region E3 is replaced in the HE stained image representing the staining state of the specimen S by HE staining is generated. Similarly, the antibody binding region identification image generation unit 757a replaces the pixel values of the pixels constituting the antibody binding region in the H single stained image with a predetermined display color, so that the antibody binding region in the H single stained image An antibody binding region identification image (antibody identification H single-stained image) is generated.

  Then, the antibody binding region identification image generation unit 757a performs processing for displaying the antibody binding region identification image generated in step b17, specifically, the antibody identification HE staining image and the antibody identification H single staining image on the display unit 72 ( Step b19). For example, the antibody binding region identification image generation unit 757a performs display processing on the display unit 72 by switching between the antibody identification HE stained image and the antibody identification H single stained image in accordance with a user operation. At this time, the HE-stained image and the H-stained image may also be appropriately switched and displayed on the display unit 72 according to the user operation. The display mode of the antibody identification HE stained image and the antibody identification H single stained image is not particularly limited, and the antibody identification HE stained image and the antibody identification H single stained image are displayed on the display unit 72 side by side. Alternatively, the HE-stained image or the H-stained image may be displayed side by side. In this way, the observation of the antibody binding region and the observation of the staining state by HE staining (morphological observation) can be performed simultaneously or while switching. That is, it is possible to observe the staining state by HE staining (morphological observation) using the HE stained image while observing the antibody binding region using the antibody identifying HE stained image or the antibody identifying H single stained image.

  As described above, according to the second embodiment, a biological tissue specimen subjected to HE staining is further subjected to immunostaining, and an antibody that has acted on a desired target molecule (antigen) is stained (fluorescent label). Obtain the fluorescence observation image and visible light image of the specimen S, and identify the antibody binding region in the visible light image by extracting the antibody binding region in the fluorescent image based on the pixel value (fluorescence intensity) of the fluorescence image can do. As described above, since the fluorescence image is obtained as an image representing only the antibody binding region, the antibody binding region in the visible light image, that is, the target molecule (antigen) is extracted by extracting the antibody binding region from the fluorescence image. ) Can be accurately identified.

  Further, by observing the specimen S with fluorescence, the specimen S is stained with indocyanine green, that is, only fluorescence emitted from indocyanine green is imaged and acquired as a fluorescence image, while the specimen S is observed with visible light. Since the HE-stained image representing the staining state by HE staining can be synthesized from the visible light image acquired in step 1, the staining for specifying the antibody binding region does not affect the staining state by HE staining. Therefore, the antibody binding region can be identified without impairing the staining state by HE staining.

  Further, the pigment amount of the sample S can be estimated based on the pixel value of each pixel of the visible light image (spectral data for each band). Then, an HE-stained image representing the staining state of the specimen S by HE staining is synthesized, and an antibody-binding region identification image (antibody-identifying HE-stained image) in which the antibody-binding region in the HE-stained image is identified and displayed with a predetermined display color Can be generated and displayed on the display unit 72. Further, an H single-stained image representing the staining state of only the dye H of the specimen S is synthesized, and an antibody binding region identification image (antibody identification H) in which the antibody binding region in the H single-stained image is identified and displayed with a predetermined display color. Single stained image) can be generated and displayed on the display unit 72. Therefore, the antibody binding region can be represented with high visibility in the HE-stained image or the H-stained image and presented to a user such as a doctor, so that the diagnostic accuracy can be improved. In addition, with the HE-stained image representing the staining state of the specimen S by HE staining, the observation of the staining state of the specimen S by HE staining (morphological observation) can be performed with good visibility.

  In Embodiment 2, an actin antibody is allowed to act on a HE-stained biological tissue specimen, and the specimen S in which the actin antibody is fluorescently labeled with indocyanine green that specifically binds to the actin antibody is exemplified. On the other hand, the present invention can be similarly applied to a case where a plurality of antibodies are allowed to act on a HE-stained biological tissue specimen, and a specimen in which each antibody is appropriately fluorescently labeled is used as an observation target. In this case, in the microscope 10a constituting the microscope system 1a, the fluorescence image capturing unit 64 may be configured by a multiband camera. Then, the fluorescence observation image of the specimen S may be taken as a multiband image, and the fluorescence image may be acquired as a multiband image. For example, on an HE-stained biological tissue specimen, after allowing an antibody that specifically binds to a protein that exists only in the vicinity of the outer periphery of the cell nucleus and an antibody that specifically binds to a protein that exists only in erythrocytes, Specimens prepared by applying fluorescent labels to these antibodies using fluorescent dyes having different fluorescence wavelengths are prepared. For example, it is assumed that the fluorescence wavelength of one fluorescent dye is 800 nm to 900 nm and the fluorescence wavelength of the other fluorescent dye is 900 nm to 1000 nm. On the other hand, the fluorescent image capturing unit 64 of the microscope 10a has at least two bandpass filters: a bandpass filter having a wavelength band of light transmitted through the wavelength band of 800 nm to 900 nm and a bandpass filter having a wavelength band of 900 nm to 1000 nm. It is composed of a multiband camera with two or more bands equipped with a filter. According to this, by performing multiband imaging of the specimen fluorescence images while switching these bandpass filters, it is possible to easily display the fluorescence images individually representing the staining state with each fluorescent dye without inhibiting each fluorescence. Can be acquired.

  Further, in Embodiment 2, indocyanine green is exemplified as the fluorescent dye, but the exemplified fluorescent dye is an example, and the type thereof is not limited thereto. That is, a fluorescent dye that specifically binds to a desired antibody, that is, an antibody that acts on a target molecule (antigen) to be observed may be appropriately selected and used.

  The second embodiment similarly applies to a case where a fluorescent dye that specifically binds to a cell component such as a cell nucleus, a fiber, or a blood vessel is used, and a specimen in which a desired cell component is fluorescently labeled is used as an observation target. Applicable. According to this, it is possible to generate an image (digital stain image) in which a desired cell component in a specimen is highlighted as if it was specially stained, and this digital stain image and the staining state by HE staining can be generated. It is also possible to perform observation (morphological observation) while switching or simultaneously.

(Embodiment 3)
FIG. 16 is a block diagram illustrating an example of the overall configuration of the microscope system 1b according to the third embodiment. In FIG. 16, the same components as those in the first embodiment are denoted by the same reference numerals. As shown in FIG. 16, the microscope system 1b is configured by connecting a microscope 10 and an image processing device 70b so that data can be transmitted and received.

  Here, in the third embodiment, as in the first embodiment, for example, LDS751 / which is a fluorescent dye that is subjected to HE staining as the first staining and specifically binds to the cell nucleus as the second staining. A biological tissue specimen stained with DNA (fluorescent label) is used as a specimen S to be observed.

  In Embodiment 3, the image processing apparatus 70b includes an input unit 71, a display unit 72, a recording unit 73b, an image processing unit 74b, and a control unit 77 that controls each unit of the apparatus.

  The recording unit 73b records an image processing program 731b for realizing the processing of the third embodiment and generating an abnormal nucleus region identification image from the visible light image and the fluorescence image.

  The image processing unit 74b includes a nuclear region specifying unit 741 serving as a region specifying unit, a spectral characteristic estimating unit 743, a dye amount estimating unit 745 serving as a dye amount estimating unit, and an abnormal nuclear region detecting unit 747b serving as a target location detecting unit. And an abnormal nucleus region identification image generating unit 749 as an observation image generating unit and an abnormal mark adding unit 753. The abnormal nucleus region detection unit 747b detects an abnormal nucleus region based on the pigment amount estimated by the pigment amount estimation unit 745 for the pixels in the nucleus region identified by the nucleus region identification unit 741. The abnormal nucleus region detection unit 747b includes a color distribution creation unit 761b. The color distribution creating unit 761b synthesizes an H single-stained image, which is a hematoxylin single-stained image, as a single-stained image synthesizing unit, and creates a color distribution of the H single-stained image.

  FIG. 17 is a flowchart illustrating a processing procedure performed by the image processing device 70b according to the third embodiment. In addition, the same code | symbol is attached | subjected about the process process similar to Embodiment 1. FIG. The process described here is realized by the operation of each unit of the image processing apparatus 70b according to the image processing program 731b recorded in the recording unit 73b.

  As shown in FIG. 17, in the third embodiment, after step a11, the color distribution creating unit 761b of the abnormal nucleus region detecting unit 747b performs each process based on the dye amount of the dye H estimated for each pixel in step a11. By calculating the pixel value (RGB value) of the pixel, an H single-stained image representing the staining state of only the dye H of the sample S is synthesized (step c12). This process can be realized by the same process as step b15 of FIG. 14 described in the second embodiment.

  Subsequently, the color distribution creation unit 761b creates a color distribution of the H single-stained image synthesized in step c12 (step c13). For example, the color distribution creation unit 761b calculates L * a * b * values from the R value, G value, and B value (color information) of each pixel of the H single stained image, and CIE 1976 (L * a * b *) Create a color distribution in space.

  Then, the abnormal nucleus region detection unit 747b detects normal / abnormality of each nucleus region identified in step a7 based on the color distribution created in step c13, and detects the abnormal nucleus region (step c14). For example, a specimen containing normal cell nuclei is prepared in advance and subjected to HE staining to obtain a visible light image. Then, by creating a color distribution in the same procedure as in steps a7 to c13, the color difference between the center of this color distribution and the normal cell nucleus region is calculated as a reference color difference that is a reference feature value. In step c14, the abnormal nucleus region detection unit 747b first calculates a color difference from the center of the color distribution created in step c13 for each nucleus region identified in step a7. Then, the abnormal nucleus region detection unit 747b uses the calculated color difference as a feature value, and compares the color difference with the reference color difference, for example, if a cell nucleus in which both differences are equal to or greater than a predetermined threshold is abnormal and does not satisfy the threshold Is determined. If the abnormal nucleus region is detected as described above, the process proceeds to step a15.

  As described above, according to the third embodiment, the same effects as those of the first embodiment can be obtained, and a color distribution is created by synthesizing the H single-stained image, and the color difference from the center corresponds. Normality / abnormality of cell nuclei can be determined. Therefore, the abnormal nucleus region can be accurately detected without being affected by the difference in the type of stain used for staining the sample S with HE (stain solution manufacturer), and the detection accuracy of the abnormal nucleus region. Can be improved.

  In the third embodiment, the color distribution in the CIE 1976 (L * a * b *) space of the H single-stained image is created. However, the color space to be created is CIE 1976 (L * a * b *). Not only the color distribution in the space but also a color space based on another color system may be created as appropriate to determine normality / abnormality of the cell nucleus.

  For example, in step c13 of FIG. 17, a color distribution in a color space based on the Munsell color system may be created based on the pixel value of each pixel of the H single stained image. In this case, a specimen containing normal cell nuclei is prepared in advance and subjected to HE staining to obtain a visible light image. Then, by synthesizing an H single-stained image in the same procedure as in steps a7 to c12 and creating a color distribution using the Munsell color system, the hue difference between the center of this color distribution and the normal cell nucleus region , Brightness difference and saturation difference are calculated as reference feature values as reference hue difference, reference brightness difference, and reference saturation difference. In step c14, the hue difference, brightness difference, and saturation difference from the center of the color distribution created in step c13 using the Munsell color system are calculated for each core region specified in step a7. Then, the calculated hue difference, brightness difference, and saturation difference are compared with the reference hue difference, reference brightness difference, and reference saturation difference. For example, when cell nuclei in which both differences are equal to or greater than a predetermined threshold are abnormal, the threshold is not reached Is determined to be normal.

  Alternatively, in step c13 of FIG. 17, a color distribution in a color space based on the CIE-XYZ color system may be created based on the pixel value of each pixel of the H single stained image. In this case, a specimen containing normal cell nuclei is prepared in advance and subjected to HE staining to obtain a visible light image. Then, an H single-stained image is synthesized in the same procedure as in steps a7 to c12, and a color distribution is created using the CIE-XYZ color system, so that the center of this color distribution and the normal cell nucleus region The chromaticity difference is calculated as a reference chromaticity difference that is a reference feature value. In step c14, the chromaticity difference with respect to each nucleus region specified in step a7 is calculated from the center of the color distribution created in step c13 using the CIE-XYZ color system. Then, the calculated chromaticity difference is compared with the reference chromaticity difference, and for example, a cell nucleus in which the difference between the two is equal to or greater than a predetermined threshold is determined to be abnormal and less than the threshold is determined to be normal.

  In the third embodiment, the color difference of the nucleus region calculated by creating the color distribution is compared with the reference color difference to determine normality / abnormality of the cell nucleus. However, the color difference of the abnormal nucleus region is learned. Then, it may be recorded in the recording unit 73b as learning data. Then, normality / abnormality of the cell nucleus may be determined using the learning data. When creating a color distribution using the Munsell color system and comparing the hue difference, brightness difference, and saturation difference, or when creating a color distribution using the CIE-XYZ color system and comparing the chromaticity difference Is the same. According to this, it is possible to detect the abnormal nucleus region with high accuracy by reducing the influence of individual differences of the individual specimens S, and to improve the detection accuracy of the abnormal nucleus region.

(Embodiment 4)
FIG. 18 is a block diagram illustrating an example of the overall configuration of the microscope system 1c according to the fourth embodiment. In FIG. 18, the same components as those in the first embodiment are denoted by the same reference numerals. As shown in FIG. 18, the microscope system 1c is configured by connecting a microscope 10 and an image processing apparatus 70c so that data can be transmitted and received.

  Here, in the fourth embodiment, as in the first embodiment, for example, LDS751 / which is a fluorescent dye that is subjected to HE staining as the first staining and specifically binds to the cell nucleus as the second staining. A biological tissue specimen stained with DNA (fluorescent label) is used as a specimen S to be observed.

  In the fourth embodiment, the image processing device 70c includes an input unit 71, a display unit 72, a recording unit 73c, an image processing unit 74c, and a control unit 77 that controls each unit of the device.

  The recording unit 73c records an image processing program 731c for realizing the processing of the fourth embodiment and generating an abnormal nucleus region identification image from the visible light image and the fluorescence image.

  The image processing unit 74c includes a nuclear region specifying unit 741 as a region specifying unit, a spectral characteristic estimating unit 743, a dye amount estimating unit 745 as a dye amount estimating unit, and an abnormal nuclear region detecting unit 747c as a target location detecting unit. And an abnormal nucleus region identification image generating unit 749 as an observation image generating unit and an abnormal mark adding unit 753. The abnormal nucleus region detection unit 747c detects an abnormal nucleus region based on the pigment amount estimated by the pigment amount estimation unit 745 for the pixels in the nucleus region identified by the nucleus region identification unit 741. The abnormal nucleus region detection unit 747c includes a morphological feature amount calculation unit 763c as a feature amount calculation unit. The morphological feature amount calculation unit 763c calculates a morphological feature amount as a feature value representing the morphological feature based on the contour of the nuclear region specified by the nuclear region specification unit 741.

  FIG. 19 is a flowchart illustrating a processing procedure performed by the image processing apparatus 70c according to the fourth embodiment. In addition, the same code | symbol is attached | subjected about the process process similar to Embodiment 1. FIG. The process described here is realized by the operation of each unit of the image processing apparatus 70c according to the image processing program 731c recorded in the recording unit 73c.

As shown in FIG. 19, in the fourth embodiment, after step a11, the morphological feature amount calculation unit 763c of the abnormal nucleus region detection unit 747c calculates the morphological feature amount of each nucleus region specified in step a7 ( Step d12). Here, the morphological feature amount calculation unit 763c calculates, for example, a size (area) and a circularity as an example of the morphological feature amount. Here, the circularity is calculated according to the following equation (8), for example. Here, the value calculated by the following equation (8) is the maximum value (= 1) when the contour shape of the core region is a perfect circle, and is obtained as a smaller value as the contour shape becomes more complicated.
Circularity = 4π × area / perimeter length (8)

  Subsequently, the abnormal nucleus region detection unit 747c determines normality / abnormality of each nucleus region specified in step a7 based on the morphological feature amount calculated in step d12 to detect the abnormal nucleus region (step d13). ). For example, the size and circularity of normal cell nuclei are acquired in advance as reference feature values that are reference feature values. In step d13, the abnormal nucleus region detection unit 747c first compares the size and circularity calculated in step d12 with the reference form feature amount. For example, the abnormal nucleus region detection unit 747c abnormally detects a cell nucleus whose difference is equal to or greater than a predetermined threshold. The case where it is less than is determined as normal. It should be noted that the normal / abnormal determination may be performed using either one of the size and the circularity, or the normal / abnormal determination may be performed using both. When both are used, a nuclear region determined to be abnormal on either side may be detected as an abnormal nuclear region, or an abnormal nucleus region may be detected if both determination results are abnormal. In this case, an abnormal nucleus region identification image may be generated by expressing the nucleus region determined to be abnormal on one side and the nucleus region determined to be abnormal on both sides with different display colors. In addition, an abnormal nucleus region identification image is generated by expressing a nucleus region determined to be abnormal as a result of determination using size and a nucleus region determined to be abnormal as a result of determination using circularity in different display colors. You may do it. In addition, the morphological feature amount is not limited to the exemplified size and circularity, other values may be calculated as appropriate, and cell nucleus abnormality / normality may be determined based on the calculated value. . If the abnormal nucleus region is detected as described above, the process proceeds to step a15.

  As described above, according to the fourth embodiment, the same effects as those of the first embodiment can be obtained, and the morphological feature quantity of the specified nucleus region is calculated, and the corresponding is based on this morphological feature quantity. Normality / abnormality of cell nuclei can be determined. Therefore, for example, an abnormal nuclear region whose form has been destroyed can be detected with high accuracy, and the detection accuracy of the abnormal nuclear region can be improved.

  In the fourth embodiment, the normal / abnormal cell nucleus is determined by comparing the morphological feature quantity such as the size and circularity of the nucleus area with the reference morphological feature quantity. However, the morphological feature quantity of the abnormal nucleus area is determined. And may be recorded as learning data in the recording unit 73c. Then, normality / abnormality of the cell nucleus may be determined using the learning data. According to this, it is possible to detect the abnormal nucleus region with high accuracy by reducing the influence of individual differences of the individual specimens S, and to improve the detection accuracy of the abnormal nucleus region.

  Further, the morphology information of the nuclear region may be acquired by classifying the visible light image, and normality / abnormality of each cell nucleus may be determined based on the morphology information of the nuclear region. The method of tissue classification is not particularly limited, and the visible light image is classified into regions for each cell component such as cell nucleus, cell membrane, and cytoplasm based on the shape information of the visible light image (tissue classification). Alternatively, each pixel of the visible light image may be classified into regions for each cell component (tissue classification) using a known clustering technique.

  In each of the above-described embodiments and modifications, the case where the HE-stained specimen is used for observation has been described. However, the morphological observation staining technique for observing the form of the specimen is known in addition to HE staining. It has been. The present invention can be similarly applied to a case where a fluorescent label is further applied to a sample stained by a staining method other than HE staining.

  In addition, the present invention is not limited to the above-described embodiments and modifications thereof, and various combinations can be made by appropriately combining a plurality of constituent elements disclosed in the embodiments and modifications. An invention can be formed. For example, some components may be excluded from all the components shown in each embodiment or each modification. Or you may form combining suitably the component shown in different embodiment and modification.

  For example, in combination with the first embodiment and the second embodiment, the HE-stained specimen is stained (fluorescent label) with a fluorescent dye that specifically binds to a desired cell component, and in addition The same applies to the case where an antibody against a target molecule (antigen) to be observed is allowed to act, and this antibody is stained (fluorescently labeled) with a fluorescent dye that specifically binds to the acted antibody. Applicable to. Alternatively, in addition to HE staining, when different cell components are stained with separate fluorescent dyes (fluorescent staining), antibodies against different target molecules (antigens) are allowed to act separately, and those antibodies are separated with separate fluorescent dyes. The same applies to staining (fluorescence staining). That is, if the fluorescent wavelengths of the fluorescent dyes are different, each dyed state can be imaged separately and the area can be specified individually, so that each area can be specified appropriately. Further, in this case, the regions of the cell constituent elements and the antibody binding regions that are individually specified may be identified and displayed with different display colors.

  Further, the normal / abnormality of the cell nucleus is determined by combining the third and fourth embodiments and creating a color distribution, and the normality / abnormality of the cell nucleus is determined by calculating the morphological feature amount. You may do it. Then, the abnormal nucleus region extracted as a result of the determination made based on the color distribution and the abnormal nucleus region extracted as a result of the determination made based on the morphological feature quantity are color-coded for identification display. Also good.

  Specifically, the nucleus area determined to be abnormal in one of the determination made using the color distribution and the determination made using the morphological feature amount is different from the nucleus area determined to be abnormal in both. You may make it carry out identification display with a color. For example, the pixel values of the pixels constituting the former nucleus region may be replaced with red, and the pixel values of the pixels constituting the latter nucleus region may be replaced with yellow to identify and display both. Similarly, when the color difference and morphological feature amount of the abnormal nucleus region are recorded as learning data, the normality / abnormality of the cell nucleus is determined using each learning data, and the determination using which learning data is abnormal. Depending on whether or not it is determined, the corresponding nuclear region may be identified and displayed with different display colors.

  As described above, the image processing apparatus, the image processing method, and the image processing program of the present invention are suitable for appropriately specifying a region of a predetermined specimen component in a specimen and improving visibility during observation. .

1, 1a, 1b, 1c Microscope system 10, 10a Microscope 33 First band pass filter 44 First dichroic mirror 61 Second dichroic mirror 64 Fluorescent image capturing unit 65 Visible light image capturing unit 70, 70a, 70b, 70c Image processing apparatus 71 Input unit 72 Display unit 73, 73a, 73b, 73c Recording unit 731, 731a, 731b, 731c Image processing program 74, 74a, 74b, 74c Image processing unit 741 Nuclear region specifying unit 743 Spectral characteristic estimation unit 745 Dye Quantity estimation unit 747, 747b, 747c Abnormal nucleus region detection unit 761b Color distribution creation unit 763c Morphological feature amount calculation unit 749 Abnormal nucleus region identification image generation unit 751 HE stained image synthesis unit 753 Abnormal mark addition unit 755a Antibody binding region specification unit 757a Antibody binding region identification image Generator 759a H single stained image synthesizing unit 77 the control unit 771 visible light image acquisition control unit 772 fluorescence image acquisition control unit S samples

Claims (11)

A staining state by the first staining of a sample in which the first staining and a predetermined staining component are stained and the second staining observed by an observation method different from the first staining is performed. First image acquisition means for acquiring a first specimen image by imaging
A second image acquisition means for acquiring a second specimen image by imaging the staining state of the specimen by the second staining;
Area specifying means for specifying an area of the predetermined specimen component in the first specimen image based on the second specimen image;
Based on the first specimen image, an observation image representing the staining state of the specimen by the first staining and representing at least a part of the predetermined specimen component in the specimen is generated. Observation image generating means for
An image processing apparatus comprising: Dye amount estimation means for estimating the dye amount of the dye used for the first staining that stains the corresponding position on the sample based on the pixel value of each pixel of the first sample image ,
The observation image generation means synthesizes a stained image that synthesizes a specimen stained image representing a staining state of the specimen by the first staining based on a dye amount of the dye in each pixel of the first specimen image. Means for generating, as the observation image, an image in which pixel values of at least some of the pixels of the region of the predetermined specimen component are replaced with a predetermined display color in the specimen-stained image. The image processing apparatus according to 1. The first staining is performed using a plurality of pigments,
The dye amount estimating means estimates a dye amount for each of the plurality of dyes,
The observation image generating means is a sample representing a staining state of only the predetermined dye of the sample based on a dye amount of the predetermined dye among the plurality of dyes in each pixel of the first sample image A single-stained image synthesizing unit that synthesizes a single-stained image, and the image obtained by replacing pixel values of at least some of the pixels in the region of the predetermined sample component in the sample single-stained image with a predetermined display color The image processing apparatus according to claim 2, wherein the image processing apparatus is generated as an image. Based on the characteristic value of the first sample image, comprising a target location detection means for detecting an identification target location from the region of the predetermined sample component in the first sample image,
The image processing apparatus according to claim 1, wherein the observation image generation unit generates an observation image in which the identification target portion can be identified. Dye amount estimation means for estimating the dye amount of the dye used for the first staining that stains the corresponding position on the sample based on the pixel value of each pixel of the first sample image ,
The image processing apparatus according to claim 4, wherein the target location detection unit detects the identification target location using a pigment amount of the pigment in each pixel of the first specimen image as the feature value. . The specimen component is a predetermined cell component constituting a cell,
The target location detection means includes feature quantity calculation means for calculating a morphological feature quantity of the region of the predetermined specimen component in the first specimen image, and the identification is performed using the morphological feature quantity as the feature value. The image processing apparatus according to claim 4, wherein a target portion is detected. The first staining is hematoxylin-eosin staining using hematoxylin and eosin as pigments;
Based on the pixel value of each pixel of the first sample image, a dye amount estimating means for estimating the dye amount of the hematoxylin and the eosin staining the corresponding position on the sample;
Based on the amount of hematoxylin dye in each pixel of the first sample image, single-stained image synthesis means for synthesizing a hematoxylin single-stained image representing the staining state of only the hematoxylin of the sample;
With
The image processing apparatus according to claim 4, wherein the target location detection unit detects the identification target location using color information of the hematoxylin single-stained image as the feature value.   The said target location detection means calculates a color difference based on the color information of the said hematoxylin single dyeing | staining image, and detects the said identification target location based on this calculated color difference. Image processing device.   The image processing apparatus according to claim 4, wherein the target location detection unit detects the identification target location by comparing the feature value with a predetermined reference feature value set in advance. A staining state by the first staining of a sample in which the first staining and a predetermined staining component are stained and the second staining observed by an observation method different from the first staining is performed. A first image acquisition step of acquiring a first specimen image by imaging
A second image acquisition step of acquiring a second specimen image by imaging the staining state of the specimen by the second staining;
An area specifying step for specifying an area of the predetermined specimen component in the first specimen image based on the second specimen image;
Based on the first specimen image, an observation image representing the staining state of the specimen by the first staining and representing at least a part of the predetermined specimen component in the specimen is generated. An observation image generation step to perform,
An image processing method comprising: On the computer,
A staining state by the first staining of a sample in which the first staining and a predetermined staining component are stained and the second staining observed by an observation method different from the first staining is performed. A first image acquisition step of acquiring a first specimen image by imaging
A second image acquisition step of capturing a second specimen image by imaging a staining state of the specimen by the second staining;
An area specifying step for specifying an area of the predetermined specimen component in the first specimen image based on the second specimen image;
Based on the first specimen image, an observation image representing the staining state of the specimen by the first staining and representing at least a part of the predetermined specimen component in the specimen is generated. An observation image generation step to perform,
An image processing program for executing

Images