JP2009014354A - Image processor and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To precisely estimate a spectral characteristic in a sample point on a sample, without generating an unintended dispersion. <P>SOLUTION: This image processor 1 is provided with a main element classifying part 7b for classifying each pixel of a teacher image, that is an image recorded with a reference sample, correlated with a reference spectral characteristic in the corresponding sample point on the reference sample in every image, based on at least a main feature amount for characterizing a main element, a representative point extracting part 7c for extracting a plurality of representative pixels out of a pixel group classified by the main element classifying part 7b, and a statistical amount calculating part 7d for calculating a statistical amount of the reference spectral characteristic used for estimating the spectral characteristic of the sample, based on the reference spectral characteristic correlated with the plurality of representative pixels extracted by the representative point extracting part 7c. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing program for processing an observation image obtained by imaging a specimen including one or more main elements.

  For biological tissue specimens, especially pathological specimens, block specimens obtained by organ excision and specimens obtained by needle biopsy are sliced to a thickness of several microns and then magnified using a microscope to obtain various findings. Is widely practiced. In particular, transmission observation using an optical microscope is one of the most popular observation methods because the equipment is relatively inexpensive and easy to handle, and has been performed for a long time. In this case, since the sliced biological specimen hardly absorbs and scatters light and is almost colorless and transparent, it is common to stain with a dye prior to observation.

  Various dyeing methods have been proposed, and the total number thereof reaches 100 or more. Particularly, regarding pathological specimens, hematoxylin-eosin staining using two dyes of blue-violet hematoxylin and red eosin ( Hereinafter, it is referred to as H & E staining).

  Hematoxylin is a natural substance collected from plants and itself has no dyeability. However, its oxide, hematin, is a basophilic dye and binds to a negatively charged substance. Since deoxyribonucleic acid (DNA) contained in the cell nucleus is negatively charged by a phosphate group contained as a constituent element, it binds to hematin and is stained blue-violet. As described above, it is not hematoxylin that has a staining property but hematin, which is an oxide thereof. However, since it is common to use hematoxylin as the name of a dye, the following is followed.

  Eosin is an acidophilic dye that binds to positively charged substances. Whether amino acids or proteins are charged positively or negatively is affected by the pH environment, and the tendency to be positively charged under acidic conditions becomes stronger. For this reason, acetic acid may be added to the eosin solution. Proteins contained in the cytoplasm are stained from red to light red by binding to eosin.

  In the specimen after H & E staining, cell nuclei, bone tissue and the like are stained blue-purple, and cytoplasm, connective tissue, erythrocytes and the like are stained red so that they can be easily visually recognized. As a result, the observer can grasp the size and positional relationship of elements constituting the tissue such as the cell nucleus, and can determine the state of the specimen morphologically.

  Since the staining of a biological tissue specimen is an operation of fixing a pigment using a chemical reaction to biological tissues originally having individual differences, it is difficult to always obtain a uniform result. Specifically, even when the specimen is reacted for the same time with the staining solution having the same concentration, the amount of the fixed dye is not always the same. Depending on the specimen, a relatively large amount of dye may be fixed, or a relatively small amount of dye may be fixed. The former is a sample that is stained darker than usual, and the latter is a sample that is lightly stained. In order to suppress such variation in staining among specimens, some facilities have a staining engineer with specialized skills. However, the dyeing engineer's craftsman's adjustment work can reduce dyeing variation within the same facility to some extent, but dyeing variation between different facilities still occurs.

  Dyeing variation is problematic in two respects. One is that when an observer visually observes a stained specimen, an uneven staining state of the specimen may lead to stress on the observer. If severe staining variability occurs, critical findings may be overlooked. Second, when a stained specimen is imaged with a camera and image processing is performed, there is a possibility that variation in staining may adversely affect image processing accuracy. For example, even if it is known that a certain lesion exhibits a specific color, it is difficult to automatically extract an image region corresponding to the lesion from an observation image obtained by imaging a specimen. This is because the staining variation disturbs the color change due to the characteristics of the lesion.

  Non-Patent Document 1 discloses a method for solving such a problem of staining variation by image processing using a multiband image. In the image processing method described in Non-Patent Document 1, a multiband image is acquired as an observation image. Based on the image data of the multiband image, first, the spectral characteristics at each sample point on the sample are determined as Wiener. Estimate by estimation. Next, using this estimated spectral characteristic, the amount of dye fixed to each sample point on the sample is estimated based on the physical model. Then, the estimated dye amount is corrected by appropriately increasing / decreasing, and image data is generated based on the corrected dye amount, thereby obtaining an observation image in which the staining state is corrected. According to this, by correcting the estimated dye amount appropriately, the observation image of the darkly stained specimen or the lightly stained specimen is corrected to an observation image exhibiting the same color as the appropriately stained specimen. be able to.

JP-A-7-120324 Japanese Patent No. 3510037 “Color Correction of Pathological Images Based on Dye Amount Quantification”, OPTICAL REVIEW, 2005, Vol.12, No.4, p.293-300 Arthur Dempster, Nan Laird, and Donald Rubin, “Maximum likelihood from incomplete data via the EM algorithm”, Journal of the Royal Statistical Society, 1977, Series B, 39 (1): 1 38

  By the way, in the image processing method described in Non-Patent Document 1, in order to perform Wiener estimation of the spectral characteristics at each sample point on the specimen, a statistical quantity of the spectral characteristics of the reference specimen including the main elements included in the specimen is required. It becomes. In the case of winner estimation, the statistic corresponds to an autocorrelation matrix or a covariance matrix of spectral transmittance as spectral characteristics. This autocorrelation matrix or covariance matrix is calculated based on the measurement result obtained by measuring the spectral transmittance at a plurality of sample points on the reference sample in advance by a spectrometer. Hereinafter, in the embodiment of the present invention, a case where an autocorrelation matrix is used for the above-described statistic will be described, and a sample point on a reference sample used for calculation of the autocorrelation matrix is referred to as a representative point.

  When the representative points are uniformly extracted from the entire area of the reference sample based on the reference sample image in which the reference sample is recorded, the main area indicating the main elements in the reference sample on the reference sample image More representative points are extracted from the corresponding main elements. Since the Wiener estimation is a technique for minimizing the mean square error, the estimation accuracy for the main element in which many representative points gather is relatively high. That is, in the Wiener estimation, the estimation accuracy is high for the main element corresponding to the main area having a large area on the reference sample image, and the estimation accuracy tends to be low for the main element corresponding to the main area having a small area. .

  Therefore, in the image processing method described in Non-Patent Document 1, when the specimen is a pathological specimen, since the cell nucleus that is important in diagnosis is small, it is difficult to accurately estimate the spectral transmittance of the cell nucleus. On the other hand, in the technique disclosed in Patent Document 1, the user extracts substantially the same number of representative points from each main element based on a reference specimen image obtained by imaging a typical pathological specimen. The estimation accuracy for the cell nucleus can be made equal to the estimation accuracy for the other main elements. Here, in addition to the cell nucleus, the main elements in the pathological specimen include cytoplasm, red blood cells, and background.

  However, in the user extraction in which the user extracts representative points, the representative points to be extracted differ from user to user, so that the spectral transmittance estimation result by winner estimation also varies from user to user. That is, when user extraction is performed, the optimality and reproducibility of the extracted representative points are not guaranteed, and there is a problem that the estimation accuracy of the spectral transmittance varies. Further, in user extraction, there is a problem that the user has to perform trial and error in order to extract the representative points, and the work requires a great deal of labor and time.

  The present invention has been made in view of the above, and provides an image processing apparatus and an image processing program capable of estimating spectral characteristics at each sample point on a sample with high accuracy without causing unintended variations. The purpose is to provide.

  In order to achieve the above object, an image processing apparatus according to the present invention is based on image data of an observation image obtained by imaging a sample including one or more main elements, on a reference sample including the one or more main elements. In the image processing apparatus for estimating the spectral characteristics of the sample using the statistical amount of the reference spectral characteristic at a plurality of sample points, an image in which the reference sample is recorded and a corresponding sample on the reference sample for each pixel A pixel classifying unit that classifies each pixel of the teacher image associated with the reference spectral characteristic at a point based on at least a main feature amount that characterizes the main element; and a plurality of pixels among the pixel groups classified by the pixel classifying unit Representative extraction means for extracting a representative pixel, and statistic calculation means for calculating the statistic based on the reference spectral characteristic associated with a plurality of the representative pixels. To.

  The image processing program according to the present invention is based on image data of an observation image obtained by imaging a specimen including one or more main elements, and a reference at a plurality of sample points on the reference specimen including the one or more main elements. An image processing apparatus that estimates spectral characteristics of the specimen using spectral characteristic statistics is an image in which the reference specimen is recorded, and the reference spectral characteristics at a corresponding specimen point on the reference specimen for each pixel. A pixel classification procedure for classifying each pixel of the associated teacher image based on at least a main feature that characterizes the main element, and a plurality of representative pixels are extracted from the pixel group classified by the pixel classification procedure A representative extraction procedure and a statistic calculation procedure for calculating the statistic based on the reference spectral characteristics associated with a plurality of the representative pixels are executed.

  According to the image processing device and the image processing program of the present invention, the spectral characteristics at each sample point on the sample can be estimated with high accuracy without causing unintended variations.

  Exemplary embodiments of an image processing apparatus and an image processing program according to the present invention are explained in detail below with reference to the accompanying drawings. In addition, this invention is not limited by this embodiment. Moreover, in description of drawing, the same code | symbol is attached | subjected and shown to the same part.

(Embodiment)
FIG. 1 is a block diagram showing a main configuration of an image processing apparatus 1 according to the present embodiment. As shown in this figure, the image processing apparatus 1 includes an observation image acquisition unit 2 that acquires an observation image obtained by imaging a specimen stained with a dye, a reference spectral characteristic acquisition unit 3 that acquires spectral characteristics of a reference specimen, Various types of input unit 4, display unit 5, and storage unit 6 for inputting, displaying, and storing various information, and the observation image acquired by the observation image acquisition unit 2 and the spectral characteristics acquired by the reference spectral characteristic acquisition unit 3 An image processing unit 7 that performs arithmetic processing and a control unit 8 that is electrically connected to each of these units and controls processing and operations of the connected units are provided.

  For example, as shown in FIG. 2, the observation image acquisition unit 2 includes a stage 11 on which a specimen 10 stained with a plurality of pigments is placed, a light source 12 that transmits and illuminates the specimen 10 through the stage 11, and a specimen 10 An imaging optical system 13 for condensing the light from the sample 10 to form an observation image of the specimen 10, a bandpass filter 14 for limiting the wavelength band of the light for forming the observation image to a predetermined range, and imaging the observation image And a camera 16 for generating an observation image.

  The camera 16 is configured by disposing a mosaic RGB filter as shown in FIG. 3 on a monochrome image pickup device using a CCD or the like. In each of the R, G, and B filter regions, the camera 16 is shown in FIG. It has a spectral sensitivity characteristic as shown. The camera 16 can image only one of R, G, and B components in each pixel. However, as disclosed in, for example, Patent Document 2, each of the R, G that is deficient by using neighboring pixel values. , B components can be interpolated. Thereby, the camera 16 acquires all R, G, and B components in each pixel. The camera 16 is not limited to the configuration using the RGB filter, and for example, a 3CCD type camera can be used. In that case, all the R, G, and B components can be acquired without performing interpolation at each pixel.

  The bandpass filter 14 includes bandpass filters 14 a and 14 b having different spectral transmittance characteristics as shown in FIGS. 5A and 5B, for example, and is held by a rotary turret 15. As a result, the bandpass filters 14a and 14b can be switched between the imaging optical system 13 and the camera 16. The observation image acquisition unit 2 performs imaging while sequentially changing the wavelength band of the light that forms the observation image by the bandpass filters 14a and 14b, so that a 6-band multiband image (6 bands) is obtained as the observation image of the specimen 10. Image). The observation image acquired by the observation image acquisition unit 2 is transferred to the control unit 8 and stored in the storage unit 6.

  Here, the observation image acquisition unit 2 acquires six bands of observation images using the two band-pass filters 14a and 14b. However, the observation image acquisition unit 2 is not limited to two and may use three or more. Thereby, a multiband image having a larger number of bands can be acquired as an observation image. For example, Patent Document 1 discloses a technique for capturing a multiband image in a frame sequential manner while rotating and switching 16 bandpass filters with a filter wheel. According to this, an observation image of 16 bands can be obtained.

  Further, although the band pass filter 14 has been described as being disposed between the imaging optical system 13 and the camera 16, it may be disposed at any position in the optical path from the light source 12 to the camera 16. Further, as a filter for limiting the wavelength band of light for forming an observation image, a transmission band variable filter may be used instead of the bandpass filter 14. For example, a liquid crystal tuner may be used instead of the bandpass filter 14 and the turret 15. Bull filters can be used.

  In this embodiment, a pathological specimen stained with two pigments of hematoxylin and eosin is used as the specimen 10 stained with a plurality of pigments, and the image processing apparatus 1 uses hematoxylin in the pathological specimen. The description will be made assuming that three types of pigment amounts are estimated: the pigment amount, the eosin pigment amount, and the erythrocyte pigment amount. In the following description, these three types of dyes are abbreviated as dye H, dye E, and dye R, respectively. However, the dyes that can be estimated by the image processing apparatus 1 are not limited to the dye H, the dye E, and the dye R, and the specimen that can be processed by the image processing apparatus 1 is not limited to the pathological specimen. Even in a state where no pigment is applied, erythrocytes have their own unique color, and after H & E staining, the color of erythrocytes and the color of eosin changed in the staining process are observed in a superimposed manner. Therefore, exactly, the combination of both is the dye R.

  The reference spectral characteristic acquisition unit 3 includes a spectrometer (not shown) instead of the camera 16 based on the configuration of the observation image acquisition unit 2 illustrated in FIG. 2 and the bandpass filter 14 is removed. . The reference spectral characteristic acquisition unit 3 is also realized as appropriate by using part of the observation image acquisition unit 2 and removing the bandpass filter 14 from the observation image acquisition unit 2 and replacing the camera 16 with a spectrometer. At that time, a reference specimen (not shown) is placed on the stage 11 instead of the specimen 10. As the reference specimen, a typical pathological specimen that includes cell nuclei, cytoplasm, erythrocytes and the like as main elements contained in the specimen 10 and is stained with H & E is used. The reference spectral characteristic acquisition unit 3 can be configured separately from the observation image acquisition unit 2.

  The reference spectral characteristic acquisition unit 3 acquires a spectral transmittance (hereinafter referred to as a reference spectral transmittance) as a spectral characteristic at each sample point on the reference specimen using a spectrometer. The reference spectral transmittance for each sample point acquired by the reference spectral characteristic acquisition unit 3 is transferred to the control unit 8 and stored in the storage unit 6 in association with the corresponding sample point position (sample point coordinates).

  The input unit 4 includes various input devices such as a keyboard and a mouse, and inputs various information such as processing parameters used for observation image processing to the control unit 8. In particular, the input unit 4 inputs parameter values used for representative point extraction or optimization described later. The display unit 5 is configured using various displays such as an FPD (Flat Panel Display), and displays various information such as an observation image, a teacher image described later, a spectral characteristic estimation result, and a pigment amount estimation result.

  The storage unit 6 is configured by using a hard disk, a ROM, a RAM, and the like, a processing program executed by the control unit 8, an observation image acquired by the observation image acquisition unit 2, and a reference spectral transmittance acquired by the reference spectral characteristic acquisition unit 3 Various information such as processing parameters used by the image processing unit 7 and processing results of the image processing unit 7 are stored. The storage unit 6 stores in advance standard spectral characteristics of the dye H, the dye E, and the dye R that are used for dye amount estimation described later.

  The image processing unit 7 is realized by, for example, a CPU, and performs various arithmetic processes on the observation image and the reference spectral transmittance stored in the storage unit 6 based on a predetermined image processing program executed by the control unit 8. In particular, the image processing unit 7 includes a reference sample image generation unit 7a that generates a reference sample image in which a reference sample is recorded based on the reference spectral transmittance acquired by the reference spectral characteristic acquisition unit 3, and each pixel of the reference sample image. Are classified based on at least a main feature amount that characterizes the main element in the sample 10, and a main element classifying unit 7b that classifies the main element, and a plurality of representative pixels from the pixel group classified by the main element classifying unit 7b And a representative point extracting unit 7c that extracts representative points by extracting.

  The image processing unit 7 includes a statistic calculation unit 7d that calculates the statistic based on the reference spectral transmittance associated with the plurality of representative pixels extracted by the representative point extraction unit 7c, and a statistic calculation unit. Based on the statistics calculated by 7d, it is determined whether or not the representative pixel needs to be updated. If it is determined that the update is necessary, the representative point is optimized by updating the extraction condition for extracting the representative pixel. Based on the image data of the observation image acquired by the representative point optimization unit 7e and the observation image acquisition unit 2, the spectral characteristics at each sample point on the sample 10 are calculated using the statistics calculated by the statistics calculation unit 7d. A spectral characteristic estimation unit 7f for estimation and a pigment amount estimation unit 7g for estimating the pigment amount at each sample point based on the spectral characteristics estimated by the spectral characteristic estimation unit 7f are provided.

  The control unit 8 is realized by a CPU, and controls processing and operation of each unit included in the image processing apparatus 1 by executing a predetermined processing program stored in the storage unit 6. For example, the control unit 8 causes the image processing unit 7 to process the observation image and the reference spectral transmittance stored in the storage unit 6 by executing a predetermined image processing program stored in the storage unit 6. The processing result by the processing unit 7 is stored in the storage unit 6 and controlled to be displayed on the display unit 5.

  Subsequently, as an image processing procedure performed by the image processing apparatus 1, the spectral characteristics of the specimen 10 are estimated based on the observation image, and the dye H, the dye E, and the dye in the specimen 10 are estimated based on the estimated spectral characteristics. A processing procedure for estimating the amount of each pigment of R will be described. FIG. 6 is a diagram schematically showing an outline of the processing procedure. As shown in this figure, the image processing apparatus 1 first acquires a reference spectral transmittance based on a reference sample, and generates a reference sample image based on the acquired reference spectral transmittance. Then, based on this reference sample image, a plurality of representative points are extracted from the reference sample and the representative points are optimized to optimize the autocorrelation matrix of the spectral transmittance as a statistic used for the Wiener estimation.

  Next, the image processing apparatus 1 estimates the spectral characteristics at each sample point on the sample 10 by Wiener estimation based on the multiband image acquired as the observation image, and acquires the estimated spectral characteristics obtained in advance. Based on the standard spectral characteristics of dye H, dye E, and dye R, the amount of each dye (the amount of H dye, the amount of E dye, and the amount of R dye) at each sample point on the sample 10 is estimated. Then, the image processing apparatus 1 displays the estimated pigment amount and the estimated spectral characteristic on the display unit 5 as the estimation result. Note that the image processing apparatus 1 appropriately stores an observation image, a reference specimen image, a reference spectral transmittance, an estimated spectral characteristic, an estimated dye amount, and the like in the storage unit 6 in the series of processing steps.

  FIG. 7 is a flowchart showing a processing procedure corresponding to FIG. This processing procedure is performed by causing the control unit 8 to execute a predetermined image processing program stored in the storage unit 6. As shown in FIG. 7, first, the observation image acquisition unit 2 performs an observation image acquisition process for acquiring a multiband image as an observation image of the specimen 10 (step S101), and the reference spectral characteristic acquisition unit 3 A reference spectral characteristic acquisition process for acquiring the reference spectral transmittance is performed (step S102).

  Next, the reference sample image generation unit 7a, the main element classification unit 7b, the representative point extraction unit 7c, the statistic calculation unit 7d, and the representative point optimization unit 7e are used for the Wiener estimation matrix based on the reference spectral transmittance. A statistic calculation process for calculating the autocorrelation matrix of the spectral transmittance is performed in cooperation (step S103), and the spectral characteristic estimation unit 7f calculates the autocorrelation calculated in step S103 based on the image data of the observed image. Using the matrix, spectral characteristic estimation processing for estimating spectral characteristics at each sample point on the specimen 10 by Wiener estimation is performed (step S104).

  Thereafter, the dye amount estimation unit 7g determines the specimen 10 for each dye based on the spectral characteristics estimated in step S104 and the standard spectral characteristics of the dyes H, E, and R stored in the storage unit 6. Dye amount estimation processing for estimating each dye amount at each sample point above is performed (step S105), and the control unit 8 displays the estimated spectral characteristics estimated in step S104, the estimated dye amount estimated in step S105, and the like. An estimation result display process to be displayed on the unit 5 is performed (step S106), and the series of processes is terminated.

  In step S106, the control unit 8 can appropriately display the observation image, the reference sample image, the reference spectral transmittance, and the like on the display unit 5 without being limited to the estimation result. Moreover, the control part 8 memorize | stores the process result, process parameter, etc. for every step in the memory | storage part 6 suitably in each step of step S101-S106.

  Next, each step shown in FIG. 7 will be described in detail. First, in the observation image acquisition process in step S101, the observation image acquisition unit 2 captures the sample 10 while sequentially switching the bandpass filters 14a and 14b, thereby, for example, generating a multiband image as an observation image as shown in FIG. get. However, in FIG. 8, monochrome display is performed. In addition, the observation image acquisition unit 2 acquires the observation image, and then observes the multiband image only of the illumination light emitted from the light source 12 as the background light multiband image in a state where the specimen 10 is removed from the stage 11. Acquired by the same imaging procedure as in the case of images. The control unit 8 causes the storage unit 6 to store the observation image acquired in step S101 and the multiband image of the background light.

  Next, the spectral characteristic estimation process in step S104 will be described. In this spectral characteristic estimation process, the spectral characteristic estimation unit 7f estimates the spectral characteristics of each sample point on the specimen 10 by Wiener estimation. Here, the sample point on the sample 10 is an observed point on the sample 10 corresponding to each pixel or each pixel group on the observation image.

  Usually, the pigment is three-dimensionally distributed in the specimen 10, but the specimen 10 cannot be captured as a three-dimensional image in the observation image, and the illumination light transmitted through the specimen 10 in the thickness direction is used as the camera 16. As a two-dimensional image formed on the image sensor. Therefore, in the observation image, each point in the plane obtained by capturing the sample 10 in a plane is observed as a sample point. In this case, the pigment amount at each sample point corresponds to the total pigment amount (integrated amount) of the pigment distributed in the thickness direction of the sample 10 at the sample point.

  Each position (coordinate) between the sample point on the sample 10 and the observation point (pixel or pixel group) on the observation image corresponding to the sample point is determined by a conversion formula based on various apparatus parameters in the observation image acquisition unit 2. Corresponding in advance, the image processing unit 7 can appropriately convert the coordinates of the sample point and the observation point using the conversion formula.

In general, the pixel value g (u, b) at the observation point position u and the band b on the observation image and the spectral transmittance t (u, λ) at the sample point on the sample 10 corresponding to the observation point position u. The relationship shown by the following formula (1) is established between them. The following equation (1) represents the wavelength λ, the spectral transmittance f (b, λ) corresponding to the band b of the bandpass filter used to capture the observation image, and the spectral sensitivity characteristics of the camera used to capture the observation image. It is expressed using s (λ), the spectral radiation characteristic e (λ) of the illumination light used for capturing the observation image, and the imaging noise n (b) in the band b. The band b is a serial number for identifying each band in the multiband image as the observation image.

When the equation (1) is discretized with respect to the wavelength, the pixel value g (u, b), spectral transmittance t (u, λ), spectral transmittance f (b, λ), spectral sensitivity characteristic s (λ), spectral radiation It is represented by the following equation (2) using matrices G (u), T (u), F, S, E, and N corresponding to the characteristic e (λ) and the imaging noise n (b), respectively.
G (u) = FSET (u) + N (2)

  Here, the matrix G (u) is B rows and 1 column, the matrix T (u) is D rows and 1 column, the matrix F is B rows and D columns, and the matrix S, based on the number of sample points D of wavelengths and the number of bands B. Is D rows D columns, matrix E is D rows D columns, and matrix N is B rows 1 columns. The matrix S is a diagonal matrix having the spectral sensitivity characteristic s (λ) as a diagonal element, and the matrix E is a diagonal matrix having the spectral radiation characteristic e (λ) as a diagonal element. In Expression (2), since the expressions related to a plurality of bands are aggregated using a matrix, the band b is not explicitly described as a variable. Also, the integration with respect to the wavelength λ has been replaced with a matrix product.

According to the Wiener estimation, the matrix T ^ (u) indicating the estimated value of the spectral transmittance at the sample point on the sample 10 corresponding to the observation point position u is expressed by the matrix R _SS , R _NN based on the equation (2). Is calculated by the following equation (3). Here, the matrix R _SS is a matrix of D rows and D columns, and is an autocorrelation matrix of the spectral transmittance of the sample to be estimated. The matrix R _NN is a matrix of B rows and B columns, and is an autocorrelation matrix of camera noise used for capturing an observation image. Note that the symbol () ^t represents a transposed matrix, and the symbol () ⁻¹ represents an inverse matrix.
_{T ^ (u) = R SS} (FSE) t ((FSE) R SS (FSE) t + R NN) -1 G (u) ··· (3)

  In the spectral characteristic estimation process in step S104, the spectral characteristic estimation unit 7f calculates an estimated spectral transmittance at each sample point on the sample 10 using Expression (3). Each estimated spectral transmittance calculated is stored in the storage unit 6. The matrices F, S, and E are measured in advance using a spectrometer or the like after the bandpass filter 14, the camera 16, and the light source 12 in the observation image acquisition unit 2 are selected, and stored in the storage unit 6. Here, the spectral transmittance of the imaging optical system 13 is approximated to “1.0”. However, when the deviation from “1.0” cannot be allowed, this spectral transmittance is also measured in advance, and the matrix E It is good to multiply by.

Further, the matrix R _SS is calculated in advance in step S103 as described later based on the reference spectral transmittance acquired in step S102 and stored in the storage unit 6. The matrix R _NN is obtained in advance by generating an autocorrelation matrix of pixel values for each band of the background light multiband image captured in step S 101, and is stored in the storage unit 6. However, in this case, it is assumed that there is no noise correlation between the bands.

Next, the pigment amount estimation processing in step S105 will be described. In general, in a substance that transmits light, a coefficient k (λ) specific to the substance and the thickness of the substance between the intensity I ₀ (λ) of the incident light and the intensity I (λ) of the emitted light for each wavelength λ. It is known that the Lambert-Beer law expressed by the following equation (4) is established using d.

In the specimen 10 subjected to H & E staining, the coefficients k _H (λ), k _E (λ), k _R (λ) for each of the dye H, the dye E, and the dye R and the position of the observation point based on the equation (4) Using the thicknesses d _H (U), d _E (U), and d _R (U) of the sample point position U corresponding to u, the following equation (5) is established.

The left side of Equation (5) is the spectral transmittance, and the coefficients k _H (λ), k _E (λ), and k _R (λ) are standard spectral characteristics specific to Dye H, Dye E, and Dye R, respectively. is there. The coefficients k _H (λ), k _E (λ), and k _R (λ) are obtained by preparing a standard specimen stained with a corresponding single dye and measuring its spectral transmittance with a spectrometer. It is obtained in advance and stored in the storage unit 6.

The thicknesses d _H (U), d _E (U), and d _R (U) are virtual thicknesses corresponding to the dye H, the dye E, and the dye R, respectively. Compared to the assumption that each dye E and dye R is dyed with a single dye, an index indicating a relative amount of dye, such as a relative amount of the dye, Become. That is, it can be said that the thicknesses d _H (U), d _E (U), and d _R (U) represent the dye amounts of the dye H, the dye E, and the dye R at the sample point position U on the sample 10, respectively. .

Taking the logarithm of both sides of equation (5), the following equation (6) is obtained.

Substituting the estimated spectral characteristic t ^ (U, λ) calculated using equation (3) as the spectral transmittance I (λ) / I ₀ (λ) on the left side of equation (6), the following equation (7 ) Here, the estimated spectral characteristic t ^ (U, λ) is obtained by converting the element t ^ (u, λ) corresponding to the wavelength λ of the matrix T ^ (u) calculated by the equation (3) with respect to the sample point position U. Is required.

Since there are three unknown variables of thicknesses d _H (U), d _E (U), and d _R (U) in equation (7), equation (7) is made simultaneous for at least three different wavelengths λ. This can be solved. Alternatively, in order to further improve the calculation accuracy, the multiple regression analysis can be performed by simultaneous equations (7) for four or more different wavelengths λ.

When the equation (7) is made simultaneous for the _three wavelengths λ ₁ , λ ₂ , and λ ₃ , the following equation (8) is obtained. Accordingly, the thicknesses d _H (U), d _E (U), and d _R (U) are calculated by the following equation (9).

In the pigment amount estimation process in step S105, the pigment amount estimation unit 7g uses the formula (9) to calculate the thicknesses d _H (U) and d _E (U) as the estimated pigment amounts at each sample point position U on the sample 10. ), D _R (U). Each calculated estimated pigment amount is stored in the storage unit 6. In the following description, the thicknesses d _H (U), d _E (U), and d _R (U) are referred to as estimated pigment amounts d _H (U), d _E (U), and d _R (U), respectively. .

  Next, the reference spectral characteristic acquisition process in step S102 will be described. In this reference spectral characteristic acquisition process, the reference spectral characteristic acquisition unit 3 acquires a reference spectral transmittance in association with each sample point position for each sample point on the reference sample. At that time, the reference spectral characteristic acquisition unit 3 acquires the reference spectral transmittance from two sample regions on the reference sample, and the control unit 8 sets the reference spectral transmittance corresponding to one of the sample regions as described later. Are used as teacher data for extracting representative points. Further, the reference spectral transmittance corresponding to the other specimen region is stored in the storage unit 6 as test data for performing representative point optimization as described later. The sample area corresponding to the test data is hereinafter referred to as a test area.

  Next, the statistic calculation process in step S103 will be described in detail. FIG. 9 is a flowchart showing a processing procedure of the statistic calculation processing. As shown in this figure, first, the reference sample image generation unit 7a generates a teacher image and a test image as a reference sample image based on the teacher data and test data acquired in step S102 (step S111). Subsequently, the main element classification unit 7b classifies each pixel of the teacher image based on at least a main feature amount that characterizes the main element in the sample 10, and thereby performs a main element classification process for classifying each main element (step S112). ).

  Then, the representative point extraction unit 7c sets an extraction condition for extracting a representative pixel for each pixel group classified in step S112 as an extraction condition for extracting a representative point for each main element (step S113). Based on the set extraction conditions, representative pixels are extracted from the pixel group, thereby extracting representative points (step S114).

Thereafter, the statistic calculation unit 7d uses the autocorrelation matrix R _SS (see formula (3)) as the statistic based on the reference spectral transmittance associated with each representative pixel extracted in step S113. Then, the representative point optimization unit 7e calculates a winner estimation processing matrix using the autocorrelation matrix R _SS calculated in step S115 (step S116).

  Subsequently, the representative point optimization unit 7e uses the Wiener estimation processing matrix calculated in step S116 based on the image data of the test image generated in step S111, and uses the spectral characteristics as the spectral characteristics in the test region of the reference sample. The transmittance is estimated (step S117), and a spectral transmittance error with respect to the reference spectral transmittance of the estimated spectral transmittance is calculated (step S118).

  Then, the representative point optimization unit 7e determines whether or not the calculated spectral transmittance error satisfies a predetermined determination condition (step S119), and determines that the determination condition is not satisfied (step S119: No), the representative pixel extraction condition, that is, the representative point extraction condition is updated (step S120). After step S120, the control unit 8 causes the representative point extraction unit 7c, the statistic calculation unit 7d, and the representative point optimization unit 7e to repeat the processing from step S114. On the other hand, when the spectral transmittance error satisfies a predetermined determination condition (step S119: Yes), the control unit 8 ends the statistic calculation process and returns to step S103.

In step S111, the reference sample image generation unit 7a calculates teacher image data and test image image data from the teacher data and test data, respectively, based on the following equation (10) based on equation (1). In the following equation (10), the pixel value g (v, b) at the pixel position v on the teacher image or test image corresponding to each sample point position V on the reference sample is acquired as teacher data or test data. It is calculated using the reference spectral transmittance t (v, λ).

  Here, the reference specimen image generation unit 7a has a spectral transmittance f (b, λ), a spectral sensitivity characteristic s (λ), and a spectral radiation characteristic e (λ). By calculating the pixel value g (v, b) using the spectral sensitivity characteristics of the light source 12 and the spectral radiation characteristics of the light source 12, a test image that is a 6-band image similar to the observation image is generated. Furthermore, by calculating each pixel value g (v, b) with the spectral transmittance f (b, λ) of the bandpass filter as “1”, a teacher image that is an RGB image is generated. The generated image data of the teacher image and the test image is stored in the storage unit 6 in association with the corresponding reference spectral transmittance for each pixel value g (v, b).

Next, the main element classification process in step S112 will be described. FIG. 10 is a flowchart showing a processing procedure of the main element classification processing. As shown in this figure, the main element classification unit 7b first maps each pixel of the teacher image to the a ^* b ^* space as the color space (step S131). The a ^* b ^* space is a two-dimensional feature space having an a ^* value and a b ^* value on each axis in the L ^* a ^* b ^* color system. The main element classification unit 7b calculates an a ^* b ^* value from the RGB values for each pixel on the teacher image, and maps each pixel to the a ^* b ^* space based on the calculated a ^* b ^* value. Accordingly, each pixel on the teacher image is a ^* b ^* value indicating the color characterizing each main element as the main feature quantity characterizing each main element in the a ^* b ^* space, for example, as indicated by a black dot in FIG. A pixel group is formed. The a ^* b ^* value of each pixel is stored in the storage unit 6 in association with the pixel position v (xy coordinates) on the teacher image.

(Step Subsequently, the main element classification unit 7b is configured to calculate a histogram in a ^* b ^* space, by dividing the histogram by the total number of pixels of the teacher image, to calculate the probability distribution in the a ^* b ^* space S132). In step S132, the main element classification unit 7b obtains a histogram as shown in FIG. 12, for example. The histogram shown in FIG. 12 corresponds to the histogram in the XII-XII cross section shown in FIG. The probability distribution calculated here is stored in the storage unit 6.

Subsequently, the main elements classification unit 7b includes a ^a * ^{b *} values at the pixel position v on the teacher ^image, using a probability distribution in the a * ^{b * space,} each group of pixels distribution function in a * ^{b *} space To calculate. Specifically, the main element classification unit 7b calculates each distribution function using an EM algorithm (Expectation Maximization Algorithm). Therefore, the main element classifying unit 7b first models a distribution function for the probability distribution for each pixel group using a mixed Gaussian distribution (GMD) (step S133).

GMD is known as a function that can be widely approximated to a general distribution of measured values. A GMD parameter θ (hereinafter referred to as GMD parameter θ) is expressed using a weighting factor α, an average μ, and a variance σ ² . In step S133, the main element classification unit 7b sets an appropriate initial value for the GMD parameter θ for each pixel group. This initial value is appropriately input from the input unit 4 by an operator of the image processing apparatus 1, for example.

Subsequently, the main element classification unit 7b calculates the GMD parameter θ that approximates the distribution function in the a ^* b ^* space by the EM algorithm for each pixel group. As disclosed in Non-Patent Document 2, the EM algorithm is widely known as a kind of maximum likelihood estimation algorithm using an iterative operation using GMD. The main element classifying unit 7b performs GMD parameter θ for each pixel group by repeating calculation of E step (Expectation Step) (step S134), M step (Maximization Step) (step S135), and determination processing (step S136) in the EM algorithm. The optimal distribution function is calculated for each pixel group using the GMD parameter θ as the maximum likelihood solution.

Specifically, in step S135, the main element classification unit 7b uses the a ^* b ^* value of each pixel calculated in step S131 as a measured value in GMD, and the total number of histograms (total number of pixels) N calculated in step S132. Is the number of measured values in the GMD, and the weighting coefficient α, average μ, and variance σ ² constituting the GMD parameter θ are sequentially updated by the following equations (11) to (13), respectively. Here, the following equations (11) to (13) are expressed using the EM algorithm repetition number t, the pixel group identification number j, and the target pixel identification number z.

Furthermore, the main element classification unit 7b sequentially updates the normal distribution function p _j , the conditional expected value p, and the evaluation value l (θ _j ) according to the following equations (14) to (16), respectively. Here, the following equations (14) to (16) are expressed using the autocorrelation matrix Σ _j , the determinant | Σ _j | of the autocorrelation matrix, and the number of dimensions d. In this embodiment, since it is a ^* b ^* space, the number of dimensions is d = 2. Note that the symbol T indicates a transposed matrix.

In the determination process in step S136, the main element classification unit 7b determines whether or not the evaluation value l (θ _j ) satisfies a predetermined convergence condition. If the convergence condition is not satisfied (step S136). : No), the process from step S134 is repeated, and when the convergence condition is satisfied (step S136: Yes), the EM algorithm is terminated and the main element classification process is terminated, and the process returns to step S112.

Here, the predetermined convergence condition, for example, the convergence value of the evaluation values l (θ _j) is within the predetermined threshold, or evaluation values l (θ _j) is the be within a predetermined threshold. Further, the convergence value of the evaluation value l (theta _j) is obtained for evaluation value l calculated as a log-likelihood (theta _j), the highest evaluation among repetitive operations of a series of EM algorithm to the current time of evaluation It is calculated as a difference obtained by subtracting the log likelihood at the current evaluation time from the log likelihood.

Through the above main element classification processing, the main element classification unit 7b obtains the optimal GMD parameter θ and the distribution function p (a ^* , b ^* : θ) that most closely approximates the probability distribution in the a ^* b ^* space. Can do. Distribution function ^{p (a *, b *:} θ) distribution function p _j for each pixel group of the ^{(a *, b *: θ} j) is, which corresponds to a molecule of the right side of equation (15) Is represented by the following equation (17).

The distribution function p _j (a ^* , b ^* : θ _j ) for each pixel group is obtained, for example, as shown in FIG. Figure 12 shows the distribution function ^{_{p j (a *, b *}} : θ j) is, XII-XII distribution function in the section ^{_{p j (a *, b *}} : θ j) of FIG. 11 corresponds to. Each pixel on the teacher image and each main element are classified by the distribution function p _j (a ^* , b ^* : θ _j ) obtained for each pixel group in this way. The distribution function p _j (a ^* , b ^* : θ _j ) calculated here is stored in association with the a ^* b ^* value at the pixel position v on the teacher image and the probability distribution in the a ^* b ^* space. Stored in the unit 6.

Next, processing for setting representative point extraction conditions in step S113 will be described. In step S113, the representative point extraction unit 7c extracts the representative pixel extraction range for each pixel group and the extraction within the extraction range as the extraction condition for extracting the representative pixel for each pixel group classified in step S112. Set the number. Specifically, the representative point extraction unit 7c uses the GMD parameter calculated in step S112 and the distribution function p (a ^* , b ^* : θ), and uses the classification feature amount range C (a ^* ) as the representative point extraction range ^. , B ^* ). Then, the classification feature amount range C _j (a ^* , b ^* ) for each pixel group classified in step S112, that is, for each main element, is used as the distribution function p _j (a ^* , b ^* : θ _j ) for each pixel group. The average μ at is the center position, and the spread from this center position is the range inside the contour line with the variance σ ² . Thereby, the classification feature amount range C _j (a ^* , b ^* ) for each pixel group is obtained as shown in FIG. 11, for example. The classification feature amount range C (a ^* , b ^* ) calculated here is stored in the storage unit 6 in association with the distribution function p (a ^* , b ^* : θ).

Further, the representative point extraction unit 7c sets, for example, random extraction as an extraction method for extracting representative pixels from the classified feature amount range C (a ^* , b ^* ). In random extraction, a plurality of combinations of a ^* values and b ^* values in the classification feature amount range C (a ^* , b ^* ) are randomly determined, and are indicated by the determined combinations of a ^* values and b ^* values. Each pixel is extracted as a representative pixel. Further, when a plurality of pixels are associated with one combination of the a ^* value and the b ^* value, extraction is also performed at random from the plurality of pixels. The extraction of the representative pixels is repeatedly performed up to the extraction number m _j set for each range in the classification feature amount range C _j (a ^* , b ^* ) of each pixel group. However, the pixels once extracted as the representative pixels are excluded from the selection targets in the series of extraction processes thereafter.

In the present embodiment, considering that the specimen is a pathological specimen, the representative point extraction unit 7c mainly uses the number of extractions within the extraction range, that is, the number of extractions m _j for each main element, and the total number of representative pixels M as the extraction number. The number M / q is divided by the number of element types q. Also, the representative point extraction unit 7c can weight the number of extractions using the weighting coefficient α when improving the estimation accuracy of a specific main element among a plurality of main elements. Specifically, the extraction number m _j is determined by the product α _j M of the weight coefficient α _j for each pixel group, that is, for each main element and the total extraction number M, and the weight coefficient α _j for the main element whose estimation accuracy is to be improved is determined. The value is larger than the weight coefficient α _j of other main elements. However, the weight coefficient α _j satisfies the following expression (18).

The representative point extraction unit 7c may use probability-dependent extraction or measurement value-dependent extraction in addition to random extraction as an extraction method for extracting representative pixels from the classification feature amount range C (a ^* , b ^* ). it can. Here, probability-dependent extraction is a method of extracting representative pixels by weighting based on the distribution function p (a ^* , b ^* : θ) calculated in step S112. That is, based on the probability of the distribution function p (a ^* , b ^* : θ), the a ^* value and b ^* value of the representative pixel in the classification feature amount range C (a ^* , b ^* ) are determined. Probability-dependent extraction can be used to extract representative points that depend on the distribution of feature quantities in the main elements. For example, the average of the teacher image corresponding to each main element in the pathological sample can be used. This can be used when it is desired to preferentially extract pixels having RGB values as representative pixels.

On the other hand, the measurement value-dependent extraction can be used when extracting representative points depending on the feature amount in the main element. For example, among the pixels on the teacher image corresponding to the main element in the pathological specimen, This can be used when a pixel having a large Euclidean distance L _y from the origin in the a ^* b ^* space represented by the following equation (19) is to be extracted preferentially as a representative pixel, or vice versa.

At this time, the extraction measurement intensity range of the representative pixel is limited, and the measurement value intensity range is added to the extraction condition. Measurement intensity range is classified feature value range _{^{C j (a *, b *}} ) maximum Euclidean distance L _y classified feature amount range as ^{_{100% C j (a *,}} b *) in the Expressed against intensity ratio of Good. For example, the classification feature quantity range C _j of the key elements ^(a *, ^{b *) of,} a * ^{b *} If the Euclidean distance L _y from the origin of the space is the minimum 0%, the measurement intensity 0% If it is desired to extract representative pixels in the range of 30% to 30%, the measured value intensity range may be specified in the range of 0% to 30%. Here, the intensity corresponds to the a ^* value and b ^* value in the a ^* b ^* space, and the intensity range corresponds to the range of the a ^* value and the b ^* value.

It should be noted that the extraction range and the number of extractions of representative pixels are not limited to the initial values set in step S113 because an optimal solution is derived by the subsequent representative point optimization process. However, in the present embodiment, considering that the specimen is a pathological specimen, the number of extracted m _j for each main element is a number M / 4 obtained by dividing the total number of extracted M by the number of main element types q = 4. This improves the convergence of the optimal solution. For the same purpose, the classification feature amount range C _j (a ^* , b ^* ) is a range in which the distribution function p _j (a ^* , b ^* : θ _j ) is a substantial proportion, for example, 90%. It is preferable that the average μ and the variance σ ² for calculating are set to initial values.

Next, representative point optimization processing in steps S116 to S120 will be described. First, in step S116, the representative point optimization unit 7e calculates a system matrix H by the following equation (20), and based on the calculated system matrix H, an estimation operator W as a winner estimation processing matrix is expressed by the following equation (21). ). Note that the matrices F, S, E, R _SS , and R _NN used in Expression (20) and Expression (21) correspond to the respective matrices used in Expression (3).
H = (FSE) ^t (20)
_{W = R SS H (HR SS} H t + R NN) -1 ··· (21)

Here, the expression (3) can be rewritten as the following expression (22) using the expression (21) with respect to the pixel position v on the test image. In step S117, the representative point optimization unit 7e calculates a matrix T ^ (v) for each of the plurality of pixels in the test image using the following equation (22), and thereby spectral transmission in the test region of the reference sample. Estimate the rate.
T ^ (v) = WG (v) (22)

Thereafter, in step S118, the representative point optimization unit 7e, based on the matrix T ^ (v) calculated in step S117 and the reference spectral transmittance t ^ (v) acquired as test data in step S102, Spectral transmittance error ΔE is calculated by the following equation (23). The spectral transmittance error ΔE is obtained as an average value of the difference between the estimated spectral transmittance and the reference spectral transmittance in the number N of pixels, as shown in Expression (23).

  In step S119, the representative point optimization unit 7e uses the spectral transmittance error ΔE calculated in step S118 as an evaluation value of the autocorrelation matrix, and determines whether or not the spectral transmittance error ΔE satisfies a predetermined determination condition. It is determined whether or not it is necessary to update the representative pixel extracted in step S114, that is, the representative point.

  Here, the predetermined determination condition is that the spectral transmittance error ΔE is within a predetermined threshold or that the spectral transmittance error ΔE is converged by repeating the processes of steps S114 to S120. As a convergence condition of the spectral transmittance error ΔE, for example, the difference between the maximum value and the minimum value of the spectral transmittance error ΔE when Steps S114 to S120 are repeated a plurality of times is within a predetermined threshold.

In step S120, the representative point optimization unit 7e updates the extraction condition by changing, for example, at least one of the extraction range of the representative pixels and the number of extractions in the extraction range. In updating the extraction range, the representative point optimization unit 7e updates the classification feature amount range C (a ^* , b ^* ) by changing the variance σ ² that defines the classification feature amount range C (a ^* , b ^* ). To do. In addition, the representative point optimization unit 7e stores the direction in which the extraction range and the number of extractions are changed as the update direction in the storage unit 6 as needed, and reflects the stored update direction in the next update. For example, when the value of the spectral transmittance error ΔE decreases with respect to the first update direction, the extraction range and the number of extractions are changed in the same update direction at the next update. Specifically, the update direction corresponds to the increase / decrease direction of the variance σ ² as the extraction probability width and the increase / decrease direction of the total extraction number M or the extraction number m _j for each main element.

  In addition, the control part 8 can also repeat the process from step S114, without making the representative point optimization part 7e update extraction conditions in step S120. In this case, the representative pixels are randomly extracted from the extraction range while leaving the extraction conditions as they are. As a result, the combination of representative pixels, that is, representative points can be changed without updating the extraction conditions. In addition, the representative point optimization unit 7e can change the representative pixel extraction method from random extraction to probability-dependent extraction method or measurement value-dependent extraction in step S120, thereby changing the combination of representative points.

  As described above, the image processing apparatus 1 according to the present embodiment is based on the image data of the observation image obtained by imaging the specimen 10 including one or more main elements, and the reference sample including the one or more main elements. An image processing apparatus that estimates the spectral characteristics of the sample 10 using the statistics of the reference spectral characteristics at a plurality of upper sample points, and is an image in which the reference sample is recorded and corresponding samples on the reference sample for each pixel A main element classifying unit 7b that classifies each pixel of the teacher image associated with the reference spectral characteristic at the point based on at least a main feature amount that characterizes the main element, and a pixel group classified by the main element classifying unit 7b A representative point extraction unit 7c that extracts a plurality of representative pixels, and a reference that is used to estimate the spectral characteristics of the sample 10 based on the reference spectral characteristics associated with the plurality of representative pixels extracted by the representative point extraction unit 7c. And it includes a statistic calculation unit 7d for calculating a statistical amount of the optical characteristics, the.

  Therefore, in the image processing apparatus 1, a plurality of representative points can be extracted from each main element in the reference sample at equal or desired ratios, and the sample 10 is based on the reference spectral characteristics at the plurality of representative points. The statistic of the reference spectral characteristic used for estimating the spectral characteristic of the present invention can be calculated, and the winner estimation processing matrix as the processing matrix used for estimating the spectral characteristic can be calculated. As a result, the image processing apparatus 1 can estimate the spectral characteristics at each sample point on the sample 10 with high accuracy without causing unintended variations.

  Further, the image processing apparatus 1 determines whether or not the representative pixel needs to be updated based on the statistic calculated by the statistic calculation unit 7d. If the image processing apparatus 1 determines that the update is necessary, extraction for extracting the representative pixel The representative point optimizing unit 7e for updating the condition is provided, the representative point extracting unit 7c re-extracts a plurality of representative pixels based on the extraction condition updated by the representative point optimizing unit 7e, and the statistic calculating unit 7d The statistic is calculated based on the reference spectral characteristics associated with the plurality of representative pixels re-extracted by the representative point extraction unit 7c. For this reason, the image processing apparatus 1 can extract the optimum representative point from the reference sample for the estimation of the spectral characteristic of the sample 10 including one or more main elements, and the reference spectral characteristic at the extracted representative point. Based on the above, an optimal statistic and a winner estimation processing matrix can be calculated. Thereby, in the image processing apparatus 1, it is possible to improve the estimation accuracy of the spectral characteristics of the specimen 10 as compared with the conventional technique.

  Since the image processing apparatus 1 estimates the dye amounts of a plurality of dyes included in the sample 10 based on the estimated spectral characteristics of the sample 10 estimated as described above, each dye is accurately detected. The amount can also be estimated.

  So far, the best mode for carrying out the present invention has been described as an embodiment. However, the present invention is not limited to the above-described embodiment, and various modifications may be made without departing from the spirit of the present invention. Is possible.

For example, in the above-described embodiment, the main element classification unit 7b uses the a ^* b ^* value indicating the color that characterizes the main element as the main feature amount that characterizes the main element in the sample 10 to each of the reference sample images. The pixels are classified. However, the pixels are not limited to a ^* b ^* values, and each pixel on the reference sample image may be classified using a value indicated by another color system such as an RGB value or an XYZ value. it can. Here, the XYZ value means a tristimulus value in the XYZ color system. Furthermore, the main feature amount is not limited to color information, and shading information, shape information, texture information, and the like can be used.

  However, since shape information and texture information cannot indicate a feature amount with a single pixel, processing such as region division of the reference sample image, for example, block division, is required. Further, when a pathological specimen is an estimation target, entropy may be used as a feature amount using a texture. Here, considering that the main element classification processing is the classification of pixels corresponding to each main element, it is not necessary to stick to classifying a single pixel as a unit, and a pixel group consisting of a plurality of pixels is a unit. Classification may be performed as follows.

Note that when using feature values other than the a ^* b ^* values, as described above, each feature value is assigned to each pixel on the teacher image in the a ^* b ^* space. By mapping and classifying each pixel on the teacher image to the feature space shown, the subsequent processing can be performed by the same processing as in the above-described embodiment.

Further, the image processing apparatus 1 can further perform the following processing in order to improve the approximation accuracy of the EM algorithm in the main element classification processing in step S112 described above. That is, first, the reference spectral characteristic acquisition unit 3 acquires the spectral transmittance of the background region as background data by the reference spectral characteristic acquisition process of step S102. Then, in step S111, the reference sample image generation unit 7a generates a background image corresponding to the background data in the same manner as the teacher image. Thereafter, the main element classification unit 7b maps each pixel of the background image to the a ^* b ^* space in the main element classification process of step S112, and a two-dimensional normal distribution on the a ^* b ^* space of the mapped background image. The parameter of the single normal distribution function that is most approximated from the maximum likelihood estimation method is calculated. This process can be performed by using the EM algorithm described above as a single normal distribution function. Next, the main element classification unit 7b determines the a ^* b ^* value range corresponding to the background region using the calculated single normal distribution function, and uses the background image from the a ^* b ^* space mapped using the teacher image. The range corresponding to is removed. By removing the background area from the color space of the teacher data in advance, the approximation target of the EM algorithm can be limited to the three areas of cell nucleus, cytoplasm, and red blood cell. In general pathological specimens, the boundary between the cytoplasm and the background often touches in the color space. By excluding the background from the processing target of the EM algorithm in advance, the distribution functions of other regions can be more accurately detected. An approximate expression can be obtained.

  In the above-described embodiment, the image processing apparatus 1 has been described as estimating the amount of dye based on the estimated spectral transmittance. However, the image processing apparatus 1 is not limited to estimating the amount of dye, and the estimated spectral transmittance and Various processes that lead to pathological diagnosis support can be performed based on the estimated pigment amount.

  In the above-described embodiment, the observation image acquisition unit 2 has been described as capturing and acquiring an observation image. However, the observation image acquisition unit 2 may acquire an observation image captured in advance by an external device. In this case, the observation image acquisition unit 2 may include a data communication interface that can input image data of the observation image from an external device or the like.

  In the above-described embodiment, the image processing apparatus 1 has been described as handling the spectral transmittance as the spectral characteristics of the specimen 10 and the reference specimen. However, the image processing apparatus 1 is not limited to the spectral transmittance and may handle the spectral reflectance. it can. In this case, the observation image acquisition unit 2 and the reference spectral characteristic acquisition unit 3 illustrated in FIG. 2 may include an illumination mechanism that performs epi-illumination on the sample 10 and the reference sample instead of the light source 12.

It is a block diagram which shows the principal part structure of the image processing apparatus concerning embodiment of this invention. It is a figure which shows the principal part structure of the observation image acquisition part with which an image processing apparatus is provided. It is a figure which shows the RGB filter of the camera with which an observation image acquisition part is provided. It is a figure which shows the spectral sensitivity characteristic of the camera with which an observation image acquisition part is provided. It is a figure which shows the spectral transmittance characteristic of the band pass filter with which an observation image acquisition part is provided. It is a figure which shows the spectral transmittance characteristic of the band pass filter with which an observation image acquisition part is provided. It is a figure which shows the outline | summary of the image processing procedure which an image processing apparatus performs. It is a flowchart which shows the image processing procedure which an image processing apparatus performs. It is a figure which shows the observation image which the observation image acquisition part acquired. It is a flowchart which shows the process sequence of a statistic calculation process. It is a flowchart which shows the process sequence of the main element classification | category process in a statistic calculation process. It is a figure explaining the process result of a main element classification | category process. It is a figure explaining the process result of a main element classification | category process.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image processing apparatus 2 Observation image acquisition part 3 Reference | standard spectral characteristic acquisition part 4 Input part 5 Display part 6 Storage part 7 Image processing part 7a Reference | standard sample image generation part 7b Main element classification | category part 7c Representative point extraction part 7d Statistics amount calculation part 7e Representative point optimization unit 7f Spectral characteristic estimation unit 7g Dye amount estimation unit 8 Control unit 10 Sample 11 Stage 12 Light source 13 Imaging optical system 14, 14a, 14b Band pass filter 15 Turret 16 Camera

Claims (13)

Based on image data of an observation image obtained by imaging a sample including one or more main elements, the statistic of the reference spectral characteristic at a plurality of sample points on the reference sample including the one or more main elements is used. In an image processing apparatus that estimates spectral characteristics,
Each pixel of the teacher image that is an image in which the reference sample is recorded and is associated with the reference spectral characteristic at the corresponding sample point on the reference sample for each pixel is used as a main feature amount that characterizes at least the main element. Pixel classification means for classifying based on;
Representative extraction means for extracting a plurality of representative pixels from the pixel group classified by the pixel classification means;
Statistic calculation means for calculating the statistic based on the reference spectral characteristics associated with a plurality of the representative pixels;
An image processing apparatus comprising: A representative that determines whether or not the representative pixel needs to be updated based on the statistics calculated by the statistic calculator, and updates the extraction condition for extracting the representative pixel when it is determined that the update is necessary. With optimization means,
The representative extraction means re-extracts the representative pixels based on the extraction conditions updated by the representative optimization means,
The said statistic calculation means calculates the said statistic based on the said reference | standard spectral characteristic matched with the said some representative pixel re-extracted by the said representative extraction means. Image processing device.   The representative optimization unit is an image in which a test area not recorded in the teacher image is recorded in the reference sample area, and the reference spectral characteristic at a corresponding sample point on the test area for each pixel. Based on the image data of the associated test image, the spectral characteristic of the test area is estimated using the statistical quantity calculated by the statistical quantity calculation means, and the estimated spectral characteristic and the test area of the test area The image processing apparatus according to claim 2, wherein a reference spectral characteristic is compared, and whether or not the representative pixel needs to be updated is determined based on the comparison result.   The representative extraction means is an extraction condition for extracting the representative pixel from at least one of the extraction range of the representative pixel and the number of extractions in the extraction range based on the classification result of the pixel classification means. The image processing apparatus according to claim 1, wherein the image processing apparatus is set as follows. The representative extraction means sets at least one of the extraction range of the representative pixel in the pixel group and the number of extractions within the extraction range based on the classification result as an extraction condition for extracting the representative pixel,
The representative optimization means, when it is determined that the representative pixel needs to be updated, updates at least one of the extraction range and the number of extractions in the extraction range based on the classification result, The image processing apparatus according to claim 2 or 3.   The pixel classification unit maps each pixel of the teacher image to a feature space based on the main feature amount, and classifies each pixel of the teacher image based on a distribution of each pixel mapped to the feature space. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus.   The image processing apparatus according to claim 6, wherein the main feature amount is at least one of color information, shading information, and texture information.   The representative extraction means extracts the representative pixel from at least one of the extraction range of the representative pixel in the pixel group and the number of extractions in the extraction range based on the distribution of each pixel mapped to the feature space. The image processing apparatus according to claim 6, wherein the image processing apparatus is set as an extraction condition for performing the operation.   The representative extraction means calculates at least one of the extraction range of the representative pixel in the pixel group and the number of extractions in the extraction range based on the distribution of each pixel mapped to the feature space and the main feature amount. The image processing device according to claim 6, wherein the image processing device is set as an extraction condition for extracting the representative pixel.   The representative extraction unit randomly extracts a plurality of the representative pixels from the pixel group based on an extraction condition for extracting the representative pixels. An image processing apparatus according to 1. The pixel classification means calculates a distribution probability of each pixel mapped to the feature space;
10. The representative extraction unit extracts a plurality of the representative pixels from the pixel group based on an extraction condition for extracting the representative pixels and the distribution probability. The image processing apparatus according to any one of the above.   The image processing apparatus according to claim 1, wherein the spectral characteristic and the reference spectral characteristic are a spectral transmittance characteristic or a spectral reflectance characteristic. Based on image data of an observation image obtained by imaging a sample including one or more main elements, the statistic of the reference spectral characteristic at a plurality of sample points on the reference sample including the one or more main elements is used. In an image processing device that estimates spectral characteristics,
Each pixel of the teacher image that is an image in which the reference sample is recorded and is associated with the reference spectral characteristic at the corresponding sample point on the reference sample for each pixel is used as a main feature amount that characterizes at least the main element. Pixel classification procedure to classify based on;
A representative extraction procedure for extracting a plurality of representative pixels from the pixel group classified by the pixel classification procedure;
A statistic calculation procedure for calculating the statistic based on the reference spectral characteristics associated with a plurality of the representative pixels;
An image processing program for executing

Images