TWI815057B - Visualization methods for cancer lesions - Google Patents

Abstract

A visualization method for cancer lesions is provided. The visualization method can extract a pre-pool layer feature map from a cancer identification deep learning model based on slide-level operations, and use a clustering algorithm to generate multiple groups and mark the groups on a slide-level image to be labeled. A class activation map extracted from the deep learning model is then used to filter the groups that should be labeled, and finally the cancer lesions are visualized.

Description

Visualization methods for cancer lesions

The present invention relates to a method for visualizing cancer lesions, and in particular, to a method for visualizing cancer lesions that can improve the effect of separating cancer cells and non-cancer cells.

In the clinical practice of clinical diagnosis of lung tumors, the patient's lungs will be scanned through low-density computed tomography (Low density CT). If there are suspicious pulmonary nodules, Core Needle Biopsy or surgery will be performed on the pulmonary nodules. After resection, the specimen is processed and made into pathological sections, which are then diagnosed by a pathologist and finally the treatment method is decided. During the diagnosis process, the pathologist must first correctly interpret whether the slices contain tumor components, and then interpret the tumor classification to issue a pathology report. This diagnostic process relies on the experience and time of the pathologist.

In the process of using slide-level image training for cancer classification, slide-level annotation (Slide-Level Annotation) can be directly used to train the deep learning model. There is no need for experts to provide detailed annotations (Patch-Level Annotation), which can save a lot of money. time and labeling resources.

Although slide-level image training can provide excellent cancer discrimination capabilities, that is, the score of judging whether cancer cells are contained in slide units can be the same as the method of cutting the whole slide into small fragments to judge cancer cells. However, its ability to display lesion areas (Leision Localization) has not reached the level of current mainstream methods.

The technical problem to be solved by the present invention is to provide a visualization method for cancer lesions that can improve the effect of separating cancer cells and non-cancer cells in response to the shortcomings of the existing technology.

In order to solve the above technical problems, one of the technical solutions adopted by the present invention is to provide a method for visualizing cancer lesions, which includes the following steps: obtaining a slide-level image to be visualized, wherein the slide-level image to be visualized is The visualized image is judged to contain cancer cells by a trained deep learning model; the slide-level image to be visualized is input into the trained deep learning model, where the trained deep learning model includes a feature extraction network and a global pool layer and a fully connected layer; the input slide-level image to be visualized is subjected to feature extraction (feature extraction) through the feature extraction network to generate a pre-pool feature map (pre-pool feature map), Wherein, the pre-pooled feature map includes a plurality of elements, each element is used to indicate whether one of the multiple features appears in one of the multiple positions of the slide-level image to be visualized; the pre-pooled feature map is The feature map decomposes one size dimension into multiple vectors to generate a vector set, where each of these vectors has multiple channel units corresponding to the features; the vector set is divided into a grouping parameter according to a grouping algorithm through a grouping algorithm. Divide into multiple clusters; convert these clusters into multiple cluster images and present them on the slide-level image to be visualized; filter out the images based on the correspondence between the clustered images on the slide-level image to be visualized Among the clusters, at least one should be labeled that corresponds to the cancer cells in the slide-level image to be visualized; and labeling the slide with the at least one labeled group based on a Class Activation Map (CAM) Awaiting visualization.

One of the beneficial effects of the present invention is that the visualization method for cancer lesions provided by the present invention can extract the pre-pooling layer feature map from the deep learning model of cancer identification based on slide-level operations and use grouping calculations. The method generates multiple clusters and labels them on the slide-level image to be labeled, and then uses the classification activation map extracted from the deep learning model to select the clusters that should be labeled, and finally visualizes the cancer lesions.

Therefore, the visualization method for cancer lesions provided by the present invention can improve the effect of the algorithm in separating cancer cells and necrotic areas, and reduce the situation where the algorithm mistakenly identifies necrotic areas as cancer cells when visualizing cancer lesions. The algorithm for visualizing the lesion area using a model trained with slide-level image operations has been improved.

In order to further understand the features and technical content of the present invention, please refer to the following detailed description and drawings of the present invention. However, the drawings provided are only for reference and illustration and are not used to limit the present invention.

The following is a specific example to illustrate the implementation of the "visualization method for cancer lesions" disclosed in the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be modified and changed based on different viewpoints and applications without departing from the concept of the present invention. In addition, the drawings of the present invention are only simple schematic illustrations and are not depictions based on actual dimensions, as is stated in advance. The following embodiments will further describe the relevant technical content of the present invention in detail, but the disclosed content is not intended to limit the scope of the present invention. In addition, the term "or" used in this article shall include any one or combination of more of the associated listed items depending on the actual situation.

FIG. 1 is a flow chart of a method for visualizing cancer lesions according to an embodiment of the present invention. Referring to Figure 1, an embodiment of the present invention provides a method for visualizing cancer lesions, which is mainly based on a deep learning model for lung cancer identification that operates at the slide level, and the deep learning model and the present invention Any method for visualizing cancer lesions can be executed by a computer system including at least a memory and a processor.

Specifically, the deep learning model for lung cancer identification is a trained deep learning model that uses a whole-slide training method to train a convolutional neural network (CNN), and the trained deep learning model is used in Pathological digital sections of lung masses were used to classify lung adenocarcinoma, squamous cell carcinoma, other cancer types and non-cancer types.

Among them, the data used for model training and learning are 7,003 lung pathological slides provided by the Taipei Medical University Hospital, including 3,876 lung adenocarcinoma (adenocarcinoma), 1,088 squamous cell carcinoma, and non-cancer ( non-cancer) 2,039 slides, as well as pathologists' clinical diagnostic information on the slides.

For example, deep learning models based on convolutional neural networks are mostly composed of several stacked layers. When a slide-level pathological image is input, the first layer will convert the image to obtain an intermediate feature map. (Intermediate Feature Map). Then, the second layer uses the previously generated feature map (not limited to the feature map generated by the previous layer) as input to convert it into another feature map. By analogy, after all layers are calculated in sequence, the last feature map This is the result of the model predicting whether this slide contains cancer cells.

According to the different calculation formulas of each layer, it can be divided into different types of layers. Common types include Convolutional Layer, Pooling Layer, Normalization Layer, and Global Pooling Layer. ), Fully-Connected Layer, etc.

Deep learning models based on convolutional neural networks can be, for example, ResNet or DenseNet, both of which adopt similar structures. Reference may be made to FIG. 2 , which is a schematic flowchart of a method for visualizing cancer lesions according to an embodiment of the present invention. As shown in Figure 2, the deep learning model may include an input layer IN, multiple hidden layers HID, and an output layer OUT, and the hidden layer HID may include a feature extraction network FEN, a global pooling layer GP, and a fully connected layer FC. , and the feature extraction network FEN may include multiple layers selected from the group consisting of the above convolution layer, pooling layer and normalization layer.

In terms of functionality, the feature extraction network FEN formed by a multi-layer structure at the beginning performs feature extraction (Feature Extraction) on the input pathological images, that is, it identifies the types of cells and tissues and retains the information in the output pre-pooled features. In the picture, unimportant features will be discarded in the process. The global pooling layer GP integrates the features captured at different locations on the picture, that is, it retains whether each feature appears anywhere on this slide. , discarding the location information of this feature; and the final fully connected layer integrates the extracted features to obtain the final prediction result. On the basis of the above, the present invention further strengthens the deep learning model's ability to visualize cancer lesions.

Referring to Figures 1 and 2 together, the visualization method for cancer lesions includes at least the following steps:

Step S100: Obtain the slide-level image IMG to be visualized. Among them, the slide-level image to be visualized is judged to contain cancer cells or is suspected to contain cancer cells through the above-mentioned trained deep learning model.

Step S101: Input the slide-level image IMG to be visualized into the trained deep learning model.

Step S102: Perform feature extraction on the input slide-level image to be visualized through a feature extraction network to generate a pre-pool feature map (pre-pool feature map) PPFM.

In detail, the pre-pooled feature map PPFM can to represent, as a Tensor of size, where The dimensions of this tensor, that is, and The dimensions correspond to the height and width of the slide-level image to be visualized, is the number of channels, which represents the maximum number of retrieved features.

Pre-pooled feature map PPFM can include multiple elements , where any element Used to indicate whether one of multiple features appears at a certain position (for example, coordinates h, w) in the slide-level image IMG to be visualized. while elements The larger the value, the more obvious the corresponding feature is.

Here, the functions performed by the global pooling layer GP and the fully connected layer FC in the trained deep learning model will be explained first. As mentioned above, the global pooling layer GP integrates features captured at different locations on the map. In other words, the global pooling layer GP integrates the size dimensions (i.e. H×W) in the pre-pooled feature map PPFM. Dimensionality reduction to produce global pooling vectors. The global pooling vector can be expressed by:

.

in, That is, the global pooling vector is a A vector of size, each element indicates whether a certain feature appears in the slide-level image to be visualized IMG.

On the other hand, the fully connected layer FC is used to pool the global vector A weighted sum is performed to produce an evaluation score. This evaluation score is used to indicate whether the slide-level image to be visualized contains cancer cells, which can be expressed by the following formula:

.

in, is the evaluation score and is a scalar quantity (Scalar), is the global pooling vector, is the first weight of the fully connected layer, is the second weight of the fully connected layer, and It is a learnable weight, which is determined during the training process of the deep learning model and is used to control the importance of each feature.

In the above-mentioned deep learning model, a Class Activation Map (CAM) can be further generated to represent the probability of being diagnosed as cancer on the slide-level image IMG to be visualized. Further reference may be made to FIG. 3 , which is a flow chart for generating a classification activation map according to an embodiment of the present invention.

Step S300: Convert the pre-pooled feature map PPFM to the size dimension ( ) into multiple vectors Vi to produce a vector set. The vector set can be expressed as , and each of the multiple vectors has multiple channel units corresponding to the features.

Step S301: Sum the vectors of the vector set using the first weight and the second weight of the fully connected layer FC to generate a summed scoring vector, which is expressed by the following formula:

.

in is the summed rating vector, is a set of vectors, W is the first weight, and b is the second weight.

Step S302: Splice the summed score vectors to generate a classification activation map. Among them, the classification activation map is a two-dimensional tensor, and its size is the size dimension ( ), and in the classification activation map, the value of each position represents the probability of being diagnosed as cancer on the slide-level image to be visualized IMG corresponding to the position. The present invention further utilizes the magnitude of the values at each position in the generated classification activation map to assist in marking the cancer lesion area.

As a strong classifier, the deep learning model will try its best to capture all the information in the training data. Therefore, in practice, in the work of distinguishing cancer, in addition to the direct correlation between cancer cells and cancer, the deep learning model must learn to capture the shape of cancer cells, as well as the cell necrosis (Necrosis) and desmoplasia (Desmoplasia) that often accompany cancer cells. It will also be recognized by the deep learning model and make a "suspected cancer" judgment, which will cause the CAM to mark such areas.

In fact, in a deep learning model trained with a slide-level training set, features such as cancer cells, necrosis, and connective tissue hyperplasia will be used by the model as different channels in the Pre-Pool Feature Map. (Channel) represents. That is, some channels on the pre-pooled feature map identify cancer cells, and other channels identify necrosis. However, in the above process of generating CAM, after these values are weighted and summed to produce the final prediction result, it is not possible to identify with a single number whether the high value is due to the identification of cancer cells or necrosis.

To this end, the present invention is based on the above premise to separate cancer cell features from other accompanying features, that is, by analyzing the distribution of each vector in the pre-pooled feature map, to obtain which channels are used to identify cancer cells of.

The vectors in the vector set corresponding to the cancer cell area have a very low intra-class dissimilarity, because the characteristics of cancer cells will cause the channel that captures the characteristics of cancer cells to have a high value, while other channels will have a very high value. For low values, any distance evaluation method, such as Euclidian distance and cosine similarity, can yield lower values; on the contrary, in the vector set corresponding to the cancer cell area and necrosis area There is a high inter-class dissimilarity between vectors because the two types of regions activate different channels respectively.

With such characteristics, the vectors of cancer cells and necrotic areas can be divided into different clusters by using a clustering algorithm to achieve the effect of separating cancer cells and necrotic areas.

Please refer to Figure 1 again. The method proceeds to step S103: compare the pre-pooled feature map PPFM with the size dimension ( ) into multiple vectors Vi to produce a vector set. This step is the same as step S300, so it will not be described again here.

Step S104: Divide the vector set into multiple clusters Gi according to the clustering parameters through a clustering algorithm. The clustering algorithm may, for example, adopt the k-means algorithm, and may, for example, use Euclidean distance as a criterion for evaluating distance (dissimilarity). In an embodiment of the present invention, the grouping parameter may be, for example, the number of these groups, such as k (k can be set to 5). The value of k needs to be manually adjusted. In principle, cancer cells and other types of cells can be separated as long as k is large enough. A too large k value may divide the cancer cell area into two or more groups.

Step S105: Convert the clusters into multiple cluster images and present them on the slide-level image to be visualized. As described previously, since a too large k value may divide the cancer cell area into two or more groups (inclusive), the areas within each group can be presented on the original image, and the final manual review can be used to confirm which of them Several groups of cancer cells should be marked in the area, and the groups identified as cancer cells should be mapped back to the corresponding positions in the original slide image to mark.

Step S106: Calculate the average classification activation map of the clusters based on the classification activation map. Please refer to FIG. 4 , which is a schematic diagram of a slide-level image to be visualized, an average classification activation map, and groups to be labeled according to an embodiment of the present invention.

Step S107: Based on the correspondence between the grouped images on the slide-level image IMG to be visualized and the average CAM, select at least one of the groupings corresponding to the cancer cells in the slide-level image IMG to be visualized. Grouping should be marked.

Step S108: Group at least one annotated group according to the classification activation map annotation in the image to be visualized at the slide level. As shown in Figure 4, the labeled groups can be imaged and superimposed on the slide-level image to be visualized.

Further reference may be made to FIG. 5 , which is a comparison between the visualization results of the method for visualizing cancer lesions of the present invention and the prior art. Parts (a) and (c) of Figure 5 are the existing CAM visualization results, while parts (b) and (d) of Figure 5 are the CAM visualization results of the present invention. As shown in the figure, compared with the existing CAM algorithm, the visualization method for cancer lesions of the present invention can reduce the area of false positive (False Positive) areas (as shown in parts (a) and (c) of Figure 5 The round box, that is, the area where normal tissue is incorrectly marked as having cancer), and maintains the True Positive area (that is, the cancer lesion is correctly marked).

[Beneficial effects of the embodiment]

One of the beneficial effects of the present invention is that the visualization method for cancer lesions provided by the present invention can extract the pre-pooling layer feature map from the deep learning model of cancer identification based on slide-level operations and use grouping calculations. The method generates multiple clusters and labels them on the slide-level image to be labeled, and then uses the classification activation map extracted from the deep learning model to select the clusters that should be labeled, and finally visualizes the cancer lesions.

Therefore, the visualization method for cancer lesions provided by the present invention can improve the effect of the algorithm in separating cancer cells and necrotic areas, and reduce the situation where the algorithm mistakenly identifies necrotic areas as cancer cells when visualizing cancer lesions. The algorithm for visualizing the lesion area using a model trained with slide-level image operations has been improved.

The contents disclosed above are only preferred and feasible embodiments of the present invention, and do not limit the scope of the patent application of the present invention. Therefore, all equivalent technical changes made by using the description and drawings of the present invention are included in the application of the present invention. within the scope of the patent.

IN: input layer HID: hidden layer OUT: output layer FEN: Feature Extraction Network GP: global pooling layer FC: fully connected layer IMG: slide-level image to be visualized PPFM: Pre-pooled feature map Vi: vector Gi: grouping

FIG. 1 is a flow chart of a method for visualizing cancer lesions according to an embodiment of the present invention.

FIG. 2 is a schematic flowchart of a method for visualizing cancer lesions according to an embodiment of the present invention.

FIG. 3 is a flow chart for generating a classification activation map according to an embodiment of the present invention.

FIG. 4 is a schematic diagram of a slide-level image to be visualized, an average classification activation map, and a grouped image according to an embodiment of the present invention.

Figure 5 is a comparison between the visualization results of the method for visualizing cancer lesions of the present invention and the prior art.

The representative diagram is a flow chart, so there are no symbols for simple explanation.

Claims (10)

A method for visualizing cancer lesions, which includes the following steps: obtaining a slide-level image to be visualized, wherein the slide-level image to be visualized is judged by a trained deep learning model to determine whether there are cancer cells; When cancer cells are detected, the slide-level image to be visualized is input into the trained deep learning model, wherein the trained deep learning model includes a feature extraction network, a global pooling layer and a fully connected layer; through the feature The acquisition network performs feature extraction on the input slide-level image to be visualized to generate a pre-pool feature map, where the pre-pool feature map includes multiple Elements, each element is used to indicate whether one of the multiple features appears in one of the multiple locations of the slide-level image to be visualized; the pre-pooled feature map is disassembled into multiple dimensions in one dimension. vectors to generate a vector set, wherein each of the vectors has multiple channel units corresponding to the characteristics; use a grouping algorithm to divide the vector set into multiple groups according to a grouping parameter; convert the groupings are multiple grouped images and are presented on the slide-level image to be visualized; based on the correspondence between the grouped images on the slide-level image to be visualized, filter out the groups corresponding to the slide-level images. At least one of the cancer cells in the image to be visualized should be labeled; and the at least one should be labeled in the slide-level image to be visualized based on a Class Activation Map (CAM). The method for visualizing cancer lesions as described in claim 1, wherein the cancer cells are lung adenocarcinoma cells or squamous cell carcinomas. The method for visualizing cancer lesions as described in claim 1, wherein the training The deep learning model includes an input layer, a plurality of hidden layers and an output layer, and the hidden layers include the feature extraction network, the global pooling layer and the fully connected layer, and the feature extraction network The path includes a plurality of layers selected from the group consisting of convolutional layers, pooling layers, and normalization layers. The method for visualizing cancer lesions as described in request 1, wherein the pre-pooled feature map is a tensor (Tensor) of size H × W × C , where H × W is the size dimension, and corresponds to The height and width of the tensor, C is the number of channels, and the H and W dimensions correspond to the height and width of the slide-level image to be visualized. The method for visualizing cancer lesions as described in claim 4, further comprising: using the global pooling layer to reduce the size dimension in the pre-pooling feature map to generate a global pooling vector. ; and perform a weighted summation of the global pooling vector through the fully connected layer to generate an evaluation score, where the evaluation score is used to indicate whether the slide-level image to be visualized contains cancer cells, and is expressed by the following formula: Z=W. E+b, where Z is the evaluation score and is a scalar, E is the global pooling vector, W is the first weight of the fully connected layer, and b is the second weight of the fully connected layer. The method for visualizing cancer lesions as described in claim 5, further comprising: summing the vectors of the vector set with the first weight and the second weight of the fully connected layer to generate a sum The total rating vector is expressed by the following formula: Z ' _hw =W. E ' _hw + b, where Z ' _hw is the total score vector, E ' _hw is the vector set, W is the first weight of the fully connected layer, and b is the second weight of the fully connected layer; add the The total score vectors are spliced to generate the classification activation map, where the classification activation map is a two-dimensional tensor whose size is the size dimension, and the value of each position of the CAM represents the pre-pooling corresponding to that position. On the feature map, the probability of identifying cancer cells. The method for visualizing cancer lesions as described in claim 6, further comprising: calculating a plurality of average classification activation maps of the clusters based on the classification activation map; and based on the clustering images to be visualized at the slide level The corresponding relationship on the image and the average classification activation map are used to filter out at least one of the clusters that should be labeled. The method for visualizing cancer lesions as described in claim 1, wherein the trained deep learning model is ResNet or DenseNet. The method for visualizing cancer lesions as described in claim 1, wherein the grouping algorithm is a k-means algorithm. The method for visualizing cancer lesions as described in claim 9, wherein the k-means algorithm uses Euclidean distance as a criterion for evaluating distance, and the clustering parameter is the number of clusters.