HK40051109B - Multiple instance learner for prognostic tissue pattern identification - Google Patents

Description

Multi-instance learner for prognostic tissue pattern recognition

Technical Field

This invention relates to the field of digital pathology, and more particularly to the field of image analysis.

Background Technology

Several image analysis methods are known that can be used to assist in the diagnostic process and to identify appropriate treatments based on the analysis of tissue sample images.

Some image analysis techniques are based on using different procedures to search for structures in images known to serve as indicators of the presence of a specific disease and/or the likelihood of successful treatment with a particular drug. For example, some drugs used in immunotherapy for cancer patients are effective only if certain immune cells are present at a certain distance from cancer cells. In this case, an attempt is made to automatically identify these objects in tissue images—that is, certain cell types or certain subcellular and supercellular structures—in order to make a statement about the presence of a disease and/or recommended treatment. A drawback of this approach is that the image analysis algorithm only identifies structures developed for that algorithm. Therefore, this type of image analysis relies on existing medical knowledge about the relationship between certain cell and tissue structures and certain diseases or their treatments. Consequently, this image analysis method is not suitable for detecting unknown predictive features about a disease and/or its treatment, and is limited by the medical knowledge available at a given time. This image analysis method is also not suitable for extending the knowledge of medical relationships, i.e., identifying features and tissue structures previously unknown to predict the presence of a certain form of disease and/or the effectiveness of a certain drug for that disease.

Other image analysis methods, particularly unsupervised machine learning methods, can also consider their predictive power in tissue patterns and features unknown to the professional community and/or imperceptible to pathologists in image analysis. This is because these features can be, for example, derived features resulting from the presence, absence, and/or expressibility of several other features. The drawback of these methods is that they often operate like black boxes. In other words, pathologists using these techniques must rely on the predictive power of these algorithms without being able to pinpoint exactly which tissue characteristics are ultimately decisive in the prediction. This can be a significant drawback, for example, in drug approval, where the patient population that will benefit from a particular treatment must be clearly identified for this purpose. Having to rely entirely or partially on this "black box" when deciding whether to administer a potentially effective but highly contagious drug to a patient, without being able to articulate the underlying "decision logic," is undesirable for both doctors and patients.

Summary of the Invention

The object of this invention is to provide an improved method for identifying tissue patterns that indicate patient-related attribute values, and a corresponding image analysis system as pointed out in the independent claims. Embodiments of the invention are given in the dependent claims. Embodiments of the invention may be freely combined with each other if they are not mutually exclusive.

In one aspect, the present invention relates to a method for identifying organizational patterns that indicate patient-related attribute values. The method includes:

- For each patient in a group of patients, at least one digital image of a tissue sample of that patient is received via an image analysis system, the at least one image being assigned one of at least two different predefined labels, each label indicating a patient-related attribute value of the patient whose tissue is depicted in the labeled image;

- Using an image analysis system, each received image is split into a set of image blocks, and each block has been assigned a label that was used to create the image;

- For each of the blocks, a feature vector is calculated using an image analysis system. This feature vector contains image features selectively extracted from the organizational patterns depicted in the block.

- A multi-instance learning (MIL) procedure is trained based on all blocks and corresponding feature vectors from all images received for all patients in the group. Each block set is processed by the MIL procedure into a block bundle with the same label. The training includes analyzing the feature vectors to compute a numerical value for each block, which indicates the predictive power of the feature vector associated with the block relative to the label of the image assigned to the derived block; and

- The image analysis system outputs an image patch report library via its GUI. This report library includes a subset of patches (which are sorted according to their respective calculated values) and/or a graphical representation of their respective values.

This approach can be advantageous because it combines the strengths of image analysis methods based on explicit biomedical expert knowledge with the strengths of machine learning methods: in machine learning, multiple instance learning (MIL) is a form of supervised learning. Instead of receiving individual sets of labeled instances, the learner receives sets of labeled packets, each containing many instances. In the simple case of multi-instance binary classification, a packet is labeled negative if all instances in it are negative. On the other hand, a packet is labeled positive if at least one instance is positive. From a set of labeled packets, the learner attempts to (i) introduce the concept that will correctly label individual instances, or (ii) learn how to label packets without introducing the concept. A convenient and simple example of MIL is given in Babenko and Boris, "Multiple instance learning: algorithms and applications" (2008). However, MIL procedures according to some embodiments also cover training based on more than two different labels (endpoints).

According to embodiments of the invention, the MIL program is used to calculate predicted values for each instance (block) in the package (preferably, all blocks of one or more images of a tissue slice from a patient with a specific label value) and is therefore also applicable to tissue patterns depicted separately in the blocks. In this step, the MIL program can identify new biomedical knowledge because, in the training data, the labels of the images and corresponding blocks serve as the endpoints of training, but not as individual features of the feature vectors derived from the blocks; the blocks are strongly (positively or negatively) correlated with the labels, and therefore the labels can be predicted. Furthermore, the predicted values calculated for individual blocks are also output along with graphical representations of related blocks in the library. For example, the blocks in the library may be sorted according to numerical values. In this case, the position of a block in the library allows pathologists or other human users to identify tissue patterns depicted in blocks found to be highly predictive of a specific label. Additionally, or alternatively, numerical values may be displayed as spatially proximate to their corresponding blocks, thereby enabling users to examine and understand tissue patterns depicted in one or more blocks that have numerical values similar to a specific label.

Therefore, the image patch library generated as output of the training phase can reveal predictive tissue signatures relative to specific patient-related attribute values. Combining the numerical representation of image patches may have the following benefits: at least in many cases, pathologists can identify and characterize predictive tissue patterns (also known as "tissue signatures") by comparing several patches in the library with those having much higher or lower values, and by comparing the tissue signatures depicted in these subsets of patches in the report library.

In a further advantageous aspect, the use of the MIL procedure, which treats image patches as instances and the entirety of all patches across all images of the same patient with specific labels assigned (e.g., "Response to drug D = True", "Microsatellite status = MSX", "HER2 expression status = +"), is particularly suitable for predicting patient-related features in the case of whole slide tissue sample images. This is because whole slide tissue samples typically cover many different tissue regions, only some of which may have any predictive value. For example, a micrometastasis may only be a few millimeters in diameter, but the slide and the corresponding whole slide image can be many centimeters long. Even though the entire image is labeled—based on empirical observation of the patient from whom the sample originated—using specific labels such as "Response to drug D = True," the tissue region surrounding a micrometastasis that includes many immune cells and predicts a positive response may only cover a few millimeters. Therefore, most patches do not include any tissue regions that are predictable relative to the image manner and the usual patient manner of labeling. The MIL procedure is particularly suitable for identifying predictive features based on data instance packages, where it is assumed that most instances have no predictive value.

According to an embodiment, the received digital image includes a digital image of a tissue sample, the pixel intensity value of which is related to the amount of a non-biomarker-specific staining agent, particularly H&E staining agent.

For example, each package block may represent a corresponding patient for whom a response to a specific drug is known. The instances contained in this patient-specific package are blocks of one or more images derived from a corresponding tissue sample from that particular patient, stained with a non-biomarker-specific staining agent (such as H&E). All tissue images of that patient, and all blocks derived therefrom, are assigned the label “Patient responds to drug D = True”.

This can be advantageous because H&E-stained tissue images represent the most common form of stained tissue images, and this type of staining alone has revealed a wealth of data that can be used to predict patient-related attribute values, such as the subtype or stage of a specific tumor. Furthermore, many hospitals include large databases of H&E-stained tissue images derived from patients treated over many years. Typically, hospitals also possess data on whether a particular patient responded to a particular treatment and/or the rate or severity of disease progression. Therefore, a large corpus of training images labeled with corresponding outcomes (e.g., whether treatment with a particular drug was successful, progression-free survival exceeding one year, progression-free survival exceeding two years, etc.) can be obtained.

According to an embodiment, the received digital image includes a digital image of a tissue sample, the pixel intensity value of which is related to the amount of biomarker-specific staining agent. The biomarker-specific staining agent is a staining agent suitable for selectively staining biomarkers contained in the tissue sample. For example, the biomarker may be a specific protein, such as HER-2, p53, CD3, CD8, etc. The biomarker-specific staining agent may be a bright-field microscopy or fluorescence microscopy staining agent conjugated to an antibody that selectively binds to the aforementioned biomarker.

For example, each package block may represent a corresponding patient whose response to a specific drug is known. The instances contained in this patient-specific package are blocks of one or more images derived from a corresponding tissue sample from that particular patient. One or more tissue samples have been stained with one or more biomarker-specific staining agents. For example, a block may be derived from one, two, or three tissue images, all depicting adjacent tissue slides from the same patient stained with a HER2-specific staining agent. According to another example, a block may be derived from a first tissue image depicting a first tissue sample stained with a HER2-specific staining agent, a second tissue image depicting a second tissue sample stained with a p53-specific staining agent, and a third tissue image depicting a third tissue sample stained with a FAP-specific staining agent. The first, second, and third tissue samples are derived from the same patient. For example, they may be sections of adjacent tissue samples. Although the three tissue images depict three different biomarkers, all tissue images are derived from the same patient, and therefore all blocks derived therefrom are assigned the label "Patient responds to drug D = True".

Training a MIL program using image patches from digital images that correlate pixel intensity values with the amount of biomarker-specific staining agents may have the following advantages: identifying the presence and location of one or more specific biomarkers in tissue can reveal highly specific and prognostic information about specific diseases and disease subtypes. Prognostic information may include positive and negative correlations between the presence of two or more observed biomarkers. For example, recommended treatments and prognoses for certain diseases (such as lung or colon cancer) have been observed to depend heavily on the mutation signature and expression profile of the cancer. Sometimes, the expression of a single biomarker alone is not predictive, but the combined expression of multiple biomarkers and/or the absence of a specific additional biomarker may have high predictive power relative to patient-specific relevant attribute values.

According to an embodiment, the received digital images include a combination of digital images of tissue samples whose pixel intensity values are related to the amount of a first biomarker-specific staining agent and a combination of digital images of tissue samples whose pixel intensity values are related to the amount of a non-biomarker-specific staining agent. The biomarker-specific staining agent is a staining agent suitable for selectively staining biomarkers contained in a tissue sample. All digital images depicting the same tissue sample and/or adjacent tissue samples from the same patient have been assigned the same label. The MIL is configured to process all blocks derived from the digital images as members of the same block package.

The advantage of this method lies in its ability to identify the presence and location of one or more specific biomarkers in tissue by combining information-rich tissue signatures revealed by H&E staining, providing highly specific and prognostic information about specific diseases and disease subtypes. Prognostic information may include positive and negative correlations in the presence of two or more observed biomarkers and/or tissue signatures visually revealed by H&E staining.

According to an embodiment, the image patches displayed in the image patch report library are derived from one or more different images among the received images. The method includes, for each of the one or more images depicted in the report patch library:

- Identify a block in a report library, said block having been derived from the image and assigned the highest score among all blocks derived from the image; according to one embodiment, the score is a numerical value calculated by the MIL for each block; according to an alternative embodiment, the score is a weight calculated by the attention-MLL for each block as described herein with respect to embodiments of the invention; according to a further embodiment, the score is a combination of the numerical value calculated by the MIL and the weight calculated by the attention-MLL for the block, whereby the combination may be, for example, a multiplication of the numerical value and the weight;

- For each of the other blocks in the image, a correlation metric is calculated by comparing the score of the other block with the score of the block with the highest score; the correlation metric is a numerical value that is negatively correlated with the difference between the score of the other block and the score of the block with the highest score.

- The correlation heatmap of the image is calculated as a function of the correlation index; therefore, the pixel color and/or pixel intensity of the correlation heatmap indicate the correlation index calculated for blocks in the image; and

- Display the correlation heatmap. For example, the correlation heatmap can be displayed as an image of the entire slide that is spatially close to the calculation of the correlation heatmap in the report block library.

For example, image regions and corresponding blocks with scores highly similar to the highest-scoring block of an image can be indicated in the correlation heatmap with a first color (e.g., "red") or a high-intensity value, and image regions and corresponding blocks whose scores differ from the highest-scoring block of the image can be represented in the correlation heatmap with a second color (e.g., "blue") or a low-intensity value, different from the first color.

This can be advantageous because the GUI automatically calculates and presents a correlation heatmap, which indicates the location and coverage of tissue regions with high predictive power (or "prognostic value") and their corresponding image patches. The correlation heatmap highlights tissue regions with high correlation indices. Patches are often just small sub-regions of the entire slide image, and such reportable patch libraries may not provide an overview of the entire tissue sample. An overview of the location and coverage of tissue patterns with high predictive correlation can be provided by a correlation heatmap, preferably combined with the original image of the entire slide tissue image in a highly intuitive and intelligent manner.

Numerical computation of relevance heatmaps based on MIL may have the following advantages: it may not require the implementation and training of attention MIL. Therefore, the system architecture can be more easily implemented.

The relevance heatmap calculated based on the weights of attention MIL may have the following advantages: in addition to the numerical value of MIL, a second numerical measure for block prognostic relevance is evaluated and represented in the relevance heatmap.

Calculating a relevance heatmap based on a combined relevance score derived from numerical values calculated by MIL and weights calculated by Attention MLL for a specific block may have the following advantages: the two independently calculated block-predicted relevance numerical measures are integrated into and represented by the combined value and the relevance heatmap based on the combined value. This can increase the accuracy of relevance tissue slice identification.

According to an embodiment, the GUI allows users to select whether the relevance heatmap is calculated based on the numerical value of the MIL, the weights of the attention MIL, or a combined score. This allows users to identify whether the outputs of the MIL and the attention MIL differ significantly regarding the predictive power of a block.

Calculating and displaying a correlation heatmap can be advantageous because it indicates the predictive power of blocks with respect to the endpoints used to train the MIL and/or attention MLL. Therefore, displaying a correlation heatmap to the user enables them to quickly identify the location and extent of blocks with tissue patterns that predict specific labels throughout the entire slide image.

According to an embodiment, the image patches displayed in the report library are selectable. The GUI is configured to calculate and display a similarity search patch library, the calculation including:

- Receive the user's selection of a specific block from the image blocks in the report library;

- Identify all blocks from all received images that depict organizational patterns similar to the selected blocks, where the similarity between the feature vectors of the selected blocks and the feature vectors of the selected blocks exceeds a threshold, by identifying all blocks from all received images that have been assigned feature vectors; and

- Display a similarity search library, which selectively includes the identified blocks.

According to an embodiment, the calculation and display of the similarity search block library further includes:

- Determine the number and/or score of blocks within the block, the blocks depicting an organizational pattern similar to the selected block, which has been assigned the same labels as the selected block; and

- Display the specified quantity and/or score in the similarity search library.

These features can be advantageous because human users can quickly determine how common a particular tissue pattern is in the examined patient group and in a subset of patients with specific labels. Therefore, human users can quickly and intuitively verify whether a particular block and the tissue pattern depicted within it truly possesses high predictive power.

For example, a user can select a block from a report library that has been assigned the highest numerical value and therefore the highest predictive power regarding image labels. After selecting a block, the user can initiate a block-based similarity search across blocks and images from many different patients, who may have been assigned labels different from the currently selected block. The similarity search is based on the comparison of feature vectors and blocks to determine similar blocks and similar tissue patterns based on similar feature vectors. This is achieved by evaluating and displaying the number and/or score of blocks (and corresponding tissue patterns) that are similar to the selected block (and its tissue pattern) but have labels different from those of the selected block (e.g., "Patient responded to drug D = false" instead of "Patient responded to drug D = true").

Therefore, pathologists can easily examine the predictive power, particularly sensitivity and specificity, of tissue patterns identified by the MIL program by selecting blocks returned by the MIL program as "high prognostic," to perform a similarity search, revealing how many blocks in the dataset with similar feature vectors have been assigned the same label as the selected block. This is a significant advantage compared to state-of-the-art machine learning applications, which also provide indications of prognostic features in tissue images, but do not allow users to identify and verify these features. Based on a report library and a similarity search library, human users can verify the proposed high-prognostic tissue patterns and can also verbally describe the common features and structures displayed and associated with similar feature vectors in all blocks with high predictive power.

The blocks in the report block library are optional, and the selection triggers a similarity search to identify and display features of other blocks with similar feature vectors/organizational patterns to the user-selected block. This allows the user to freely select any image from the report block library that interests them. For example, a pathologist might be interested in an organization pattern and corresponding block with the highest predictive power (the highest value calculated by the MIL), as described above. Alternatively, a pathologist might be interested in artifacts that typically have particularly low predictive power (particularly low values). Furthermore, a pathologist might be interested in a particular organization pattern for any other reason, such as because it reveals some side effects of a drug or any other relevant biomedical information. The pathologist is free to select any block from the corresponding report block library. Thus, the pathologist triggers a similarity search, and the results are calculated and displayed in the form of a similarity block library. The display and GUI can be automatically refreshed after the similarity search is completed.

According to some embodiments, the calculation and display of the similarity search library includes the calculation and display of a similarity heatmap. The heatmap encodes similar blocks and their corresponding feature vectors using color and/or pixel intensity. Image regions and blocks with similar feature vectors are represented in the heatmap by similar colors and/or high or low pixel intensities. Therefore, users can quickly obtain an overview of the distribution of a specific tissue pattern signature across the entire slide image. The heatmap can be easily refreshed simply by selecting a different block, as this selection automatically induces a recalculation of feature vector similarity based on the feature vector of the newly selected block.

According to an embodiment, the similarity search library includes a similarity heatmap. The method includes creating the similarity heatmap via a sub-method, which includes:

- Select a block from the report library;

- For each of the other blocks in some or all of the received images, a similarity score is calculated about the selected block by comparing the feature vectors of other blocks derived from the same image and other images with the feature vector of the selected block;

- For each block in the image used to calculate the corresponding similarity score, calculate a corresponding similarity heatmap as a function of the similarity score, whereby the pixel color and/or pixel intensity of the similarity heatmap indicates the similarity between the block in the image and the selected block; and

- Displays a similarity heatmap.

According to an embodiment, the image patches displayed in the similarity search library are also selectable.

Similarity heatmaps provide valuable overview information, allowing human users to easily perceive the prevalence of a specific tissue pattern in a particular tissue or in tissue samples from a patient subgroup with a specific label. Users can freely select any block from the search library, thereby inducing a recalculation of the similarity heatmap based on the feature vector assigned to the currently selected block, and automatically refreshing the GUI containing the similarity heatmap.

According to one embodiment, image blocks in the report library and/or similarity search block library are grouped based on blocks derived from patient tissue sample images. According to an alternative embodiment, image blocks in the report library and/or similarity search block library are grouped based on tags assigned to images from which blocks are derived.

Typically, all images derived from the same patient will have the same label, and all blocks from those images of a specific patient will be processed by MIL as members of the same "package". However, in certain special cases, different images from the same patient may be assigned different labels. For example, if the first image depicts the patient's first metastasis, and the second image depicts the same patient's second metastasis, and the observation is that the first metastasis responds to treatment with drug D and disappears, while the second metastasis continues to grow, then patient-related attribute values may be assigned image-wise rather than patient-wise. In this case, each patient may have multiple package blocks.

According to another example, images of patient tissue samples taken before and after treatment with a specific drug, and the endpoints (labels) used for training the MIL and/or applying the trained MIL, are the attribute value "tissue state = after treatment with drug D" or the attribute value "tissue state = before treatment with drug D". Training the MIL based on these patient-related attribute values has the advantage of identifying tissue patterns that indicate the drug's activity and morphological effects on the tumor. Such identified tissue patterns related to drug effects can validate and explore the drug's mode of action and potential drug side effects.

According to an embodiment, the method further includes: computationally increasing the number of block packets by creating additional block sets, each additional block set being processed by a MIL program as an additional block packet, the block packet being assigned the same label as the tissue image from which the source blocks were generated. The creation of the additional block sets specifically includes: applying one or more artifact generation algorithms to at least a subset of the blocks to create new blocks including artifacts. Furthermore, or alternatively, the creation of additional block packets may include increasing or decreasing the resolution of at least a subset of the blocks to create new blocks with finer or coarser granularity than their corresponding source blocks.

For example, a subset can be obtained for each patient by randomly selecting some or all blocks from one or more tissue images obtained from the patient. The artifact generation algorithm simulates image artifacts. Image artifacts can be, for example, types of artifacts generated during tissue preparation, staining, and/or image acquisition (e.g., edge artifacts, over-staining, under-staining, dust, speckle artifacts (simulated by Gaussian blur, etc.)). Furthermore, or alternatively, artifacts can be general noise types (e.g., simulated by occlusion, color jitter, Gaussian noise, salt-and-pepper noise, rotation, flipping, skew distortion, etc.).

Creating additional block packs can have the advantage of generating additional training data from a limited available training dataset. This additional training data represents image data, the quality of which may be degraded by common distortions, artifacts, and noise that frequently occur during sample preparation and image acquisition. Therefore, an expanded training dataset can ensure that overfitting of the MIL procedure's underlying model is avoided during training.

According to an embodiment, the method further includes calculating a cluster of blocks obtained from one or more received digital images, wherein blocks are grouped into clusters based on the similarity of their feature vectors. Preferably, clusters are calculated for each patient. This means that if the feature vectors of the blocks are sufficiently similar, blocks from different images of different tissue slides depicting the same patient can be grouped into the same cluster.

According to other embodiments, clusters are calculated together for all blocks originating from all patients.

In both of these methods of clustering blocks (all blocks from different patients together or all blocks from each patient), blocks that appear similar to each other (i.e. have similar feature vectors) are clustered into the same cluster.

For example, in the case of "all blocks from different patients," the result of aggregation might be to generate, for example, 64 groups (clusters) of blocks for all patients. Each of these 64 clusters contains similar blocks derived from different patients. Conversely, in the case of per-patient aggregation, each patient would have its own 64 clusters.

If clusters are created for each patient, it's possible that the patient images contain no fat patches or very few fat patches. In this case, "fat clusters" might not be created because there isn't enough data to learn clusters around the "fat" feature vector. However, performing clustering on all patches from all patients together has the following advantages: more clusters/tissue types can be identified using the largest amount of available data: in "all patient patches" clusters, clusters of "fat" tissue patterns are likely to be identified because at least some patients' biopsies contain some fat cells. Therefore, there is a sufficient probability that the number of fat cells depicting patches in the dataset will create fat cell clusters (also applicable to patients with very few fat cells). If clusters are created on all patches from all patients together, and one cluster represents fat cells, then all patches containing fat cells from all patients will be grouped into that cluster. This means that for a specific patient/package, all patches with fat cells will be grouped into said cluster, and if cluster sampling is used for a package, a certain number of patches belonging to said cluster (from the current patient/package) are selected.

Clustering blocks can be advantageous because this operation can reveal the number and/or type of tissue patterns observable in a particular patient. According to some embodiments, the GUI includes user-selectable elements that allow the user to trigger the clustering of blocks and the presentation of block clusters within a cluster library view. This can help the user intuitively and quickly understand the important types of tissue patterns observed in a specific tissue sample from a patient.

According to an embodiment, training the MIL program includes repeatedly sampling the block set to select a subset of blocks from the block set, and training the MIL program based on the subset of blocks.

As used herein, the term "sampling" is a technique employed in data analysis or training machine learning algorithms that involves selecting a specific number of L samples from a plurality of N data items (instances, blocks) in a dataset (a population of blocks obtained from one or more images of a patient). According to an embodiment, "sampling" involves selecting a subset of data items from the number of N data items based on a probability distribution that is assumed to statistically represent the population of N blocks in the training dataset. This allows for a more accurate understanding of the characteristics of the entire population. The probability distribution represents the statistical assumption that guides the machine learning process and makes "learning from data" feasible.

According to some embodiments, sampling is performed by randomly selecting a subset of blocks to provide a block packet for sampling.

According to an embodiment, the aggregation and sampling are combined as follows: sampling includes selecting each block from a cluster of blocks obtained for the patient, such that the number of blocks in each subset of the blocks created in the sampling corresponds to the size of the cluster taken from said blocks.

For example, 1,000 blocks can be created from digital tissue images of a specific patient. Aggregation creates a first cluster showing a background tissue slide area comprising 300 blocks, a second cluster showing a stromal tissue area comprising 400 blocks, a third cluster showing metastatic tumor tissue comprising 200 blocks, a fourth cluster showing specific staining artifacts comprising 40 blocks, and a fifth cluster showing tissue with microvessels comprising 60 blocks.

According to one embodiment, sampling includes selecting a specific portion of blocks from each of the clusters, such as 50%. This would mean 150 blocks from cluster 1, 200 blocks from cluster 2, 100 blocks from cluster 3, 20 blocks from cluster 4, and 30 blocks from cluster 5.

According to a preferred embodiment, sampling involves selecting an equal number of blocks from each cluster. This sampling method has the advantage of drawing the same number of block/organization pattern examples from different types of clusters, thus making the training dataset more balanced. This can increase the accuracy of the trained MIL and/or the trained attention MIL if the desired predictive features are rare in the training dataset.

The combination of clustering and sampling can be particularly advantageous because sampling can increase the data base for training without inadvertently "losing" a few blocks that actually have high predictive power. Typically, in the context of digital pathology, the vast majority of a tissue sample does not include tissue areas modified or prognostic by specific disease or other patient-related attributes. For example, only a small subregion of a tissue sample may actually contain tumor cells, while the rest may show normal tissue. By first performing block clustering and then selecting blocks from each cluster, it is ensured that at least some blocks show prognostic tissue patterns, for example, ensuring that tumor cells or microvessels are always part of the sample.

Feature extraction methods

According to an embodiment, the calculation of the feature vector for each of the blocks includes receiving patient-related data of the patient in the block that depicts a tissue sample therein, and representing the patient-related data in the form of one or more features in the feature vector, the patient-related data being specifically selected from the group consisting of: genomic data, RNA sequence data, known diseases of the patient, age, sex, concentration of metabolites in body fluids, health parameters, and current medications.

According to an embodiment, the computation of the feature vector is performed by trained machine learning logic, particularly by a trained fully convolutional neural network including at least one bottleneck layer.

According to an embodiment, the trained machine learning logic for feature extraction (“Feature Extraction MLL”) is trained in a supervised manner using an MLL of the type including a bottleneck, such as UNET. The “UNET” architecture is described by Olaf Ronneberger, Philipp Fischer, and Thomas Brox in “U-Net: A Convolutional Network for Biomedical Image Segmentation,” Department of Computer Science and BIOSS Centre for Biosignals, University of Freiburg, Germany (arXiv:1505.04597v1, May 18, 2015). This document is available for download from Cornell University Library at https://arxiv.org/abs/1505.04597.

For example, a feature extraction MLL can be trained to perform tissue image segmentation tasks, whereby the fragments to be identified include two or more of the following tissue image fragment types: tumor tissue, healthy tissue, necrotic tissue, tissue containing specific objects such as tumor cells, blood vessels, stroma, lymphocytes, etc., and background regions. According to some embodiments, the feature extraction MLL is trained in a supervised manner using a classification network (such as ResNet, ImageNet, or SegNet) to classify image patches with specific predetermined categories or objects.

After training, the feature extraction MLL is split into an "encoder" part (including an input layer, one or more intermediate layers, and a bottleneck layer) and a "decoder," i.e., the output generation part. According to embodiments of the invention, the "encoder" part is used to reach the bottleneck layer of the trained MLL to extract and compute the feature vector for each input block. The bottleneck layer is a layer of the neural network that contains significantly fewer neurons than the input layer. For example, the bottleneck layer may be a layer containing fewer than 60% or even less than 20% of the "neurons" of the input layer. Depending on the network architecture, the number and proportion of neurons in different layers can vary considerably. The bottleneck layer is a hidden layer.

As an example, the feature extraction MLL network has a UNET-based network architecture. It has an input layer with 512*512*3 (512x512 RGB) neurons and a bottleneck layer with 9*9*128 neurons. Therefore, the number of neurons in the bottleneck layer is approximately 1.5% of the number of neurons in the input layer.

As an example, the network for feature extraction MLL has a ResNet architecture, which implements supervised or unsupervised learning algorithms. The input layer contains 512x512x3 neurons, and the bottleneck layer and the corresponding feature vectors output by the bottleneck layer typically contain 1024 or 2048 elements (neurons/digits).

According to the embodiments, feature extraction is performed by a feature extraction procedure module based on the ResNet-50 architecture (He et al., 2016) trained on the ImageNet Natural Image dataset. Pierre Courtiol, Eric W. describes some detailed examples of feature extraction from images based on this architecture. Tramel, Marc Sanselme, and Gilles Wainrib, “Histopathological Classification and Disease Localization Using Global Labels Only: A Weakly Supervised Approach,” arXiv:1802.02212, submitted February 1, 2018, is available online at Cornell University Library at https://arxiv.org/pdf/1802.02212.pdf.

According to an embodiment, the output generated by one layer of a trained feature extraction MLL for a specific block is used as the feature vector extracted from the block by the MIL procedure. This layer may be, in particular, a bottleneck layer. According to an embodiment, the feature extraction MLL is trained in an unsupervised or self-supervised manner, as described in Mathilde Caron and Piotr Bojanowski and Armand Joulin and Matthijs Douze, “Deep Aggregation for Unsupervised Learning of Visual Features,” available electronically at https://arxiv.org/abs/1807.05520, CoRR, 1807.05520, 2018.

Alternatively, MLL can be extracted from features trained by Spyros Gidaris, Praveer Singh, and Nikos Komodakis: “Unsupervised Representation Learning by Predicting Image Rotation,” February 15, 2018. The electronic version of the ICLR 2018 conference is available at https://openreview.net/forum?id=S1v4N2l0-.

Alternatively, MLLs can be extracted from trainable features, according to Elad Hoffer and Nir Ailon. “Semi-supervised deep learning through metric embeddings,” ICLR 2017, November 4, 2016. The electronic version is available at https://openreview.net/forum?id=r1R5Z19le.

The dataset used to train the Feature Extraction MLL can be another dataset of tissue images and/or a set of tissue images later used to train the MIL procedure. During the training phase, the Feature Extraction MLL does not evaluate or otherwise use any labels associated with the training images because the Feature Extraction MLL is trained to identify tissue types and corresponding image fragments, rather than to identify patient-related attribute values (which serve as the endpoint of the MIL procedure's learning phase).

Feature extraction method based on proximity similarity labels

According to an embodiment, the feature vector is computed by a feature extraction machine learning logic (“Feature Extraction MLL”), which has been trained on a training dataset including labeled block pairs, whereby each label represents the similarity of two organizational patterns depicted by the block pair and is computed as a function of the spatial distance between the two blocks in the block pair.

According to a preferred embodiment, each tag represents the similarity of two organizational patterns depicted by the block pair, and is calculated as a function of the spatial distance between the two blocks in the block pair, thereby using spatial distance as the sole measure of the similarity between the two blocks.

According to a preferred embodiment, labels are automatically assigned to block pairs in the training dataset.

This approach may be beneficial for several reasons: the spatial proximity of two image regions is a consistently and inherently available feature in every digital image of a tissue sample. The problem is that the spatial proximity of the image and the corresponding tissue region itself typically does not reveal any relevant information relative to biomedical problems, such as tissue type classification, disease classification, prediction of the persistence of a specific disease, or image segmentation tasks. The applicant surprisingly observed that the spatial proximity of two image regions (“blocks”) conveys an accurate indicator of the similarity between the two image regions, at least whether a large number of blocks and their corresponding distances were analyzed during the training phase of the MLL. Therefore, by leveraging the inherently available information of the “spatial proximity” of two blocks to automatically assign tissue pattern similarity labels to two compared blocks, a large annotated dataset that can be automatically provided for training an MLL can be provided. The trained MLL can be used to automatically determine whether two images or image blocks received as input depict similar or dissimilar tissue patterns. However, this dataset can also be used for other and more complex tasks, such as image similarity search, image segmentation, tissue type detection, and tissue pattern aggregation. Therefore, the applicant surprisingly observed that the information conveyed by the spatial proximity of blocks can be used to automatically create annotated training data, allowing the training of MLLs that reliably determine image similarity, and further, the training of MLLs that output feature vectors, which can be used by additional data processing units for multiple complex image analysis tasks in digital pathology. None of these methods require manual annotation of training data by domain experts.

When training images containing many different tissue patterns (e.g., "non-tumor" and "tumor") are split into many different blocks, the smaller the distance between two blocks, the higher the probability that the two compared blocks depict the same tissue pattern, e.g., "non-tumor". However, there are some block pairs (e.g., the first block "tumor", the other "non-tumor") next to the boundary between two different patterns depicting different tissue patterns. These block pairs generate noise because they depict different tissue patterns, even though they are very close to each other spatially. The applicant surprisingly observed that the noise generated by block pairs crossing the boundary between different tissue patterns, combined with the simplifying assumption (spatial proximity indicates the similarity of the depicted tissue patterns), does not significantly reduce the accuracy of the trained MLL. In fact, the applicant observed that the accuracy of the MLL trained according to embodiments of the present invention can outperform existing benchmark methods.

A further advantage is that training data can now be created rapidly and fully automatically for many different image sets. Currently, there is a lack of available annotated datasets to capture the natural and real-world variability in histopathological images. For example, even existing large datasets (such as Camelyon) contain only one staining (hematoxylin and eosin) and one type of cancer (breast cancer). Histopathological image textures and object shapes can vary considerably across images from different cancer types, different tissue staining methods, and different tissue types. Furthermore, histopathological images contain many different textures and object types (e.g., matrix, tumor-infiltrating lymphocytes, blood vessels, fat, healthy tissue, necrosis, etc.) with different domain-specific meanings. Therefore, embodiments of the present invention can allow the automatic creation of annotated datasets for each of many different cancer types, cancer subtypes, staining methods, and patient groups (e.g., treated/untreated, male/female, older/younger than a threshold age, biomarker positive/biomarker negative, etc.). Thus, embodiments of the present invention can allow the automatic creation of annotated training data and the training of corresponding MLLs based on this training data, such that the trained MLLs are adapted to accurately address biomedical problems in many different patient groups in a highly specific manner. In contrast to existing techniques that train MLLs on manually annotated breast cancer datasets to provide suboptimal results for colon cancer patients, embodiments of the present invention allow for the creation of MLLs separately for each of different patient groups.

According to embodiments, the labels indicating the degree of similarity between two organizational patterns are binary data values, meaning they can have a value that is one of two possible options. For example, the label could be "1" or "Similar," indicating that the two blocks depict similar organizational patterns. Alternatively, the label could be "0" or "Dissimilar," indicating that the two blocks depict different organizational patterns. According to other embodiments, the labels can be more granular, for example, they could be data values selected from a finite set of three or more data values, such as "Dissimilar," "Similar," and "Highly Similar." According to still other embodiments, the labels can be more granular and can be numerical, where the magnitude of the numerical value is positively correlated with the degree of similarity. For example, the numerical value can be computed as a function of the linear and inverse transformation of the spatial distance between a pair of blocks into a numerical value representing the similarity of organizational patterns. The larger the spatial distance, the smaller the numerical value indicating the similarity of organizational patterns. Various MLL architectures exist that can handle and utilize different types of labels (e.g., ordinal or numerical) from the training dataset. The type of MLL is chosen to handle automatically generated labels from the training dataset.

According to an embodiment, an MLL trained on an automatically annotated training dataset and adapted for feature extraction is trained using a supervised learning algorithm. Supervised learning is about finding a mapping that transforms a set of input features into one or more output data values. These output data values are provided as labels during training, such as binary option labels "similar" or "dissimilar," or as numerical values that quantify similarity. In other words, during training, the data values to be predicted are explicitly provided to the MLL model in the form of training data labels. A challenge with supervised learning is the need to label the training data to define the output space for each sample.

According to an embodiment, at least some or all of the block pairs respectively depict two tissue regions contained in the same tissue slice. Each of the tissue slices is depicted in a corresponding one in the received digital image. The distance between the blocks is calculated in a 2D coordinate system defined by the x and y dimensions of the received digital image from which the blocks of the pair are derived.

According to an embodiment, block pairs are generated by randomly selecting block pairs within each of a plurality of different images. This random selection ensures that the spatial distance between blocks in each pair will be different. Similarity labels, for example, are calculated in a numerical form inversely proportional to the distance between the two blocks and assigned to each pair.

According to other embodiments, block pairs are generated by selecting at least some or all of the blocks in each received image as starting blocks; for each starting block, all or a predefined number of "nearby blocks" are selected, where "nearby blocks" are blocks within a first circle centered on the starting block, where the radius of the circle is the same as a first spatial proximity threshold; for each starting block, all or a predefined number of "distant blocks" are selected, where "distant blocks" are blocks outside a second circle centered on the starting block, where the radius of the circle is the same as a second spatial proximity threshold; the predefined number of selections can be performed by randomly selecting this number of blocks within the corresponding image region. The first proximity threshold and the second proximity threshold can be the same, but preferably, the second proximity threshold is greater than the first proximity threshold. For example, the first proximity threshold can be 1 mm and the second proximity threshold can be 10 mm. Then, a first set of block pairs is selected, whereby each block pair includes a starting block and a nearby block located within the first circle. Each block pair in the first set is assigned a label for a "similar" organization pattern. Furthermore, a second set of block pairs is selected, whereby each pair in the set includes a starting block and one of the "distant blocks". Each block pair in the second set is assigned a label for a "dissimilar" organization pattern. For example, this embodiment can be used to create “binary” tags that are “similar” or “different”.

According to embodiments, the distance between blocks is measured within a 2D coordinate system defined by the x-axis and y-axis of a digital image from which the blocks are derived. These embodiments can be used in situations where multiple tissue sample images are available, depicting tissue samples from different patients and/or different regions within the same patient, where these different regions are far apart from each other or where the precise location of the two regions relative to each other is unknown. In this case, the spatial proximity between blocks is measured only within a 2D pixel plane defined by the digital image. Based on the known resolution factor of the image acquisition device (e.g., a camera for a microscope or a slide scanner), the distance between blocks in the original image can be used to calculate the distance between tissue regions in the tissue samples depicted by the two blocks.

According to an embodiment, at least some or all of the block pairs depict two tissue regions contained in two different tissue slices from a set of adjacent tissue slices. Each of the tissue slices is depicted in a corresponding one in the received digital image. The received image (which depicts tissue slices from a set of adjacent tissue slices) is aligned with each other in a 3D coordinate system. The distance between the blocks is calculated in the 3D coordinate system.

For example, some or all of the received digital images can depict tissue samples of slices within a tissue block of adjacent tissue slices. In this case, the digital images can be aligned with each other in a common 3D coordinate system such that the position of the digital images in the 3D coordinate system reproduces the position of the tissue slices depicted separately within the tissue block. This allows for the determination of block distances in the 3D coordinate system. The selection of “nearby” and “far” blocks can be performed as described above for the 2D coordinate system case, the only difference being that at least some of the blocks in the block pairs are derived from different images in the received images.

According to some embodiments, the annotated training data includes block pairs derived from the same digital image as well as block pairs derived from different images that have been aligned with each other in a common 3D coordinate system. This can be beneficial because, when only a small number of images of the corresponding tissue samples are available, considering the third dimension (representing the spatial proximity of blocks of tissue regions in different tissue samples) can greatly increase the number of blocks in the training data, where tissue samples belong to the same cell block, such as a 3D biopsy cell block.

According to an embodiment, each block depicts a tissue or background area having a maximum edge length of less than 0.5 mm, preferably less than 0.3 mm.

Smaller patch sizes offer several advantages: a reduction in the number and area fraction of patches describing mixtures of different tissue patterns. This helps reduce noise generated by patches depicting two or more distinct tissue patterns and by patch pairs adjacent to the “tissue pattern boundary” depicting two distinct tissue patterns. Furthermore, smaller patch sizes allow for the generation and labeling of a large number of patch pairs, thereby increasing the amount of labeled training data.

According to an embodiment, the automatic generation of block pairs includes: generating a first set of block pairs using a first spatial proximity threshold; separating two tissue regions depicted by the two blocks of each block pair in the first set by a distance less than the first spatial proximity threshold; generating a second set of block pairs using a second spatial proximity threshold; and separating two tissue regions depicted by the two blocks of each block pair in the second set by a distance greater than the second spatial proximity threshold. For example, this can be achieved by selecting multiple starting blocks, calculating a first circle and a second circle based on the first and second spatial proximity thresholds around each starting block, and selecting a starting block and either a "nearby block" (first set) or a "distant block" (second set), as described above with respect to embodiments of the invention.

According to an embodiment, the first spatial proximity threshold and the second spatial proximity threshold are the same, for example, 1 mm.

According to a preferred embodiment, the second spatial proximity threshold is at least 2 mm larger than the first spatial proximity threshold. This may be advantageous because, in the case of the organizational pattern gradually changing from one pattern to another, the difference between the organizational pattern depicted in the "distant block" and the organizational pattern depicted in the "nearby" block can be clearer and the learning effect can be improved.

According to an embodiment, the first spatial proximity threshold is a distance of less than 2 mm, preferably less than 1.5 mm, and particularly 1.0 mm.

Alternatively, the second spatial proximity threshold is a distance greater than 4 mm, preferably greater than 8 mm, and particularly 10.0 mm.

These distance thresholds refer to the distance between a depicted tissue region (or slice background region) and the corresponding block in a digital image. Based on the known magnification of the image acquisition device and the resolution of the digital image, this distance can be converted within the 2D or 3D coordinate system of the digital image.

For example, distances between blocks (and the tissue regions depicted within them) can be measured, such as between the centers of two blocks in a 2D or 3D coordinate system. According to alternative implementation variations, distances can be measured between the edges of two blocks (image region edges) that are closest to each other in a 2D or 3D coordinate system.

It has been observed that the aforementioned thresholds provide labeled training data that allows for the automatic generation of trained MLLs capable of accurately identifying similar and dissimilar tissue patterns in breast cancer patients. In some other implementation examples, the first and second spatial proximity thresholds may have other values. Particularly when using different sets of received digital images displaying different tissue types or cancer types, the first and second spatial proximity thresholds may have values other than the distance thresholds provided above.

According to an embodiment, the method further includes creating a training dataset for training a feature extraction MLL. The method includes receiving a plurality of digital training images, each depicting a tissue sample; splitting each of the received training images into multiple blocks (“feature extraction training blocks”); automatically generating block pairs, each block pair having been assigned a label indicating the degree of similarity between two tissue patterns depicted in the two blocks of the pair, wherein the degree of similarity is calculated as a function of the spatial proximity of the two blocks in the pair, where distance is positively correlated with dissimilarity; training the machine learning logic – MLL – using the labeled block pairs as training data to generate a trained MLL, the trained MLL having learned to extract feature vectors from digital tissue images, the digital tissue images representing images in such a way that similar images have similar feature vectors and dissimilar images have dissimilar feature vectors; and using the trained MLL or components thereof as a feature extraction MLL for computing the feature vectors of the blocks.

This approach can be beneficial because labels for the training dataset can be automatically created based on the inherent information contained in each digital pathology image. Annotated datasets can be created to train a feature extraction MLL, which is particularly well-suited to the current biomedical problem being addressed, simply by selecting the appropriate training images. All further steps, such as splitting, labeling, and machine learning steps, can be performed fully or semi-automatically.

According to an embodiment, the trained MLL is a Siamese neural network, comprising two neuronal subnetworks connected through their output layers. One of the subnetworks of the trained Siamese neural network is stored separately on a storage medium and used as a component of the trained MLL for computing feature vectors.

According to an embodiment, the tags processed by the MIL program are selected from the group consisting of: indications that a patient has responded to a specific drug; indications that a patient has developed metastasis or a specific form of metastasis (e.g., micrometastasis); indications that a cancer patient has shown pathological complete remission (pCR) to a specific treatment; indications that a patient has cancer with a specific morphological state or microsatellite state; indications that a patient has developed an adverse reaction to a specific drug; genetic characteristics, particularly gene signature; and/or RNA expression profile.

These labels can aid in diagnosis and finding suitable medications to treat diseases. However, the labels described above are merely examples. Other patient-related attributes can also be used as labels (i.e., endpoints for training MIL programs), as mentioned above. The term "patient-related" can also include treatment-related, since the effectiveness of a specific treatment for a disease is also relevant to the patient receiving that treatment.

Combination of MIL program and Attention MLL

According to embodiments of the invention, the MIL procedure is combined with an attention-based MLL to compute a numerical value indicating the predictive power of a particular block relative to a label assigned to an image from which the block is derived. For example, this combination can be performed during the training of the MIL procedure, as described in embodiments of the method and corresponding system depicted in FIG. 6. According to another example, this combination can be performed during the training of the MIL procedure, as described in embodiments of the method and corresponding system depicted in FIG. 7.

According to an embodiment, the attention MLL is a machine learning logic adapted to compute weights that indicate the predictive power of a block’s feature vector relative to the labels of the image from which the block is derived, and these weights may be provided as input to the MLL or may be combined with numerical values from the MLL output.

According to embodiments, both the MIL program and the attention MLL program learn to recognize feature vectors and corresponding blocks (and thus, the depicted tissue patterns) with predictive capabilities relative to patient-related attribute values. The attention MLL program can be implemented as a part, for example, a submodule of the MIL program.

According to some embodiments, the attention MIL program implements a permutation-invariant transformation operation, which the MIL program uses to aggregate the predictive power of the bag's label relative to the labels encoded in all feature vectors of the blocks within the bag. This permutation-invariant transformation generates a single aggregated value for the bag based on all blocks. According to embodiments, the difference between the aggregated value and the actual label assigned to the bag is also considered as a form of "loss" of the MIL program that will be minimized during backpropagation. The permutation-invariant transformation operation is used by the MIL during the training phase and also by the overtrained MIL program during the testing phase.

Permutation-invariant transformations allow for specifying how information encoded in all blocks of a packet should be considered during the training phase.

According to the embodiment, the permutation-invariant transformation operation is a maximization operation. This can be beneficial because the predictive model generated during MIL training strongly reflects the organizational pattern described in the block, which has the feature vector with the highest predictive power relative to the bag label. The model is unaffected by the negative impact of organizational regions/blocks unrelated to the label. However, the maximization operation will ignore all information contained in all blocks except the highest-scoring block. Therefore, the predictive power of potentially relevant block/organizational patterns may be missed.

According to an embodiment, the permutation-invariant transformation operation is an averaging operation, such as the arithmetic mean or median of the values, representing the predictive power of each individual feature vector relative to a particular label. This can be beneficial because the predictive model generated during the training of the MIL takes into account the organizational patterns depicted in all blocks. However, considering organizational patterns and corresponding blocks that are not actually relevant to the occurrence of a particular label can lead to a deterioration and reduction in the predictive accuracy of the trained MIL.

According to the embodiments, the permutation-invariant transformation operation of the MIL program is the AVERAGE or MEDIAN operation.

According to one embodiment, the permutation-invariant transformation operation is an averaging operation, such as the arithmetic mean or median of the values, representing the predictive power of each individual feature vector relative to a particular label, and the attention MIL is used to compute the weights for each in the block. The weights computed for a particular block and its corresponding feature vectors represent the "attention" that MIL will draw for that block during the training phase.

The combination of the "average" permutation-invariant transformation operation and the attention MIL operation with weight configurations tailored to specific blocks may offer the following advantages: the benefits provided by the AVERAGE operation (considering the information conveyed across all blocks) can be used without accepting its drawbacks (the impact of irrelevant organizational patterns on the training of the MIL program's predictive model). This can allow for improved accuracy of the predictive model trained on the MIL program by selectively/primarily learning from blocks assigned higher weights, balancing less important blocks during the learning process.

Combining the attention MLL procedure and the MIL procedure as described herein with respect to embodiments of the invention offers the following advantages: The attention MLL procedure (particularly when implementing permutation-invariant transformation operations other than the MAX operation, such as AVERAGE or MEDIAN) allows the MIL procedure to learn from multiple instances (blocks) in each iteration. For example, unlike the example of the MAX operation, which is a sparse method that selects only one instance of all bags for learning in each iteration. Generally, the AVERAGE or MEDIAN operation is not preferred because it can lead to degradation of the model learned by the MIL procedure due to the feature vectors of blocks that lack predictive power. However, if the feature vectors of these blocks have been assigned low weights based on independent estimates from the attention MLL, the training process of the MIL procedure may benefit from using AVERAGE or MEDIAN instead of the MAXIMUM operation as a permutation-invariant transformation.

For example, using attention-based MLL when training a MIL program can be performed as described in Maximilian Ilse, Jakub M. Tomczak, and Max Welling, “Attention-Based Deep Multi-Instance Learning,” February 2018, available electronically at https://arxiv.org/abs/1802.04712.

According to one embodiment, the GUI is configured to create and render a heatmap of weights computed by an attention MLL procedure for all blocks derived from a specific digital image. The weights are normalized, for example, to a range of 0-1, and then the normalized weights of the blocks are color-coded. The more similar the weights of the blocks, the more similar the colors of the attention MLL-based heatmap.

Attention MLL program that provides weighted values

According to an embodiment (see, for example, Figure 6), the method includes computing, for each block, a numerical value indicating the predictive power of the feature vector associated with the block in the form of a weighted value. Each weighted value for a block is calculated as a function of the weights calculated by the attention MLL for the block and the value calculated by the MIL for the block. In particular, the weighted value can be calculated by multiplying the weights calculated by the attention MLL by the value of the corresponding block.

Attention MLL program that provides weighted feature vectors

According to an embodiment, the method includes computing a feature vector in the form of a weighted feature vector for each block. The weighted feature vector is computed as a function of weights computed by the attention MLL for the block and a feature vector computed by a feature extraction procedure for the block. In particular, the weights provided by the attention MLL for a particular block can be multiplied by the feature vector of that block.

According to another embodiment, training the MIL is performed such that the numerical value output by the MIL for a specific block relative to a specific label, indicating the predictive power of the block relative to the label of the bag (image), is multiplied by the weights calculated by the attention MLL for that block. During backpropagation, the weights affect the adaptability of the MIL's predictive model. The effect of a specific feature vector on the predictive model of the MIL learned during training is positively correlated with the weights calculated by the attention MLL for that specific block.

According to one embodiment, training the MIL is performed such that the weights provided by the attention MLL, along with the feature vectors, are used as input to the MIL program. The MIL is trained such that it learns more from blocks with higher weights on their feature vectors than from blocks with lower weights. In other words, the impact of blocks and their feature vectors on the predictive model of the MIL learned during training is positively correlated with the weights computed by the attention MLL for a particular block.

Using attention-based MIL to compute weights for each feature vector can be advantageous because the MIL will learn more from a few blocks with high predictive potential and less from the majority of blocks showing unrelated tissue slices. As a result, the accuracy of the trained MIL program increases.

Further embodiments

According to an embodiment, the method further includes:

- For each patient in another group of patients, at least one additional digital image of the patient's tissue sample is received via the image analysis system, each additional image having been assigned one of the predefined labels;

-The image analysis system splits each received additional image into additional image block sets, each block being assigned a label to the image used to create the additional block;

- For each of the additional blocks, the image analysis system calculates additional feature vectors, which contain image features selectively extracted from the additional blocks and from the organizational patterns depicted therein;

- Apply a trained Multiple Instance Learning (MIL) procedure to the additional blocks and corresponding additional feature vectors of all additional images received for all patients in the additional group, so as to compute, for each of the additional blocks, a numerical value indicating the probability that an image from which the additional block is derived has been assigned a specific label, the numerical value being computed as a nonlinear transformation function of the learned feature vectors of the additional block; and

- An additional image block report library is output via the GUI of the image analysis system. The additional report library contains multiple additional blocks, which are sorted according to their corresponding calculated values and/or contain graphical representations of their corresponding values.

This can be advantageous because the trained MIL program can be easily applied to new image data, thereby simplifying the analysis and interpretation of new images relative to target patient-related attributes, for example by automatically presenting blocks of new images that are identified by the trained MIL program as having high predictive power relative to such patient-related attributes through an automated report library.

According to embodiments, the MIL program learns during the training phase to transform feature vectors into values that can represent probabilities of a specific label. A label can represent a category (e.g., a patient who responds to treatment with a specific drug D) or a numerical endpoint value (e.g., a number or percentage value indicating the degree of response). This learning can be mathematically described as learning a non-linear transformation function that transforms the feature values into one of the labels provided during training. According to some embodiments, at test time, some minor structural changes are applied to the trained MIL program (such as disabling Dropout layers, etc.) and no sampling of test data occurs. The main change in applying the trained MIL program at test time is that the MIL program analyzes all instances (blocks) in the test data packet to calculate a final value indicating the predictive power of each block and the multiple labels provided during the training phase. Finally, the final value is calculated for the entire image or a specific patient by aggregating the values calculated from the blocks of images into multiple labels. The final result of applying the trained MIL program to one or more images of a patient is one of the labels with the highest probability (e.g., "The patient will respond to treatment with drug D!"). In addition, one of the blocks with the highest predictive power relative to the label can be presented in the report image block library, which is structurally equivalent to the report image block library described above for the training phase.

According to an embodiment, the method further includes automatically selecting or enabling a user to select one or more "high predictive power blocks". A "high predictive power block" is a block whose numerical value indicates that the predictive power of its feature vector relative to a specific label exceeds a high predictive power threshold; and/or

In addition, or alternatively, the method also includes automatically selecting or enabling the user to select one or more "artifact blocks". An artifact block is a block whose value indicates that its feature vector has a predictive power of less than a minimum predictive power threshold relative to a particular label or depicts one or more artifacts.

In response to the selection of one or more high-predictability blocks and/or artifact blocks, the MIL program is automatically retrained, thereby excluding the high-predictability blocks and the artifact blocks from the training set.

These features may have the advantage that a retrained MIL program might be more accurate because excluded artifact patches are no longer considered during retraining. Therefore, by retraining the MIL program based on a simplified version of the training dataset that does not contain artifact patches, any bias in the learning transformation caused by patches describing artifacts in the training dataset can be avoided and eliminated.

Allowing users to remove highly predictive blocks from the training dataset may be counterintuitive, but it still offers important benefits: sometimes, the predictive power of certain organizational patterns relative to certain labels is self-evident.

For example, tissue sections containing tumor cells expressing many lung cancer-specific biomarkers are certainly important prognostic markers for the presence of lung cancer. However, pathologists may be more interested in some less obvious tissue patterns, such as the presence and/or location of non-tumor cells, such as FAP+ cells.

In another example, MIL training is used to identify smoking-induced tissue patterns in lung cancer, which may have predictive potential relative to the label "patients show low response to treatment with specific drug D". MIL can calculate the highest numerical value/predictive power corresponding to a first tissue pattern that includes smoking-induced residues. Removing blocks showing tissue regions with smoking-induced residues may reveal another tissue pattern with moderate predictive power. When the feature vector includes the patient's genetic and/or physiological attribute values, the predictive power of these additional features may become more relevant after the block with the highest value is "blacklisted". These genetically or physiologically relevant predictive features can also be reflected in specific tissue patterns, thus allowing pathologists to identify and understand genetically or physiologically relevant attributes by examining corresponding blocks in the resulting block library generated after retraining MIL based on a training block set that does not contain blacklisted blocks.

Therefore, when all blocks showing tumor cells as the most important prognostic factor are removed, and the MIL program is retrained based on the remaining training dataset, the retrained MIL will be able to more reliably identify less prominent but still important prognostic factors and tissue patterns.

In another aspect, the present invention relates to an image analysis system for identifying tissue patterns that indicate patient-related attribute values. The image analysis system includes:

- At least one processor;

- A volatile or non-volatile storage medium comprising digital tissue images of tissues from a group of patients, wherein for each patient in the group, at least one digital image of a tissue sample of that patient is stored in the storage medium, and at least one image is assigned one of at least two different predefined labels, each label indicating a patient-related attribute value of the tissue in the labeled image depicting the patient.

- An image splitting module, which can be executed by at least one processor and configured to split each of an image into a set of image blocks, each block having been assigned a label to the image used to create the block;

- A feature extraction module, which can be executed by at least one processor and is configured to compute a feature vector for each block in the block, the feature vector including image features selectively extracted from the organizational patterns depicted in the block;

- A multi-instance learning (MIL) program, which can be executed by at least one processor and is configured to receive all blocks and corresponding feature vectors of all images of all patients in the group during the training phase of the MIL program. The MIL program is configured to process each block set into a block bundle with the same label during the training phase. The training includes analyzing the feature vectors to compute a numerical value for each block in the block, which indicates the predictive power of the feature vector associated with the block relative to the label assigned to the image from which the block is derived.

- A GUI generation module, executable by at least one processor and configured to generate and output a GUI containing a report library of image patches, the report library containing a subset of patches sorted according to their respective calculated values and/or including graphical representations of their respective values; and

-Suitable for displays of GUIs with image block report libraries.

As used herein, a “tissue sample” is a 3D assembly of cells that can be analyzed by the methods of the present invention. The 3D assembly can be a slice of an ex vivo cell block assembly. For example, the sample can be prepared from tissue collected from a patient, such as liver, lung, kidney, or colon tissue samples from a cancer patient. The sample can be a whole tissue or a TMA slice on a microscope slide. Methods for preparing slide-fixed tissue samples are well known in the art and are applicable to the present invention.

Tissue samples can be stained using any reagent or biomarker, such as dyes or staining agents, histochemicals, or immunohistochemicals that react directly with specific biomarkers or various cell types or cell compartments. Not all staining agents/reagents are compatible. Therefore, the type of staining agent used and the order of its application should be carefully considered, but this can be readily determined by those skilled in the art. Such histochemicals can be chromophores detectable by transmission microscopy or fluorophores detectable by fluorescence microscopy. Typically, samples containing cells can be incubated with a solution containing at least one histochemical that will react directly with or bind to the target chemical groups. Some histochemicals are often co-incubated with a mordant or metal for staining. Cells containing a sample can be incubated with a mixture of at least one histochemical that stains the target component and another histochemical used as a counterstain and binding to regions outside the target component. Alternatively, a mixture of multiple probes can be used in staining, providing a method for identifying specific probe sites. Procedures for staining samples containing cells are well known in the art.

As used herein, an "image analysis system" is a system, such as a computer system, suitable for evaluating and processing digital images, particularly images of tissue samples, to help users assess or interpret the images and/or extract biomedical information implicit or explicitly contained within them. For example, the computer system could be a standard desktop computer system or a distributed computer system, such as a cloud system. Typically, computerized histopathological image analysis takes single-channel or multi-channel images captured by a camera as its input and attempts to provide additional quantitative information to aid in diagnosis or treatment.

Embodiments of the present invention can be used to determine which patient subgroups within a larger patient population are likely to benefit from a particular drug. Personalized medicine (PM) is a new field of medicine that aims to provide effective, tailored treatment strategies based on an individual's genomic, epigenomic, and proteomic characteristics. PM not only attempts to treat patients but also prevents them from suffering the side effects of ineffective treatments. Some mutations that frequently occur during tumor development can cause resistance to certain treatments. Therefore, a patient's mutation profile, which can be revealed at least partially through tissue images of tissue samples stained with biomarkers specifically, will allow a trained MIL (Mutual Injection and Treatment) program to definitively determine whether a particular treatment is effective for an individual patient. Currently, it is necessary to determine whether a prescription drug is effective for a patient through trial and error. The trial-and-error process can produce many side effects, such as undesirable and complex drug interactions, frequent changes in prescription drugs, long delays before an effective drug is identified, disease progression, etc. PM is based on dividing individuals into subgroups that respond differently to therapeutic agents targeting their specific diseases. For example, some ALK kinase inhibitors are useful drugs for treating approximately 5% of NSCLC lung cancer patients with elevated ALK gene expression. However, after a period of time, kinase inhibitors become ineffective due to mutations in the ALK gene or other genes downstream of the ALK signaling cascade. Therefore, intelligent molecular characterization of lung cancer patients allows for optimized use of certain mutation-specific drugs through patient stratification. Thus, the "patient group" from which training or test images are obtained could be a group such as "100 breast cancer patients," "100 HER+ breast cancer patients," "200 colon cancer patients," etc.

The term "digital image" as used here refers to a digital representation of a two-dimensional image, typically in binary form. Generally, an organized image is a raster-type image, meaning that the image is a raster ("matrix") of pixels, each assigned at least one intensity value. Some multi-channel images may have pixels with one intensity value for each color channel. A digital image contains a fixed number of rows and columns of pixels. A pixel is the smallest individual element in an image, holding an outdated value representing the brightness of a given color at any given point. Pixels are typically stored in computer memory as raster images or raster maps (two-dimensional arrays of small integers). These values are usually transmitted or stored in compressed form. Digital images can be acquired, for example, via digital cameras, scanners, coordinate measuring machines, microscopes, slide scanning devices, etc.

The “labels” used here are data values, such as strings or numbers, that represent and indicate relevant patient attribute values. Examples of labels could be “Patient response to drug D = True”, “Patient response to drug D = False”, “Progression-free survival = 6 months”, etc.

The term "image block" as used herein refers to a sub-region of a digital image. Typically, blocks created from a digital image can have any shape, such as circles, ellipses, polygons, rectangles, squares, etc., and can overlap or not. According to a preferred embodiment, the blocks generated from the image are rectangular, and preferably overlapping blocks. The advantage of using overlapping blocks is that organizational patterns that would otherwise be fragmented by the block generation process are also represented in the package. For example, the overlap of two overlapping blocks can cover 20-30%, such as 25% of the area of a single block.

According to an embodiment, the image block library, such as the image block report library and/or the image similarity search block library, is a grid-style organization of blocks on a GUI, wherein the spatial organization of blocks in the image block library is independent of their spatial arrangement in the images from which the blocks are derived.

The "feature vector" used here is a data structure containing information describing the important features of an object. The data structure can be one-dimensional or multi-dimensional, where data values of a specific type are stored in corresponding positions within the structure. For example, the data structure can be a vector, array, matrix, etc. A feature vector can be considered an n-dimensional vector representing the numerical features of an object. In image analysis, features can take many forms. A simple feature representation of an image is the raw intensity value of each pixel. However, more complex feature representations are also possible. For example, features extracted from an image or image patch can also be SIFT descriptor features (Scale Invariant Feature Transform). These features capture the generality of different line directions. Other features can indicate the contrast, gradient direction, color composition, and other aspects of an image or image patch.

The term "heatmap" as used herein is a graphical representation of data in which individual values contained in a matrix are represented by color and/or intensity values. According to some embodiments, the heatmap is opaque and includes at least some structures of a tissue slide image upon which the heatmap is created. According to other embodiments, the heatmap is semi-transparent and appears as a cover layer on top of the tissue image used to create the heatmap. According to some embodiments, the heatmap indicates each of a plurality of similarity scores or ranges of similarity scores by corresponding color or pixel intensity.

As used in this article, “biomarker-specific staining agents” are selective staining agents for specific biomarkers, such as specific proteins like HER, but not for other biomarkers or tissue components in general.

As used herein, “non-biomarker-specific staining agents” are staining agents with more general binding behavior. Non-biomarker-specific staining agents do not selectively stain individual protein or DNA sequences, but rather stain larger groups of substances and subcellular and supracellular structures with specific physical or chemical properties. For example, hematoxylin and eosin are non-biomarker-specific staining agents. Hematoxylin is a basic/positive deep blue or purple staining agent. Hematoxylin binds to basophilic substances (such as DNA and RNA, which are acidic and negatively charged). DNA/RNA in the cell nucleus and RNA in rough endoplasmic reticulum ribosomes are acidic because the phosphate backbone of nucleic acids is negatively charged. These backbones form salts with positively charged basic dyes. Therefore, dyes like hematoxylin bind to DNA and RNA and stain them purple. Eosin is an acidic and negative red or pink staining agent. Eosin binds to acidophilic substances, such as positively charged amino acid side chains (e.g., lysine, arginine). In some cells, most proteins in the cytoplasm are basic because the amino acid residues of arginine and lysine give them a positive charge. These form salts with negatively charged acidic dyes such as eosin. Therefore, eosin binds to these amino acids/proteins and stains them pink. This includes cytoplasmic filaments, intracellular membranes, and extracellular fibers in muscle cells.

The “attention machine learning logic program” used in this paper is an MLL trained to assign weights to specific parameters, whereby the weights indicate importance and the attention that other programs might spend analyzing these parameters. The idea behind attention MLLs is to simulate the human brain’s ability to selectively focus on a subset of available data that is particularly relevant to the current context. Using attention MLLs, for example in the field of text mining, weights and computational resources are selectively assigned to specific words that are particularly important for extracting meaning from a sentence. Not all words are equally important. Some are more representative of a sentence than others. By training an attention MLL on a training dataset, an attention model can be generated that indicates how much attention can be paid to “important” words in a sentence vector. According to one embodiment, the trained attention MLL is adapted to compute the weight of each feature value in each feature vector examined, as well as to compute a weighted sum of all feature values in each feature vector. This weighted sum reflects the entire feature vector of the block.

According to an embodiment, an attention MLL is an MLL that includes a neural attention mechanism adapted to equip a neural network with the ability to focus on a subset of its inputs (or features): it selects specific inputs. Let x∈Rd be the input vector, z∈Rk be the feature vector, a∈[0,1]k be the attention vector, g∈Rk be the attention glimpse, and fφ(x) be the attention network with parameter φ.

Typically, attention is implemented as

ag＝fφ(x),＝a⊙z，

Here, ⊙ represents element-wise multiplication, and z is the output of another neural network fθ(x) with parameter θ. We can talk about soft attention, which multiplies the features by a (soft) mask of values between zero and one, or hard attention, when these values are restricted to exactly zero or one, i.e., a∈{0,1}k. In the latter case, we can directly index the feature vector using a hard attention mask: g-＝z[a] (in Matlab notation), which changes its dimension, now g-∈Rm and m≤k.

As used herein, the term "intensity information" or "pixel intensity" is a measure of the amount of electromagnetic radiation ("light") captured at or represented by pixels of a digital image. As used herein, the term "intensity information" may include additional, related information, such as the intensity of a particular color channel. MLLs can use this information to computationally extract derived information such as gradients or textures contained in a digital image, and may implicitly or explicitly extract derived information from a digital image during training and/or during feature extraction by a trained MLL. For example, the statement "pixel intensity values of a digital image are correlated with the intensity of one or more specific staining agents" may imply that intensity information (including color information) allows the MLL and may also allow the user to identify regions of a tissue sample stained with one of the one or more specific staining agents. For example, pixels depicting a sample region stained with hematoxylin may have high pixel intensity in the blue channel, and pixels depicting a sample region stained with fast red may have high pixel intensity in the red channel.

As used in this paper, a "fully convolutional neural network" is a neural network composed of convolutional layers, without any fully connected layers or multilayer perceptrons (MLPs) typically found at the ends of the network. Fully convolutional networks learn filters at each layer. Even the decision layers at the ends of the network learn filters. Fully convolutional networks attempt to learn representations and make decisions based on local spatial inputs.

According to an embodiment, a fully convolutional network is a convolutional network with only layers of the following form, whose activation function generates an output data vector y_{_ij} at position (i, j) in a specific layer that satisfies the following properties:

y _ij =f _ks ({x _{si+δi, sj+δj} } _{0≤δi, δj≤k} )

Where x _{_ij} is the data vector at position (i; j) in a specific layer, y _{_ij} is the data vector at position (i _{; j)} in the following layer, y_ij is the output of the network activation function, k is called the kernel size, s is the stride or subsampling factor, and f _{_ks} determines the layer type: matrix multiplication of convolution or average pooling, spatial maximum of max pooling, or other types of layers such as element-wise nonlinear activation function. This functional form remains invariant under combination, and the kernel size and stride follow transformation rules:

While general deep networks compute nonlinear functions, only networks with this type of layer compute nonlinear filters, also known as deep filters or fully convolutional networks. FCNs naturally operate on inputs of arbitrary size and produce outputs with corresponding (possibly resampled) spatial dimensions. For a more detailed description of the characteristics of several fully convolutional networks, see Jonathan Long, Evan Shelhamer, and Trevor Darrell, “Fully Convolutional Networks for Semantic Segmentation,” CVPR 2015.

As used in this article, "Machine Learning Logic (MLL)" is a program logic, such as software like a trained neural network or support vector machine, that has been trained or can be trained during training and—as a result of the learning phase—has learned to perform some prediction and/or data processing tasks based on the provided training data. Therefore, MLL can be at least partially unspecified program code that is implicitly learned and modified during data-driven learning processes that build one or more implicit or explicit models from sample inputs. Machine learning can be supervised or unsupervised. Effective machine learning is often difficult because finding patterns is challenging and there is often insufficient training data available.

As used herein, the term "biomarker" is a molecule that can be measured in a biological sample as an indicator of tissue type, normal or pathogenic process, or response to therapeutic intervention. In one particular embodiment, biomarkers are selected from proteins, peptides, nucleic acids, lipids, and carbohydrates. More specifically, biomarkers can be specific proteins, such as EGRF, HER2, p53, CD3, CD8, Ki67, etc. Some biomarkers are characteristic of specific cells, while others have been identified as being associated with specific diseases or conditions.

Image analysis of tissue samples to determine the stage of a specific tumor may require staining the samples with multiple biomarker-specific staining agents. Biomarker-specific staining of tissue samples typically involves the use of primary antibodies that selectively bind to the target biomarker. In particular, these primary antibodies, along with other components of the staining protocol, can be expensive, thus cost may exclude the use of available image analysis techniques in many applications, especially high-throughput screening.

Typically, tissue samples are stained with background staining (“reverse staining”), such as hematoxylin staining or a combination of hematoxylin and eosin staining (“H&E” staining), to reveal large-scale tissue morphology and cell and nucleus boundaries. In addition to background staining, a variety of biomarker-specific stains can be applied depending on the biomedical question being answered, such as tumor classification and staging, detecting the number and relative distribution of certain cell types in tissues, etc.

Attached Figure Description

In the following embodiments, the invention is explained in more detail by way of example only, with reference to the accompanying drawings, wherein:

Figure 1 depicts a flowchart of a method according to an embodiment of the present invention;

Figure 2 depicts a block diagram of an image analysis system according to an embodiment of the present invention;

Figure 3 depicts a GUI with a report image block library according to an embodiment of the present invention;

Figure 4 depicts a GUI for a similarity-based image patch library according to an embodiment of the present invention;

Figure 5 depicts the network architecture of the feature extraction MLL program according to an embodiment of the present invention;

Figure 6 depicts a possible system architecture for combining MIL procedures and attention MLL;

Figure 7 illustrates another possible system architecture for combining MIL procedures and attention MLL;

Figure 8 illustrates the spatial distances of blocks in 2D and 3D coordinate systems;

Figure 9 illustrates the architecture of a twin neural network according to an embodiment of the present invention;

Figure 10 illustrates the feature extraction MLL implemented as a truncated Siamese neural network;

Figure 11 illustrates a computer system that uses feature vectors for similarity-based search in an image database;

Figure 12 shows pairs of "similar" and "dissimilar" blocks labeled based on their spatial proximity; and

Figure 13 shows the feature vector based on similarity search results, which is extracted by feature extraction MLL trained on proximity-based similarity labels.

Detailed Implementation

Figure 1 illustrates a flowchart of a method according to an embodiment of the present invention. This method can be used, for example, to predict patient-related attribute values, such as, for example, biomarker status, diagnosis, treatment outcome, microsatellite status (MSS) of a specific cancer (such as colorectal cancer or breast cancer), micrometastases in lymph nodes, and pathological complete remission (pCR) in diagnostic biopsies. The prediction is based on digital images of histological slides using deep learning based on—preferably—assumption-free—feature extraction.

Method 100 can be used to train a weakly supervised deep learning computer algorithm designed to identify and extract previously unknown predictive histological signatures. This method allows for the identification of tissue patterns that indicate patient-related attribute values.

Tissue specimens from patients can be provided, for example, in the form of FFPET tissue blocks. Tissue blocks need to be obtained from patients with predetermined and known endpoints (e.g., survival, response, genetic signature, etc.) for use as labels.

The tissue block is sectioned and placed on a microscope slide. The section is then stained with one or more histology-associated staining agents, such as H&E and/or various biomarker-specific staining agents. Images are taken from the stained tissue sections, for example, using a scanning microscope on a slide.

In the first step 102, the image analysis system (e.g., as described with reference to FIG2) receives at least one digital image 212 of a tissue sample from each of the patients in a group of patients.

Retrieving images may include reading images from a database. For example, images could be tissue sample images from many years ago. The advantage of older image datasets is that the outcomes of many relevant events, such as treatment success, disease progression, and side effects, are known simultaneously and can be used to create a training dataset that includes tissue images labeled with these known events. Alternatively, images can be received directly from an image acquisition system, such as a microscope or slide scanner. Labels can be assigned manually or automatically to the received images. For example, a user can configure the software of a slide scanner so that the acquired images are automatically labeled with specific tags during their acquisition process. This can be helpful in scenarios where tissue sample images from a large number of patients with the same patient-related attribute values/endpoints are acquired sequentially, such as 100 tissue images from a first group of 100 breast cancer patients known to have responded to a specific drug D, and 120 tissue images from a second group of 120 breast cancer patients known not to have shown such a response. The user may have to set the labels to be assigned to the captured images only once before acquiring the first group of images, and then set the labels a second time before acquiring the second group of images.

For each patient, one or more images are retrieved. For example, the same tissue sample may be stained multiple times according to different staining schemes, thereby acquiring an image for each staining scheme. Alternatively, several adjacent tissue sample sections may be stained with the same or different staining schemes, and an image may be acquired for each of the tissue sample slides. Each of the received images has been assigned one of at least two different predefined labels. Each label indicates a patient-related attribute value of the patient whose tissue is depicted in the labeled image. Attribute values can be of any type, such as Boolean values, numbers, strings, ordered parameter values, etc.

Next, in step 104, the image analysis system splits each received image into a set of image blocks 216. Therefore, each block has been assigned a label that was originally assigned to the image used to create the block.

For example, the image dataset released as the basis for the 2016 "CAMELYON16" challenge can be used as a training dataset. The CAMELYON16 dataset consists of 270 whole-slide images of H&E-stained lymph node tissue sections from breast cancer patients, provided as the training image dataset (160 normal tissue images and 110 tumor metastasis images). This dataset is available at https://camelyon16.grand-challenge.org/data/ . At 10x magnification, the images in this dataset can be used to generate 1,113,403 RGB blocks from non-background regions of size 256x256 pixels, with no overlap between blocks.

According to one embodiment, the received image and the generated blocks are multi-channel images. The number of blocks can be increased to enrich the training dataset by creating modified copies of existing blocks with different sizes, magnification levels, and/or including some simulated artifacts and noise. In some cases, multiple packets can be created by resampling instances in a packet as described herein with respect to embodiments of the invention and placing selected instances in additional packets. This “sampling” can also have a positive effect on enriching the training dataset.

In some cases, feature vectors can be clustered into N clusters, and M instances (blocks) can be randomly selected from each cluster and placed into a pseudo-packet to generate a variant of the clusters of instances in the pack.

Next, in step 106, the image analysis system computes a feature vector 220 for each block. The feature vector includes image features selectively extracted from the tissue patterns depicted in the block. Optionally, the feature vector may also include genetic features or other patient or patient-related data from which the image and the patient of the corresponding block are derived. According to some embodiments, feature extraction is performed by a trained feature extraction MLL. The feature extraction MLL can generate feature vectors for each block in the training dataset while preserving feature-vector-label relationships. However, other embodiments may use an explicitly programmed feature extraction algorithm to provide a variety of features that describe the tissue regions depicted in the block from which the feature vectors are computed.

Next, in step 108, a multi-instance learning (MIL) program 226 is trained based on all blocks and corresponding feature vectors from all images received for all patients in the group. Thus, the MIL program processes each block set as a bundle of blocks with the same label. Training includes analyzing the feature vectors 220 of the blocks in the training dataset and calculating a value 228 for each block. This value indicates the predictive power of the feature vector associated with the block relative to the label assigned to the image from which the block is derived. In other words, this value represents the predictive power of a particular feature vector, i.e., the "predictive value/power," for the occurrence/observation of the label assigned to the block. Since the features of the feature vector have been extracted entirely or at least partially from the image information contained in the corresponding block, the feature vector represents the optical properties of the tissue region depicted in that block. Therefore, the feature vector can be considered an electronic tissue signature.

For example, a MIL program can be trained to predict one or more possible labels for a specific tissue region and/or a MIL program can be trained to regress labels if floating-point label prediction is required. In some cases, an additional attention MIL is trained to understand which feature vectors are most relevant to the predicted labels. In other cases, weights computed by the attention MIL are multiplied by the feature vector values for each slide. As a result of the multiplication, a feature vector with weighted feature values is obtained for each block and its feature vector, and is used as input to the MIL program during training. In other embodiments, weights computed by the attention MIL are multiplied by values computed by the MIL for each block's feature vector. This creates a weighted value that serves as an indicator of the predictive power of a particular block and its feature values relative to the label. The weighted value can be compared to the underlying facts during training to evaluate the accuracy of the trained MIL program. In some cases, averaging, min, max-min-max pooling (or combinations thereof) can be applied to the feature vectors obtained by the MIL program as block-specific results during training of its permutation-invariant transformation operations.

Next, in step 110, the image analysis system outputs an image patch report library 206 via a GUI 232 generated by the image analysis software. An example GUI includes the report image patch library depicted in Figure 3. The report library comprises subsets of patches, which are thereby sorted according to their corresponding calculated numerical values. Additionally or alternatively, the report image patch library includes graphical representations of the numerical values associated with the respective patches.

Finally, as a result of the training phase, a trained MIL program was obtained. This trained MIL program can be applied to image patches exported from other patient cohorts.

For testing purposes, the available dataset can be split into a subset (including, for example, approximately 75% of the images) for use as the training dataset, and another subset (including, for example, approximately 25% of the images) for use as the test dataset. The trained MIL program has been observed to achieve high prediction values for relevant fields of use (FOVs). These include organizational patterns, which have not yet been considered to affect pCR predictions.

Therefore, embodiments of the present invention can allow the use of a wealth of data available in the drug development process, derived from histological and clinical imaging, genomics and sequencing, real-world data, and diagnostic methods. This method can allow for the extraction of novel insights and the development of new technologies.

In the context of pathological and histological analysis, the task of manually identifying predictive latent tissue textures or tissue-related signatures can be daunting due to the shearing amount of information available in multi-channel, multi-staining, multi-modal, high-magnification images, each containing billions of pixels. Therefore, such exploration is often based on the exploration of human-generated hypotheses and is thus limited to the boundaries of pre-existing knowledge about tumors and biological mechanisms, as well as the complexity and labor demands of manually examining large numbers of high-magnification histological images. Embodiments of the present invention can allow the revelation of hidden information in microscopic pathological histological images, enabling both machine learning logic and humans to interpret and identify them as highly predictive features.

According to an embodiment, the trained MIL can be used to stratify patient groups. This means segmenting patients based on factors other than a given treatment. Stratification can be performed based on patient-related attributes that are not used as labels when training the MIL or attention MIL. For example, such patient-related attributes could be age, sex, other demographic factors, or specific genetic or physiological traits. The GUI allows users to select a subgroup of patients whose tissue images are used to train the MIL based on any of the said patient-related attributes that are not used as labels, and selectively compute the predictive accuracy of the trained MIL on that subgroup. For example, the subgroup could consist of female patients or patients over 60 years of age. The accuracy selectively obtained for a given subgroup, such as female/male or patients over 60/under 60, may reveal specific high or low accuracy of the trained MIL in certain subgroups. This may allow for confounding variables (variables other than those being studied by researchers), making it easier for researchers to detect and interpret relationships between variables and identify patient groups that will benefit most from a particular drug.

Figure 2 depicts a block diagram of an image analysis system 200 according to an embodiment of the present invention.

Image analysis system 200 includes one or more processors 202 and volatile or non-volatile storage media 210. For example, the storage media may be a hard disk drive, such as an electromagnetic or flash drive. It may be magnetic, semiconductor-based, or optical data storage. The storage media may be a volatile medium, such as main memory, that only temporarily contains data.

The storage medium includes multiple labeled digital images 212 of tissue samples from patients with known endpoints.

The image analysis system includes a splitting module 214 configured to split each image 212 into multiple blocks. The blocks are grouped into packages 216, whereby all blocks within the same package are typically derived from the same patient. The package label is a known endpoint for the patient, and all blocks within the package are assigned a package label.

Feature extraction module 218 is configured to extract multiple image features from each of blocks 216. In some embodiments, feature extraction module 218 may be a trained MLL or the encoding portion of a trained MLL. The extracted features are stored as feature vectors 220 associated with the blocks from which the feature vectors are derived in storage medium 210. Optionally, the feature vectors may be enriched with patient features derived from other sources, such as genomic data or microarray data.

Optionally, the image analysis system may include a sampling module 215 adapted to select image samples (subsets) for training and test the trained MIL on the remaining image patches. The sampling module may first perform clustering on the patches based on their feature vectors before performing sampling.

Optionally, the image analysis system may include an attention MLL procedure 222 configured to compute weights for each feature vector and the corresponding block. The weights may be used, along with the feature vectors, as input to the MIL procedure 226 during training, or to weight the values returned by the MIL for each block as a result of the MIL procedure training.

The image analysis system includes a multi-instance learning procedure (MIL procedure 226). During training, MIL procedure 226 receives feature vectors 220 (or weighted feature vectors 224 generated by attention MIL 222) and labels assigned to the corresponding blocks. As a result of training, the trained MIL procedure 226 is provided. Furthermore, for each block, a numerical value 228 is calculated, which indicates the predictive power of the block and the organization pattern of the labels assigned to the block as described in this paper. These values may also be referred to as "numerical block relevance scores".

The image analysis system further includes module 230, which is configured to generate a GUI 232 displayed on screen 204 of the image analysis system.

The GUI includes a report block library 206, which comprises at least some blocks and numerical values 228 calculated for these blocks. The numerical values 228 may be explicitly displayed, for example as overlays on the corresponding blocks, and/or implicitly, for example, sorted according to their respective numerical values 228 in the order of the blocks' sorting. When the user selects one of the blocks, a thermal image of the entire slide from which the image of the block is derived is displayed. In other embodiments, a thermal image may be displayed in addition to the default report block library 206.

Each of program modules 214, 215, 218, 222, 226, and 230 can be implemented as a submodule of a large MIL training framework software application. Alternatively, one or more modules can each represent an independent software application interoperable with other programs and modules of the image analysis system. Each module and program can be software written, for example, in Java, Python, C#, or any other suitable programming language.

Figure 3 depicts a GUI 300 with a report image block library according to an embodiment of the present invention. The report library (the block matrix below row labels 302, 304, 306, and 308) allows the user to explore tissue patterns identified by the MIL program for high predictive power for specific labels. The library includes one of the blocks with the highest value relative to a target specific label, such as "responsive to drug D treatment = true" calculated by the MIL. Blocks are grouped based on tissue slide images from which the blocks are derived and categorized within their groups according to their corresponding values, which indicate the predictive power of the block relative to a specific label assigned to the image used to train the MIL. Furthermore, the library may include the overall predictive accuracy for each of the blocks in the library, which may have been automatically determined after training. Additionally, or alternatively, the report library may include the label assigned to the corresponding image and the predictive accuracy for each packet obtained for that label. For example, "baseline fact = 0" could represent the label "patient responds to drug D," and "baseline fact = 1" could represent the label "patient does not respond to drug D." When using Attention MLL to calculate weights, ranking can also be based on a combined score value calculated for each block, as described herein with respect to embodiments of the invention, for a combination (e.g., a product) of the weights of the blocks generated by Attention MLL and the values calculated by MIL. The highest value of all blocks for a particular image calculated by MIL is displayed as the "predicted value" at the top of the block group derived from said image.

In the depicted library, block row 302 shows six blocks for the first patient. The first assigned highest value (prognostic value) in the block indicates the predictive power of the specific tissue slide/entire slide image relative to the label. The first block of each slide group may additionally or alternatively be assigned the highest combined value of all blocks derived from the specific tissue slide image (derived from the values provided by MIL and the weights calculated by Attention MLL).

As shown in Figure 3, the highest value can be displayed at the top of the highest score block for each patient.

A report block library that includes only a subset of the blocks with the highest predictive power may be advantageous, as pathologists do not need to examine the entire slide. Instead, the pathologist's attention is automatically directed to a small sub-region (block) of each full slide image whose tissue pattern has been identified as having the highest predictive power relative to the target label.

According to the embodiment depicted in Figure 3, the report image block library displays image blocks derived from H&E staining images. The report image block library is organized as follows:

Row 302 includes six blocks assigned the highest values (indicating predictive ability, i.e., prognostic values) calculated by the MIL procedure, these blocks being among all blocks derived from a specific whole slide image 312 of the first patient. According to other embodiments, sorting is performed based on either the same score value calculated by the MIL or a derivative value of the MIL-calculated value. For example, the derivative value could be a combined score calculated as a combination of the value calculated by the MIL for the block and a weight calculated by the Attention-MLL for said block. For example, this combination could be the product of the value and the weight. According to other embodiments, blocks are sorted only according to the weights calculated by the Attention-MLL, and the MIL-calculated values are displayed to the user in different ways, e.g., as numbers covering the corresponding blocks or presented in a form spatially close to the corresponding blocks.

The corresponding whole slide image 312 of the first patient’s tissue sample, some of which are shown in the blocks in row 312, is spatially close to the selected set 312 of the highly correlated blocks.

Furthermore, the optional relevance heatmap 322 displayed highlights all entire slide image regions where the values calculated by the MIL are similar to the values of a block in image 312, with the highest value indicating the predictive power calculated. In this case, one of the blocks with the highest calculated values (e.g., the block at the first position in row 312) is automatically identified and selected as the basis for calculating the relevance heatmap 322. According to an alternative embodiment, the relevance heatmap 322 does not represent the similarity of a block's value to the highest value calculated for all blocks in the image, but rather the similarity of the block to the highest combined score calculated for all blocks in the image. The combined score can be a combination, for example, a product of the weights calculated by the attention MLL for the block and a value calculated by the MIL indicating the predictive power of the block relative to the image label. According to a further embodiment, the relevance heatmap 322 represents the similarity of the weights of the block calculated by the attention MLL to the highest weights calculated by the attention MLL for all blocks in the image.

Column 304 comprises six blocks assigned the highest values calculated by the MIL program, derived from a specific whole slide image 314 of the second patient. The corresponding whole slide image 314 spatially approximates the selected set of highly correlated blocks. Furthermore, the displayed correlation heatmap 324 highlights all whole slide image regions whose corresponding values calculated by the MIL are highly similar to the blocks in the whole slide image 314 with the highest values calculated by the MIL.

Column 306 comprises six blocks assigned the highest values calculated by the MIL program, derived from a specific whole slide image 316 of the third patient. The corresponding whole slide image 316 spatially approximates the selected set of highly correlated blocks. Furthermore, the displayed correlation heatmap 326 highlights all whole slide image regions whose corresponding values calculated by the MIL are highly similar to the blocks in the whole slide image 316 with the highest values calculated by the MIL.

Column 308 comprises six blocks assigned the highest values calculated by the MIL program, derived from a specific whole slide image 318 of the patient. The corresponding whole slide image 318 is spatially close to the selected set of highly correlated blocks. Furthermore, the displayed correlation heatmap 328 highlights all whole slide image regions whose corresponding values calculated by the MIL are highly similar to the blocks in the whole slide image 318 with the highest values calculated by the MIL.

According to embodiments, the relevance heatmap presented in the report block library indicates predictive power, attention-based weights, or a combination thereof. In the depicted example, bright pixels in the heatmap depict regions in the image where blocks have high predictive values, high attention-based weights, or a combination thereof. According to embodiments, the calculation of the relevance heatmap includes determining whether a block's score (e.g., a numerical, weighted, or combined value) is higher than a minimum percentage value of the score of the highest-scoring block in the image. If yes, the corresponding block in the relevance heatmap is represented by a first color or "bright" intensity value, such as "255". If not, the individual blocks in the relevance heatmap are represented by a second color or "dark" intensity value, such as "0".

Users can select each block in the report block library to start a similarity search (e.g., by double-clicking the block or by clicking to select the block and then selecting the GUI element "Search"), and a similarity search block library will be displayed, as shown in Figure 4.

The "Blacklist" and "Retrain" elements in the optional GUI element set 310 allow users to define a blacklist of blocks and retrain the MIL program based on all blocks except those in the blacklist and those highly similar to them. For example, the blacklist can include a manually selected set of blocks with particularly low numerical values (predicted values), such as because they contain artifacts, or with particularly high numerical values (excluding blocks with very high predictive power increases MIL's ability to identify additional, previously unknown organizational patterns that are also predictive relative to the target label). The image analysis system can be configured to automatically identify all blocks whose feature vectors are more than a minimum similarity threshold to the feature vectors of blocks added to the blacklist in response to a user adding a specific block to the blacklist. The identified blocks are also automatically added to the blacklist. When the user selects the Retrain GUI element, MIL will be retrained based on all blocks in the training dataset, except those in the blacklist.

Figure 4 depicts a GUI 400 with a similarity search image patch library according to an embodiment of the present invention. The similarity search is triggered by the user's selection of 430 patches in the report library.

The search identifies a subset of the six most similar blocks within each block generated from the entire slide images 412-418, for example, based on the similarity of the compared feature vectors. The blocks identified in the similarity search are grouped by each entire slide image or by each patient and sorted in descending order according to their similarity to the block 430 (“query block”) that triggered the similarity search.

The entire slide images 412-418 and similarity heatmaps 422-428 indicate the location of the block whose feature vector (and thus the depicted tissue pattern) is most similar to the feature vector of the selected block.

Optionally, the similarity search block library also includes one or more of the following data:

- Labels are assigned to the images from which the depicted blocks are derived; one label depicted in Figure 4 is "Fundamental Fact: 0";

- The prediction accuracy of each packet (image) relative to the packet's label, calculated by the MIL program;

- The count of similar blocks in the entire slide image and/or the percentage (fraction) of similar blocks compared to dissimilar blocks (e.g., by thresholding).

- The average, median, or histogram of similarity values for all blocks in the entire slide image.

Figure 5 depicts a network architecture 600 for a feature extraction MLL procedure according to an embodiment of the present invention, which supports a supervised learning method for feature vector generation. A deep neural network consisting of a series of autoencoders 604 is trained based on multiple features extracted hierarchically from image patches. The trained network is later able to perform classification tasks, such as classifying tissue depicted in an image patch into one of the categories of “stromal tissue,” “background slide region,” “tumor cells,” “metastatic tissue,” etc., based on optical features extracted from the image patch. The network architecture includes a bottleneck layer 606, which has far fewer neurons than the input layer 603 and may be followed by further hidden layers and classification layers. According to one example, the bottleneck layer comprises approximately 1.5% of the number of neurons in the input layer. There may be hundreds or even thousands of hidden layers between the input layer and the bottleneck layer, and the features extracted by the bottleneck layer may be referred to as “deep bottleneck features” (DBNF).

Figure 6 depicts a possible system architecture for combining the MIL procedure and the attention MLL. According to the depicted embodiment, training the MIL procedure involves training an attention machine learning logic program 222 based on feature vectors 220, 708-714 and labels 216, 702-706 for all blocks of all received images to compute weights for each block. The weights computed by the attention MLL indicate the predictive power of the feature vectors and the corresponding blocks relative to the patient-related attribute values indicated by the block's label. The machine learning system depicted in Figure 6 then computes a combined prediction value for each block obtained from the received training images. The combined prediction value is a function of the numerical value computed by the MIL for the block and the weights computed by the attention MLL for the block. The combined numerical value can be, for example, the product or average of the MIL numerical value and the attention MLL weights. The combined numerical value indicates the predictive power of the feature vectors and the corresponding blocks relative to the patient-related attribute values indicated by the block's label. A calculated loss value then indicates the difference between the combined prediction value obtained for a specific label and the actual label assigned to the block. Based on the calculated loss value, a model of the MIL procedure is then iteratively adapted using backpropagation.

Figure 7 illustrates another possible system architecture for combining the MIL procedure and the attention MLL. Training the MIL procedure involves training an attention machine learning logic procedure 222—an attention MLL procedure—based on feature vectors 220 and labels 216 for all blocks of all received images to compute weights for each block. Weights indicate the predictive power of the feature vectors and the corresponding blocks relative to patient-related attribute values represented by the block's label. The machine learning system depicted in Figure 7 then computes a weighted feature vector for each block as a function of the weights computed by the attention MLL for that block and the feature vector extracted from that block. This weighted feature vector is input into the MIL, enabling it to compute values for each block using the weighted feature vectors instead of the feature vectors initially extracted from the corresponding blocks and optional other data sources. The MIL procedure then computes a loss value indicating the difference between the numerical value obtained for a particular label and the actual label assigned to the block. During training, the MIL iteratively adapts its model using backpropagation based on the computed loss value.

Figure 8 illustrates the spatial distances of blocks in 2D and 3D coordinate systems used to automatically assign similarity labels to block pairs based on similarity labels derived automatically from the spatial proximity of the blocks. Therefore, a training dataset for training the Feature Extraction MLL is provided, which does not require manual annotation of images or blocks by domain experts.

Figure 8A illustrates the spatial distances of blocks in a 2D coordinate system defined by the x and y axes of a digital tissue sample training image 800. The training image 800 depicts a tissue sample from a patient. After obtaining the tissue sample from the patient, the sample is placed on a microscope slide and stained with one or more histology-associated staining agents, such as H&E and/or various biomarker-specific staining agents. The training image 800 is obtained from the stained tissue sample, for example, using a slide scanning microscope. According to some implementation variations, at least some of the received training images are derived from different patients and/or from different tissue regions (biopsy) from the same patient and therefore cannot be aligned with each other in a 3D coordinate system. In this case, the block distances can be calculated in a 2D space defined by the x and y coordinates of the image as described below.

The training image 800 is divided into multiple blocks. For illustrative purposes, the block size in Figure 8A is larger than the usual block size.

The training dataset can be automatically labeled using the following method: First, a starting block 802 is selected. Then, a first circular region surrounding this starting block is determined. The radius of the first circle is also called the first spatial proximity threshold 808. All blocks within the first circle, such as block 806, are considered "nearby" blocks of the starting block 802. Furthermore, a second circular region surrounding this starting block is determined. The radius of the second circle is also called the second spatial proximity threshold 810. All blocks outside the second circle, such as block 804, are "far away" blocks relative to the starting block 802.

Then, a first set of block pairs is created, where each block pair in the first set includes a starting block and a “nearby” block of the starting block. For example, this step may include creating as many block pairs as the number of nearby blocks contained in the first circle. Alternatively, this step may include randomly selecting a subset of available nearby blocks and creating a block pair for each of the selected nearby blocks by adding the starting block to the selected nearby blocks.

Create a second set of block pairs. Each block pair in the second set includes a starting block and a “distant” block relative to the starting block. For example, this step may include creating as many block pairs as there are distant blocks contained in the image 800 outside the second circle. Alternatively, this step may include randomly selecting a subset of the available distant blocks and creating a block pair for each of the selected distant blocks by adding the starting block to the selected distant blocks.

Then, another block within image 800 can be used as a starting block, and the above steps can be performed similarly. This means redrawing the first and second circles using the new starting block as the center. Thus, nearby and distant blocks are identified with respect to the new starting block. The first block set is supplemented with pairs of nearby blocks identified based on the new starting block, while the second block set is supplemented with pairs of distant blocks identified based on the new starting block.

Then, another block within image 800 can be selected as the starting block, and the above steps can be repeated to further supplement the first and second block sets with more block pairs. New starting block selections can be performed until all blocks in the image have been selected as starting blocks or until a predetermined number of blocks have been selected as starting blocks.

For each block pair in the first set, such as 812, assign the label "similar". For each block pair in the second set, such as 814, assign the label "dissimilar".

Figure 8B illustrates the spatial distances of blocks in a 3D coordinate system defined by the x and y axes of digital tissue sample image 800 and the z-axis corresponding to the stacking height of images 800, 832, and 834 aligned with each other, according to the relative positions of tissue slices from tissue blocks depicted by training images 800, 832, and 834, respectively. The training images depict tissue samples derived from individual tissue blocks from a specific patient. The depicted tissue samples belong to a stack of multiple adjacent tissue slices. For example, this stack of tissue slices can be prepared ex vivo from an FFPET tissue block. The tissue block is sliced, and the slices are placed on a microscope slide. The slices are then stained, as described with reference to Figure 8A for image 800.

Since the tissue samples within this stack are derived from a single tissue block, digital images 800, 832, and 834 can be aligned in a common 3D coordinate system, whereby the z-axis is orthogonal to the tissue slices. The z-axis is an axis orthogonal to the tissue slices. The distance in the z-direction of the images corresponds to the distance of the tissue slices depicted by the images. If two blocks in a pair are derived from the same image, the block distance of the pair is calculated in 2D space. Furthermore, block pairs can be created where the blocks are derived from different images aligned with each other in a common 3D coordinate system. In this case, the distance between the two blocks in the pair is calculated using the 3D coordinate system.

Each of the aligned digital images is divided into multiple blocks. For illustrative purposes, the block size in Figure 8B is larger than the usual block size.

The training dataset can be automatically labeled using the following method: First, select the starting block 802. Then, identify and label the block pairs including the starting block and nearby blocks, as well as the block pairs including the starting block and distant blocks, as described below.

A first 3D sphere is determined surrounding the starting block. For illustrative purposes, only a cross-section of the first sphere is shown. The radius of the first sphere is also referred to as a first spatial proximity threshold 836. All blocks within the first sphere, such as block 806 in image 800 and block 840 in image 834, are considered “nearby” blocks of the starting block 802. Furthermore, a second sphere is determined surrounding the starting block. The radius of the second sphere is also referred to as a second spatial proximity threshold 838. All blocks outside the second sphere, such as block 804 in image 800 and block 842 in image 834, are “far” blocks relative to the starting block 802.

Create a first set of block pairs, where each block pair in the first set includes a starting block and a "nearby" block of the starting block. For example, this step may include creating as many block pairs as the number of nearby blocks contained in the first sphere. Alternatively, this step may include randomly selecting a subset of available nearby blocks and creating a block pair for each of the selected nearby blocks by adding the starting block to the selected nearby blocks.

Create a second set of block pairs. Each block pair in the second set includes a starting block and a “distant” block relative to the starting block. For example, this step may include creating as many block pairs as the distant blocks contained in images 800, 832, 834 outside the second sphere. Alternatively, this step may include randomly selecting a subset of available distant blocks and creating a block pair for each of the selected distant blocks by adding the starting block to the selected distant blocks.

Then, another block within image 800 or images 832, 834 can be used as a starting block and the above steps can be performed similarly. This means that the first and second spheres are redrawn using the new starting block as the center. Thus, nearby and distant blocks with respect to the new starting block are identified. The first block set is supplemented with pairs of nearby blocks identified based on the new starting block, while the second block set is supplemented with pairs of distant blocks identified based on the new starting block.

The above steps can be repeated until each block of each of the received images 800, 832, 834 is selected as the starting block (or until another termination criterion is met), thereby further supplementing the first block pair set and the second block pair set with further block pairs.

For each block pair in the first set, such as 812 and 813, assign the label "similar". For each block pair in the second set, such as 814 and 815, assign the label "dissimilar".

The circle- and sphere-based distance calculations shown in Figures 8A and 8B are merely examples for calculating distance-based similarity labels, in which case the binary labels should be "similar" or "dissimilar". Other methods could be used, such as calculating the Euclidean distance between two blocks in a 2D or 3D coordinate system and calculating the numerical similarity that is negatively correlated with the Euclidean distance between the two blocks.

Since the number of pixels corresponding to one millimeter of tissue depends on various factors, such as the magnification of the image capture device and the resolution of the digital image, this paper will specify all distance thresholds for the real physical object being depicted, namely the tissue sample or the slide covering the tissue sample.

Figure 9 illustrates the architecture of a Siamese neural network trained according to an embodiment of the invention, for providing feature vector-based block aggregation from a subnetwork suitable for performing feature vector-based similarity search and/or extracting biomedically significant feature vectors from image patches. The Siamese neural network 900 is trained on an automatically labeled training dataset comprising block pairs with proximity-based similarity labels, such as those automatically created with reference to Figures 8A and/or 8B.

The Siamese neural network 900 consists of two identical subnetworks 902 and 903 connected at their output layer 924. Each network includes input layers 905 and 915, adapted to receive a single digital image (e.g., a block) 954 and 914 as input. Each subnetwork includes multiple hidden layers 906, 916, 908, and 918. A one-dimensional feature vector 910 and 920 is extracted from one of the two input images through a corresponding subnetwork. Therefore, the final hidden layers 908 and 918 of each network are adapted to compute the feature vector and provide it to the output layer 924. The processing of the input images is strictly separate. This means that the subnetwork processes only the input image 954, and the subnetwork processes only the input image 914. When the output layer compares the two vectors to determine vector similarity, the only point where the information conveyed in the two input images is combined is in the output layer, thus determining the similarity of the organizational patterns depicted in the two input images.

According to an embodiment, each subnetwork 902, 903 is based on a modified ResNet-50 architecture (He et al., Deep Residual Learning for Image Recognition, 2015, CVPR’15). According to an embodiment, the ResNet-50 pre-trained subnetworks 902, 903 are pre-trained on ImageNet. The last layer (typically outputting 1,000 features) is replaced with fully connected layers 408, 418, with dimensions equal to the desired size of the feature vectors, such as size 128. For example, the last layer 908, 918 of each subnetwork can be configured to extract features from the penultimate layer, whereby the penultimate layer can provide a significantly larger number of features than the last layer 908, 418 (e.g., 2048). According to an embodiment, the optimizer, for example, uses the Adam optimizer with default parameters in PyTorch (learning rate 0.001, beta 0.9, 0.999), and a batch size of 256 is used during training. For data augmentation, random horizontal and vertical flips and/or random rotations up to 20 degrees, and/or color jitter enhancements with brightness, contrast, saturation, and/or hue values of 0.075 can be applied to blocks to increase the training dataset.

When a Siamese neural network is trained on automatically labeled pairs of images, the goal of the learning process is to ensure that similar images have similar outputs (feature vectors), while dissimilar images have dissimilar outputs. This can be achieved by minimizing a loss function, such as a function that measures the difference between the feature vectors extracted by the two sub-networks.

According to an embodiment, a loss function is used to train a twin neural network based on block pairs, such that the similarity of feature vectors extracted by the two subnetworks for the two blocks of the pair is correlated with the similarity of the organizational patterns depicted in the two blocks of the pair.

For example, a Siamese neural network can be, as described by Bromley et al. in “Signature Verification Using a Time-Delayed Neural Network with ‘Siamese Neural’, 1994, NIPS’ 1994.” Each subnetwork of the Siamese neural network is adapted to extract a multidimensional feature vector from a corresponding one of two image patches provided as input. The network is trained on multiple patch pairs that have been automatically labeled with proximity-based organizational pattern similarity tags, with the goal that patches depicting similar organizational patterns should have outputs (feature vectors) that are close to each other, and patch pairs depicting dissimilar organizational patterns should have outputs that are far from each other. According to one embodiment, this is achieved by performing a contrastive loss, such as dimensionality reduction by learning an invariant mapping as described by Hadsell et al., 2006, CVPR’06. During training, the contrastive loss is minimized. The contrastive loss CL can be calculated, for example, according to…

CL = (1-y)²(f1-f2) + y*max(0, m-L²(f1-f2)), where 1 and 2 are the outputs of two identical subnetworks, and y is the basic fact label of the block pair: 0 if they are labeled "similar" (first block pair set), and 1 if they are labeled "dissimilar" (second block pair set).

Training the Siamese neural network 900 involves feeding the network 900 multiple pairs of automatically labeled similar blocks 812, 813 and dissimilar blocks 814, 815. Each input training data record 928 includes the two blocks of the block pair and their automatically assigned spatial proximity-based labels 907. The proximity-based labels 403 are provided as “ground facts.” The output layer 924 is adapted to compute predicted similarity labels for two input images 904, 914 as a function of the similarity between two compared feature vectors 908, 918. Training the Siamese neural network includes a backpropagation process. Any deviation between the predicted label 926 and the input label 907 is considered an “error” or “loss” measured in the form of a loss function. Training the Siamese neural network involves iteratively minimizing the error computed by the loss function using backpropagation. For example, the Siamese neural network 900 can be implemented as described by Bromley et al. in “Signature Verification Using a Time-Delayed Neural Network with ‘Siamese Neural Networks’,” 1994, NIPS 1994.

Figure 10 illustrates the feature extraction MLL950, implemented as a truncated Siamese neural network, for example, as described with reference to Figure 9.

The feature extraction MLL 950 can be obtained, for example, by storing one of the subnetworks 902 and 903 of a trained Siamese neural network 900, respectively. In contrast to the trained Siamese neural network, the subnetworks 90 and 903 used as the feature extraction MLL require only a single image 952 as input and do not output similarity labels. Instead, they output a feature vector 910 that selectively includes a defined set of features identified during the training of the Siamese neural network 900 as having specific characteristics for a particular organizational pattern, and is particularly suitable for determining the similarity of organizational patterns depicted in two images by extracting and comparing this specific set of features from the two images.

Figure 11 illustrates a computer system 980 using feature vector-based similarity search in an image database. For example, similarity search can be used to compute a search block library, an example of which is depicted in Figure 4. The computer system 980 includes one or more processors 982 and a trained feature extraction MLL 950, which may be a subnetwork of a trained Siamese neural network (“truncated Siamese neural network”). System 980 is adapted to perform image similarity search using the feature extraction MLL to extract feature vectors from the search image and from each of the search images (blocks), respectively.

For example, the computer system may be a standard computer system or a server comprised of or operationally coupled to the database 992. For example, the database may be a related BDSM, comprising hundreds or thousands of whole slide images depicting tissue samples from multiple patients. Preferably, for each image in the database, the database includes a corresponding feature vector that has been extracted from said image in the database by the feature output MLL 950. Preferably, the calculation of the feature vector for each image in the database is performed in a single preprocessing step before any such request is received. However, the feature vectors of images in the database may also be dynamically calculated and extracted in response to a search request. The search may be limited to blocks derived from a particular digital image, such as blocks within a single whole slide image used to identify tissue patterns similar to those depicted in the search image 986. The search image 986 may be, for example, a block included in a library of report blocks selected by the user.

The computer system includes a user interface that enables user 984 to select or provide a specific image or image patch for use as a search image 986. A trained feature extraction MLL 950 is adapted to extract feature vectors 988 (“search feature vectors”) from the input image. A search engine 990 receives the search feature vectors 988 from the feature output MLL 950 and performs a vector-based similarity search in an image database. The similarity search involves comparing the search feature vector with each of the feature vectors of the images in the database to compute a similarity score as a function of the two compared feature vectors. The similarity score indicates the degree of similarity between the search feature vector and the feature vectors of the images in the database, and thus indicates the similarity of the organizational patterns depicted in the two compared images. The search engine 990 is adapted to return and output search results 994 to the user. The search results may be, for example, one or more images in the database that compute the highest similarity score.

For example, if the search image 986 is a known image patch depicting breast cancer tissue, the system 980 can be used to identify multiple other patches (or an entire slide image including such patches) that depict similar breast cancer tissue patterns.

Figure 12 shows two block matrices, each consisting of three columns, with six block pairs per column. The first (top) matrix shows the first set of block pairs (A), which consists of blocks that are close to each other and have been automatically assigned the label "similar." The second (bottom) matrix shows the second set of block pairs (B), which consists of blocks that are far apart and have been automatically assigned the label "dissimilar." In some cases, blocks labeled "similar" appear dissimilar, while blocks labeled "dissimilar" appear similar. This noise arises because at the boundary where two different organizational patterns meet, two nearby blocks may depict different organizational patterns, and even distant organizational regions may depict the same organizational pattern. This is an inherent noise expected during the dataset generation process.

The applicant has observed that, despite this noise, the feature extraction MLL trained on an automatically labeled dataset is able to accurately identify and extract features, thus clearly distinguishing between similar and dissimilar block pairs. The applicant hypothesizes that the observed robustness of the trained MLL to this noise is based on the fact that the area of a region boundary is typically smaller than the area of the region's non-boundary areas.

According to an embodiment, the quality of the automatically generated training dataset is determined by evaluating the similarity of block pairs in the first step using a previously trained similarity network or an ImageNet pre-trained network, then generating similarity labels based on the spatial proximity of the blocks described herein with respect to embodiments of the invention in the second step, and then correcting the pairwise labels, wherein a strong discrepancy is observed between the similarity of the two blocks determined in the first step and the similarity of the two blocks determined in the second step.

Figure 13 shows the feature vectors of the similarity-based search results, extracted by a feature extraction MLL trained with proximity-based similarity labels. The five tumor query blocks are designated A, B, C, D, and E. These query blocks are used for the image retrieval task to identify and retrieve five blocks excluding the query slides (A1-A5, B1-B5, C1-C5, D1-D5, E1-E5), sorted from lowest to highest distance, using feature vectors extracted using a feature extraction MLL trained on data automatically labeled with proximity-based tags. The target category (e.g., tumor) comprises only 3% of the searched blocks. Even though some retrieved blocks appear very different from the query blocks (e.g., C3 and C), all retrieved blocks except A4 have been verified by expert pathologists to contain tumor cells (i.e., the correct category retrieval).

Reference Number List

100 Methods

Steps 102-110

200 Image Analysis System

202 Processor

204 Monitor

206 Image Tile Library

208 The entire glass slide is heated to map

210 Storage medium

212 Digital Images

214 Module Splitting

216 Packets of blocks marked with tags

218 Feature Extraction Module

220 Eigenvectors

222 Attention Machine Learning Logic Program

224 Eigenvector Weights

226 Multi-Instance Learning Program

Numerical correlation score of 228 blocks

230 GUI Generation Module

232 GUI

300 A GUI containing the report block library

302 First Subset of Similar Blocks, First Organization Pattern

304 represents the second subset of similar blocks representing the second organizational pattern.

306 represents the third subset of similar blocks representing the third organizational pattern.

308 represents the fourth subset of similar blocks in the fourth organizational pattern.

310 Selectable GUI element set

312 Image of the entire slide

314 Image of the entire slide

316 Image of the entire slide

318 Image of the entire slide

322 Relevance Heatmap

324 Relevance Heatmap

326 Relevance Heatmap

328 Relevance Heatmap

400 A GUI containing a similarity search block library

402 First Subset of Similar Blocks, First Organization Pattern

404 represents the second subset of similar blocks in the second organizational pattern.

406 represents the third subset of similar blocks representing the third organizational pattern.

408 represents the fourth subset of similar blocks representing the fourth organizational pattern.

410 Selectable GUI element set

412 Image of the entire slide

414 Image of the entire slide

416 Image of the entire slide

418 Image of the entire slide

422 Similarity Heatmap

424 Similarity Heatmap

426 Similarity Heatmap

428 Similarity Heatmap

430 Query block

950 Network Architecture for Feature Extraction MLL

602 Image block used as input

603 Input Layer

604 Multi-story

606 Bottleneck Layer

800 Digital tissue image cut into multiple blocks

802 Block T1

804 Block T2

806 T3 blocks

808 First spatial proximity threshold (2D)

810 Second spatial proximity threshold (2D)

812 Label "similar" block pairs

813 Label "similar" block pairs

814 Label "dissimilar" block pairs

815 Label "dissimilar" block pairs

816 Training Data

832 Digital tissue image aligned with image 300

834 A digitally organized image aligned with image 332

836 First Spatial Proximity Threshold (3D)

838 Second Spatial Proximity Threshold (3D)

840 T4 units

842 T5 blocks

900 Siamese Neural Network

902 Subnetwork

903 Subnetwork

904 First Input Block

905 Input layer of the first network N1

906 Hidden Layer

907 Similarity tags based on proximity (“measurement”)

908 Suitable for hidden layers that compute feature vectors for the first input block

910 Feature vector extracted from 904 in the first input block

914 Second Input Block

915 Input layer of the second network N2

916 Hidden Layer

918 Suitable for hidden layers that compute feature vectors for the second input block

920 Feature vector extracted from the second input block 914

922 Input block pairs

924 Output layer connecting networks N1 and N2

926 Predicted similarity tags

928 Individual data records of the training dataset

950 Feature Extraction MLL

952 Individual input image/block

954 Eigenvectors

980 Computer System

982 Processor

984 users

986 Individual input image/block

988 Search Feature Vector

990 Feature Vector-Based Search Engine

992 A database containing multiple images or blocks.

994 The returned similarity search results

Claims (25)

1. A method (100) for identifying organizational patterns that indicate patient-related attribute values, the method comprising: - For each patient in a group of patients, at least one digital image (212) of a tissue sample of the patient is received (102) by an image analysis system (200), the at least one image being assigned one of at least two different predefined labels, each label indicating a patient-related attribute value of the patient in which the tissue is depicted in the labeled image; -The image analysis system splits (104) each received image into a set of image blocks (216), and each block is assigned a label to the image used to create the block; - For each block in the block, the image analysis system calculates (106) a feature vector (220), the feature vector containing image features selectively extracted from the organizational patterns depicted in the block; - A multi-instance learning (MIL) program (226) is trained (108) based on the blocks and corresponding feature vectors of the images received for all patients in the group, each block set being processed by the MIL program into a block bundle with the same label, the training including analyzing the feature vectors (220) to compute a value (228) for each block, the value indicating the predictive power of the feature vector associated with the block relative to the label assigned to the image from which the block was derived; and - via the GUI (232) of the image analysis system, an image block report library (206) is output (110) containing subsets of the blocks sorted according to their respective computed values. The image patches displayed in the report library are selectable. The GUI is configured to calculate and display a similarity search block library, the calculation including: - Receive the user's selection of a specific block from the image blocks in the report library; - Identify all blocks from all received images that depict an organizational pattern similar to the selected block, wherein the similarity between the feature vectors of the selected block and the feature vectors of the selected block exceeds a threshold, by identifying all blocks from all received images that have been assigned feature vectors; - Display the similarity search block library, which selectively includes the identified blocks; and - Determine the number and/or score of blocks within the identified blocks that have been assigned the same label as the selected block, wherein the displayed similarity search block library further includes the determined number and/or score. 2. The method according to claim 1, wherein the received digital image comprises: - Digital images of tissue samples, with pixel intensity values correlated with the amount of non-biomarker-specific staining agent; and/or - A digital image of a tissue sample, the pixel intensity value of which is correlated with the amount of a biomarker-specific staining agent adapted to selectively stain biomarkers contained in the tissue sample; -The following combinations: • A digital image of the tissue sample, wherein the pixel intensity value of the digital image is correlated with the amount of a first biomarker-specific staining agent, and • A digital image of a tissue sample, wherein the pixel intensity value of the digital image is correlated with the amount of a non-biomarker-specific staining agent, the biomarker-specific staining agent being adapted to selectively stain for biomarkers contained in the tissue sample. All digital images depicting the same tissue sample and/or adjacent tissue samples from the same patient have been assigned the same label, and the MIL is configured to treat all blocks derived from the digital images as members of the same block package. 3. The method according to claim 2, wherein the non-biomarker-specific staining agent is hematoxylin staining agent or H&E staining agent. 4. The method according to claim 1, wherein the image blocks displayed in the image block report library (300) are one or more images (312, 314, ...) from the received images. Derived from 316, 318), the method includes targeting each of one or more images in the report block library: - Identify a block in the report library that has been exported from the image and has been assigned the highest score among all blocks exported from the image. The score is a value calculated by the MIL for each block, or a weight calculated by the Attention MLL for each block, or a combination of the value and the weight calculated by the MIL and the Attention MLL for the block. - For each of the other blocks in the image, a correlation index is calculated by comparing the score of the other block with the score of the block with the highest score, wherein the correlation index is a value that is negatively correlated with the difference between the score of the other block and the score of the block with the highest score; - Calculate (208) correlation heatmaps (322, 324, 326, 328) of the images (312, 314, 316, 318) as a function of the correlation index, wherein the pixel color and/or pixel intensity of the correlation heatmap indicate the correlation index calculated for the block in the image; and - Display the aforementioned correlation heatmap. 5. The method of claim 1, wherein the image blocks in the report library are grouped based on patients from whom the blocks are derived, and/or wherein the image blocks in the report library are grouped based on labels assigned to images from which the blocks are derived. 6. The method of claim 1, further comprising: - The number of block packets is increased computationally by creating additional block sets, each of which is processed by the MIL program as an additional block packet that has been assigned the same labels as the tissue image from which the source blocks were generated. 7. The method of claim 6, wherein the creation of the additional block set comprises: • Apply one or more artifact generation algorithms to at least a subset of the blocks to create new blocks containing the artifacts, and/or • Increase or decrease the resolution of at least a subset of blocks to create new blocks that are finer or coarser in granularity than their corresponding source blocks. 8. The method of claim 1, further comprising: - Calculate the block clusters obtained from one or more received digital images, where blocks are grouped into clusters based on the similarity of their feature vectors. 9. The method of claim 1, wherein training the MIL program comprises: repeatedly sampling the block set to select a subset of blocks from the block set; and training the MIL program based on the subset of blocks. 10. The method of claim 8, wherein training the MIL program comprises: repeatedly sampling the block set to select a subset of blocks from the block set; and training the MIL program based on the subset of blocks, wherein the sampling comprises selecting blocks from each of the block clusters obtained for the patient, such that the number of blocks in a subset of each block created in the sampling corresponds to the size of the cluster from which the blocks are obtained. 11. The method of claim 1, wherein the calculation of the feature vector for each of the blocks includes receiving patient-related data of a patient in which a tissue sample is depicted in the block, and representing the patient-related data in the form of one or more features of the feature vector. 12. The method of claim 11, wherein the patient-related data is selected from the group consisting of: genomic data, RNA sequence data, known diseases of the patient, age, sex, concentration of metabolites in body fluids, health parameters, and current medications. 13. The method of claim 1, wherein the computation of the feature vector is performed by trained machine learning logic (950). 14. The method of claim 13, wherein the computation of the feature vector is performed by a trained fully convolutional neural network comprising at least one bottleneck layer (606). 15. The method of claim 1, wherein the feature vector is computed by feature extraction machine learning logic trained on a training dataset containing labeled block pairs, each label representing the similarity of two organizational patterns depicted by the block pairs and computed as a function of the spatial distance between the two blocks in the block pairs. 16. The method of claim 15, further comprising: - Receive multiple digital training images (800, 832, 834) each depicting a tissue sample; - Divide each of the received training images into multiple blocks; - Automatically generate block pairs (812, 813, 814, 815, 922), each block pair has been assigned a label (907) indicating the degree of similarity between the two organizational patterns depicted in the two blocks of the pair, wherein the degree of similarity is calculated as a function of the spatial proximity (d1, d2, d3, d4) of the two blocks of the pair, wherein the distance is positively correlated with dissimilarity; - Using the labeled block pairs as training data, train machine learning logic MLLs (900, 902, 903, 950) to generate trained MLLs (900) that have learned to extract feature vectors from digitally organized images, which represent images in a manner where similar images have similar feature vectors and dissimilar images have dissimilar feature vectors; and - The trained MLL or its components are used to compute the feature vector of the block. 17. The method of claim 16, wherein the trained MLL is a Siamese neural network (900) comprising two neuronal subnetworks (902, 903) connected by their output layers (924), the method further comprising: - Store one subnetwork (902) of the trained Siamese neural network (900) separately on a storage medium; and - Use the stored subnetwork as a component of the trained MLL(900) to compute the feature vector of the block. 18. The method of claim 1, wherein the label is selected from the group consisting of: - An indication that the patient has responded to a specific drug; - The patient has developed indications of metastasis or a specific form of metastasis; - Cancer patients show indications of pathological complete remission (pCR) in response to specific therapies; - This indicates that the patient has cancer with a specific morphological state or microsatellite state; - The patient has indicated that they have developed an adverse reaction to a specific drug; -Genetic attributes; and/or -RNA expression profile. 19. The method of claim 18, wherein the transfer is a microtransfer. 20. The method of claim 18, wherein the genetic property is a gene signature. 21. The method according to claim 1, wherein training the MIL program comprises: -Based on the feature vectors (220) of all blocks in all received images and the labels, train the attention machine learning logic program (222) — the attention MLL program. A weight is calculated for each of the blocks, the weight indicating the predictive power of the feature vector and the corresponding block relative to the patient-related attribute values represented by the label of the block; - For each block in the block, a combined predicted value is calculated, which is a function of the value calculated by the MIL for the block and the weights calculated by the attention MLL for the block, the combined value indicating the predictive power of the feature vector and the corresponding block relative to the patient-related attribute values represented by the label of the block; - Calculate the loss value indicating the difference between the combined predicted value obtained for a specific label and the actual label assigned to the block; and - Based on the calculated loss value, backpropagation is used to adapt the model of the MIL program. 22. The method according to claim 1, wherein training the MIL program comprises: -Based on the feature vectors (220) of all blocks in all received images and the labels, train the attention machine learning logic program (222) — the attention MLL program. A weight is calculated for each of the blocks, the weight indicating the predictive power of the feature vector and the corresponding block relative to the patient-related attribute values represented by the label of the block; - For each block in the block, a weighted feature vector is calculated as a function of the sum of the weights calculated by the attention MLL for the block and the feature vector extracted from the block; - Input the weighted feature vector into the MIL so that the MIL can use the weighted feature vector as the feature vector to calculate the value for each block in the block; - Calculate the loss value indicating the difference between the numerical value obtained for a specific tag and the actual tag assigned to the block; and - Based on the calculated loss value, backpropagation is used to adapt the model of the MIL program. 23. The method of claim 1, further comprising: - For each patient in another group of patients, at least one additional digital image of the patient's tissue sample is received via the image analysis system, each additional image having been assigned one of the predefined labels; -The image analysis system splits each received additional image into additional image block sets, each block being assigned a label to the image used to create the additional block; - For each of the additional blocks, the image analysis system calculates additional feature vectors, which contain image features selectively extracted from the additional blocks and from the organizational patterns depicted therein; - Apply a trained Multiple Instance Learning (MIL) procedure to the additional blocks and corresponding additional feature vectors of all additional images received for all patients in the additional group, so as to compute, for each of the additional blocks, a numerical value indicating the probability that an image from which the additional block is derived has been assigned a specific label, the numerical value being computed as a nonlinear transformation function of the learned feature vectors of the additional block; and - The image analysis system outputs an additional image block report library via its GUI. The additional report library contains multiple additional blocks, which are sorted according to their respective calculated values and/or include graphical representations of their respective values. 24. The method of claim 1, further comprising: - Automatic selection or enabling the user to select one or more "high predictive power blocks", wherein a high predictive power block is a block whose value (228) exceeds a high predictive power threshold, the value indicating the predictive power of its feature vector relative to a specific label in the labels; and/or - Automatic selection or enabling the user to select one or more "artifact blocks", wherein an artifact block is a block whose value (228) is below the minimum predictive power threshold or depicts one or more artifacts, the value indicating the predictive power of its feature vector relative to a specific label in the label; - In response to the selection of one or more high-predictability blocks and/or artifact blocks, the MIL program is automatically retrained, thereby excluding the high-predictability blocks and the artifact blocks from the training set. 25. An image analysis system (200) for identifying tissue patterns that indicate patient-related attribute values, the image analysis system comprising: -At least one processor (202); - A volatile or non-volatile storage medium (210) containing digital tissue images (212) of a group of patients, wherein for each patient in the group of patients, at least one digital image of a tissue sample of the patient is stored in the storage medium, the at least one image being assigned one of at least two different predefined labels, each label indicating a patient-related attribute value of the patient in which the tissue is depicted in the labeled image; - Image splitting module (214), which can be executed by the at least one processor and is configured to split each of the images into a set of image blocks (216), each block being assigned a label to the image used to create the block; - Feature extraction module (218), which can be executed by the at least one processor and is configured to compute a feature vector (220) for each block in the block, the feature vector containing image features selectively extracted from the organizational patterns depicted in the block; - A multi-instance learning (MIL) program (226), which can be executed by the at least one processor and is configured to receive, during the training phase of the MIL program, all blocks and corresponding feature vectors (220) of all images of all patients in the group, the MIL program being configured to process each block set into a block bundle with the same label during the training phase, the training including analyzing the feature vectors to compute a numerical value (228) for each block in the block, the numerical value indicating the predictive power of the feature vector associated with the block relative to the label assigned to the image from which the block is derived; and - A GUI generation module (230), which can be executed by the at least one processor and is configured to generate and output a GUI (232) containing an image patch report library (206), the report library containing a subset of the patches, the subset of patches being sorted according to their respective calculated values; and - Display (204), the display being adapted to display the GUI having the image block report library, The image patches displayed in the report library are selectable. The GUI is configured to calculate and display a similarity search block library, the calculation including: - Receive the user's selection of a specific block from the image blocks in the report library; - Identify all blocks from all received images that depict an organizational pattern similar to the selected block, wherein the similarity between the feature vectors of the selected block and the feature vectors of the selected block exceeds a threshold, by identifying all blocks from all received images that have been assigned feature vectors; - Display the similarity search block library, which selectively includes the identified blocks; and - Determine the number and/or score of blocks within the identified blocks that have been assigned the same label as the selected block, wherein the displayed similarity search block library further includes the determined number and/or score.