WO2019091509A1 - Method for analyzing the skin appearance of a human being in the context of skin cancer screening and apparatus for carrying out same - Google Patents

Abstract

The invention relates to an improved method for examining the skin by means of microscopic images while using a convolutional neural network (CNN). The aim of the invention is also to refine said method by additionally analyzing clinical images together with the microscopic images. Another aim of the invention is to devise an apparatus which is particularly suitable for carrying out said method. The advantages obtained by the invention are, in particular, that a rule-independent analysis of lesions can be carried out on the basis of the microscopic and clinical images that are produced in daily dermatological practice in the context of digital skin cancer screening. The invention is independent of the digital recording media, image formats and image resolutions used. The combination of microscopic and clinical images in the context of an analysis by means of a convolutional neural network (CNN) takes the analytical process as such to a new level and allows parallel analysis or individual analysis using the microscopic image or the clinical image.

Description

 description

Method for evaluating the skin image of a human in the context of skin cancer screening and device for its operation

The invention relates to a method for examining the skin of a subject, in particular in the context of a skin cancer screening. The invention further relates to an apparatus for carrying out the method.

There has recently been a significant increase in the incidence of skin cancer. The reason for this is in particular the trend to expose oneself increasingly to intensive sunshine. The most common fatal skin disease is the so-called malignant melanoma. Early diagnosis of skin disease and immediate therapy, which usually involves the surgical removal of the diseased tissue, provides good prospects for cure. However, the problem is the early detection. The most modern methods for the early detection of malignant melanomas are the digital reflected light microscopy as well as the analysis of individual nevi and the analysis of the entire skin image to show new and changed lesions, the so-called Total Body Mapping. A combination of reflected-light microscopy and total body mapping represents the gold standard of skin cancer screening and serves to detect changes in pigmented skin lesions even earlier and to avoid unnecessary excisions.

The typical method for the early detection of malignant melanomas is the monitoring of suspicious naevi over time using digital reflected light microscopy. Images are taken with a special video camera and dermatoscopy attachment (eg FotoFinder medicam®) in 20-70x magnification and stored in a database. Follow-up monitoring allows early identification of developing melanomas and removes them at a stage where the chances of recovery are considered high. By comparing repeat exposures at intervals of three to six months, changes are diagnosed in time to specifically excise suspicious lesions. The evaluation of the lesions is the sole responsibility of the dermatologist and depends on his expertise. On average, dermatologists achieve a sensitivity of approx. 80% and a specificity of approx. 80% in early detection of skin cancer.

The "sensitivity" (true positive rate of a test) refers to the proportion of test-positive persons among all patients in a sample, d. H. the likelihood of diagnosing the patient as ill with a diagnostic test. A high sensitivity is sought when a disease with high security is to be excluded.

The "specificity" (true-negative rate of a test) describes the proportion of test-negative persons among all non-sufferers of a random sample, ie the probability with one diagnostic test to correctly identify non-sufferers. A high specificity is sought if a disease is to be confirmed with great certainty.

 The distinction between benign birthmarks and malignant melanoma requires years of clinical experience. Even specialists can not always decide with certainty on the basis of the clinical picture, whether it is a melanoma or a conspicuous birthmark. For the light microscopic assessment of birthmarks, a number of different scores (ABCD Rule, 7 Point Rule, 3 Point Rule) have been developed in recent years to assist in the diagnosis of melanoma, especially for dermatologists in training, and less experienced in this field Dermatologists and difficult to diagnose lesions. The evaluation of birthmarks for their malignancy is further supported by numerous computer-assisted expert systems. For example, the "Tübinger Mole Analyzer" (developed under the direction of the University Skin Clinic Tübingen) is a tool to underpin the diagnosis of the dermatologist.Microscopic images of lesions are evaluated according to certain image recognition parameters and these values are compared with values of a reference database (Score) gives the dermatologist further clues to establishing his final diagnosis.

 First of all, the image analysis software uses a filter to remove any image interference caused by the accidental absorption of small air bubbles or caused by hair. The next step is a fully automated edge search. The software describes the lesion as precisely as possible using a specially developed procedure, taking into account gradients and changes in brightness. In a study carried out at the University Skin Clinic Tübingen, the most important image-analytical parameters were investigated, which enable a distinction between melanoma and birthmark. With a complex model that includes the size of the lesion, special color values and a texture measure, probability statements can be used to diagnose the melanoma. Two models were adapted: one for small lesions that are fully imaged, the other for large lesions protruding beyond the edge of the image. For 605 small lesions the sensitivity is 80 percent, the specificity 82 percent; Sensitivity is 88 percent and specificity 83 percent for the 232 large lesions. Thus, the system achieves approximately the same diagnostic quality as that of a specialist. Although the specificity and sensitivity demonstrated in a study reached good levels, this expert system is disadvantageous in that the use of the expert system is only validated for one recording medium (camera). Although lesions recorded with different camera models (video camera, digital camera, etc.) can also be evaluated, the results are not validated.

US Pat. No. 7,994,651 discloses a device and a method that by means of a special computer-aided imaging system (MelaFind®) can for the first time non-invasively record, display and evaluate data of a conspicuous liver spot from deeper skin layers of up to 2.5 mm. The multispectral use of polarized light (10 different wavelengths from 430 to 950 nm) makes it possible to detect conspicuous growth patterns below the skin surface extremely precisely (resolution up to 20 pm) and the degree of such To assess conspicuousness extremely precisely (sensitivity of 98.3%). However, an approval study for the US market found that the specificity is relatively low, ie the system often classifies lesions as malignant, which later turned out to be harmless in the histological evaluation. This system offers neither the physician nor the patient added value in skin cancer screening, as almost every birthmark is classified as a risk mother.

Disadvantage of all expert systems is that they are highly specialized in a narrow field of knowledge and do not recognize the limits of their competence. So if incorrect images are entered into an expert system, it can give diagnostic recommendations or risk assessments that are simply wrong because the source material was inappropriate. In this respect, an expert system always requires a competent user.

WO / 2011/087807 discloses a system and method for diagnosing melanoma from digital images taken by a remote user with a smartphone or digital camera and transmitted to an image analysis server communicating with a distributed network. The image analysis server includes a trained learning machine for classifying images of malignant and benign skin lesions. The user-provided image is pre-processed to extract size, shape, and color. These features are then processed with the trained learning machine to classify the suspicious lesion. The classification result is post-processed and a risk score is determined, which is transmitted to the remote user. A database associated with the server contains geographic matching reference information, i. The remote user is provided with contact information from doctors for the treatment of skin cancer.

The use of so-called artificial intelligence for pattern processing and diagnostic recommendations has been known since the early 1960's. Based on neurophysiology, the information architecture of the human and animal brain was analyzed. Modeling in the form of artificial neural networks illustrated how complex pattern processing can be made from a simple basic structure. Neuroinformatics has become a scientific discipline for studying these techniques. Unlike expert systems, this type of learning is not based on the derivation and application of rules, because the special abilities of the human brain are not reducible to a rule-based concept of intelligence.

The human brain is easy at image recognition. It takes no effort, for example, to distinguish a lion and a tiger from each other, to read a sign or to recognize the face of a human being. It just seems because the human brain is incredibly good at understanding images. By contrast, computer-aided image recognition is a complex task. In recent years, machine learning has made tremendous progress in tackling this difficult task. A form of artificial intelligence called Convolutional Neural Network (deep convolutional neural network) achieves one Reasonable performance in severe visual recognition tasks, with results in some areas that match or even surpass human performance.

 Typical models of a Convolutional Neural Network are QuocNet, AlexNet, Inception (GoogLeNet), BN-lnception-v2. Research has shown steady progress in image processing by validating results against ImageNet. ImageNet is a database of images used for research projects. Each picture is assigned to a noun. The nouns are hierarchically arranged by the Word Net project. Each noun has an average of more than 500 images.

 ImageNet has been used to train Convolutional Neural Networks (CNN) since its release in 2009 at the IEEE Conference on Computer Vision and Pattern Recognition, and represents an academic benchmark for image recognition.

Based on the good results of the Convolutional Neural Network in general image recognition, medicine, and in particular dermatology, is working to improve early detection of skin cancer with the help of a Convolutional Neural Network (CNN).

The aim is to work with clinical images of the skin to determine the malignancy of a lesion with the help of a CNN. The disadvantage here is that the available image material on clinical images is often poorly and not histologically confirmed in terms of their quality, i. to train a CNN is only partially suitable.

It is therefore the object of the invention to propose a method for the examination of the skin which has been improved on the background described above on the basis of microscopic images involving a CNN. This method should also be refined by the complementary evaluation of clinical images together with the microscopic images. A further object of the invention is to provide a device which is particularly suitable for carrying out said method.

The advantages achieved by the invention are in particular that a rule-independent assessment of lesions can be based on the microscopic and clinical images that are created in the daily dermatological practice in the context of digital skin cancer screening. The invention is independent of the digital recording media, image formats and image resolutions used. The combination of microscopic and clinical images as part of an evaluation using a Convolutional Neural Network (CNN) brings the evaluation process as such to a new level and enables a parallel evaluation or a single evaluation based on the microscopic image or the clinical picture.

With regard to the method, the object is achieved according to the invention by the features of claim 3 et seq. Further advantages and features of the invention will become apparent from the following description of a preferred embodiment with reference to the accompanying drawings.

Showing:

Fig. 1 is a schematic flow diagram of part of the method according to the invention; FIG. 2 shows a schematic flow diagram of a further part of the method according to the invention; FIG.

 3 shows a schematic representation of a device which is particularly suitable for carrying out said method;

The method according to the invention provides the following steps according to FIG. 1:

First, digital images of an examination body (patient) are recorded on the basis of a plurality of predetermined positions of the examination body with respect to the imaging system by the imaging system (S1 a). This process step is called Total Body Mapping.

 In a next method step, microscopic images are created by the user to the lesions found using a digital dermatoscope (S1 b). It is up to the user whether he creates microscopic images of all or only selected lesions. The microscopic images are assigned to the lesions in the total body mapping images by the prescriptive procedure (Si c).

In a further method step (S2), the microscopic images are transferred to the evaluation module containing a CNN, which is based, for example, on the GoogleNet Inception v4 CNN architecture. For this, the microscopic images are processed as follows:

The Inception architecture currently requires standardized delivery of 229 x 229 pixel images. The microscopic images are usually in a rectangular format and are recorded in a higher resolution. After the microscopic images can come from different recording devices, in addition, the resolution of each recording may vary. Adjusting the image ratio to a 1: 1 ratio for the CNN module can be achieved by cropping the image to a 1: 1 aspect ratio or by filling the rectangular image area to a 1: 1 aspect ratio. However, it is disadvantageous that when cutting important image information of the lesion could be lost if it was not centered, so was taken in the center. However, filling the image area to an image ratio of 1: 1 has the disadvantage that the image becomes even larger and is filled with image information that is not relevant for the evaluation, since it is subsequently added. On the other hand, the resolution must then be reduced to 229 x 229 pixels, so that by filling and Subsequent compressing still an additional quality loss takes place. In the case of both aforementioned processing steps, important image information would therefore be lost for the evaluation.

The method step on which this invention is based therefore provides a combined procedure. First, the image is filled to an image ratio of 1: 1 (so-called padding). The image is filled with black pixels on a square image format. Subsequently, this image is cropped (so-called cropping), so that they can be taken over in terms of format and resolution of the CNN evaluation module. Depending on the size and resolution, a large number of image crops with a picture ratio of 1: 1 and a resolution of 229 x 229 pixels are created from a microscopic image. As a result, no loss of quality must be accepted. If it is a melanocytic lesion that is completely imaged in the image area of the microscopic image, the lesion can be excised from the image by means of automatic edge recognition and buffered in a new data set. The aforementioned processing steps S2a and S2b are then subsequently carried out on this data record, with the advantage that less image information is lost in the process.

A CNN consists of one or more convolutional layers, often with a sub-sampling layer followed by one or more fully connected layers as in a conventional neural network. The design of a CNN is based on the visual cortex in the brain. The visual cortex contains many cells responsible for detecting light in small overlapping subregions of the visual field. These are called receptor fields. The cells act as local filters for the input area, while more complex cells have larger receptor fields. The folding layer in a CNN performs the same function as the cells in the visual cortex. Each element of a layer receives inputs from different elements that are in close proximity to the previous layer and thus are quasi-local receptor fields. With local receptor fields, elementary traits such as edges, endpoints, corners, etc. can be extracted, which are then combined by means of higher layers. In the conventional pattern / image recognition model, a manually developed feature extraction tool collects relevant information from the input and removes irrelevant components. The extraction tool is followed by a trainable classifier, a conventional neural network, which classifies feature vectors into classes. In a CNN, on the other hand, folding layers assume the function of the feature extraction tool. These were not developed manually, but a training process has determined the appropriate filter weights. The convolution layers can extract local features because they restrict the receive fields of the hidden layers to the local area.

The CNN structure used in this invention consists of 27 layers, the first layer representing the input image and the last layer the classification and a confidence value. In the layers in between, certain filter methods are applied to different pixel settings to extract as much information from the images as possible. Some filters extract certain averages or maximums. The model is generalized over the penultimate layer. In order to use a CNN network for the detection and evaluation of skin lesions, one must train the CNN network on it. CNN networks work best with large datasets. For this purpose, a very well-trained data set (over 1.2 million images) with excellent performance with regard to general image recognition is used and adapts to the new task for the detection of skin lesions. This method is referred to as "transfer learning." For this purpose, the existing model is initialized with adjusted weighting values, so that it is finally able to classify a lesion.To ensure that this method works reliably, the specific training of the invention was used Over 25,000 dermatoscopic images of skin lesions were used, which were diagnosed as either malignant or benign by histology or by follow-up without change.

 These image sections produced according to S2a and S2b are now transferred by the CNN evaluation module (S3) and evaluated with regard to their malignancy (S4a). The image sections are in each case entered several times into the CNN evaluation module in different orientations, i. in different angles and rotations.

These partial results are then calculated to a final result (S4b).

The final result is then displayed in the form of a weight value for the affected lesion on the output unit for the user.

In the context of the evaluation according to S4a and S4b, it is irrelevant in which magnification factor the microscopic images were taken. For example, dermatoscopic images with 20x optical magnification can be evaluated in the same way as images with 70x magnification.

After a microscopic image is always associated with a clinical image, a partial detail is created and stored by the processing unit in a supplementary method step (see FIG. 2) from the clinical image. The extraction is performed by an image recognition and processing function (software) installed on the processing unit. If this detail is subsequently displayed in full resolution, the dermoscopic structures of the lesion, such as color, diameter, structure, and limitation, can be easily recognized on the basis of this clinical image. Subsequently, this record is provided for training the CNN by being entered into the CNN according to the usual procedure (S6).

In this respect, the inventive method can also be used in addition to clinical images, because after appropriate training effort based on the clinical pictures of a lesion already made a risk assessment in terms of malignancy.

In order to confer further advantages to the method, it is particularly desirable if, in addition to the malignancy of the lesion, it is also determined whether the lesion is a melanocytic or a non-melanocytic lesion. Having the demarcation between melanocytic and non-melanocytic lesions is expensive for the doctor, a corresponding evaluation by the method according to the invention provides a clear added value for the medical diagnosis.

 In dermatology, the term melanocytic lesion refers to a skin area that differs in color from its surroundings. Responsible for the mostly brownish color are the pigment cells of the skin - the melanocytes. This lesion is called a nevus.

Non-melanocytic lesions are understood to be due to various factors related to changes in the epidermis. The causes are especially chronic light damage or infection with certain viruses in question. Compared to the melanocytic lesions, they are delineated in a multi-step procedure. Common non-melanocytic lesions include, for example, actinic keratosis or Bowen's disease, a precursor to the spinal biome, and basal cell carcinomas.

 Actinic keratoses may be dark in color, so the doctor may initially assume a melanocytic lesion.

The method is analogous to the method for determining the weighting value in terms of malignancy. As shown in Figure 1, the microscopic images of the lesion are prepared for transfer to the CNN and transferred to it. The CNN has previously been trained on the basis of training data sets by means of images of melanocytic and non-melanocytic lesions and can issue a weighting value in this regard.

The object according to the invention is achieved by a computerized device which is equipped with a receiving unit for receiving clinical images of skin lesions as well as the associated microscopic images of the skin lesion of a human or animal. Further, the device has a processing unit configured to process the received clinical images of skin lesions and the associated microscopic images of the skin lesions, ie to prepare the images for delivery to a Convolutional Neural Network (CNN). Further, the device has a provisioning unit configured to hold a Convolutional Neural Network (CNN). Since the operation of a Convolutional Neural Network (CNN) requires significant computing power, the provisioning unit is typically spatially separate from the receiving and processing unit and the various units are connected via a network line or the Internet. The receiving and processing unit can for example consist of a commercially available Total Body Mapping System (eg FotoFinder bodystudio ATBM), as disclosed in DE 20 2013 007 374U1. It uses a high-resolution digital camera (resolution 24 megapixels or higher) to create the clinical images of the human skin of individual body regions. The processing unit uses the high-resolution images to extract individual lesions from the clinical image and provide them in their full resolution for further processing. The extraction is performed by an image recognition and processing function (software) installed on the processing unit. If this detail is then displayed in full resolution, you can already use this clinical picture of the dermoscopic structures of the lesion such as color, diameter, structure, limitation are well recognized. Since experience shows that the resolution of future digital cameras will continue to increase, the result of this procedure will increasingly approximate or correspond to a dermatoscopic image. The extracted images are saved as a new record and linked to the associated clinical record. These two data records can then be transferred to the CNN for further evaluation. The data set of the clinical image can be used to train the CNN, the result of the evaluation of the associated microscopic image being transmitted in the form of the weighting value. This weighting value defines the classification of the clinical picture as benign or malignant.

 In a further advantageous embodiment, the histological findings for the lesion can also be transferred. If a lesion is classified by the doctor as malignant and surgically removed, the malignancy of the lesion is usually clarified as part of a histological examination. The result of the histological examination can be linked to the record of the microscopic image and thus also to the clinical recording of the lesion, thus increasing the validity of the data set as a training dataset for the CNN.

Findings a) Dermatologist level of skin cancer with deep neural networks "Andre Esteva, Brett Kuprel (..), Nature 542, 1 15-1 18 02 February 2017)

 b) Ciresan, D., Meier, U., Schmidhuber, J.,:. Multi-column deep neural networks for image classification. In: Conference on Computer Vision and Pattern Recognition, IEEE (2012) 3642-3649;

 c) Krizhevsky, A., Sutskever, I., Hinton, G.E .: Imagenet classication with deep convolutional neural networks. In: Advances in Neural Information Processing Systems. Volume 1. (2012) 4;

Claims

claims A computerized image data processing apparatus, the computerized apparatus comprising:  a receiving unit configured to receive clinical images of skin lesions and associated microscopic images of the skin lesion of a human or animal,  a processing unit configured to process the received clinical images of skin lesions and the associated microscopic images of the skin lesions, and  a deployment unit configured to maintain a convolutional neural network (CNN) for processing the clinical images of skin lesions and associated microscopic images of the skin lesions, and  using a Convolutional Neural Network (CNN)), skin lesions are classified according to the microscopic images for their malignancy and  determined a weight value for the lesion and  then provide the clinical images of the lesion by reference to the determined weight value of the associated microscopic image to train the CNN. 2. A computerized device according to claim 1, characterized in that  classified using a Convolutional Neural Network (CNN) skin lesions based on the micrographs regarding their melanocytosis and  a weighting value for the lesion is determined. 3. A method of processing image data, the method comprising:  Receiving a microscopic image of the skin lesion of a human or animal providing a convolutional neural network (CNN) for processing the received image data and  Process microscopic image using the Convolutional Neural Network (CNN) to classify the malignancy of the skin lesion and  Calculate and output a weighting value for the lesion. 4. A method of processing image data, the method comprising: Receiving a microscopic image of a human or animal skin lesion; providing a convolutional neural network (CNN) to process the received image data; and Processing the received image data using the Convolutional Neural Network (CNN) to determine the melanocyte of the lesion. and  Calculate and output a weighting value for the lesion. 5. A method according to claim 3, wherein the method comprises:  Receiving a magnified portion of the clinical image of the lesion and the associated microscopic image of the skin lesion;  Providing a Convolutional Neural Network (CNN) to process the received and associated image data in parallel as well  Train the Convolutional Neural Network (CNN) on the basis of the clinical picture and the associated microscopic image as well as the determined weight value. A method according to claims 3, 4 and 5 wherein the method comprises:  Padding the image to be transferred to the Convolutional Neural Network (CNN) to an aspect ratio of 1: 1 (so-called padding) and  Cropping the image (so-called cropping) into partial images with a resolution suitable for transfer to a Convolutional Neural Network (CNN). A method according to claim 6, wherein the method comprises:  Cropping the image (so-called cropping) into partial images with a resolution of 223 x 223 pixels. 8. The method according to any one of claims 3 to 5, characterized in that the weighted total value is stored in the processing unit.