TWI882800B - Ai-based method, computer program, and computer readable medium for identifying genera and species of mold - Google Patents

Abstract

The present invention relates to an AI-based method, computer program, and computer readable medium for identifying genera and species of mold. First, an image of mold to be recognized is obtained, and then morphological characters of the mold to be recognized are acquired through image processing. These morphological characters are then input into a convolutional neural network (CNN) model, which is established by CNN learning based on sample morphological characters of multiple sample mold images. The CNN model identifies and classifies the genus of the mold to be recognized based on the morphological characters thereof, and then the image of the mold to be recognized is input into an object detection model. The object detection model is established by object detection learning after frame selecting a to-be-detected object on each sample mold image as a label. The object detection model identifies and classifies the species of the mold to be recognized within the genus of the mold identified and classified by the CNN model, based on the image of the mold to be recognized. This assists laboratory personnel in quickly identifying the genus and species of the mold and thereby obtaining the name of the mold.

Description

Method, computer program, and computer-readable medium for identifying genus and species of mold using artificial intelligence

The present invention is a method, computer program, and computer-readable medium for identifying the genus and species of mold using artificial intelligence, and particularly refers to a two-stage invention for identifying and classifying the genus and species of mold using a convolutional neural network model (CNN model) to identify and classify the mold genus of the mold to be identified, and then using an object detection model (Object Detection) to identify and classify the mold species of the mold to be identified under the classified mold genus.

Taiwan's hot, humid and warm climate provides a favorable environment for the growth of mold. In addition to the daily environment including food, clothing, daily necessities, furniture, and housing that may be moldy, mold infection in the human body can sometimes cause fatal consequences.

For the treatment of fungal infection, the current clinical identification method of fungal species in hospitals is mainly for inspectors to identify them by morphology under a microscope. The morphology of the fungus to be identified is based on the image obtained by slide culture, and the morphological characteristics of the fungus to be identified are used as the identification and classification of the fungus genus, species, and name.

However, mold growth takes a long time, which makes the process of using slide culture to obtain images very time-consuming. In addition, the identification of mold morphological characteristics requires the use of experienced inspectors with mold testing expertise. The shortage of inspectors and lack of experience often lead to misjudgments, affecting the accuracy of identification and classification.

As shown in Figure 5, in clinical applications, a simple tape sticking method is used to directly obtain the mold strain and obtain the mold image on the tape, thereby replacing the time-consuming slide culture technology to improve the timeliness. However, the use of the tape sticking method may damage the appearance of the mold strain, affecting its identification rate.

In clinical applications, mass spectrometry will be further used to assist in the classification of fungal species, but the success rate of mass spectrometry identification is not high. The inventors have measured that the success rate in identifying Aspergillus is about 86%, the success rate in identifying Cladosporium is about 40.5%, the success rate in identifying Trichophyton is about 83.3%, and the success rate in identifying Penicillium is about 44.3%. Therefore, mass spectrometry identification is of limited help in accurately identifying and classifying fungal species.

With the development of artificial intelligence, the use of artificial intelligence to assist inspectors in identifying mold species has improved accuracy. For example, the "Classification of fungal genera from microscopic images using artificial intelligence" proposed by Md Arafatur Rahman et al., which used a convolutional neural network (CNN) model to identify 89 fungal genera; the "Automatic Fungi Recognition: Deep Learning Meets Mycology" proposed by Lukáš Picek et al., which trained a CNN model using the Danish Fungi 2020 (DF20) dataset that produces finely classified data, and produced species-level labels with low recognition error output; the "Detection of mold on the food surface using YOLOv5" proposed by Fahad Jubayer et al., which used the YOLO object detection model for mold detection.

As shown in the sixth and seventh figures, since the identification and classification of molds mainly adopts morphology, although CNN can quickly identify the mold genus based on morphological features, the mold species under the same mold genus have smaller differences in morphological features. The inventors have tested that the accuracy of using CNN to identify and classify mold species is low. Although object detection (YOLO) can detect objects by selecting them (for example, selecting morphological features of details such as spores or spore sacs as objects), AI recognition requires a lot of computing resources, and only using object detection to select objects in the hope of finding the correct mold species under many different mold genera based on morphological features is like looking for a needle in a haystack, and it is easy to make identification errors.

In order to make artificial intelligence identify the genus and species of molds based on morphology like humans, and to improve the accuracy of identification and classification, the present invention proposes a method for identifying the genus and species of molds using artificial intelligence, the steps of which include: a processing unit obtains an image of a mold to be identified, and obtains at least one morphological feature of the mold to be identified by image processing; the morphological feature is input into a convolutional neural network model (CNN model), the convolutional neural network model is established based on at least one sample morphological feature of a plurality of sample mold images through convolutional neural network learning, and the convolutional neural network model identifies and classifies a mold genus of the mold to be identified based on the morphological feature. The mold image to be identified is input into an object detection model (Object Detection). The object detection model is established by selecting the object to be detected as a label on the plurality of sample mold images through object detection learning. The object to be detected is the detailed morphological features of the mold. The object detection model identifies and classifies a mold species of the mold to be identified according to the mold image to be identified under the classified mold genus.

Furthermore, the fungus to be identified was obtained by the tape sticking method.

Furthermore, a plurality of images of the mold to be identified under different viewing fields are obtained for identification and classification. Furthermore, a first identification threshold is set on the processing unit, and the processing unit calculates the proportion of the plurality of images of the mold to be identified that are classified into the same mold genus, and finds the mold genus with the highest proportion. When the proportion of the mold genus with the highest proportion is higher than the first identification threshold, the mold genus is used as the classification result of the mold to be identified. When the proportion is lower than the first identification threshold, an unidentifiable message is output; preferably, the first identification threshold is set at 60% or 70%. Furthermore, a second identification threshold is set on the processing unit, and the processing unit calculates the ratio of the multiple mold images to be identified that are classified into the same mold species, and finds the mold species with the highest ratio. When the ratio of the mold species with the highest ratio is higher than the second identification threshold, the mold species is used as the classification result of the mold to be identified. When the ratio is lower than the second identification threshold, an unidentifiable message is output; preferably, the second identification threshold is set at 60% or 70%.

Furthermore, the morphological characteristics include one or a combination of color, texture, shape, size and growth pattern.

Furthermore, the convolutional neural network model is established by combining transfer learning with the convolutional neural network and adding attention mechanism learning.

Furthermore, the object detection model is selected from one of the following: YOLO model, Faster-RCNN model, and SSD model (Single Shot Multi-box Detector).

The present invention further proposes a computer program for installation in a computer, and the computer program executes the aforementioned method of identifying the genus and species of mold using artificial intelligence.

The present invention further proposes a computer-readable medium that stores the aforementioned computer program.

Based on the above technical features, the following effects can be achieved:

1. It solves the problem of misjudgment caused by the shortage of professional inspectors and lack of experience, and improves the accuracy of identification and classification of mold genera and mold species.

2. The fungus to be identified is obtained using a simple tape sticking method, replacing the time-consuming slide culture method. For clinical applications in medicine, it can improve the timeliness of testing.

3. Obtaining multiple images of the fungus to be identified from different fields of view for identification and classification can avoid misclassification caused by using a single image and improve accuracy.

4. The two-stage identification procedure of "first identify the genus, then identify the species" is adopted. When applied in clinical medicine, if the classification result of the identified fungus genus is unidentifiable, it will be directly classified as others (MOLD) and the fungus species identification will no longer be performed. If the fungus genus is identified, the fungus species can be further identified by object detection.

5. Combining CNN and object detection can achieve the effect of quickly and accurately classifying mold genera and mold species. Among them, CNN first identifies the mold genus, which is like narrowing the search range from a large ocean to a small pool. Then object detection identifies the detailed morphological features of the mold species (such as the morphological features of spores or sporangia) in the small pool of the same mold genus, and the recognition accuracy is higher.

6. When CNN incorporates transfer learning and adds an attention mechanism, the recognition accuracy can be improved by 3.4% to 23.2%.

1: Convolutional neural network model

2: Object detection model

3: Image of mold to be identified

4: User interface

5: Processing unit

6: Identification and classification results

[Figure 1] is a functional block diagram of an embodiment of the present invention.

[Figure 2] is a flow chart of an embodiment of the present invention.

[Figure 3] is a microscopic image of multiple mold images to be identified under different viewing fields in an embodiment of the present invention.

[Figure 4] is a schematic diagram of the user interface of an embodiment of the present invention.

[Fifth Figure] is a microscopic image of the mold to be identified in the embodiment of the present invention, comparing its appearance using the slide culture method and the tape sticking method.

[Figure 6] is a micrograph comparing the morphological characteristics of different mold genera.

[Figure 7] is a micrograph comparing the morphological characteristics of different mold species under the same mold genus.

The following embodiments are only used to assist in explaining the method, computer program, and computer-readable medium of the present invention for identifying the genus and species of mold using artificial intelligence, and are not intended to limit the present invention.

Referring to the first and second figures, the present embodiment includes the following steps: based on multiple sample morphological features of multiple sample mold images, a convolutional neural network model 1 (CNN model) is established using convolutional neural network learning, wherein the sample morphological features include color, texture, shape, size, etc. The feature values of these sample morphological features can be obtained by known image processing, for example, the feature values of color and texture can be obtained based on pixel values (such as RGB values), and the feature values of shape and size can be obtained by using geometric contour recognition. In the present embodiment, the convolutional neural network model 1 is further combined with transfer learning by the convolutional neural network and an attention model is added. The initial convolutional neural network was built based on a learning mechanism, and the multiple sample mold images based on which the initial convolutional neural network was trained came from the Google image database. A total of 4108 images were used for training, including 1488 Aspergillus images (36.2%), 779 Cladosporium images (19%), 522 Penicillium images (12.7%), 487 Trichophyton images (11.9%), and 832 other mold images (20.3%). Among them, Aspergillus niger, which is common in clinical patient specimens, is the most common. niger) with 302 photos (accounting for 20.3% of Aspergillus), Aspergillus fumigatus with 238 photos (accounting for 16% of Aspergillus), Aspergillus terreus with 227 photos (accounting for 15.3% of Aspergillus), Aspergillus versicolor with 117 photos (accounting for 7.9% of Aspergillus), Aspergillus flavus with 111 photos (accounting for 7.5% of Aspergillus), other Aspergillus (Aspergillus sp.) has 493 images (accounting for 31.9% of Aspergillus); in the learning of convolutional neural networks, the training set accounts for 64%, the validation set accounts for 16%, and the test set accounts for 20%.

After selecting the objects to be detected on the multiple sample mold images as labels, an object detection model 2 (Object Detection) is established using object detection learning. For example, spores or spore sacs on the sample mold images are selected as the objects to be detected and used as labels for object detection learning. The object detection model 2 can be selected from one of the following: YOLO model, Faster-RCNN model, and SSD model (Single Shot Multi-box Detector). This embodiment uses YOLOv7 for testing. The feature values of object detection learning come from the color, texture, shape, size and growth mode of the objects to be detected. In object detection learning, the training set accounts for 64%, the validation set accounts for 16%, and the test set accounts for 20%.

After obtaining the convolutional neural network model 1 and the object detection model 2, they can be used to identify and classify the genus and species of a mold to be identified.

As shown in the third figure, the mold to be identified is obtained by the tape sticking method, and a plurality of mold images 3 of the mold to be identified in different fields of view are obtained by the photographic unit. The third figure shows four mold images 3 of the mold to be identified in different fields of view.

Referring to the first, second and fourth figures, the image 3 of the mold to be identified is input into a processing unit 5 through a user interface 4, wherein the processing unit 5 may be a cloud server provided by a server, the user interface 4 may be a network interface, or the processing unit 5 may be a stand-alone computer, or the user interface 4 may be a stand-alone computer interface. The processing unit 5 obtains the morphological features of the mold to be identified by image processing, including color, texture, shape, size, etc., and then the convolution neural network model 1 identifies and classifies a mold genus of the mold to be identified according to the morphological features. The morphological features are equivalent to the mold genus classified by the inspector according to morphology, so the convolution neural network model 1 can effectively identify and classify the correct mold genus. Since the present embodiment uses a plurality of mold images 3 to be identified under different fields of view for identification and classification, a first identification threshold is set on the processing unit 5. The first identification threshold is set at 60% or 70%. In the present embodiment, it is set at 60%. The processing unit 5 calculates the proportion of the plurality of mold images 3 to be identified classified into the same mold genus by the convolutional neural network model 1, and finds the mold genus with the highest proportion. When the proportion is higher than the first identification threshold, the mold genus is used as the identification and classification result 6 of the mold to be identified. When the proportion is lower than the first identification threshold, the mold genus is used as the identification and classification result 6 of the mold to be identified. When the value is set, an unrecognizable message is output. According to the test, the overall identification and classification accuracy of mold genera is about 94.9%, of which the identification and classification accuracy of Aspergillus is about 98.1%, the identification and classification accuracy of Cladosporium is about 96.3%, the identification and classification accuracy of Penicillium is about 92.5%, the identification and classification accuracy of Trichophyton is about 92.6%, and the identification and classification accuracy of other mold genera is about 90.6%.

Referring to Table 1 below, this embodiment compares the difference in recognition accuracy when using a convolutional neural network and when using an attention model in combination. It can be found that when various convolutional neural networks are combined with an attention model, the recognition accuracy can be improved by 3.4% to 23.2%.

After obtaining the mold genus of the mold to be identified, the object detection model 2 classifies the mold to be identified as a mold species according to the mold image 3 under the mold genus classified by the convolution neural network model 1. Since the morphological features of the same mold genus are relatively small, inspectors often classify mold species according to the morphology of spores or sporangia. Therefore, the object detection frame selects spores or sporangia as the object to be detected, which can effectively identify the correctly classified mold species. Similarly, since the present embodiment uses multiple mold images 3 to be identified under different fields of view for identification and classification, a second identification threshold is set on the processing unit 5. The second identification threshold is set at 60% or 70%. In the present embodiment, it is set at 60%. The processing unit 5 calculates the proportion of the multiple mold images 3 to be identified that are classified into the same mold species by the object detection model 2, and finds the mold species with the highest proportion. When the proportion is higher than the second identification threshold, the mold species is used as the identification and classification result 6 of the mold to be identified. When the proportion is lower than the second identification threshold, an unidentifiable message is output.

Referring to Table 2 and Table 3 below, on the object detection model 2, this embodiment further obtains a total of 135 images 3 of molds to be identified under different views of 36 molds to be identified.

Table 2 below shows that each of the 135 mold images 3 to be identified is identified and classified by the object detection model 2, and the recognition accuracy under different thresholds set in the object detection model 2 itself. It can be found that the recognition accuracy of the mold species of the mold to be identified under different thresholds set in the object detection model 2 itself can reach more than 90%.

Table 3 below shows the recognition accuracy of each to-be-identified mold when multiple to-be-identified mold images 3 with different fields of view are input for identification and classification and different second identification thresholds are set. It can be found that when each to-be-identified mold is identified and classified with multiple to-be-identified mold images 3 with different fields of view and the second identification threshold is set at 60% or 70%, the recognition accuracy of the mold species of the to-be-identified mold is 100%.

This embodiment subsequently stores the mold image 3 to be identified in the learning database of the convolutional neural network and object detection as a sample mold image, so that the identification and classification accuracy of the convolutional neural network model and the object detection model can be continuously improved.

The computer program installed to implement the above-mentioned method of using artificial intelligence to identify the genus and species of mold can be stored in the cloud for downloading, or can be stored in a computer-readable medium.

Combined with the description of the above embodiments, the operation, use and effects of the present invention can be fully understood. However, the above embodiments are only preferred embodiments of the present invention and cannot be used to limit the scope of implementation of the present invention. In other words, simple equivalent changes and modifications made according to the scope of the patent application and the content of the invention description are all within the scope of the present invention.

Claims (12)

A method for identifying the genus and species of mold using artificial intelligence, the steps include: A processing unit obtains an image of a mold to be identified, and obtains at least one morphological feature of the mold to be identified by image processing; the morphological feature is input into a convolutional neural network model (CNN model), the convolutional neural network model is established based on at least one sample morphological feature of a plurality of sample mold images through convolutional neural network learning, and the convolutional neural network model identifies and classifies a mold genus of the mold to be identified based on the morphological feature; The mold image to be identified is input into an object detection model (Object Detection), the object detection model is established by selecting the object to be detected as a label on the plurality of sample mold images through object detection learning, the object to be detected is the detailed morphological features of the mold, and the object detection model identifies and classifies a mold species of the mold to be identified according to the mold image to be identified under the classified mold genus. A method for identifying the genus and species of fungi using artificial intelligence as described in claim 1, wherein the fungi to be identified are obtained by a tape sticking method. A method for identifying the genus and species of fungi using artificial intelligence as described in claim 1, wherein a plurality of images of the fungi to be identified under different fields of view are obtained for identification and classification. A method for identifying the genus and species of mold using artificial intelligence as described in claim 3, wherein a first identification threshold is set on the processing unit, the processing unit calculates the ratio of the multiple mold images to be identified that are classified into the same mold genus, and finds the mold genus with the highest ratio. When the ratio of the mold genus with the highest ratio is higher than the first identification threshold, the mold genus is used as the classification result of the mold to be identified. When the ratio is lower than the first identification threshold, an unidentifiable message is output. A method for identifying the genus and species of mold using artificial intelligence as described in claim 4, wherein the first identification threshold is set at 60% or 70%. A method for identifying the genus and species of mold using artificial intelligence as described in claim 3, wherein a second identification threshold is set on the processing unit, the processing unit calculates the ratio of the multiple mold images to be identified that are classified into the same mold species, and finds the mold species with the highest ratio. When the ratio of the mold species with the highest ratio is higher than the second identification threshold, the mold species is used as the classification result of the mold to be identified. When the ratio is lower than the second identification threshold, an unidentifiable message is output. A method for identifying the genus and species of mold using artificial intelligence as described in claim 6, wherein the second identification threshold is set at 60% or 70%. A method for identifying the genus and species of mold using artificial intelligence as described in claim 1, wherein the morphological characteristics include one or a combination of color, texture, shape, and size. A method for identifying the genus and species of mold using artificial intelligence as described in claim 1, wherein the convolutional neural network model is established by combining a convolutional neural network with transfer learning and adding an attention mechanism learning. A method for identifying the genus and species of mold using artificial intelligence as described in claim 1, wherein the object detection model is selected from one of the following: YOLO model, Faster-RCNN model, and SSD model (Single Shot Multi-box Detector). A computer program is installed in a computer and executes the method of identifying the genus and species of mold by artificial intelligence as described in any one of claims 1 to 10. A computer-readable medium stores the computer program of claim 11.