JP2022033035A - Intelligent identification method and system for calcium oxalate crystals based on microscopic images - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an intelligent identification method for calcium oxalate crystals, which solves problems of error, time and work efficiency due to manual identification by an operator. SOLUTION: This is a step of acquiring training data of calcium oxalate crystals, and the training data includes a step including a plurality of crystal types of calcium oxalate crystals and training data using a deep convolution neural network. The step of obtaining a deep convolutional neural network model for identifying calcium oxalate crystals and the feature matching of the microscopic image of the collected sample using the deep convolutional neural network model are performed, and the crystal type of the calcium oxalate crystal contained in the collected sample is performed. This includes a step of generating and outputting the identification result of the calcium oxalate crystal of the collected sample according to the crystal type of the calcium oxalate crystal contained in the collected sample. [Selection diagram] Fig. 1

Description

The present invention relates to the technical field of plant Chinese herbal medicine identification, and particularly to an intelligent identification method and system of calcium oxalate crystals based on microscopic images.

Chinese medicine is a drug that plays a role in prevention, treatment, diagnosis, rehabilitation and health maintenance under the guidance of Chinese medicine theory. Chinese herbs are mainly derived from medicinal plants, medicinal animals and natural medicines including medicinal minerals and their processed products. Of these, medicinal plants account for 87% of Chinese herbal resources, and it can be seen that medicinal plants are extremely important in the field of Chinese herbs.

Calcium oxalate crystals are an important chemical component in medicinal plants. Calcium oxalate crystals are widely present in plant cells as common cytoactive substances and are usually of multiple types such as columnar crystals, clustered crystals, acicular crystals, prismatic crystals and sandy crystals. Has a crystal form of. The structural morphology of calcium oxalate crystals is stable, and the presence or absence of crystals and the different morphologies, sizes and distributions of crystals show specificity in the tissue structure of biopharmaceuticals of various families, genera and species. As is clear from previous studies, the rhizomes of plants of the Liliaceae, Orchid, Satoimo, and Yam families contain needle-like crystals and needle-like bundle crystals, and plants of the Ranunculaceae, Polygonaceae, and Araliaceae families. Cluster-like crystals are included, and sand-like crystals are often contained in plants of the family Polygonaceae and Liliaceae, and prismatic crystals are often contained in plants of the family Polygonaceae and Polygonaceae. As can be seen from the above contents, the microstructural information of calcium oxalate crystals can be an important basis for identifying and studying Chinese herbal medicine varieties under a microscope and classifying and searching unknown crude drugs. In recent years, the specificity of calcium oxalate crystals has also been used in the field of herbal testing, providing technical support for identifying the authenticity of herbal medicines and drinks.

Usually, a method for microscopic identification of Chinese herbs is required to identify calcium oxalate crystals. The microscopic identification method is one of the four main means for identifying Chinese herbal medicines, and the microscopic identification of Chinese herbal medicines is the use of a microscope for crude drug (drinking piece) sections, powders, surfaces, dissociated tissues and sheets, and drinking pieces. This is a method of observing a pharmaceutical product containing a powder and identifying crude drugs according to characteristics such as tissues, cells and inclusions. This identification method is traditional, efficient, fast, economical, environmentally friendly, especially useful when the active ingredient of Chinese herbal medicine is not clear, in qualitative, quantitative, and even localization. It has a predetermined effect. Microidentification of calcium oxalate crystals needs to be determined according to the characteristics of the relevant substances in plant tissues, cells and inclusions.

However, in most conventional methods for microscopic identification of Chinese herbal medicines for calcium oxalate crystals, the operator manually operates the microscope and observes the microscopic characteristics of related substances in plant tissues, cells and inclusions under the microscope. The operator manually draws this feature and performs feature matching with a conventional specific calcium oxalate crystal to artificially identify the calcium oxalate crystal in the medicinal plant. In such a method, since the operator manually operates the method, there are problems that an error due to subjective factors is large, drawing takes time, and work efficiency is low.

The present invention solves the problems of the prior art that the Chinese herbal medicine micro-identification method for calcium oxalate crystal requires manual identification by an operator, large errors due to subjective factors, long drawing time, and low work efficiency. To provide an intelligent identification method, system, and readable storage medium for calcium oxalate crystals based on microscopic images.

In order to achieve the above object, according to the first aspect of the present invention, the present invention proposes an intelligent identification method of a calcium oxalate crystal based on a microscopic image, and the intelligent identification method of the calcium oxalate crystal is described.
It is a step to acquire training data of calcium oxalate crystals, and the training data includes a step containing multiple types of crystal types of calcium oxalate crystals.
Steps to train training data using a deep convolutional neural network to obtain a deep convolutional neural network model for identifying calcium oxalate crystals,
A step to obtain the crystal type of calcium oxalate crystals contained in the sample by performing feature matching on the microscopic image of the sample using a deep convolutional neural network model.
It includes a step of generating and outputting the identification result of the calcium oxalate crystal of the collected sample according to the crystal type of the calcium oxalate crystal contained in the collected sample.

Preferably, the step of acquiring training data for calcium oxalate crystals is
Steps to select multiple crude drugs containing various crystal types of calcium oxalate crystals,
Steps to take microscopic images of calcium oxalate crystals in each herbal medicine,
Steps to mark the crystal type of calcium oxalate crystals in the microscopic image,
Includes steps to generate training data for each calcium oxalate crystal using microscopic images and crystal types of each calcium oxalate crystal.

Preferably, the deep convolutional neural network includes a convolutional layer, a pooling layer and a fully connected layer.
The above step of training training data using a deep convolutional neural network to obtain a deep convolutional neural network model is
The step of inputting the microscopic image of the calcium oxalate crystal included in the training data into the convolution layer,
A step of performing a convolution operation on a microscopic image of a calcium oxalate crystal using a multi-layer convolution layer to obtain a characteristic image of the calcium oxalate crystal.
A step of performing feature compression on a feature image of a calcium oxalate crystal using a pooling layer to obtain crystal features corresponding to the feature image, and
A step of connecting the crystal features of all feature images corresponding to the same microscopic image using a fully connected layer and determining the crystal type of the microscopic image and the crystal features corresponding to the crystal type with a classifier.
Includes steps to create a deep convolutional neural network model for each type of calcium oxalate crystal using the crystal type and crystal characteristics of the microscopic image.

Preferably, the step of performing a convolution operation on a microscopic image of a calcium oxalate crystal using a plurality of convolution layers to obtain a characteristic image of the calcium oxalate crystal is the step.

Preferably, prior to the step of feature matching the microscopic images of the sample taken using a deep convolutional neural network model, the calcium oxalate crystal identification method is performed in a normal light source field and a polarized light source field of view using an automatic microscopic image collector. Including the step of taking each microscopic image of the collected sample of
The step of taking microscopic images of the sample collected in the normal light source field of view and the polarized light source field of view using the automatic microscope image collecting device is
A step of controlling the stage of the microscope using a control box so that horizontal motion is performed in a three-dimensional coordinate system according to a predetermined motion trajectory.
A Tep that controls the stage of the microscope using a control box to perform vertical movement in a three-dimensional coordinate system according to a predetermined movement trajectory, and tracks and focuses the sample collected on the stage in real time.
It comprises taking multiple microscopic images of a sample taken at predetermined time intervals using a digital image pickup device.

Preferably, the step of generating and outputting the identification result of the calcium oxalate crystal of the collected sample according to the crystal type of the calcium oxalate crystal contained in the collected sample is
When the deep convolutional neural network model acquires the crystal type of the calcium oxalate crystal contained in the collected sample, the crystal type information of the collected sample and the stitch crystal characteristic image of the collected sample are generated and output, or
If the deep convolutional neural network model does not acquire the crystal type of the calcium oxalate crystals contained in the sample, it includes the step of generating and outputting a stitch microscopic image of the sample.

Preferably, the step of generating a stitched microscopic image of a sample is
The step of calculating the transformation matrix between the image pixel coordinates and the spatial physical coordinates of each microscope image corresponding to the same sample collected using the stitch algorithm, and
A step of stitching all microscopic images of the same sample taken under the same coordinate system according to the transformation matrix to obtain a stitched microscopic image is included.

Preferably, after the step of generating and outputting the identification result of the calcium oxalate crystal of the collected sample, the method is described.
Steps to mark the crystal type of calcium oxalate crystals contained in the sample,
It further includes a microscopic image of the sample taken and the step of adding the marked crystal type to the training data.

According to the second aspect of the present invention, the present invention
It is a data acquisition module for acquiring training data of calcium oxalate crystals, and the training data includes a data acquisition module containing multiple types of crystal types of calcium oxalate crystals.
A neural network module for training training data using a deep convolutional neural network to obtain a deep convolutional neural network model for identifying calcium oxalate crystals, and
A feature matching module for acquiring the crystal type of calcium oxalate crystals contained in the sample by performing feature matching on the microscopic image of the sample using the deep convolutional neural network model.
A result generation module for generating the identification result of the calcium oxalate crystal of the collected sample according to the crystal type of the calcium oxalate crystal contained in the collected sample.
Further provided is an intelligent identification system for calcium oxalate crystals based on microscopic images, including a result output module for outputting the identification results of calcium oxalate crystals in a sample.

Preferably, the neural network module is
An image input submodule for inputting microscopic images of calcium oxalate crystals contained in the training data into the convolutional layer,
A convolution operation submodule for performing a convolution operation on a microscopic image of a calcium oxalate crystal using a multi-layer convolution layer and obtaining a characteristic image of the calcium oxalate crystal,
A feature compression submodule for performing feature compression on a feature image of calcium oxalate crystals using a pooling layer to obtain crystal features corresponding to the feature image,
A feature connection submodule for connecting the crystal features of all feature images corresponding to the same microscope image using a fully connected layer,
A crystal classification submodule for determining the crystal type of the microscopic image and the crystal characteristics corresponding to the crystal type by the classifier,
Includes a modeling submodule for creating a deep convolutional neural network model for each type of calcium oxalate crystal using the crystal type and crystal characteristics of the microscopic image.

Preferably, the system is
It also includes an imaging module for capturing microscopic images of the sample taken in the normal light source field of view and the polarized light source field of view using an automatic microscope image collector.
The image shooting module is
A motion control submodule for controlling the stage of a microscope using a control box to perform horizontal motion in a three-dimensional coordinate system according to a predetermined motion trajectory.
A focus control submodule for controlling the stage of the microscope using a control box to perform vertical movement in a three-dimensional coordinate system according to a predetermined motion trajectory, and tracking and focusing the sample collected on the stage in real time. When,
Includes an imaging submodule for taking multiple microscopic images of a sample taken at predetermined time intervals using a digital imaging device.

According to a third aspect of the present invention, a computer-readable storage medium is further provided, and the computer-readable storage medium stores a program of an intelligent identification method for calcium oxalate crystals, and the oxalate is stored. When executed by a processor, the program for intelligent identification of calcium crystals implements the steps of the calcium oxalate identification method described in any of the above technical solutions.

The technical solution proposed in this application obtains training data for calcium oxalate crystals containing multiple crystal types of calcium oxalate crystals and then trains the training data using a deep convolutional neural network. And to obtain a deep convolutional neural network model that can identify the characteristics of various types of calcium oxalate crystals for identifying calcium oxalate crystals, then harvested using a trained deep convolutional neural network model. Feature matching was performed on the microscopic image of the sample to obtain the crystal type of the calcium oxalate crystal contained in the collected sample, and the calcium oxalate crystal of the collected sample was obtained according to the crystal type of the calcium oxalate crystal contained in the collected sample. Generate and output the identification result.

As described above, the intelligent identification method of calcium oxalate crystals according to the present application obtains a deep convolution neural network model by training using an artificial intelligence method, and automatically characterizes the crystal characteristics of calcium oxalate crystals in the collected sample. The type of calcium oxalate crystal is automatically identified by matching the crystals, and the identification structure of the calcium oxalate crystal of the collected sample is output. In the artificial identification method, the operator manually operates the calcium oxalate crystal. It solves the problems of the prior art such as large error due to subjective factors, long drawing time, and low work efficiency.

In order to more clearly explain the technical solution of the embodiment or the prior art of the present invention, the drawings necessary for the description of the embodiment or the prior art will be briefly described below, but the drawings in the following description are clearly described. Are only a few embodiments of the invention, and one of ordinary skill in the art can obtain other drawings based on the structures shown in these drawings without the need for creative effort.
It is a schematic diagram of the application scenario according to the Example of this invention. It is a schematic flowchart of an example of the intelligent identification method of the calcium oxalate crystal based on the microscope image by the Example of this invention. It is a schematic flowchart of the training data acquisition method by the Example shown in FIG. It is a schematic flowchart of the training method of the neural network by the Example shown in FIG. It is a schematic flowchart of the method of generating the identification result by an Example shown in FIG. It is a schematic flowchart of the intelligent identification method of the calcium oxalate crystal based on another microscopic image by the Example of this invention. It is a schematic flowchart of the microscope image taking method by an Example shown in FIG. It is a schematic diagram of the movement path of the microscope by the example shown in FIG. 7. 6 is a microscopic image of calcium oxalate columnar crystals in a normal optical field according to an embodiment of the present invention. It is a microscope image of the calcium oxalate columnar crystal in the polarized field by the Example of this invention. 6 is a microscopic image of calcium oxalate clustered crystals in a normal optical field according to an embodiment of the present invention. 6 is a microscopic image of calcium oxalate clustered crystals in a polarized field according to an embodiment of the present invention. It is a microscopic image of a calcium oxalate needle-like crystal in a normal optical field according to an embodiment of the present invention. 6 is a microscopic image of a calcium oxalate needle-like crystal in a polarized field according to an embodiment of the present invention. 6 is a microscopic image of a calcium oxalate prismatic crystal in a normal optical field according to an embodiment of the present invention. 6 is a microscopic image of a calcium oxalate prismatic crystal in a polarized field according to an embodiment of the present invention. It is a microscope image of the calcium oxalate sand-like crystal in a normal light field of view by the Example of this invention. It is a microscope image of the calcium oxalate sand-like crystal in the polarized field by the Example of this invention. It is a structural schematic diagram of the intelligent identification system of the calcium oxalate crystal based on the microscope image by the Example of this invention. It is a structural schematic diagram of the neural network module according to the Example shown in FIG. It is a structural schematic diagram of the image capturing module according to the embodiment shown in FIG. The realization of the object of the present invention, the features and advantages of the functions will be further described with reference to the drawings with examples.

It should be noted that the specific examples described here are merely for interpreting the present invention and do not limit the present invention.

The main solving problems of the examples of the present invention are as follows. In the prior art, the Chinese herbal medicine micro-identification method for calcium oxalate crystals is to identify the type of calcium oxalate crystal by observing with an operator and manually drawing the microscopic features and manually matching the features of the calcium oxalate crystal. Is common. Such a method causes problems such as a large error due to subjective factors, a long drawing time, and low work efficiency.

In order to solve the above problem, with reference to FIG. 1, FIG. 1 is a schematic diagram of an application scenario according to an embodiment of the present invention. The application scenario realizes an intelligent identification method for calcium oxalate crystals based on microscopic images according to the following examples of the present invention. As shown in FIG. 1, the application scenario includes a microscope and an automatic microscope image collector, which mainly includes a control box, a digital imaging device, and a computer, of which controls. The box controls and moves the stage of the microscope, the digital imaging device is connected to the microscope to collect images of the microscope, and the computer is electrically connected to the digital imaging device and the control box, respectively, for image processing and control box. Used to send microscope control commands to. Here, the microscope acquires images in two visual field scenarios, a normal light source and a polarized light source, respectively.

Specifically, the computer transmits a host device control signal 3 to the control box, the control card of the control box sends a motor drive signal 4 to the microscope according to the host device control signal 3, and three stepping motors in the microscope. The microscope stage (where the control drug or sample is placed) moves axially in the three vertical directions of X, Y, and Z, respectively, and the stepping motor of the microscope. Feeds the table position signal 5 to the control box in real time, and the control card of the control box controls the card communication signal 2 alternately with the computer to feed back and adjust the movement of the stage of the microscope in real time. Tracks and acquires a microscope image of a control drug or sample taken in real time while moving the stage of the microscope, feeds the microscope image back to the computer in the form of a video signal 1, and the computer uses a deep convolution neural network to obtain the microscope image. To obtain, for example, a deep convolutional neural network model that identifies the features of the calcium oxalate crystals, or to match the microscopic features of the sample sample, thus identifying the calcium oxalate crystals.

In order to achieve the above object, with reference to FIG. 2, FIG. 2 is a schematic flowchart of an example of an intelligent identification method for calcium oxalate crystals based on a microscopic image according to an embodiment of the present invention. As shown in FIG. 2, the method for intelligently identifying calcium oxalate crystals based on the microscopic image includes steps S110 to S140.

S110: The training data of calcium oxalate crystals is acquired, and the training data includes a plurality of crystal types of calcium oxalate crystals.

The training data trains a deep convolutional neural network, and the calcium oxalate crystal has multiple crystal types, so that the deep convolutional neural network can be used for various crystal types of calcium oxalate through the training data. The characteristics of the crystal can be identified. The training data is also test data, and the deep convolutional neural network is identified by comparing the type of calcium oxalate crystal marked by the training data with the type of calcium oxalate crystal identified by the deep convolutional neural network. The accuracy can be improved.

As shown in FIG. 3, the step of acquiring the training data of the calcium oxalate crystal specifically includes steps S111 to S114.

S111: A plurality of crude drugs containing various crystal types of calcium oxalate crystals are selected.

Calcium oxalate crystals have multiple types, different biopharmaceuticals have different types of calcium oxalate crystals, for example, Yakan contains calcium oxalate columnar crystals, Gokahi contains calcium oxalate clustered crystals, Hange contains calcium oxalate acicular crystals, Koukahi contains calcium oxalate prismatic crystals, and Dikoppi contains calcium oxalate sandy crystals.

S112: A microscopic image of calcium oxalate crystals in each crude drug is taken.

Since the types of calcium oxalate crystals contained in each biopharmaceutical are different, taking a microscopic image of the calcium oxalate crystals in each biopharmaceutical yields microscopic images of different types of calcium oxalate crystals, followed by a deep convolution neural network. Microscopic features of different types of calcium oxalate crystals are obtained through.

S113: Mark the crystal type of calcium oxalate crystals in the microscopic image.

Since various crystal types of calcium oxalate crystals have different crystal characteristics, by marking the crystal type of the calcium oxalate crystal, the deep convolution neural network identifies the crystal characteristics of the calcium oxalate crystal according to the different crystal types. It is also possible to check whether the marked crystal type is accurately identified by the deep convolution neural network and improve the identification accuracy of the deep convolution neural network.

S114: Using the microscopic image and crystal type of each calcium oxalate crystal, training data of each calcium oxalate crystal is generated respectively.

The training data includes at least a microscopic image and crystal type of each calcium oxalate crystal, thus the deep convolutional neural network identifies and extracts crystal features within this image according to the microscopic image and marks the crystal type. Matching and confirmation can be performed, thereby improving the identification accuracy of the deep convolutional neural network.

See FIGS. 9-A-9-J for microscopic images of the relevant calcium oxalate crystals. Of these, 9-A is a microscope image of calcium oxalate columnar crystals taken in a normal light source scenario, and 9-B is a microscope image of calcium oxalate columnar crystals taken in a polarized light source scenario, and the above two microscopes. The image is obtained by taking an image of Yakan. 9-C is a microscopic image of calcium oxalate clusters taken in a normal light source scenario, 9-D is a microscopic image of calcium oxalate clusters taken in a polarized light source scenario, and the above two microscopic images are This is an image of Gokahi. 9-E is a microscope image of calcium oxalate needle-like crystals taken in a normal light source scenario, 9-F is a microscope image of calcium oxalate needle crystals taken in a polarized light source scenario, and the above two microscope images are This is a photograph of Hange. 9-G is a microscopic image of calcium oxalate prismatic crystals taken in a normal light source scenario, 9-H is a microscopic image of calcium oxalate prismatic crystals taken in a polarized light source scenario, and the above two microscopic images are This is an image of Koukahi. 9-I is a microscopic image of calcium oxalate sandy crystals taken in a normal light source scenario, 9-J is a microscopic image of calcium oxalate sandy crystals taken in a polarized light source scenario, and the above two microscopic images are This is an image of Zikoppi.

S120: Training data is trained using a deep convolutional neural network to obtain a deep convolutional neural network model for identifying calcium oxalate crystals.

The deep convolution neural network acquires a microscopic image of a calcium oxalate crystal in the training data, then extracts features from the microscopic image, and uses a preset classifier to correspond the crystal features of the calcium oxalate crystal to the crystal type. To obtain crystal characteristics corresponding to various crystal types of calcium oxalate crystals, and to obtain a deep convolutional neural network model that identifies various crystal types of calcium oxalate crystals. In this way, it is possible to easily identify the crystal characteristics and crystal type of the input calcium oxalate crystal.

Here, the deep convolutional neural network includes a convolutional layer, a pooling layer and a fully connected layer. As shown in FIG. 4, the step of training the training data using the deep convolutional neural network specifically includes steps S121 to S125.

S121: The microscopic image of the calcium oxalate crystal contained in the training data is input to the convolution layer.

When the microscopic image of the calcium oxalate crystal is input to the convolutional layer, the convolutional layer performs a convolution operation on the microscopic image and converts it into a corresponding feature image. The microscopic images of the calcium oxalate crystals are a microscopic image of a calcium oxalate columnar crystal, a microscopic image of a calcium oxalate cluster-like crystal, a microscopic image of a calcium oxalate needle-like crystal, a microscopic image of a calcium oxalate prismatic crystal, and Shu. Includes microscopic images of calcium oxalate sandy crystals.

S122: Using a plurality of convolutional layers, a convolution operation is performed on a microscopic image of a calcium oxalate crystal to obtain a characteristic image of the calcium oxalate crystal.

The convolutional layer has a "receptive field" having a predetermined size, and the depth of this "receptive field" is the same as the depth of the input image. A feature map of is obtained, the convolutional layer contains multiple convolutions, thus allowing the convolutional layer to repeat the convolution operation multiple times on the original microscopic image, thereby providing deeper levels of oxalic acid. A characteristic image of calcium crystals can be obtained.

Specifically, the method for obtaining a characteristic image of the calcium oxalate crystal by performing a convolution operation on the microscopic image of the calcium oxalate crystal using the plurality of convolutional layers is as follows.

の顕微鏡画像に畳み込み操作を行い、シュウ酸カルシウム結晶の特徴画像を得て、式中、

ディープ畳み込みニューラルネットワークに非線形ファクタを追加して、畳み込み層が要件を満たす特徴画像を抽出することを可能とする。

By performing a convolution operation on the microscope image of, a characteristic image of calcium oxalate crystals was obtained, and in the formula,

A non-linear factor is added to the deep convolutional neural network to allow the convolutional layer to extract feature images that meet the requirements.

Here, the formula for calculating the number of output dimensions of the convolution layer is as follows.

S123: Using the pooling layer, feature compression is performed on the feature image of the calcium oxalate crystal to obtain crystal features corresponding to the feature image.

The pooling layer serves to compress the input feature image, making the feature image smaller, reducing the complexity of network calculations, and performing feature compression to extract the main features on the feature image. The feature is a crystal feature corresponding to the feature image.

式中、プーリング層ｐｏｏｌのカーネルのサイズはＦ_２である。 Here, the formula for calculating the number of output dimensions of the pooling layer is as follows.

_In the equation, the size of the kernel of the pooling layer pool is F2.

S124: A fully connected layer is used to connect the crystal features of all feature images corresponding to the same microscopic image, and the classifier determines the crystal type of the microscopic image and the crystal features corresponding to the crystal type.

The fully connected layer connects all the crystal features on the same microscope image, inputs the output crystal features to the classifier, and determines the crystal type of the microscope image and the crystal features corresponding to the crystal type by the classifier.

S125: A deep convolutional neural network model is created for each type of calcium oxalate crystal using the crystal type and crystal characteristics of the microscope image.

Each deep convolutional neural network model can identify the crystal features of the corresponding calcium oxalate crystals, and after identifying the crystal features, match the crystal features with the corresponding calcium oxalate type. For example, a neural network model of calcium oxalate columnar crystals, a neural network model of calcium oxalate clustered crystals, using the crystal types and crystal characteristics of microscopic images of Yakan, Gokahi, Hange, Koukahi, and Zikoppi control biopharmaceuticals, respectively. A neural network model of calcium oxalate needle-like crystals, a neural network model of calcium oxalate prismatic crystals, and a neural network model of calcium oxalate sandy crystals were created, respectively, thereby forming calcium oxalate columnar crystals and clustered crystals. , Needle-shaped crystals, prismatic crystals, and sand-like crystals, respectively, to identify the crystal features of the microscopic images.

In the technical solution according to the embodiment of the present application, a plurality of folding layers are used to perform a folding operation on a microscopic image of a calcium oxalate crystal to obtain a characteristic image of the calcium oxalate crystal, and then a pooling layer. The feature image of the calcium oxalate crystal is feature-compressed using, the crystal type corresponding to the feature image is obtained, and the crystal type of the microscopic image and the crystal feature corresponding to the crystal type are further determined by a classifier. Create a deep convolutional neural network model for each type of calcium oxalate crystal, identify the crystal characteristics of the corresponding type calcium oxalate crystal by the deep convolutional neural network model, and collect by identifying the crystal characteristics. Determine the type of calcium oxalate crystals contained in the sample.

S130: Using a deep convolutional neural network model, feature matching is performed on the microscopic image of the collected sample, and the crystal type of the calcium oxalate crystal contained in the collected sample is obtained.

Different deep convolutional neural network models identify different crystal types of calcium oxalate crystals and provide the crystal characteristics of different crystal types of calcium oxalate crystals. In this way, when the microscopic image of the collected sample is input to the deep convolutional neural network, the deep convolutional neural network model matches the crystal features with the microscopic image, and the oxalate contained in the collected sample according to the crystal characteristics. Determine the type of calcium oxalate crystals. For example, the microscopic image of the collected sample is taken with any one of calcium oxalate columnar crystals, clustered crystals, acicular crystals, prismatic crystals, and sandy crystal models identified by different deep convolutional neural network models. When matched, the corresponding crystal type information is generated.

S140: The identification result of the calcium oxalate crystal of the collected sample is generated and output according to the crystal type of the calcium oxalate crystal contained in the collected sample.

Once the crystal type of the calcium oxalate crystal contained in the sample is obtained, the corresponding calcium oxalate crystal identification result is generated, which enables rapid and accurate identification of the calcium oxalate crystal.

As shown in FIG. 5, the steps for generating and outputting the identification result of the calcium oxalate crystal of the collected sample according to the crystal type of the calcium oxalate crystal contained in the collected sample are specifically steps S141 to S143. including.

S141: Whether the crystal type of the calcium oxalate crystal contained in the collected sample has been determined. The deep convolution neural network model stores the correspondence between the crystal characteristics of the calcium oxalate crystal and the crystal type. When the crystal characteristics of the calcium oxalate crystal in the collected sample are matched in step S130, the shu contained in the collected sample is stored. The crystal type of calcium oxalate crystals can be determined.

S142: When the deep convolution neural network model acquires the crystal type of the calcium oxalate crystal contained in the collected sample, the crystal type information of the collected sample, the stitch crystal characteristic image of the collected sample, and the characteristic image of the sample crystal of the collected sample are obtained. Generate and output. When the crystal type of the calcium oxalate crystal contained in the collected sample is acquired, only the image of the crystal characteristics of the collected sample may be stitched, and in this way, a clearer image is displayed by the user. For example, if the deep convolutional neural network model detects a clustered crystal type calcium oxalate crystal, the deep convolutional neural network model stitches an image of the calcium oxalate clustered crystal, thereby giving the user a clear and detailed image. Is displayed.

Or,
S143: When the deep convolutional neural network model does not acquire the crystal type of the calcium oxalate crystal contained in the sample, a stitch microscope image of the sample is generated and output. If the deep convolutional neural network model does not acquire the crystal type of the calcium oxalate crystal contained in the collected sample, it is found that the collected sample does not contain the calcium oxalate crystal, and thus all of the collected sample. When the stitch microscopic image is generated, the entire morphology of the collected sample is displayed, and the related operator can clearly search and understand the microstructure of the collected sample.

Here, the method of generating a stitch microscopic image of the collected sample specifically includes the following steps.

That is, it includes a step of calculating a conversion matrix between the image pixel coordinates and the spatial physical coordinates of each microscope image corresponding to the same sample sample using a stitch algorithm. Normally, it is necessary to acquire multiple microscope images for the same sample, and the image pixel coordinates of each microscope image are different. Therefore, in order to obtain a stitch microscope image of the sample, image pixels are used using a stitch algorithm. All microscopic images need to be stitched by calculating the transformation matrix between the coordinates and the spatial physical coordinates in real space.

It comprises the step of stitching all microscopic images of the same sample taken under the same coordinate system according to the transformation matrix to obtain a stitched microscopic image. By using the transformation matrix, all microscopic images can be stitched under the same coordinate system according to the actual spatial and physical coordinates, stitched microscopic images can be obtained, and the morphology of the entire sample to be collected can be obtained. Further, when there is a problem that the stitch result pixels of two adjacent images are displaced, matching optimization is performed by a template matching method based on an image grayscale feature in order to obtain a complete stitch microscope image. Since the method of acquiring the stitch crystal feature image is the same, it will not be described in detail here.

As described above, in the intelligent identification method of calcium oxalate crystal based on the microscopic image according to the embodiment of the present application, training data of calcium oxalate crystal including the calcium oxalate crystal of a plurality of crystal types is acquired, and the following In addition, the training data was trained using a deep convolutional neural network to obtain a deep convolutional neural network model capable of identifying the characteristics of various types of calcium oxalate crystals for identifying calcium oxalate crystals. In addition, feature matching was performed on the microscopic image of the collected sample using a trained deep convolution neural network model to obtain the crystal type of the calcium oxalate crystal contained in the collected sample, and the calcium oxalate contained in the collected sample was obtained. The identification result of the calcium oxalate crystal of the collected sample is generated and output according to the crystal type of the crystal.

The intelligent identification method of calcium oxalate crystals according to the examples of the present application obtains a deep convolution neural network model by training using an artificial intelligence method, and automatically characterizes the crystal characteristics of calcium oxalate crystals in a sample. By matching, the type of the calcium oxalate crystal is automatically identified, and the identification structure of the calcium oxalate crystal of the collected sample is output. In the artificial identification method, the operator manually operates it, so that it is subjective. It solves the problems of the conventional technology such as large error due to various factors, time-consuming drawing, and low work efficiency.

Further, as shown in FIG. 6, before the step S130 of performing feature matching on the microscopic image of the collected sample using the deep convolutional neural network model, the calcium oxalate crystal identification method further includes step S200.

S200: A microscope image of a sample collected in a normal light source field of view and a polarized light source field of view is taken by using an automatic microscope image collecting device.

In the conventional method of collecting a calcium oxalate crystal microscope image, it is often the case that only a microscope image of a calcium oxalate crystal is taken in a normal light source field of view. However, in such an image collection method, there is uneven illumination in the imaging process, so that a part of the calcium oxalate crystal may be confused with the other structure of the collected sample. Therefore, the calcium oxalate crystal is used. It is difficult to identify completely and accurately, and it is easy to cause a misjudgment. Therefore, in the embodiment of the present application, the microscope image of the sample collected in the normal light source field of view and the polarized light source field of view can be taken by using the automatic microscope image collecting device, and the two light source fields of view are combined in this way. A clear calcium oxalate crystal structure can be obtained, and the identification accuracy of the calcium oxalate crystal can be improved.

As shown in FIGS. 1 and 7, the method of taking the microscope images of the sample collected in the normal light source field of view and the polarized light source field of view by using the microscope image automatic collecting device is specifically described in steps S210 to S230. include.

S210: The stage of the microscope is controlled by using a control box so as to perform horizontal motion in a three-dimensional coordinate system according to a predetermined motion trajectory.

S220: The stage of the microscope is controlled by using a control box so as to perform vertical motion in a three-dimensional coordinate system according to a predetermined motion trajectory, and the sample collected on the stage is tracked and focused in real time.

S230: Using a digital image pickup device, microscopic images of the collected sample are taken a plurality of times at predetermined time intervals.

In a technical solution according to an embodiment of the present application, the motion control card of the control box controls and accurately motions three stepping motors, whereby the stages are three perpendicular to each other, X, Y, Z, respectively. It moves in the axial direction of the direction, and here, as shown in FIG. 8, the slide glass (or stage) of the microscope moves horizontally due to the movement in the X-axis and the Y-axis to collect, and the stage is in the Z-axis. By moving vertically along the direction, the sample collected is tracked and focused in real time, displaying a clear image throughout the observation process. As the stage moves along the three axial directions perpendicular to each other, X, Y and Z, the digital imaging device takes multiple microscopic images of the sample taken at predetermined time intervals and multiple samples of the same sample. It is advantageous for the computer to acquire the microscopic image of the same sample in real time and repeat the process multiple times for the same sample. Here, the image acquisition step is the same for the two light source fields of view of ordinary light and polarized light.

The specific movement path of the microscope is shown in FIG. Point A is the initial coordinate point of the digital imaging device, the microscope slide glass is first adjusted to point A, then moves along the horizontal direction of the Y axis, and then moves along the vertical direction of the X axis. As such, n (n is a positive integer greater than 0) is repeated, thereby covering the entire collection area by horizontally moving the control drug or sample. As the microscope moves in the X-axis and Y-axis, microscope images on the microscope slides are collected, and as the stage moves in the Z-axis direction, the microscope is tracked and focused in real time and over the course of the process. Multiple clear microscopic images are taken.

Further, as a preferred embodiment, in the intelligent identification method of the original calcium oxalate crystal, after the original step S140 of generating and outputting the identification result of the calcium oxalate crystal of the collected sample, the method of the embodiment of the present application is performed. Further includes a step of marking the crystal type of the calcium oxalate crystals contained in the collected sample and a step of adding the microscopic image of the collected sample and the marked crystal type to the training data.

In the technical solution according to the embodiment of the present application, after marking the crystal type of the calcium oxalate crystal contained in the collected sample, the microscopic image of the collected sample and the marked crystal type are added to the training data. Strengthen the training sample of the deep convolutional neural network and enhance the identification ability of the deep convolutional neural network.

Further, based on the same concept as the embodiment of the above method, the embodiment of the present invention further provides an intelligent identification system for calcium oxalate crystals based on a microscope image for realizing the above method of the present invention. The embodiments of the system have at least all the beneficial effects of the technical solutions of the embodiments, as the principle of the problem to be solved is similar to the method described above, and will not be described in detail herein.

With reference to FIG. 10, FIG. 10 is a structural schematic diagram of an intelligent identification system for calcium oxalate crystals based on microscopic images according to an embodiment of the present invention. As shown in FIG. 10, the intelligent identification system for calcium oxalate crystals based on the microscopic image is
It is a data acquisition module 110 for acquiring training data of calcium oxalate crystals, and the training data includes a data acquisition module 110 including a plurality of crystal types of calcium oxalate crystals.
A neural network module 120 for training training data using a deep convolutional neural network to obtain a deep convolutional neural network model for identifying calcium oxalate crystals, and
A feature matching module 130 for performing feature matching on a microscopic image of a sample sample using a deep convolutional neural network model and acquiring the crystal type of calcium oxalate crystals contained in the sample sample.
A result generation module 140 for generating the identification result of the calcium oxalate crystal of the collected sample according to the crystal type of the calcium oxalate crystal contained in the collected sample, and
It includes a result output module 150 for outputting the identification result of the calcium oxalate crystal of the collected sample.

As described above, in the intelligent identification system for calcium oxalate crystals based on the microscopic images according to the examples of the present application, the data acquisition module 110 acquires training data for calcium oxalate crystals, and the training data is obtained from a plurality of crystal types. The above-mentioned calcium oxalate crystal is contained, and then the neural network module 120 trains the training data acquired by the data acquisition module 110 using the deep convolution neural network, and the deep convolution for identifying the calcium oxalate crystal is performed. Obtaining a neural network model, the deep convolution neural network model identifies features of various types of calcium oxalate crystals, and then the feature matching module 130 is harvested using a trained deep convolution neural network model. Feature matching is performed on the microscopic image of the sample to obtain the crystal type of the calcium oxalate crystal contained in the collected sample, and then the result generation module 140 and the result output module 150 are the calcium oxalate crystals contained in the collected sample. The identification result of the calcium oxalate crystal of the collected sample is generated and output according to the crystal type.

The intelligent identification system for calcium oxalate crystals according to the examples of this application obtains a deep convolution neural network model by training using an artificial intelligence method, and automatically characterizes the crystal characteristics of calcium oxalate crystals in the collected sample. By matching, the type of the calcium oxalate crystal is automatically identified, and the identification structure of the calcium oxalate crystal of the collected sample is output. In the artificial identification method, the operator manually operates it, so that it is subjective. It solves the problems of the conventional technology such as large error due to various factors, time-consuming drawing, and low work efficiency.

As shown in FIG. 11, the neural network module 120 is
An image input submodule 121 for inputting a microscopic image of calcium oxalate crystals contained in the training data into the convolutional layer, and
A convolution operation submodule 122 for performing a convolution operation on a microscopic image of a calcium oxalate crystal and obtaining a characteristic image of a calcium oxalate crystal using a plurality of convolution layers, and a convolution operation submodule 122.
A feature compression submodule 123 for performing feature compression on a feature image of a calcium oxalate crystal using a pooling layer to obtain crystal features corresponding to the feature image, and
A feature connection submodule 124 for connecting crystal features of all feature images corresponding to the same microscope image using a fully connected layer,
A crystal classification submodule 125 for determining the crystal type of the microscopic image and the crystal characteristics corresponding to the crystal type by the classifier, and
Includes a model creation submodule 126 for creating a deep convolutional neural network model for each type of calcium oxalate crystal using the crystal type and crystal characteristics of the microscopic image.

Further, as a preferred embodiment, refer to FIG. 12, and the intelligent identification system for calcium oxalate crystals according to this embodiment uses a normal light source field of view using an automatic microscope image acquisition device in addition to each structural module shown in FIG. It also includes an imaging module 160 for capturing each microscopic image of the sample taken in the polarized light field of view.
The image capturing module 160 is
A motion control submodule 161 for controlling the stage of the microscope using a control box so as to perform horizontal motion in a three-dimensional coordinate system according to a predetermined motion trajectory.
A focus control submodule for controlling the stage of the microscope using a control box to perform vertical movement in a three-dimensional coordinate system according to a predetermined motion trajectory, and tracking and focusing the sample collected on the stage in real time. 162 and
Includes an imaging submodule 163 for taking multiple microscopic images of a sample taken at predetermined time intervals using a digital imaging device.

In addition, the examples of the present application further provide a computer-readable storage medium, in which the computer-readable storage medium stores a program of an intelligent identification method for calcium oxalate crystals based on a microscope image, and the microscope is stored. The image-based intelligent identification method program for calcium oxalate crystals, when executed by a computer, implements the steps of the intelligent identification method for calcium oxalate crystals described in any of the above technical solutions.

Since the specific examples of the computer-readable storage medium of the present invention are substantially the same as the examples of the intelligent identification method for calcium oxalate crystals based on the above-mentioned microscopic images, detailed description thereof will be omitted here.

As will be apparent to those of skill in the art, embodiments of the present invention may be provided as methods, systems, or computer program products. Accordingly, the invention may be in the form of a complete hardware embodiment, a complete software embodiment, or a combination of software and hardware embodiments. Further, the present invention is a computer program implemented in one or more computer-usable storage media including, but not limited to, magnetic disk memory, CD-ROM, optical memory, etc., including program code available by the computer. It may be in the form of a product.

The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be noted that a combination of each flow and / or block of the flowchart and / or the block diagram and the flow and / or the block of the flowchart and / or the block diagram can be realized by a computer program instruction. These computer program instructions are applied to the processors of general purpose computers, dedicated computers, embedded processors and other programmable data processing devices to form machines, thereby executing on computers or the processors of other programmable data processing devices. The instructions given constitute a device for realizing a function specified by one flow or a plurality of flows and / or a block of a block diagram.

These computer program instructions may be stored in a computer-readable memory that guides the computer or other programmable data processing device to operate in a particular manner, thus in the computer-readable memory. The stored instructions constitute a product including an instruction device, which realizes a function specified by one flow or multiple flows and / or one block of a block diagram.

These computer program instructions may be installed on a computer or other programmable data processing device to perform a series of operational steps on the computer or other programmable device to generate the processing achieved by the computer. , Thereby an instruction executed on a computer or other programmable device realizes the function specified by one or more flows and / or one block of the block diagram. I will provide a.

It should be noted that in the claims, no reference code in parentheses limits the claim. The term "contains" does not exclude cases that include members or steps not described in the claims. The term "one" or "one" described between the members does not exclude the case where a plurality of such members are present. The present invention may be realized by hardware including a plurality of different members and a properly programmed computer. In the claims in which the units of several devices are listed, some of these devices may be embodied in the same hardware. The use of terms such as "first", "second", and "third" does not indicate any order. These terms may be interpreted as names.

Although preferred embodiments of the present invention have been described, those skilled in the art will be able to make further changes and modifications to these embodiments once they have a basic concept of creation. For this reason, the appended claims should be construed as including preferred embodiments and all modifications and amendments within the scope of the invention.

Obviously, one of ordinary skill in the art can make various changes and modifications to the present invention without departing from the gist and scope of the present invention. Therefore, if these modifications or modifications of the present invention belong to the claims of the present invention and the same technical scope thereof, the present invention is intended to include these changes and modifications.

Claims (10)

An intelligent identification method for calcium oxalate crystals based on microscopic images.
It is a step of acquiring the training data of the calcium oxalate crystal, and the training data includes a step including the calcium oxalate crystal of a plurality of crystal types and a step.
Steps to train the training data using a deep convolutional neural network to obtain a deep convolutional neural network model for identifying calcium oxalate crystals.
The step of performing feature matching on the microscopic image of the collected sample using the deep convolutional neural network model and acquiring the crystal type of the calcium oxalate crystal contained in the collected sample, and
An intelligent identification method comprising a step of generating and outputting an identification result of a calcium oxalate crystal of the collected sample according to a crystal type of the calcium oxalate crystal contained in the collected sample. The step of acquiring training data for calcium oxalate crystals is
Steps to select multiple crude drugs containing various crystal types of calcium oxalate crystals,
Steps to take microscopic images of calcium oxalate crystals in each herbal medicine,
The step of marking the crystal type of calcium oxalate crystals in the microscopic image,
The calcium oxalate crystal according to claim 1, comprising a step of generating training data for each of the calcium oxalate crystals using a microscopic image and a crystal type of each calcium oxalate crystal. Intelligent identification method. The deep convolutional neural network includes a convolutional layer, a pooling layer and a fully connected layer.
The step of training the training data using a deep convolutional neural network to obtain a deep convolutional neural network model is
A step of inputting a microscopic image of calcium oxalate crystals contained in the training data into the convolution layer, and
A step of performing a convolution operation on a microscopic image of the calcium oxalate crystal using the plurality of convolutional layers to obtain a characteristic image of the calcium oxalate crystal.
A step of performing feature compression on a feature image of the calcium oxalate crystal using the pooling layer to obtain crystal features corresponding to the feature image, and
A step of connecting the crystal features of all feature images corresponding to the same microscope image using a fully connected layer and determining the crystal type of the microscope image and the crystal features corresponding to the crystal type by a classifier.
The calcium oxalate crystal according to claim 1, comprising a step of creating a deep convolutional neural network model for each type of calcium oxalate crystal using the crystal type and crystal characteristics of the microscopic image. Intelligent identification method. The step of performing a convolution operation on a microscopic image of the calcium oxalate crystal using a plurality of convolutional layers to obtain a characteristic image of the calcium oxalate crystal is

Before the step of feature matching microscopic images of sample samples using a deep convolutional neural network model,
It also includes the steps of taking microscopic images of the sample taken in the normal light source field of view and the polarized light source field of view using an automatic microscope image collector.
The above step of taking a microscope image of a sample collected in a normal light source field of view and a polarized light source field of view using an automatic microscope image collecting device is described in the above step.
A step of controlling the stage of the microscope using a control box so that horizontal motion is performed in a three-dimensional coordinate system according to a predetermined motion trajectory.
A step of controlling the stage of the microscope using a control box to perform vertical motion in a three-dimensional coordinate system according to a predetermined motion trajectory, and tracking and focusing the sample collected on the stage in real time.
The intelligent identification method for a calcium oxalate crystal according to claim 1, comprising: taking a microscopic image of the collected sample a plurality of times at predetermined time intervals using a digital image pickup apparatus. The step of generating and outputting the identification result of the calcium oxalate crystal of the collected sample according to the crystal type of the calcium oxalate crystal contained in the collected sample is
When the deep convolutional neural network model acquires the crystal type of the calcium oxalate crystal contained in the collected sample, the crystal type information of the collected sample and the stitch crystal characteristic image of the collected sample are generated and output, or
If the deep convolutional neural network model does not acquire the crystal type of the calcium oxalate crystal contained in the sample, the claim comprises a step of generating and outputting a stitch microscopic image of the sample. Item 2. The method for intelligently identifying calcium oxalate crystals according to Item 1. The step of generating a stitched microscopic image of a sample is
The step of calculating the transformation matrix between the image pixel coordinates and the spatial physical coordinates of each microscope image corresponding to the same sample collected using the stitch algorithm, and
The calcium oxalate crystal according to claim 6, comprising stitching all microscopic images of the same sample sample under the same coordinate system according to the conversion matrix to obtain the stitched microscopic images. Intelligent identification method. An intelligent identification system for calcium oxalate crystals based on microscopic images.
It is a data acquisition module for acquiring training data of calcium oxalate crystals, and the training data includes a data acquisition module including the calcium oxalate crystals of a plurality of crystal types.
A neural network module for training the training data using a deep convolutional neural network to obtain a deep convolutional neural network model for identifying calcium oxalate crystals, and
A feature matching module for performing feature matching on a microscopic image of a sample collected using the deep convolutional neural network model and acquiring the crystal type of calcium oxalate crystals contained in the sample, and a feature matching module.
A result generation module for generating the identification result of the calcium oxalate crystal of the collected sample according to the crystal type of the calcium oxalate crystal contained in the collected sample, and the result generation module.
An intelligent identification system comprising: a result output module for outputting the identification result of the calcium oxalate crystal of the collected sample. The neural network module is
An image input submodule for inputting a microscopic image of calcium oxalate crystals contained in the training data into the convolutional layer, and an image input submodule.
A convolution operation submodule for performing a convolution operation on a microscopic image of the calcium oxalate crystal using the plurality of convolution layers to obtain a characteristic image of the calcium oxalate crystal, and a convolution operation submodule.
A feature compression submodule for performing feature compression on the feature image of the calcium oxalate crystal using the pooling layer to obtain crystal features corresponding to the feature image, and
A feature connection submodule for connecting the crystal features of all feature images corresponding to the same microscope image using a fully connected layer,
A crystal classification submodule for determining the crystal type of the microscope image and the crystal characteristics corresponding to the crystal type by the classifier, and
8. The eighth aspect of claim 8 comprises a model creation submodule for creating a deep convolutional neural network model for each type of calcium oxalate crystal using the crystal type and crystal characteristics of the microscopic image. Intelligent identification system for calcium oxalate crystals. It also includes an imaging module for capturing microscopic images of the sample taken in the normal light source field of view and the polarized light source field of view using an automatic microscope image collector.
The image capturing module is
A motion control submodule for controlling the stage of a microscope using a control box to perform horizontal motion in a three-dimensional coordinate system according to a predetermined motion trajectory.
A focus control submodule for controlling the stage of the microscope using a control box to perform vertical movement in a three-dimensional coordinate system according to a predetermined motion trajectory, and tracking and focusing the sample collected on the stage in real time. When,
The calcium oxalate crystal according to claim 8, which comprises an imaging submodule for taking a microscope image of the collected sample a plurality of times at a predetermined time interval using a digital image pickup apparatus. Intelligent identification system.

Images