TWI770754B - Neural network training method electronic equipment and storage medium - Google Patents

Abstract

This application relates to a neural network training method, electronic equipment, and storage medium for training a neural network model, and classifying images according to the neural network model obtained by the training. The method includes: obtaining position information of a target area in a sample image and Category information; segment to obtain at least one sample image block according to the location information of the target area in the sample image; classify at least one sample image block according to the category information to obtain N types of sample image blocks; input N types of sample image blocks to the nerve Training in the network.

Description

Neural network training method, electronic device and storage medium

The invention relates to the field of computer technology, in particular to a neural network training method, electronic equipment and storage medium.

Methods such as machine learning have a wide range of applications in the field of image processing, for example, they can be applied to the classification and image detection of ordinary images or 3D images. For example, in the processing of medical images, machine learning methods can be used to determine the type of disease and detect diseased areas.

In the processing of medical images, the classification and detection of lung medical images (eg, lung computed tomography (CT)) play an important role in the screening and diagnosis of pneumonia, lung cancer and other lesions. Early detection of lesion characteristics before surgery is clinically important and can guide clinical decision-making. However, due to the lack of typical radiological features of early-stage cancer (clear air bubbles, pleural retraction, etc.), it is difficult for specialists or radiologists to accurately differentiate the subtypes of ground-glass nodules (Ground-Glass Nodule) from CT images. , GGN) category.

Embodiments of the present invention provide a neural network training method, electronic device, and storage medium.

An embodiment of the present invention provides a neural network training method, the neural network training method is used for training a neural network model, and images are classified according to the neural network model obtained by training, and the method includes: obtaining a sample image position information and category information of the target area in the sample image; according to the position information of the target area in the sample image, at least one sample image block is obtained by segmentation; according to the category information, the at least one sample image block is classified, N types of sample image blocks are obtained, where N is an integer, and N≥1; the N types of sample image blocks are input into the neural network for training.

According to the neural network training method of the embodiment of the present invention, the fine classification of the sample image blocks can be obtained, and the neural network can be trained, so that the neural network can finely classify the images and improve the classification efficiency and accuracy.

In some embodiments of the present invention, the sample image is a medical image picture.

In some embodiments of the present invention, the obtaining the location information and category information of the target area in the sample image includes: locating the target area on the medical image picture to obtain the location information of the target area; the pathological picture associated with the medical image picture, the pathological picture is a diagnosed picture including pathological information; according to the pathological information of each target area on the pathological picture, the target area on the medical image picture is determined category information.

In some embodiments of the present invention, inputting the N types of sample image blocks into a neural network for training includes: inputting any sample image block into the neural network for processing to obtain a sample image block The classification prediction information and prediction target area of the loss; train the neural network according to the classification loss and the segmentation loss.

In some embodiments of the present invention, determining the classification loss according to the category prediction information and the category information of the sample image block includes: determining the first classification loss according to the category prediction information and the category information of the sample image block. a classification loss; a second classification loss is determined according to the class prediction information and the class information of the class center of the class to which the sample image block belongs; weighted summation is performed on the first classification loss and the second classification loss process to obtain the classification loss.

In this way, the category features of the sample image blocks of the same category can be more aggregated during training, and the feature distance between the category information of sample image blocks of different categories can be larger, which is helpful to improve the classification performance and improve the classification performance. Accuracy.

In some embodiments of the present invention, determining the segmentation loss according to the location information of the prediction target area and the sample image block includes: according to the number of primitives in the prediction target area, the segmentation loss is included in the sample image block. The first weight of the prediction target area and the second weight of the sample background area in the sample image block are determined; according to the first weight, the second weight, the prediction target area and all The location information of the sample image block is used to determine the segmentation loss.

In some embodiments of the present invention, the first weight of the prediction target area and the sample image block are determined according to a first proportion of the number of primitives in the prediction target area in the sample image block The second weight of the sample background area in the sample image block includes: determining the second weight of the sample background area in the sample image block according to the first proportion of the number of primitives in the predicted target area in the sample image block. ratio; determining the second ratio as the first weight, and determining the first ratio as the second weight.

In this way, the error of the target area and the error of the non-target area can be balanced, which is beneficial to the optimization of network parameters and improves the training efficiency and training effect.

In some embodiments of the present invention, the category information includes: preinvasive adenocarcinoma atypical adenoma hyperplasia nodules, adenocarcinoma in situ nodules, minimally invasive adenocarcinoma nodules, and invasive adenocarcinoma nodules.

In some embodiments of the present invention, the neural network includes a common feature extraction network, a classification network and a segmentation network, and the method further includes: inputting image blocks to be processed into the common feature extraction network for processing , obtain the target feature of the to-be-processed image block, wherein the common feature extraction network includes M common feature extraction blocks, and the input feature of the i-th common feature extraction block includes the first i-1 common feature extraction blocks. The output feature of , i and M are integers and 1<i≤M; input the target feature into the classification network for classification processing, and obtain the category information of the image block to be processed; input the target feature into the The segmentation network performs segmentation processing to obtain the target area in the to-be-processed image block.

In this way, the target feature can be obtained through the shared feature extraction network, and the shared feature extraction block of the shared feature extraction network can obtain the output features of all previous shared feature extraction blocks, and input its own output features to all subsequent shared feature extraction blocks. Feature extraction block. It can strengthen the gradient flow in the network, alleviate the phenomenon of gradient disappearance, and at the same time improve the feature extraction and learning ability, which is conducive to the more fine classification and segmentation of the input image blocks to be processed. And the finer category information and target area of the image block to be processed can be obtained, which improves the image processing efficiency.

In some embodiments of the present invention, inputting the image block to be processed into the common feature extraction network for processing to obtain the target feature of the image block to be processed includes: performing a first step on the image block to be processed. Feature extraction processing to obtain the first feature of the to-be-processed image block; inputting the first feature into a first common feature extraction block to obtain the output feature of the first common feature extraction block, and applying the first feature to the first common feature extraction block The output features of the first common feature extraction block are output to the subsequent M-1 common feature extraction blocks; the output features of the first j-1 common feature extraction blocks are input to the jth common feature extraction block to obtain the Output features of the j shared feature extraction blocks, where j is an integer and 1<j<M; perform the second feature extraction process on the output features of the Mth shared feature extraction block to obtain the first feature of the to-be-processed image block. Second feature; perform pooling processing on the second feature to obtain the target feature.

In some embodiments of the present invention, the method further includes: preprocessing the image to be processed to obtain a first image; locating a target area on the first image to determine The position information of the target area; according to the position information of the target area in the first image, at least one image block to be processed is obtained by segmentation.

An embodiment of the present invention provides a neural network training device, the neural network training device is used for training a neural network model, and classifying images according to the neural network model obtained by training, the device includes: an acquisition module, is configured to obtain position information and category information of the target area in the sample image; the first segmentation module is configured to obtain at least one sample image block by segmentation according to the position information of the target area in the sample image; the classification module, is configured to classify the at least one sample image block according to the category information to obtain N types of sample image blocks, where N is an integer and N≥1; the training module is configured to classify the N types of sample image blocks The image patches are fed into the neural network for training.

An embodiment of the present invention provides an electronic device, including: a processor; a memory configured to store a computer program executable by the processor; wherein the processor is configured to execute the above neural network training method through the computer program.

An embodiment of the present invention provides a storage medium, where a computer program is stored in the storage medium, and the computer program is configured to execute the above neural network training method when running.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. Other features and aspects of the present invention will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Various exemplary embodiments, features and aspects of the present invention will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures denote components that have the same or similar functions. While various aspects of the embodiments are shown in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" in this article is only a relationship to describe related objects, which means that there can be three relationships, for example, A and/or B, which can mean that A exists alone, A and B exist at the same time, and B exists alone. three conditions. In addition, the term "at least one" herein refers to any combination of any one of a plurality or at least two of a plurality, for example, including at least one of A, B, and C, and may mean including those composed of A, B, and C. Any one or more elements selected in the collection.

In addition, in order to better illustrate the present invention, numerous details are given in the following embodiments. It will be understood by those skilled in the art that the present invention may be practiced without certain details. In some instances, methods, means, components and circuits well known to those skilled in the art have not been described in detail so as not to obscure the subject matter of the present invention.

Before further detailed description of the embodiments of the present invention, defects in the related art are described first.

In the related art, methods such as machine learning are widely used in the field of image processing, for example, they can be applied to the fields of general image or three-dimensional image classification and image detection.

Lung cancer is one of the most common malignant tumors in my country, and its mortality rate ranks first in cancer deaths in urban or rural areas, male or female, among which adenocarcinoma accounts for about 40% of all lung cancers. Screening using medical images (eg, lung CT and low-dose helical CT), an increasing number of early-stage lung adenocarcinomas are found and manifest as Ground-Glass Nodule (GGN), adenocarcinomas classified as Atypical Adenomatous Hyperplasia Of Preinvasive Adenocarcinoma (AAHOPA), Adenocarcinoma In Situ (AIS), Minimally Invasive Adenocarcinoma (MIA) and Invasive Adenocarcinoma (Invasive Adenocarcinoma) , IA). GGN categories of adenocarcinoma include atypical adenomatous hyperplastic nodules of preinvasive adenocarcinoma, adenocarcinoma in situ nodules, minimally invasive adenocarcinoma nodules, and invasive adenocarcinoma nodules. Survival decreases significantly with increasing tumor size, suggesting that early detection and diagnosis is an effective and crucial approach to reducing patient mortality. Therefore, early detection of aggressive features before surgery will be clinically important and can guide clinical decision-making.

In medical image processing, the classification and detection of lung medical images (eg, lung CT) plays an important role in the diagnosis of medical screening and diagnosis of pneumonia, lung cancer, etc. In the related art, machine learning and other methods can be used to determine the disease category and detect the lesion area, for example, to predict whether the input image of the nodule belongs to a malignant tumor or a benign tumor. However, the prediction result is not subdivided in the related art. . Moreover, due to the lack of typical radiological features of early-stage cancer (clear bubbles, pleural retraction, etc.), it is clinically difficult for specialists or radiologists to accurately differentiate the subtype GGN categories from CT images. In this context, AI-based computer-aided diagnosis is a more effective method to assess the invasiveness of nodules and is expected to play an important role in clinical assessment tasks.

Referring to FIG. 1 , FIG. 1 is a schematic diagram of a system architecture of a neural network training method provided by an embodiment of the present invention. As shown in FIG. 1 , the system architecture includes a CT instrument 100 , a server 200 , a network 300 and a terminal device 400 , in order to support an exemplary application, the CT apparatus 100 can be connected to the terminal device 400 through the network 300, the terminal device 400 can be connected to the server 200 through the network 300, and the CT apparatus 100 can be used to collect CT images, such as X-ray CT It is a terminal that can scan a certain thickness of a certain part of the human body, such as a CT scanner or a gamma ray CT scanner. The terminal device 400 may be a notebook computer, a tablet computer, a desktop computer, a dedicated message device, or other device with a screen display function. The network 300 may be a wide area network or a local area network, or a combination of the two, using wireless links for data transmission.

Based on the neural network training method provided by the embodiment of the present invention, the server 200 can cut each pathologically proven lung nodule area in the acquired training medical image pictures into small image blocks through the designed three-dimensional classification framework, and then cut the area into small image blocks. The image blocks are classified to obtain training data, and the training data is logged into the neural network for training, so that the neural network can finely classify the training medical image pictures, and after the training is completed, a trained neural network model is obtained. The medical image picture may be a CT image of the lungs of a patient or a medical examiner collected by a CT instrument 100 in a hospital, a medical examination center or other institutions. The server 200 can obtain the medical image pictures collected by the CT apparatus 100 from the terminal device 400 as the training medical image pictures, can also obtain the training medical image pictures from the CT apparatus, and can also obtain the training medical image pictures from the Internet.

The server 200 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server based on cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize the calculation, storage, processing and sharing of data. In the embodiment of the present invention, when the server 200 is a cloud server, the provided artificial intelligence cloud service may include a neural network model, and the neural network is trained based on the finely classified training data, so that the neural network can perform medical imaging analysis. Pictures are finely classified.

As an example, after receiving the medical image picture (eg, lung CT image) to be processed, the server 200 classifies and divides the medical image picture according to the trained neural network, and obtains the finely classified lesion area. . Then, the server 200 returns the obtained finely classified lesion area to the terminal device 400 for display, so that the medical staff can view it.

In some embodiments of the present invention, after the training of the server 200 is completed, the trained neural network may be sent to the terminal device 400, and the terminal device 400 will process the collected medical image pictures (eg, lung CT images) to be processed. ) for classification, segmentation, etc., to obtain finely classified lesion areas, and display the obtained finely classified lesion areas on its own display screen for the convenience of medical staff to view.

In some embodiments of the present invention, the system architecture of the neural network training method includes a CT instrument 100, a network 300 and a terminal device 400. The terminal device 400 trains the training medical image pictures to obtain a trained neural network. , and then the terminal device 400 performs classification, segmentation, etc. on the collected medical image pictures (such as lung CT images) to obtain a finely classified lesion area, and the obtained finely classified lesion area is placed in its own It is displayed on the display screen for the medical staff to see.

An embodiment of the present invention provides a neural network training method, the method is applied to a neural network training device, and the neural network training device may be a server for training a neural network model. The model classifies images. The method provided by the embodiment of the present invention may be implemented by a computer program, and when the computer program is executed, each step in the neural network training method provided by the embodiment of the present invention is completed. In some embodiments, the computer program can be executed by a processor. Fig. 2 is an implementation flowchart of a neural network training method provided by an embodiment of the present invention. As shown in Fig. 2, the method includes: Step S11, obtaining location information and category information of the target area in the sample image; Step S12, according to the location information of the target area in the sample image, segment to obtain at least one sample image block; Step S13, classifying the at least one sample image block according to the category information to obtain N types of sample image blocks, where N is an integer, and N≥1; Step S14, inputting the N types of sample image blocks into a neural network for training.

According to the neural network training method provided by the embodiment of the present invention, the fine classification of the sample image blocks can be obtained, and the neural network can be trained, so that the neural network can finely classify the images and improve the classification efficiency and accuracy.

In some embodiments of the present invention, the neural network training method may be performed by a terminal device or other processing device, where the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, Cellular phones, wireless phones, Personal Digital Assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. Other processing devices may be servers, cloud servers, or the like. In some embodiments of the present invention, the neural network training method can be implemented by the processor calling the computer program stored in the memory.

In some embodiments of the present invention, the sample image is a medical image, such as a lung CT image. The sample image block may be an image block in the sample image that includes the target area. In an example, the sample image may be an annotated (eg, category annotation and segmentation annotation) three-dimensional medical image, and the sample image patch may be an image patch containing nodules in the three-dimensional medical image.

In some embodiments of the present invention, in step S11, the location information and category information of the target area in the sample image can be determined to obtain sample image blocks for training the neural network, and the sample image blocks can be labeled . Step S11 may include: locating the target area on the medical image picture to obtain location information of the target area; obtaining a pathology picture associated with the medical image picture; Pathological information, to determine the category information of the target area on the medical image. The pathological picture is a diagnosed picture including pathological information, which may be obtained from a medical image database, or sent to the neural network training device after being manually marked by a professional such as a doctor on the terminal.

In some embodiments of the present invention, the sample image may be resampled to obtain a three-dimensional image with a resolution of 1×1×1. And segment the 3D image. For example, in a 3D medical image of the lungs, there may be some areas outside the lung parenchyma, while lesions such as pulmonary nodules often exist in the lung parenchyma, so the map where the lung parenchyma is located is cut out (ie segmented). image block and normalize the image block. The target area (for example, the lesion area) in the normalized three-dimensional image can be located to obtain the position information of the target area. For example, the location information of the target area may be determined by a convolutional neural network used for localization, or the location information of the target area may be confirmed by a professional such as a doctor, etc. The embodiment of the present invention does not limit the location method.

In some embodiments of the present invention, the medical image picture may have a related pathology picture, which may be used to determine the category of the lesion in the medical image picture. For example, the category of the lesion may include a ground-glass nodule (GGN). Adenocarcinoma is divided into Atypical Adenomatous Hyperplasia Of Preinvasive Adenocarcinoma (AAHOPA), Adenocarcinoma In Situ (AIS), Minimally Invasive Adenocarcinoma (MIA) and Invasive Adenocarcinoma Cancer (Invasive Adenocarcinoma, IA), the embodiment of the present invention does not limit the types of lesions.

In some embodiments of the present invention, the pathological information of each target area can be obtained according to the pathological picture. For example, the pathological picture can be a professionally diagnosed picture, which can have an analysis description of each lesion, and can be based on the pathological picture. Obtain the pathological information of each target area, and then determine the category information of each target area on the medical image.

In some embodiments of the present invention, an image block including a lesion area can be cut out in a medical image picture, that is, a sample image block is cut out, and N types of sample image blocks are obtained according to the category information of the target area. For example, after the statistics of the nodule size, the size of the sample image block can be determined as 64 × 64 × 64, and after cropping and classification, four types (AAHOPA, AIS, MIA and IA) sample image blocks are obtained.

In some embodiments of the present invention, since the number of medical image pictures is small, the labeling is difficult, and the cost is high, if the three-dimensional image is divided into multiple two-dimensional images, spatial information will be lost, resulting in performance degradation. The sample image blocks can be rotated, translated, mirrored, zoomed, etc., to expand the number of samples, and using the amplified sample image blocks to train the neural network can improve the generalization ability of the neural network and prevent excessive fit. In some embodiments of the present invention, positive and negative samples may also be balanced, in an example, the number of samples of benign nodules such as atypical adenoma hyperplasia, adenocarcinoma in situ, and minimally invasive adenocarcinoma, and malignant nodules such as invasive adenocarcinoma If there is a large gap, a small number of samples can be amplified by the above method to balance the number of positive and negative samples. The embodiments of the present invention do not limit the manner of amplifying the number of samples.

In some embodiments of the invention, the sample image patches may be fed into the neural network in batches. Wherein, step S14 may include: inputting any sample image block into the neural network for processing to obtain category prediction information and prediction target area of the sample image block; at least according to the category prediction information and the sample map The classification loss is determined according to the category information of the image block; the segmentation loss is determined according to the predicted target area and the position information of the sample image block; the neural network is trained according to the classification loss and the segmentation loss.

In some embodiments of the present invention, the neural network may include a common feature extraction network, a classification network, and a segmentation network. The feature extraction of the sample image block can be performed through the shared feature extraction network to obtain the sample target feature of the sample image block, and the category prediction information of the sample image block can be obtained through the classification network. There may be errors in the category prediction information. The sample prediction category information and the category annotation information of the sample image block determine the classification loss of the neural network.

In some embodiments of the present invention, determining the classification loss according to the category prediction information and the annotation information of the sample image block includes: determining the first classification loss according to the category prediction information and the annotation information of the sample image block. a classification loss; a second classification loss is determined according to the class prediction information and the class information of the class center of the class to which the sample image block belongs; weighted summation is performed on the first classification loss and the second classification loss process to obtain the classification loss.

In some embodiments of the present invention, the annotation information of the sample image block may include category annotation information, for example, the category annotation information may be information indicating the category of the nodule in the sample image block. In an example, the category prediction information may be category information represented by a vector or the like, and a probability dictionary may be used to determine that the to-be-processed image block represented by the vector belongs to the probability distribution of each category, and then the probability distribution of the to-be-processed image block may be determined. category. Alternatively, the vector of the category prediction information may directly represent the probability of the image block to be processed. In an example, each element of the vector represents the probability of the category to which the image block to be processed belongs.

，例如，可根據softmaxloss損失函數來計算第一分類損失

。在示例中，可通過以下公式（1）來確定第一分類損失

：

（1）； 其中，

表示第i個樣本圖像塊的類別預測資訊，

表示第i個樣本圖像塊所屬的類別，n表示類別數量。

表示全連接層中第

個類別的權重，

表示全連接層中第j個類別的權重，m表示每個批次輸入神經網路的樣本圖像塊的數量，

表示第i個樣本圖像塊所屬的類別偏置項，

表示第j個類別的偏置項。In some embodiments of the present invention, the first classification loss may be determined according to the category prediction information and the category annotation information of the sample image block, for example, the feature distance between the vector of the category prediction information and the vector of the category annotation information (eg , Euclidean distance, cosine distance, etc.), and determine the first classification loss according to the feature distance

, for example, the first classification loss can be calculated according to the softmaxloss loss function

. In an example, the first classification loss can be determined by the following formula (1)

:

(1); wherein,

represents the category prediction information of the i-th sample image block,

Indicates the category to which the i-th sample image patch belongs, and n represents the number of categories.

represents the first in the fully connected layer

class weights,

represents the weight of the jth category in the fully connected layer, m represents the number of sample image patches input to the neural network in each batch,

represents the category bias term to which the i-th sample image block belongs,

represents the bias term for the jth class.

訓練後的分類網路的分類效果不佳。In some embodiments of the present invention, using the above-mentioned first classification loss for training can expand the inter-class feature distances of class information of different classes, thereby enabling the classification network to distinguish sample image blocks of different classes. However, the differences between multiple classes of nodules in the lung were not significant (eg, nodules in adenocarcinoma in situ and minimally invasive adenocarcinoma did not differ greatly in shape), and the shapes varied between two classes of nodules (eg, invasive glandular Malignant nodules such as cancer have different shapes), thus resulting in a small feature distance between classes and a large feature distance within class information, resulting in only the use of the first classification loss

The classification performance of the trained classification network is not good.

In some embodiments of the present invention, aiming at the above problem, the classification network can be trained through the second classification loss. In an example, the class information of the class centers of each of the multiple sample image blocks may be determined, for example, the weighted average of the class information of the class centers of the multiple sample image blocks may be performed, or the class information of the sample image blocks may be determined. The information is clustered to obtain class center features, etc. The embodiment of the present invention does not limit the category information of the class center.

，例如，可根據centerloss損失函數來計算第二分類損失

。通過第二分類損失

訓練分類網路，可縮小同類樣本圖像塊的類別資訊的類內特徵距離，使得同類的特徵資訊在特徵空間中更集中，有利於確定樣本圖像塊的類別。在示例中，可通過以下公式（2）來確定第二分類損失

：

（2）； 其中，

為第i個樣本圖像塊所屬的類別的類中心的類別標注資訊。In some embodiments of the present invention, the second classification loss may be determined according to the class prediction information of the sample image block and the class annotation information of the class center of the class to which it belongs. For example, the feature distance between the class prediction information and the class information of the class center can be determined and the second classification loss can be determined according to the feature distance

, for example, the second classification loss can be calculated according to the centerloss loss function

. Pass the second classification loss

Training the classification network can reduce the intra-class feature distance of the category information of the same sample image block, so that the same type of feature information is more concentrated in the feature space, which is beneficial to determine the category of the sample image block. In the example, the second classification loss can be determined by the following formula (2)

:

(2); wherein,

Class annotation information for the class center of the class to which the i-th sample image patch belongs.

In some embodiments of the present invention, the classification loss may be determined jointly by the first classification loss and the second classification loss. For example, the first classification loss and the second classification loss can be weighted and summed to obtain the classification loss. For example, the weight ratio of the first classification loss and the second classification loss is 1:0.8, and the classification loss can be obtained after the weighted summation is performed according to the above weight ratio. This embodiment of the present invention does not limit the weight ratio.

In this way, the category features of the sample image blocks of the same category can be more aggregated during training, so that the distance between the category information of the sample image blocks of different categories is larger, which is helpful to improve the classification performance and improve the classification performance. Accuracy.

In some embodiments of the present invention, the target feature of the sample may be segmented through a segmentation network to obtain the prediction target area in the image block of the sample. The prediction target area may have errors, and the segmentation loss may be determined according to the error between the prediction target area and the labeling target area of the sample image block, and then the training is performed through the segmentation loss.

In some embodiments of the present invention, determining the segmentation loss according to the prediction target area and the annotation information of the sample image block includes: according to the number of primitives in the prediction target area, the number of pixels in the sample image block The first weight of the prediction target area and the second weight of the sample background area in the sample image block are determined; according to the first weight, the second weight, the prediction target area and all The annotation information of the sample image block is used to determine the segmentation loss.

In some embodiments of the present invention, the annotation information includes annotated segmentation regions, and the segmentation loss can be determined directly according to the error between the predicted target region and the annotated segmentation regions. However, the diameter of the nodule is usually between 5 millimeters (millimeter, mm) and 30 mm, and the proportion of the nodule in the sample image block is different from other areas, resulting in the primitives between the target area and the non-target area. Unbalanced numbers can make the error of predicting the target region account for a small proportion of the segmentation loss, which is not conducive to the optimization and adjustment of the neural network, resulting in low training efficiency and poor training effect.

In some embodiments of the present invention, the weighting process may be performed according to the primitives of the target area and the primitives of the non-target area (ie, the sample background area). In an example, the first weight of the prediction target area and the second weight of the sample background area in the sample image block may be determined according to the first proportion of the number of primitives of the prediction target area in the sample image block. Furthermore, when determining the segmentation loss, weighting processing is performed on the primitives of the above two regions to balance the loss of the target region and the loss of the non-target region.

In some embodiments of the present invention, the first weight of the prediction target area and the sample image block are determined according to a first proportion of the number of primitives in the prediction target area in the sample image block The second weight of the sample background area in the sample image block includes: determining the second weight of the sample background area in the sample image block according to the first proportion of the number of primitives in the predicted target area in the sample image block. ratio; determining the second ratio as the first weight, and determining the first ratio as the second weight.

In some embodiments of the present invention, the sample image block may include a predicted target area and a background area, and the proportion of the number of primitives in the predicted target area may be counted to determine the proportion of the sample background area. For example, if the first proportion of the number of primitives in the predicted target area is 0.2, the second proportion of the number of primitives in the sample background area is 0.8. The embodiment of the present invention does not limit the first ratio and the second ratio.

In some embodiments of the present invention, in order to balance the predicted target area and the sample background area, the second ratio is determined as the first weight of the predicted target area, and the first ratio is determined as the second weight of the sample background area. For example, if the first proportion of the number of primitives in the predicted target area is 0.2, the first weight of the predicted target area is 0.8, and the second proportion of the number of primitives in the sample background area is 0.8, then the second weight of the sample background area is 0.2.

。例如，可根據weightedDiceloss損失函數來計算分割損失

。在示例中，可通過以下公式（3）確定分割損失

：

（3）； 其中，

，

=1時，表示第k個圖元位置為預測目標區域，

=0時，表示第k個圖元位置為樣本背景區域，

表示分割網路在第k個圖元位置的輸出，W表示第一權重，Y表示第k個圖元位置的分割標注。In some embodiments of the present invention, the segmentation loss may be determined according to the first weight, the second weight, the prediction target area, and the labeling target area of the sample image block. In an example, the segmentation loss can be determined according to the difference between the predicted target area and the target area in the annotation information. For example, the primitive points in the predicted target area can be weighted, the weight is the first weight, and the sample background area can be weighted. The primitive points are weighted, the weight is the second weight, and the weighted segmentation loss is determined

. For example, the segmentation loss can be calculated according to the weightedDiceloss loss function

. In the example, the segmentation loss can be determined by the following formula (3)

:

(3); wherein,

,

When =1, it means that the k-th primitive position is the prediction target area,

When =0, it means that the k-th primitive position is the sample background area,

Represents the output of the segmentation network at the kth primitive position, W represents the first weight, and Y represents the segmentation annotation at the kth primitive position.

In this way, the error of the target area and the error of the non-target area can be balanced, which is beneficial to the optimization of network parameters and improves the training efficiency and training effect.

：

（4）； 其中，

表示

的權重，

表示

的權重，

表示

的權重，例如，

=1.2，

=0.8，

=2，本發明實施例對分類損失和分割損失的權重不做限制。In some embodiments of the invention, the combined network loss for the common feature extraction network, the segmentation network and the classification network may be determined from the classification loss and the segmentation loss. For example, the classification loss and segmentation loss can be weighted and summed to obtain the comprehensive network loss. In the example, the comprehensive network loss can be determined according to the following formula (4)

:

(4); wherein,

express

the weight of,

express

the weight of,

express

weights, for example,

=1.2,

=0.8,

=2, the weights of the classification loss and the segmentation loss are not limited in this embodiment of the present invention.

In some embodiments of the present invention, the network parameters of the above-mentioned neural network can be adjusted inversely through the integrated network loss. For example, the network parameters can be adjusted through the gradient descent method, so that the network parameters can be optimized and the segmentation and classification accuracy can be improved. accuracy.

In some embodiments of the present invention, the above-mentioned training method can be repeatedly performed for many times, and the training is performed according to a set learning rate. In the example, in the first 20 training epochs, training can be performed using a learning rate of 0.001*1.1 ^x (where x represents the training epoch), and in the subsequent training, the training can be performed at the 40th, 80th and 120th... to halve the learning rate. In the initial stage of training, the training efficiency can be improved, the network parameters can be greatly optimized, and the learning rate can be gradually reduced in the subsequent training, the network parameters can be finely adjusted, the accuracy of the neural network can be improved, and the accuracy of classification processing and segmentation processing can be improved. accuracy.

In some embodiments of the present invention, the training can be completed when the training conditions are met, and the shared feature extraction network, segmentation network and classification network after training can be obtained. The training conditions may include training times, that is, when the preset training times are reached, the training conditions are satisfied. The training conditions may include that the comprehensive network loss is less than or equal to a preset threshold or converges in a preset interval, that is, when the comprehensive network loss is less than or equal to a preset threshold or converges in a preset interval, it can be considered that the neural network has The accuracy meets the requirements for use, and the training can be completed. The embodiment of the present invention does not limit the training conditions.

In some embodiments of the present invention, the trained neural network may be tested after the training is completed. For example, a three-dimensional image block including a nodule region in a three-dimensional medical image of the lung can be input into the above neural network, and the accuracy of the output segmentation results and classification results can be counted, for example, compared with the annotation information of the three-dimensional image block. , to determine the accuracy of the segmentation results and classification results, and then the training effect of the neural network can be determined. If the accuracy rate is higher than the preset threshold, it can be considered that the training effect is good, the performance of the neural network is good, and it can be used in the process of obtaining the category of the image block to be processed and segmenting the target area. If the accuracy rate does not reach the preset threshold, it can be considered that the training effect is not good, and other sample image blocks can be used to continue training.

In some embodiments of the present invention, the trained neural network can obtain the category and target region of the image block to be processed under the condition that the target region and category in the image block to be processed are unknown, or can be obtained in the image block to be processed. When the type of the image block is known, only the target area in the image block to be processed is acquired, or the type of the image block to be processed can also be acquired when the target area in the image block to be processed is known. The embodiments of the present invention do not limit the use method of the neural network.

In some embodiments of the present invention, the neural network trained by the above training method can be used to determine the lesion area and lesion category in the image block to be processed. The neural network includes a common feature extraction network, a classification network and a segmentation network, and the method further includes: inputting the image blocks to be processed into the common feature extraction network for processing to obtain target features of the image blocks to be processed , wherein the shared feature extraction network includes M shared feature extraction blocks, the input feature of the i-th shared feature extraction block includes the output features of the first i-1 shared feature extraction blocks, i and M are integers and 1< i≤M; input the target feature into the classification network for classification processing to obtain the category information of the image block to be processed; input the target feature into the segmentation network for segmentation processing to obtain the image block to be processed in the target area.

In this way, a common feature extraction network is used to obtain target features, and the common feature extraction block of the common feature extraction network can obtain the output features of all previous common feature extraction blocks, and input its own output features to all subsequent common features. Extract blocks. It can strengthen the gradient flow in the network, alleviate the phenomenon of gradient disappearance, and at the same time improve the ability of feature extraction and learning, which is conducive to the finer classification and segmentation of the input image blocks to be processed. And the finer category information and target area of the image block to be processed can be obtained, which improves the image processing efficiency.

In some embodiments of the present invention, the to-be-processed image block may be a partial area in the to-be-processed image. In an example, a partial area may be cropped from the image to be processed, for example, a cropped area including the target object. For example, the image to be processed is a medical image picture, and a region including the lesion can be cut out in the medical image picture. For example, the image to be processed may be a three-dimensional medical image of the lung (for example, a CT image of the lung), and the image block to be processed may be a region of a lesion (for example, a region with nodules) clipped from the image to be processed 3D image block. The embodiments of the present invention do not limit the types of images to be processed and image blocks to be processed.

In some embodiments of the present invention, in a medical image (for example, a three-dimensional medical image of the lungs), the size and resolution of the medical image are relatively high, and in the medical image, there are many areas of normal tissue, therefore The medical image picture can be preprocessed, and the region including the lesion can be trimmed to improve the processing efficiency.

In some embodiments of the present invention, the method further includes: preprocessing the image to be processed to obtain a first image; locating a target area on the first image to determine the target in the first image The location information of the area; according to the location information of the target area in the first image, at least one image block to be processed is obtained by segmentation.

In some embodiments of the present invention, the image to be processed may be preprocessed first to improve processing efficiency. For example, preprocessing such as resampling and normalization can be performed. In an example, a three-dimensional medical image of the lung may be resampled to obtain a three-dimensional image with a resolution of 1×1×1 (ie, each primitive represents the content of a 1mm×1mm×1mm cube). The size of the resampled 3D image can be cropped. For example, in the 3D medical image of the lungs, there may be some non-pulmonary areas, and the area where the lungs are located can be cropped to save computation and improve processing efficiency. .

In an example, the trimmed three-dimensional image may be normalized, and the primitive values of each primitive in the three-dimensional image may be normalized to a value range of 0 to 1, so as to improve processing efficiency. After normalization processing, the first image is obtained. The embodiments of the present invention do not limit the preprocessing method.

In some embodiments of the invention, the target area in the first image may be detected. For example, the region of interest in the first image may be detected by a convolutional neural network for position detection. In an example, a convolutional neural network may be utilized to detect regions including nodules in a three-dimensional medical image of the lungs.

In some embodiments of the present invention, the target area may be cropped to obtain image blocks to be processed, for example, a region including nodules in a three-dimensional medical image of the lungs may be cropped to obtain image blocks to be processed. In an example, the size of the image block to be processed can be determined according to the size of the nodule, and then trimmed. Crop to obtain one or more image blocks to be processed.

In some embodiments of the present invention, the neural network can be used to determine the category information of the image block to be processed, and segment the target area, for example, the image block to be processed is cropped from a three-dimensional medical image of the lungs Image blocks including nodules. The type of nodule in the image block to be processed (eg, AAHOPA, AIS, MIA, and IA) can be determined by a neural network, and the region where the nodule is located can be segmented.

In some embodiments of the present invention, a common feature extraction network can be used to extract target features of image blocks to be processed for classification and segmentation processing. Inputting the to-be-processed image block into a shared feature extraction network for processing to obtain the target feature of the to-be-processed image block may include: performing a first feature extraction process on the to-be-processed image block to obtain the first feature of the to-be-processed image block ; Input the first feature into the first common feature extraction block, obtain the output feature of the first common feature extraction block, and output the output feature of the first common feature extraction block to the subsequent M-1 common feature extraction blocks ; Input the output features of the first j-1 common feature extraction blocks to the jth common feature extraction block to obtain the output features of the jth common feature extraction block; Carry out the second output feature of the Mth common feature extraction block The feature extraction process obtains the second feature of the image block to be processed; the second feature is pooled to obtain the target feature.

In some embodiments of the present invention, the first feature extraction process may be performed first, for example, through a network including a three-dimensional convolutional layer (Three Dimensional Convolutional Layer), a batch normalization layer (Normalization), and an activation layer (Activation Layer) The module performs the first feature extraction process to obtain the first feature. The embodiment of the present invention does not limit the network level for performing the first feature extraction process.

In some embodiments of the present invention, the common feature extraction network may include multiple common feature extraction blocks, and the common feature extraction block may include multiple network layers, such as convolution layers, startup layers, etc. The network level included in the extraction block is not limited. The first feature may be processed by a plurality of common feature extraction blocks. In an example, the number of common feature extraction blocks is M, and the first feature can be input into the first common feature extraction block, that is, the first common feature extraction block can use the first feature as an input feature, and the input feature Perform feature extraction processing to obtain output features. The output features of the first shared feature extraction block can be shared by all subsequent shared feature extraction blocks, that is, the output features of the first shared feature extraction block can be up to the subsequent M-1 shared features The extraction block is used as the input feature of the subsequent M-1 shared feature extraction blocks.

In some embodiments of the present invention, the input feature of the second common feature extraction block is the output feature of the first common feature extraction block, and after the second common feature extraction block performs feature extraction on its input feature, the The output features are output to the subsequent 3rd to Mth common feature extraction blocks as input features of the 3rd to Mth common feature extraction blocks.

In some embodiments of the present invention, the input feature of the third common feature extraction block is the output feature of the first common feature extraction block and the output feature of the second common feature extraction block, and the output feature of the first common feature extraction block The features and the output features of the second common feature extraction block can be input to the third common feature extraction block after feature fusion (for example, by computing the mean, maximum, etc., or by keeping all feature channels) The input feature of the third common feature extraction block can be the output feature of the first common feature extraction block and the output feature of the second common feature extraction block after fusion), or the third common feature extraction block can be directly The output features of the first common feature extraction block and the output features of the second common feature extraction block are both used as input features (for example, the third common feature extraction block can include a feature fusion layer, which can be used for feature fusion processing, Or keep all feature channels, and directly perform subsequent processing on the features of all feature channels, that is, perform subsequent processing on the output features of the first common feature extraction block and the output features of the second common feature extraction block), and Perform feature extraction processing on the input features (for example, directly perform feature extraction processing on the features of all feature channels, or perform feature extraction processing on the fused features), and obtain the output features of the third common feature extraction block. The output features of the third common feature extraction block can be output to the fourth to Mth common feature extraction blocks as input features of the fourth to Mth common feature extraction blocks.

In some embodiments of the present invention, taking the jth (j is an integer and 1<j<M) common feature extraction block as an example, the output features of the first j-1 common feature extraction blocks can be used as input features to input to the j shared feature extraction blocks. After feature fusion of the output features of the first j-1 common feature extraction blocks, the fused features can be used as the input features of the jth common feature extraction block, or the output of the first j-1 common feature extraction blocks can be directly fused. The feature is used as the input feature of the j-th common feature extraction block (for example, fused within the j-th common feature extraction block, or directly perform subsequent processing on the features of all feature channels, that is, extract the first j-1 common features block output features for subsequent processing). The jth common feature extraction block can perform feature extraction on its input features, obtain the output features of the jth common feature extraction block, and use the output features as the input of the j+1th to Mth common feature extraction blocks feature.

In some embodiments of the present invention, the Mth common feature extraction block may obtain the output features of the Mth common feature extraction block according to the output features of the first M-1 common feature extraction blocks. The second feature extraction process can be performed through subsequent network layers of the shared feature extraction network. For example, the Nth shared feature can be extracted by a network module including a 3D convolution layer, a batch normalization layer, and a startup layer. The output feature of the block is subjected to a second feature extraction process to obtain the second feature. The embodiment of the present invention does not limit the network level for performing the second feature extraction process.

In some embodiments of the present invention, the second feature may be pooled, for example, the target feature may be obtained by pooling the second feature through an average pooling layer. This embodiment of the present invention does not limit the type of pooling processing.

In some embodiments of the present invention, the above process may be performed multiple times, eg, may include multiple common feature extraction networks. The first common feature extraction network can take the first feature as the input feature, and after the feature extraction process of the common feature extraction block, the second feature extraction process and the pooling process, the output feature of the first common feature extraction network can be obtained. , the second shared feature extraction network can use the output features of the first shared feature extraction network as the input features, and after the feature extraction processing of the shared feature extraction block, the second feature extraction processing and the pooling processing, the second shared feature extraction network can be obtained. The output features of two common feature extraction networks...may be processed by multiple common feature extraction networks, and the output feature of the last (eg, 4th) common feature extraction network is used as the target feature. The embodiment of the present invention does not limit the number of shared feature extraction networks.

In this way, the target feature can be obtained through the shared feature extraction network, and the shared feature extraction block of the shared feature extraction network can obtain the output features of all previous shared feature extraction blocks, and input its own output features to all subsequent shared feature extraction blocks. Feature extraction block. It can strengthen the gradient flow in the network, alleviate the phenomenon of gradient disappearance, and at the same time improve the ability of feature extraction and learning, which is conducive to the finer classification and segmentation of the input image blocks to be processed.

In some embodiments of the present invention, the category information of the to-be-processed image block can be determined according to the target feature. For example, the to-be-processed image block is an image block including nodules and other lesions in a three-dimensional medical image of the lung. Features determine the class of the nodule. In an example, it may be determined whether the nodule is classified as preinvasive adenocarcinoma atypical adenoma hyperplasia, adenocarcinoma in situ, minimally invasive adenocarcinoma, or invasive adenocarcinoma.

In some embodiments of the present invention, the target feature can be classified and processed through a classification network to obtain the classification information of the image block to be processed. In an example, the classification network may include multiple network layers, such as convolutional layers, global average pooling layers (Global Average Pooling), fully connected layers, etc. Classification processing, and can output category information. The category information may be category information represented by a vector or the like, and a probability dictionary or the like may be used to determine that the image block to be processed represented by the vector belongs to the probability distribution of each category, and then the category information of the to-be-processed image block may be determined. Alternatively, the vector of category information can directly represent the probability of the image block to be processed. In an example, each element of the vector represents the probability of the category to which the image block to be processed belongs. For example, (0.8, 0.1, 0.1) can represent the probability of the image block to be processed The probability of the processed image block belonging to the first category is 0.8, the probability of belonging to the second category is 0.1, and the probability of belonging to the third category is 0.1, and the category with the highest probability can be determined as the category of the image block to be processed , that is, the category information of the image block to be processed can be determined as the first category. The embodiment of the present invention does not limit the representation method of category information.

In some embodiments of the present invention, the category information of the to-be-processed image block can be determined according to the target feature. For example, the to-be-processed image block is an image block including nodules and other lesions in a three-dimensional medical image of the lung. Features determine the location of nodules and segment the region where they are located.

In some embodiments of the present invention, segmentation processing may be performed through a segmentation network to obtain the target area in the image block to be processed, for example, the target area may be segmented. In an example, the segmentation network may include multiple network levels, eg, upsample layers, fully connected layers, and the like. In an example, the target feature is a feature map obtained by performing feature extraction, pooling, etc. on the image block to be processed in a common feature extraction network, and the resolution of the target feature may be lower than that of the image block to be processed. Upsampling can be performed through the upsampling layer to reduce the number of feature channels of the target feature and improve the resolution, so that the feature map output by the segmentation network has the same resolution as the image block to be processed. For example, if the feature extraction network is shared for four times of pooling, the upsampling layer can be used for four times of upsampling, so that the feature map output by the segmentation network has the same resolution as the image block to be processed. The target area can be segmented in the output feature map of the segmentation network, for example, the target area where the nodule is located can be determined by contour lines or contour surfaces. The embodiment of the present invention does not limit the network level of the divided network.

In some embodiments of the present invention, after the target area is segmented in the to-be-processed image block, the position of the target area in the to-be-processed image may also be determined. For example, the position of the target area in the image to be processed can be restored according to the position of the image block to be processed in the image to be processed and the position of the target area in the image block to be processed. In an example, in a lung medical image, the location of the nodule in the image block to be processed can be segmented, and the location of the nodule in the lung medical image can be restored.

According to the neural network training method provided by the embodiment of the present invention, the fine classification of the sample image blocks can be obtained, and the neural network is trained, so that the neural network can finely classify the images, and the classification efficiency and accuracy are improved; The target feature can be obtained through the shared feature extraction network. The shared feature extraction block of the shared feature extraction network can obtain the output features of all the previous shared feature extraction blocks, and input its own output features to all subsequent shared feature extraction blocks. Strengthen the gradient flow in the network, alleviate the phenomenon of gradient disappearance, and improve the feature extraction and learning ability at the same time, which is conducive to more fine classification and segmentation of the input image blocks to be processed; class information and target area, improve image processing efficiency; and can make the class information of sample image blocks of the same class more aggregated during training, so that the feature distance between the class information of sample image blocks of different classes is larger ; It can balance the error of the target area and the error of the non-target area, which helps to improve the classification performance and improve the classification accuracy.

FIG. 3 is a schematic diagram of an application of the neural network training method provided by an embodiment of the present invention. As shown in FIG. 3 , the sample image 31 is a medical image picture, and the sample image block 32 is clipped from the medical image picture including a lesion ( For example, nodules) image blocks. Also, the sample image block may have category annotations, for example, the sample image block may include four categories of AAHOPA, AIS, MIA, and IA.

In some embodiments of the present invention, the sample image blocks 32 may be input into the neural network 33, and the common feature extraction network 331 included in the neural network 33 performs feature extraction on each batch of sample image blocks to obtain the The sample target features are obtained, and the category prediction information of the sample image block is obtained through the classification network 332 included in the neural network 33, and the classification loss of the neural network can be determined by formula (1) and formula (2). In some embodiments of the present invention, the segmentation network 333 included in the neural network 33 can obtain the predicted target area in the sample image block 32, and can determine the segmentation loss of the neural network according to formula (3). The segmentation loss and the classification loss can be weighted and summed to obtain the comprehensive network loss of the neural network, and the neural network can be trained by the comprehensive network loss. The trained neural network can be used to determine lesion regions and lesion categories in image patches of medical images.

In some embodiments of the present invention, the to-be-processed image may be a three-dimensional lung medical image (eg, a lung CT image), and the to-be-processed image block may be a cropped case region in the to-be-processed image (eg, 3D image patch of the region with nodules).

In some embodiments of the present invention, the three-dimensional medical image can be resampled to obtain a three-dimensional image with a resolution of 1×1×1, and the area where the lung is located can be cut out, and then the area where the lung is located can be normalized. unify. In some embodiments of the present invention, the area where the nodule is located in the area where the lung is located can be detected, and a plurality of image blocks to be processed including the area where the nodule is located can be cut out according to the size of 64×64×64.

In some embodiments of the present invention, feature extraction processing may be performed on multiple image blocks to be processed in batches to obtain target features of the image blocks to be processed. For example, the first feature extraction process may be performed first, for example, the first feature extraction process may be performed through a network module including a three-dimensional convolution layer, a batch normalization layer, and a startup layer to obtain the first feature.

In some embodiments of the present invention, the first feature may be input into a common feature extraction network. The common feature extraction network may include multiple common feature extraction blocks. In the example, the number of common feature extraction blocks is M, the first feature can be input into the first common feature extraction block for processing, and the output feature of the first common feature extraction block can be the subsequent M-1 common features Extract blocks. The input features of the second common feature extraction block are the output features of the first common feature extraction block, and the second common feature extraction block can output its output features to the subsequent third to Mth common features Extract blocks. The input features of the third common feature extraction block are the output features of the first common feature extraction block and the output features of the second common feature extraction block, and the output features of the third common feature extraction block can be output to the fourth common feature extraction block. to M-th common feature extraction blocks. Similarly, the output features of the first j-1 common feature extraction blocks can be input to the jth common feature extraction block, and the output features of the jth common feature extraction block can be output to the j+1th to Mth common feature extraction blocks. Feature extraction block. The Mth common feature extraction block can obtain the output features of the Mth common feature extraction block according to the output features of the first M-1 common feature extraction blocks, and perform the second feature extraction process, for example, by including a three-dimensional convolution layer. The network modules of the batch normalization layer and the startup layer perform second feature extraction processing on the output feature of the Nth shared feature extraction block to obtain the second feature. In some embodiments of the present invention, a pooling (eg, average pooling (Average Pooling)) process may be performed on the second feature to obtain the target feature.

In some embodiments of the present invention, the above process may be performed multiple times (eg, 4 times), for example, may include multiple common feature extraction networks. The target features can be obtained through the processing of multiple cascaded shared feature extraction networks.

In some embodiments of the present invention, the classification network may perform classification processing on the target features to obtain the classification information of the image blocks to be processed. For example, the classification network can obtain the category information of the image blocks to be processed through convolutional layers, global average pooling layers, and fully connected layers.

In some embodiments of the present invention, the segmentation network may perform segmentation processing on the target feature to obtain the target area (ie, the area where the nodule is located). In the example, the segmentation network performs up-sampling processing four times through the upsampling layer, so that the output feature map of the segmentation network has the same resolution as the image block to be processed, and can be displayed in the output feature map of the segmentation network. Segment out the target area.

In some embodiments of the present invention, the above-mentioned neural network can obtain the category and target region of the image block to be processed when the target region and category in the image block to be processed are unknown (for example, the region where the nodule is located can be segmented , and obtain the category of the nodule), or if the category of the image block to be processed is known, only the target area in the image block to be processed (for example, the area where the nodule is located) can be obtained, or it can be When the target area in the image block is known, obtain the category of the image block to be processed (eg, determine the category of the nodule).

In some embodiments of the present invention, the image processing method can be used to segment and classify case areas in medical images such as lung CT images, improve clinical work efficiency, reduce missed diagnosis and misdiagnosis, and can also be used for other The image is classified and the target area is segmented, and the embodiment of the present invention does not limit the application field of the image processing method.

It can be understood that the above method embodiments mentioned in the present invention can be combined with each other to form a combined embodiment without violating the principle and logic. In addition, the embodiments of the present invention also provide devices, electronic devices, computer-readable storage media, and programs, all of which can be used to implement any of the methods provided by the embodiments of the present invention. For the corresponding technical solutions and descriptions, refer to the corresponding records in the Methods section. Those skilled in the art can understand that, in the above method embodiments, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the execution order of each step should be determined by its function and possible internal logic .

4 is a schematic diagram of a neural network training apparatus provided by an embodiment of the present invention. As shown in FIG. 4 , the apparatus includes: an acquisition module 11, configured to acquire position information and category information of a target area in a sample image; The first segmentation module 12 is configured to obtain at least one sample image block by segmentation according to the position information of the target area in the sample image; the classification module 13 is configured to classify the at least one sample image according to the category information. The image blocks are classified to obtain N types of sample image blocks, where N is an integer, and N≥1; the training module 14 is configured to input the N types of sample image blocks into a neural network for training.

In some embodiments of the present invention, the sample image is a medical image picture.

In some embodiments of the present invention, the acquisition module 11 is further configured to: locate the target area on the medical image picture to obtain the position information of the target area; acquire the pathological picture associated with the medical image picture ; the pathological picture is a diagnosed picture including pathological information; according to the pathological information of each target region on the pathological picture, the category information of the target region on the medical image picture is determined.

In some embodiments of the present invention, the training module 14 is further configured to: input any sample image block into the neural network for processing to obtain category prediction information and prediction target area of the sample image block; at least Determine the classification loss according to the category prediction information and the category information of the sample image block; determine the segmentation loss according to the predicted target area and the position information of the sample image block; determine the segmentation loss according to the classification loss and the A segmentation loss to train the neural network.

In some embodiments of the present invention, the training module 14 is further configured to: determine the first classification loss according to the category prediction information and the category information of the sample image block; Class information of the class center of the class to which the sample image block belongs to determine the second classification loss; weighted summation processing is performed on the first classification loss and the second classification loss to obtain the classification loss.

In some embodiments of the present invention, the training module 14 is further configured to: determine the size of the prediction target area according to the first proportion of the number of primitives in the prediction target area in the sample image block. the first weight and the second weight of the sample background area in the sample image block; the segmentation is determined according to the first weight, the second weight, the prediction target area and the position information of the sample image block loss.

In some embodiments of the present invention, the training module 14 is further configured to: determine the sample image block according to a first proportion of the number of primitives in the predicted target area in the sample image block and determining the second proportion as the first weight, and determining the first proportion as the second weight.

In some embodiments of the present invention, the category information includes: preinvasive adenocarcinoma atypical adenoma hyperplasia nodules, adenocarcinoma in situ nodules, minimally invasive adenocarcinoma nodules, and invasive adenocarcinoma nodules.

In some embodiments of the present invention, the neural network includes a common feature extraction network, a classification network and a segmentation network, and the apparatus further includes: an obtaining module configured to input the image block to be processed into the common feature The feature extraction network performs processing to obtain the target feature of the image block to be processed, wherein the common feature extraction network includes M common feature extraction blocks, and the input features of the i-th common feature extraction block include the first i-1 The output features of the shared feature extraction block, i and M are integers and 1<i≤M; the classification module is configured to input the target feature into the classification network for classification processing, and obtain the image block to be processed. Category information; a segmentation module, configured to input the target feature into the segmentation network for segmentation processing to obtain the target area in the to-be-processed image block.

In some embodiments of the present invention, the obtaining module is further configured to: perform a first feature extraction process on the to-be-processed image block to obtain a first feature of the to-be-processed image block; The feature is input to the first common feature extraction block, the output features of the first common feature extraction block are obtained, and the output features of the first common feature extraction block are output to the subsequent M-1 common feature extraction blocks ; Input the output features of the first j-1 common feature extraction blocks to the jth common feature extraction block to obtain the output features of the jth common feature extraction block, where j is an integer and 1<j<M; The second feature extraction process is performed on the output feature of the Mth shared feature extraction block to obtain the second feature of the to-be-processed image block; the second feature is pooled to obtain the target feature.

In some embodiments of the present invention, the device further includes: a preprocessing module, configured to preprocess the image to be processed to obtain a first image; a positioning module, configured to perform preprocessing on the first image. The target area is positioned to determine the position information of the target area in the first image; the second segmentation module is configured to divide and obtain at least one of the target areas according to the position information of the target area in the first image. Process image blocks.

In some embodiments, the functions or modules included in the apparatus provided in the embodiments of the present invention may be configured to execute the methods described in the above method embodiments, and for implementation, reference may be made to the above method embodiments.

Embodiments of the present invention further provide a computer-readable storage medium, on which a computer program is stored, and the computer program is configured to execute the above method when running. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

An embodiment of the present invention further provides an electronic device, including: a processor; a memory configured to store a computer program executable by the processor; wherein the processor is configured to execute the above method through the computer program. The electronic device may be provided as a terminal, server or other form of device.

Embodiments of the present invention further provide a computer program product, including computer-readable codes. When the computer-readable codes are run on a device, a processor in the device executes the instructions of the neural network training method provided in any of the foregoing embodiments. .

Embodiments of the present invention further provide another computer program product configured to store computer-readable instructions, which, when executed, cause the computer to perform the operations of the neural network training method provided by any of the above embodiments.

FIG. 5 is a schematic diagram of an electronic device provided by an embodiment of the present invention. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like. 5, the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812 , sensor component 814 , and communication component 816 .

The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above. Additionally, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

The memory 804 is configured to store various types of data to support the operation of the electronic device 800 . Examples of such data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory Memory (Read Only Memory, ROM), magnetic memory, flash memory, disk or CD.

Power supply components 806 provide power to various components of electronic device 800 and may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a touch panel (TouchPanel, TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.

Audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (Microphone, MIC) configured to receive external audio signals when the electronic device 800 is in an operating mode, such as a calling mode, a recording mode, and a voice recognition mode. The received audio signal may be stored in memory 804 or transmitted via communication component 816 . In some embodiments, audio component 810 also includes a speaker configured to output audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules. The peripheral interface modules may be keyboards, click wheels, buttons, and the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors configured to provide various aspects of status assessment for electronic device 800 . For example, the sensor assembly 814 can detect the open/closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, the sensor assembly 814 can also detect the electronic device 800 or Changes in the position of a component of the electronic device 800 , presence or absence of user contact with the electronic device 800 , orientation or acceleration/deceleration of the electronic device 800 and changes in the temperature of the electronic device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. The sensor assembly 814 may also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, which may be used in imaging applications . In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 may access wireless networks based on communication standards, such as WiFi, 2G or 3G, or a combination thereof. In some embodiments of the present invention, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In some embodiments of the present invention, the communication component 816 further includes a near field communication (Near Field Communication, NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (Bluetooth, BT) technology and other technologies to achieve.

In some embodiments of the present invention, the electronic device 800 may be configured by one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Process, DSP), digital signal processing devices (Digital Signal Processing) Process Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation, Configured to perform the above method.

In some embodiments of the present invention, a non-volatile computer-readable storage medium, such as a memory 804 including computer program instructions, is also provided. The computer program instructions can be executed by the processor 820 of the electronic device 800 to complete the above method.

FIG. 6 is a schematic diagram of another electronic device provided by an embodiment of the present invention. For example, the electronic device 1900 may be provided as a server. 6, electronic device 1900 includes processing component 1922, and may also include one or more processors, and memory resources represented by memory 1932 configured to store instructions executable by processing component 1922, such as applications. An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Additionally, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input output (I/O) Interface 1958. Electronic device 1900 may operate based on an operating system stored in memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

In some embodiments of the present invention, a non-volatile computer-readable storage medium, such as a memory 1932 including computer program instructions, is also provided. The computer program instructions can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

The present invention may be a system, method and/or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon configured to cause a processor to implement various aspects of the present invention.

A computer-readable storage medium may be a tangible device that holds and stores instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. Computer-readable storage media may include: portable computer disks, hard disks, random access memory (Random Access Memory, RAM), ROM, erasable programmable read-only memory (EPROM or flash memory) , SRAM, Compact Disk-Read Only Memory (CD-ROM), Digital Video Disc (DVD), Memory Sticks, Floppy Disks, Mechanical Encoding Devices, such as A punched card or a raised structure in a groove with instructions stored thereon, and any suitable combination of the above. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or Electrical signals carried by wires.

The computer-readable program instructions described herein may be downloaded from computer-readable storage media to various computing/processing devices, or downloaded to external computers or external storage over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network device. Networks may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. A network interface card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for computer-readable storage stored in each computing/processing device in the medium.

The computer program instructions configured to perform the operations of the present invention may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or in one or more programs Source or object code written in any combination of design languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or the like programming language. The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely remotely. run on a client computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network or a wide area network, or it can be connected to an external computer (for example, using an Internet service provider to Internet connection). In some embodiments, by utilizing the state information of computer readable program instructions to personalize custom electronic circuits, such as programmable logic circuits, FPGAs, or Programmable Logic Arrays (PLAs), the electronic circuits can execute Computer readable program instructions to implement various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

These computer readable program instructions may be provided to the processor of a general purpose computer, special purpose computer or other programmable data processing device to produce a machine for execution of the instructions by the processor of the computer or other programmable data processing device When, means are created that implement the functions/acts specified in one or more of the blocks in the flowchart and/or block diagrams. These computer readable program instructions may also be stored on a computer readable storage medium, the instructions causing the computer, programmable data processing device and/or other equipment to operate in a particular manner, so that the computer readable medium storing the instructions Included is an article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

Computer readable program instructions can also be loaded into a computer, other programmable data processing device, or other equipment, so that a series of operational steps are performed on the computer, other programmable data processing device, or other equipment to generate a computer Processes of implementation such that instructions executing on a computer, other programmable data processing apparatus, or other device implement the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions that contains one or more logic configured to implement the specified Executable instructions for the function. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by dedicated hardware-based systems that perform the specified functions or actions. implementation, or may be implemented in a combination of special purpose hardware and computer instructions.

The computer program product can be implemented in hardware, software or a combination thereof. In one embodiment of the embodiment of the present invention, the computer program product may be embodied as a computer storage medium, and in another embodiment, the computer program product may be embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the various embodiments, the practical application or technical improvement over the related art, or to enable others of ordinary skill in the art to understand the various embodiments disclosed herein.

Industrial Applicability The invention relates to a neural network training method, electronic equipment and storage medium. The method includes: acquiring position information and category information of a target area in a sample image; dividing and obtaining at least one sample image block according to the position information of the target area ; According to the category information, classify at least one sample image block to obtain N types of sample image blocks; Input the N types of sample image blocks into the neural network for training. According to the neural network training method of the embodiment of the present invention, the fine classification of the sample image blocks can be obtained, and the neural network can be trained, so that the neural network can finely classify the images and improve the classification efficiency and accuracy.

100: CT instrument 200: Server 300: Internet 400: Terminal Equipment 11: Get the mod 12: The first segmentation module 13: Classification module 14: Training module 31: Sample Image 32: Sample image block 33: Neural Networks 331: Shared Feature Extraction Network 332: Classification Network 333: Divide the network 800: Electronics 802: Process component 804: memory 806: Power Components 808: Multimedia Components 810: Audio Components 812: Input/Output Interface

814: Sensor Assembly

816: Communication Components

820: Processor

1900: Electronic equipment

1922: Processing components

1926: Power Components

1932: Memory

1950: Web Interface

1958: Input and output interface

S11~S14: Steps

The accompanying drawings, which are incorporated into and constitute a part of this specification, illustrate embodiments consistent with the present invention, and together with the description, serve to explain the technical solutions of the present invention. 1 is a schematic diagram of a system architecture of a neural network training method provided by an embodiment of the present invention; Fig. 2 is a kind of realization flow chart of the neural network training method provided by the embodiment of the present invention; 3 is a schematic diagram of an application of a neural network training method provided by an embodiment of the present invention; 4 is a schematic diagram of a neural network training device provided by an embodiment of the present invention; FIG. 5 is a schematic diagram of an electronic device provided by an embodiment of the present invention FIG. 6 is a schematic diagram of another electronic device provided by an embodiment of the present invention.

S11~S14: Steps

Claims (13)

A neural network training method, wherein the neural network training method is used for training a neural network model, and images are classified according to the neural network model obtained by training, the method comprises: obtaining the position of a target area in a sample image information and category information; according to the location information of the target area in the sample image, at least one sample image block is obtained by segmentation; according to the category information, the at least one sample image block is classified to obtain N types of sample images Like blocks, N is an integer, and N

1; input any sample image block into the neural network for processing, and obtain the category prediction information of the sample image block; determine the classification loss according to the category prediction information and the category information of the sample image block; The classification loss is determined based on the first classification loss and the second classification loss; the first classification loss is determined according to the class prediction information and the class information of the sample image block; the second classification loss is determined according to the class prediction information and the class information of the class center of the class to which the sample image block belongs; and the neural network is trained at least according to the classification loss. The method according to claim 1, wherein the sample image is a medical image picture. The method according to claim 2, wherein the obtaining the location information and category information of the target area in the sample image includes: locating the target area on the medical image picture to obtain the location information of the target area; Acquire a pathological picture associated with the medical image picture, where the pathological picture is a diagnosed picture including pathological information; according to the pathological information of each target area on the pathological picture, determine the pathological picture on the medical image picture. category information for the target area of . The method according to claim 1, wherein inputting any sample image block into the neural network for processing to obtain category prediction information of the sample image block includes: inputting any sample image block into the neural network for processing. The neural network performs processing to obtain the category prediction information and prediction target area of the sample image block; at least according to the classification loss, training the neural network includes: according to the prediction target area and the sample image block The location information is determined, and the segmentation loss is determined; according to the classification loss and the segmentation loss, the neural network is trained. The method according to claim 1 or 4, wherein determining the classification loss according to the category prediction information and the category information of the sample image block includes: comparing the first classification loss and the second classification loss A weighted summation process is performed to obtain the classification loss. The method according to claim 4, wherein determining the segmentation loss according to the prediction target area and the position information of the sample image block includes: according to the number of primitives in the prediction target area, in the sample image The first proportion in the block, determine the first weight of the prediction target area and all the The second weight of the sample background area in the sample image block; the segmentation loss is determined according to the first weight, the second weight, the prediction target area and the position information of the sample image block. The method according to claim 6, wherein the first weight of the prediction target area and the sample are determined according to the first proportion of the number of primitives in the prediction target area in the sample image block The second weight of the sample background area in the image block includes: determining the sample background area in the sample image block according to the first proportion of the number of primitives in the predicted target area in the sample image block the second ratio; determining the second ratio as the first weight, and determining the first ratio as the second weight. The method according to any one of claims 1 to 7, wherein the category information comprises: preinvasive adenocarcinoma atypical adenoma hyperplasia nodules, adenocarcinoma in situ nodules, minimally invasive adenocarcinoma nodules and invasive adenocarcinoma nodules. The method according to any one of claims 1 to 7, wherein the neural network includes a common feature extraction network, a classification network and a segmentation network; the method further comprises: inputting the image blocks to be processed into the The common feature extraction network performs processing to obtain the target feature of the image block to be processed, wherein the common feature extraction network includes M common feature extraction blocks, and the input feature of the i-th common feature extraction block includes the Output features of i-1 common feature extraction blocks, i and M are integers and 1<i

M; Input the target feature into the classification network for classification processing to obtain the category information of the to-be-processed image block; input the target feature into the segmentation network for segmentation processing to obtain the to-be-processed image Like the target area in the block. The method according to claim 9, wherein inputting the image block to be processed into the common feature extraction network for processing to obtain the target feature of the image block to be processed includes: performing a first feature extraction process to obtain a first feature of the image block to be processed; inputting the first feature into a first common feature extraction block to obtain an output feature of the first common feature extraction block, and Outputting the output features of the first common feature extraction block to the subsequent M-1 common feature extraction blocks; inputting the output features of the first j-1 common feature extraction blocks to the jth common feature extraction block to obtain The output feature of the jth shared feature extraction block, where j is an integer and 1<j<M; the second feature extraction process is performed on the output feature of the Mth shared feature extraction block to obtain the to-be-processed image The second feature of the block; the second feature is pooled to obtain the target feature. The method according to claim 9, further comprising: preprocessing the image to be processed to obtain a first image; locating a target area on the first image to determine the target in the first image The location information of the area; according to the location information of the target area in the first image, segmentation is obtained at least one of the image blocks to be processed. An electronic device, comprising: a processor; a memory configured to store a computer program executable by the processor; wherein the processor is configured to: execute the computer program described in any one of request items 1 to 11 through the computer program method. A computer-readable storage medium, in which a computer program is stored, the computer program is configured to execute the method described in any one of claim items 1 to 11 when running.

Images