CN106203327A - Lung tumor identification system and method based on convolutional neural networks - Google Patents

Abstract

The invention provides a lung tumor recognition system and method based on a convolutional neural network. Among them, the lung tumor recognition system based on convolutional neural network includes: receiving multiple stacked images collected along preset dimensions; The image blocks in the stacked image are merged and convolved to obtain the feature image block, and each feature image block is down-sampled; the down-sampled feature image blocks are up-sampled; and each pixel in the image is predicted based on the heat Points correspond to the probability that at least one upsampled feature image block may be true or false, and draw a heat prediction map. The invention effectively improves the recognition rate for the local area of the stacked image with spatial correlation.

Description

Lung tumor recognition system and method based on convolutional neural network

technical field

The present invention relates to an image processing technology, in particular to a lung tumor recognition system and method based on a convolutional neural network.

Background technique

In recent years, convolutional neural networks have demonstrated their advantages in image feature recognition tasks. For example, during the construction of a 2D image to a 3D image, a convolutional neural network is used to perform bad block screening on 2D image blocks. Most of the current convolutional neural networks use deep learning methods to classify images as true or false. As a model with a slightly special structure, the fully convolutional neural network has made great progress in local recognition tasks, including object bounding box detection, prediction of key parts and key points of objects, etc. These prediction methods still use the method of learning the classifier to improve the recognition and classification of image blocks. The disadvantage of this method is that: since the local features identified by the full convolutional neural network are located in an image, there is no way to correlate the relevant information of the image to be tested, so that the recognition accuracy is limited by the bottleneck.

Contents of the invention

In view of the shortcomings of the prior art described above, the purpose of the present invention is to provide a lung tumor identification system and method based on a convolutional neural network, which is used to solve the problem of identifying a part of an image in the prior art. The problem of low accuracy.

In order to achieve the above object and other related objects, the present invention provides an image heat prediction system, including: an image receiving device for receiving multiple stacked images collected along preset dimensions; a convolutional network-based down-sampling device for According to the same movement rule of the same window, each of the stacked images is divided into blocks, and the image blocks in each stacked image corresponding to the same window position are merged and convolved to obtain a feature image block, and each feature image block is performed. Down-sampling; up-sampling means for performing up-sampling processing on each feature image block after down-sampling, so that each feature image block after up-sampling corresponding to at least the adjacent window position has pixel point overlap; heat prediction means, using The color of each pixel in the heat prediction map is drawn by using the probability that at least one feature image block after upsampling is true or false; wherein, each pixel in the heat prediction map belongs to at least one of the upsampled feature image blocks.

In some embodiments, the convolutional network-based down-sampling device further includes: a normalization unit, configured to perform normalization processing on the multiple stacked images, and then move according to the same window to normalize The subsequent multiple stacked images are divided into blocks.

In some embodiments, the convolutional network-based down-sampling device includes: at least one set of structures consisting of filters and down-sampling units; wherein, when the number of the structures is multiple, each set of structures is mutually Cascading; wherein, the filter connected to the image receiving device has a receiving channel consistent with the number of stacked images, and the receiving channel transmits a stacked image; Each image block is merged and convolved to obtain the feature image block corresponding to the position of the window; the receiving channel of the filter in other groups is cascaded with the output channel of the previous filter, which is used to perform full convolution of the received feature image block product; the output channel of each filter is provided with the down-sampling unit, which is used to down-sample the received feature image block, and send the down-sampled feature image block to the next-level filter or the upper The receiving channel of the sampling device.

In some embodiments, the filter is used to Combine and convolute the image blocks corresponding to the same window position in each stacked image; where W _k is the kth convolution kernel, and b _k is the offset added to each channel corresponding to the kth convolution kernel , c is the number of channels of the stacked image, (u, v) is the size of W _k , (i, j) is the pixel position in the image block.

In some embodiments, the filter includes at least one of the following: a color-based filter, a line-based filter, and a grayscale-based filter.

In some implementations, the upsampling device is configured to perform upsampling based on quadratic interpolation on the received feature image block according to the window size.

In some embodiments, the up-sampling device includes: an up-sampling unit, configured to fill in a preset expansion window based on each pixel value in the received feature image block, so as to obtain an up-sampled feature image block a smoothing processing unit, configured to convolve the once upsampled feature image block based on a preset convolution kernel consistent with the window size, to obtain a second upsampled feature image block corresponding to the window size.

In some implementations, the heat prediction device is configured to perform a process on each pixel of the heat prediction map corresponding to each received feature image block based on a pre-learned loss function including true value weights and false value weights. The popularity evaluations are performed one by one, and the popularity prediction map is drawn based on the obtained popularity evaluations.

In some implementations, the loss function is a cross-entropy function obtained based on pre-learning and including true value weights and false value weights.

In some embodiments, the multiple pieces of stacked image data are consecutive odd-numbered images.

Based on the above purpose, the present invention also provides a convolutional neural network-based lung tumor recognition system, including: an image library for storing multiple stacked lung CT images in a spatial order; image heat prediction as described in any one of the above system; the image heat prediction system outputs a heat prediction map corresponding to the multiple lung CT stack images according to the received multiple lung CT stack images; the control device is used to judge whether the image heat prediction system has extracted All the stored lung CT stack images, if not, control the image heat prediction system to extract multiple lung CT stack images in sequence until it is determined that the image extraction device has extracted all the stored lung CT stack images CT stacked images so far.

In some implementations, the control device extracts multiple CT stacked images in stages based on the span of one image.

Based on the above purpose, the present invention also provides an image heat prediction method, including: receiving multiple stacked images collected along preset dimensions; The down-sampling device based on the convolutional network divides each of the stacked images into blocks according to the same movement rule of the same window, and merges and convolutes the image blocks in each stacked image corresponding to the same window position to obtain the feature image block , and each feature image block is down-sampled; the down-sampled feature image blocks are up-sampled so that at least the feature image blocks corresponding to the adjacent window positions and the up-sampled feature image blocks have pixel overlap; The probability that at least one feature image block after upsampling is true or false is used to draw the color of each pixel in the heat prediction map; wherein, each pixel in the heat prediction map belongs to at least one feature image after upsampling piece.

In some embodiments, after receiving the multiple stacked images collected along the preset dimension, it also includes: performing normalization processing on the multiple stacked images; moving the normalized images according to the same window Block multiple stacked images.

In some implementations, the convolutional network-based down-sampling device includes: at least one set of structures consisting of filters and down-sampling units; wherein, when the number of the structures is multiple, each set of structures is mutually Cascading; wherein, the filter connected to the image receiving device has a receiving channel consistent with the number of stacked images, and the receiving channel transmits a stacked image; the filter combines each stacked image corresponding to the same window position The image blocks are merged and convolved to obtain the feature image block corresponding to the window position; the receiving channel of the filter in other groups is cascaded with the output channel of the previous filter, and the received feature image block is fully convoluted; The down-sampling unit on the output channel of each filter stage down-samples the received feature image blocks, and sends the down-sampled feature image blocks to the next-stage filter or performs an up-sampling operation.

In some embodiments, the filter combines and convolutes the image blocks corresponding to the same window position in each stacked image to obtain the feature image block corresponding to the window position, including: the filter according to Combine and convolute the image blocks corresponding to the same window position in each stacked image; where W _k is the kth convolution kernel, and b _k is the offset added to each channel corresponding to the kth convolution kernel , c is the number of channels of the stacked image, (u, v) is the size of W _k , (i, j) is the pixel position in the image block.

In some embodiments, the filter includes at least one of the following: a color-based filter, a line-based filter, and a grayscale-based filter.

In some embodiments, the performing upsampling processing on the downsampled feature image blocks includes: performing upsampling based on quadratic interpolation on the received feature image blocks according to the window size.

In some implementations, the performing upsampling based on quadratic interpolation on the received feature image block according to the window size includes: filling in a preset extended window based on each pixel value in the received feature image block , to obtain a feature image block that has been upsampled once; based on a preset convolution kernel that is consistent with the window size, the feature image block that has been upsampled once is convolved to obtain a second upsampled image corresponding to the window size feature image blocks.

In some implementations, the probability that at least one feature image block after upsampling is true or false is used to draw the color of each pixel in the heat prediction map; wherein, each pixel in the heat prediction map The point belongs to at least one feature image block after upsampling, including: based on the loss function including the true value weight and the false value weight obtained through pre-learning, each pixel point of the heat prediction map corresponds to each feature image block received. The popularity evaluations are performed one by one, and the popularity prediction map is drawn based on the obtained popularity evaluations.

In some implementations, the loss function is a cross-entropy function obtained based on pre-learning and including true value weights and false value weights.

In some embodiments, the multiple pieces of stacked image data are consecutive odd-numbered images.

Based on the above purpose, the present invention also provides a method for identifying lung tumors based on convolutional neural networks, which includes: pre-storing multiple stacked lung CT images in a spatial order; according to any of the image heat prediction methods described above, according to Output the heat prediction map corresponding to the multiple received lung CT stack images; determine whether the image heat prediction system has extracted all the stored lung CT stack images, if not, then The image heat prediction system is controlled to extract a plurality of stacked lung CT images in sequence in space until it is determined that the image extraction device has extracted all the stacked lung CT images.

In some implementations, the control device extracts multiple CT stacked images in stages based on the span of one image.

As mentioned above, the convolutional neural network-based lung tumor identification system and method of the present invention have the following beneficial effects: each feature image block that can represent the common features of multiple stacked images is extracted through the convolutional neural network, thus obtaining Each image includes the feature information of the plane and the dimension outside the plane, and then by upsampling each feature image block and predicting the heat map, the recognition rate of the local area of the stacked image with spatial correlation is effectively improved. At the same time, setting the down-sampling unit in the convolutional neural network is beneficial to reduce the calculation amount of the convolutional neural network and effectively improve the efficiency of tumor recognition. In addition, a secondary interpolation method is adopted during upsampling, which can smooth the color transition of each image block and improve recognition accuracy. In addition, the use of an odd number of consecutive stacked images not only helps to preserve the feature information of the spatial dimension, but also ensures the balanced evaluation of the front and back images on the middle image. In addition, because the loss function can only reflect whether the model training is benign, it cannot be used as the final accurate result indicator. After the experiment, calculate and compare the position of each tumor and calculate FalsePositive and False Negative. The present invention has dropped to close to 5% on False Negative and 20% on FalsePositive. It can be seen that, compared with the prior art, the present invention has achieved a breakthrough.

Description of drawings

FIG. 1 is a schematic structural diagram of the image heat prediction system of the present invention.

FIG. 2 is a schematic diagram of image blocks in each stacked image combined and convolved by filters in the image heat prediction system of the present invention.

FIG. 3 is a schematic structural diagram of another image heat prediction system of the present invention.

Fig. 4 is a schematic structural diagram of the convolutional neural network-based lung tumor recognition system of the present invention.

FIG. 5 is a flow chart of the image heat prediction method of the present invention.

Fig. 6 is a flow chart of the convolutional neural network-based lung tumor identification method of the present invention.

FIG. 7 is a flow chart of another convolutional neural network-based lung tumor identification method of the present invention.

Figure 8 shows the comparison between the grayscale image of the lung CT stack image, the artificially marked tumor image, and the lung tumor heat prediction map identified by the image heat prediction system from left to right. .

detailed description

Embodiments of the present invention are described below through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation modes, and various modifications or changes can be made to the details in this specification based on different viewpoints and applications without departing from the spirit of the present invention.

See Figures 1-8. It should be noted that the diagrams provided in this embodiment are only schematically illustrating the basic idea of the present invention, and only the components related to the present invention are shown in the diagrams rather than the number, shape and shape of the components in actual implementation. Dimensional drawing, the type, quantity and proportion of each component can be changed arbitrarily during actual implementation, and the component layout type may also be more complicated.

Embodiment one

As shown in FIG. 1 , the present invention provides a schematic structural diagram of an image heat prediction system. The image heat prediction system includes software and hardware installed in computer equipment. Wherein, the hardware in the computer device includes: an input unit, a processing unit, a storage unit, a cache, and a display unit, etc., wherein the processing unit may include a chip or an integrated circuit dedicated to a convolutional neural network and include A computer program for the convolutional neural network algorithm. The processing unit allocates the operation of each hardware through the time sequence set by the program, so as to execute the functions of the following devices. Wherein, the computer equipment includes, but is not limited to: a single server, a server cluster in which multiple servers cooperate to operate, and the like.

The image popularity prediction system 1 includes: an image receiving device 11 , a convolutional network-based downsampling device 12 , an upsampling device 13 , and a popularity prediction device 14 .

The image receiving device 11 is used for receiving multiple stacked images collected along preset dimensions. Here, the image receiving device 11 may include a processing unit, a cache, and an interface connected to an image library storing stacked images. The processing unit in the image receiving device 11 reads multiple stacked images pre-collected along preset dimensions from the image library through the interface according to the sequence instruction of the program. Wherein, the plurality of stacked images are read from a set of images taken along a preset time dimension or space dimension and which can be overlapped together to reflect the correlation of features of the taken dimensions.

For example, in the CT image library, the image receiving device 11 reads a plurality of consecutive CT stacked images from the CT image library, where the number of the CT stacked images is an odd number. In order to facilitate subsequent heat prediction on the received multiple stacked images, the multiple stacked images have the same size.

In a specific example, the multiple stacked images are, for example, a lung CT image with a size of h×512×512, where h is the number of CT scan layers.

The convolutional network-based down-sampling device 12 is used to divide each of the stacked images into blocks according to the same window and the same moving rule, and merge image blocks in each stacked image corresponding to the same window position. to obtain feature image blocks, and down-sample each feature image block.

The convolutional network in the down-sampling device 12 based on the convolutional network is a convolutional neural network (Convolutional Neural Network, referred to as CNN), and the convolutional neural network includes a convolutional layer (alternatingconvolutional layer) and a pool layer (pooling layer) ). Wherein, the convolutional layer can be regarded as filters that will be described in detail later, and the pooling layer can be regarded as a downsampling unit that will be described in detail later.

Here, the convolutional network-based down-sampling device 12 includes: a processing unit capable of processing convolutional network algorithms, and a matching cache. The processing unit may be shared with the processing unit in the image receiving device 11, or may refer to a unit including a chip or an integrated circuit dedicated to convolutional neural networks. If the convolutional network-based down-sampling device 12 includes the above chip or integrated circuit. In order to improve the image processing speed, the chip or integrated circuit is connected to the image receiving device 11 through a hardware image receiving channel. Each receiving channel receives each stacked image provided by the image receiving device 11 in parallel.

Specifically, the convolutional network-based down-sampling device 12 uses the above-mentioned hardware to form a structural form including a filter and a down-sampling unit. Wherein, the number of the filter can be one or more. Each filter and downsampling unit may be integrated in the chip/integrated circuit. The image receiving device 11 is connected to at least one filter via each receiving channel. Wherein, the filter includes at least one of the following: a color-based filter, a line-based filter, and a grayscale-based filter. Wherein, the window size of the window is smaller than the image size of the stacked image. The moving rule of the window in each stacked image is uniform. The processing unit in the down-sampling device 12 based on the convolutional network first starts from the same starting pixel point (such as [0,0]) in each stacked image, and follows the same moving rule (such as a moving rule with a span of 1) ), block each stacked image.

For example, the processing unit divides the stacked image A into image blocks a11, a12, ..., a1n, a21, a22, ..., and amn; divides the stacked image B into image blocks b11, b12, ..., b1n, b21, b22, ... , and bmn; divide the stacked image C into image blocks c11, c12, ..., c1n, c21, c22, ..., and cmn; divide the stacked image D into image blocks d11, d12, ..., d1n, d21, b22, ..., and dmn; Divide the stacked image E into image blocks e11, e12, ..., e1n, e21, e22, ..., and emn. Among them, m is the number of rows of the image block, and n is the number of columns of the image block.

It should be noted that, according to the actual heat prediction requirements, the window can perform carpet-like block processing on each stacked image according to the movement rule with a span of 1 pixel row/pixel column, or it can target the preset image area Perform block processing. In existing or future technical solutions, if there is a block method based on the inspiration of this embodiment, it falls within the scope of the present invention. Next, the processing unit sends the image blocks with the same superscript (ie superscript [m, n]) in each stacked image to the same filter for merging and convolution. Wherein, the combined convolution refers to convolving each image block with the convolution kernel in the filter and merging into a feature image block, so that the feature information reflected by each pixel in the filtered feature image block It is possible to comprehensively represent the features of each input image block.

For example, as shown in Figure 2, the filter receives image blocks with the same subtitle as [m,n] in each stacked image, and convolves each image block with the convolution kernel in the filter, and each The results of pixel convolution are summed to obtain pixel feature information that can comprehensively reflect the features of each image block.

Alternatively, the blocking process may be part of a filter. The convolutional network in the convolutional network-based down-sampling device 12 is formed by stacking several layers, and generally the output of the next layer is the input of a higher layer. The picture is input from the bottom layer, and the output of the top layer is the final result. The structure of a more special neural network can be a directed graph, which is used to complete some special tasks. The input of each filter layer is a three-dimensional array h, w, d, where h and w are the dimensions of the input, and d represents the special feature map or channel. More specifically, for the input image, h and w are the height and width of the image, and d is the number of color channels of the input image (the number of color channels of the regular RGB image is 3, and the number of grayscale images is 1). When each stacked image enters the filter according to the input and output paths of the signal, the size of the convolution kernel in the filter is the window size. In this embodiment, according to the full convolution filtering method, each stacked image is input into the filter, and the filter performs combined convolution on the divided image blocks corresponding to the same window position according to the window movement rule.

In an optional solution, the convolutional network-based down-sampling device 12 includes: a normalization unit, configured to perform normalization processing on the multiple stacked images, and then move them according to the same window and the same According to the rule, the normalized multiple stacked images are divided into blocks.

Specifically, the normalization unit normalizes the pixel values of each stacked image to be between [0,1]. The normalized stacked images are then fed into the filter.

In another option, the filter is used to Combine and convolute the image blocks corresponding to the same window position in each stacked image; where W _k is the kth convolution kernel, and b _k is the offset added to each channel corresponding to the kth convolution kernel , c is the number of channels of the stacked image, (u, v) is the size of W _k , (i, j) is the pixel position in the image block.

In order to better extract feature information that can comprehensively represent each image block, there are multiple filters in the convolutional network-based down-sampling device 12 , and each filter is cascaded. As shown in Figure 3.

For example, Filter11, Filter12 and Filter13 are used to extract the color, line and grayscale features of the image block respectively. And the filter Filter11 is connected with the image receiving device 11 to receive the stacked images of each receiving channel. Filters Filter12 and Filter13 are sequentially cascaded after Filter11, and perform correlation filtering for lines and grayscales on the feature image blocks output by Filter11.

Next, the down-sampling unit down-samples each feature image block output by the filter in a manner such as selecting an extreme value or an average value from a preset pixel point group. Among them, the size of the convolution kernel of the filter in each level structure is preferably not larger than the size of the pixel group in the downsampling unit, that is, 0≤u≤s; 0≤v≤s. Among them, (u, v) is the size of the convolution kernel, and (s, s) is the size of the pixel grouping in the downsampling unit.

It can be found that after passing through a downsampling layer with a window size of s, the number of channels of the data remains unchanged, and the length and width are respectively reduced to the original 1/s, so that the total data size is reduced to the original 1/s ² . Another advantage of the downsampling layer is that if the downsampling layer is not added, the convolutional neural network will be particularly sensitive to the position of the object, and even a small movement can cause great deviations in the intermediate results of the network. After adding the down-sampling layer, the convolutional neural network can better adapt to the impact of displacement, and keep the intermediate results from large deviations. After the data size is continuously reduced, due to the reduction of computational overhead, we can appropriately increase the number of feature maps in the deep layer of the network, so that the convolutional neural network can learn more image features.

In addition, in order to improve computing efficiency, each filter can use multiple CPUs (or GPUs) in the server to perform combined convolution on the image blocks at each window position in parallel.

For example, the down-sampling unit is grouped according to 4×4 pixel points, averages the feature values of every 4×4 pixel points in the received feature image block, and assigns the average value to the down-sampled image block corresponding pixels.

In order to increase the processing speed of the subsequent up-sampling process, an optional solution is that the convolutional network-based down-sampling device 12 further condenses the feature information of each image block.

Specifically, the convolutional network-based down-sampling device 12 includes: multiple sets of structures composed of filters and down-sampling units, and each set of structures is cascaded with each other.

Wherein, the filter connected to the image receiving device 11 has a receiving channel consistent with the number of stacked images, and the receiving channel transmits a stacked image; The blocks are merged and convolved to obtain the feature image block corresponding to the window position. The receiving channels of the filters in the other groups are cascaded with the output channels of the previous filter to perform full convolution on the received feature image blocks. The down-sampling unit is provided on the output channel of each stage filter, which is used to down-sample the received feature image block, and send the down-sampled feature image block to the next-stage filter or up-sampling device 13 receive channels.

For example, the convolutional network-based down-sampling device 12 includes four levels of the aforementioned structures Structure1, Structure2, Structure3 and Structure4. Among them, each level structure contains multiple filters and a downsampling unit, and the filters in each structure are also connected in cascade, and the downsampling unit receives the feature image block output by the last filter in the group and performs downsampling deal with. The filter connected to the image receiving device 11 in Structure 1 performs combined convolution on the image blocks corresponding to the same window position in each stacked image, and sends each feature image block after combined convolution to the next filter for further processing. Feature extraction until the last filter in Structure1 finishes filtering each feature image block, and then send each feature image block to the down-sampling unit for down-sampling. The structure Structure1 sends the downsampled feature image blocks to the filter in the structure Structure2 in order to extract features one by one, and after the last filter in the structure Structure2 has filtered each feature image block, the downsampling in the structure The unit down-samples each feature image block again. After each stacked image is extracted and down-sampled by the convolutional network-based down-sampling device 12, the size of each feature image block is much smaller than the size of the original image block, but the feature information of the original image block is still retained.

Next, each characteristic image block is sent to the upsampling device 13 . The up-sampling device 13 is configured to perform up-sampling processing on the down-sampled feature image blocks, so that at least the up-sampled feature image blocks corresponding to adjacent window positions have pixel overlapping.

Specifically, the up-sampling device 13 may divide each pixel according to the size difference between the received characteristic image block and the original image block, and according to the area in the original image block corresponding to each pixel in each characteristic image block. The value of the point is copied to the corresponding field.

For example, the upsampling device 13 utilizes for upsampling. Wherein, the size of the region is (s, s). It can be seen that each input pixel is copied into a rectangle of size s*s in the output, such upsampling maintains the spatial structure of the original data while enlarging the data scale.

In order to make the final heat prediction map more accurately describe the common features in each stacked image, an optional solution is that the up-sampling device 13 performs binary-based processing on the received feature image blocks according to the window size. Upsampling for subinterpolation.

Specifically, the upsampling device 13 includes: an upsampling unit and a smoothing processing unit.

Wherein, the up-sampling unit is configured to fill in a preset expansion window based on each pixel value in the received feature image block, so as to obtain a feature image block that is up-sampled once.

The upsampling unit can use the formula: Repeatedly filling each pixel point in the received feature image block into the corresponding area of the preset expansion window. Wherein, the size of the expanded window is smaller than the size of the aforementioned window used to divide the image into blocks. Considering that i/s and j/s may be real numbers, this example adopts quadratic interpolation of the nearest four pixels to approximate the value of each pixel.

The smoothing processing unit is configured to convolve the once-upsampled feature image block based on a preset convolution kernel consistent with the window size, to obtain a second-upsampled feature image block corresponding to the window size.

The upsampling method using the upsampling unit and the smoothing processing unit is intuitively superior to directly upsampling the image block, but the effect of the actual experiment is almost the same as the former. The reason behind it is that if the upsampling layer is followed by a convolutional layer and the filter core size of the convolutional layer is larger than s, then the above-mentioned secondary interpolation can be obtained by learning the filter parameters after this filter. Therefore, the above-mentioned secondary interpolation is in a sense included in the latter convolutional layer, although it is not completely equivalent. The receptive fields of the above filter layers are obviously interleaved, and the parameters of each filter are shared within the layer, so the GPU-based forward propagation and back propagation calculations will become very efficient.

Here, the smoothing processing unit may only include a convolution kernel consistent with the size of the window, or perform processing such as weighted average on each feature image block on the basis of convolution. Each feature image block processed by the smoothing processing unit may have 1 span according to the window position sequence.

Here, the feature image block after the upsampling process can be regarded as reflecting the heat of the image block at the corresponding window position of each stacked image.

Next, the heat prediction device 14 is used to draw the color of each pixel in the heat prediction map by using the probability that the up-sampled at least one feature image block is true or false; wherein, each pixel in the heat prediction map The pixel points belong to at least one feature image block after upsampling.

Here, the popularity prediction device 14 may share hardware such as processing units and buffers in the aforementioned devices, and receive the characteristic image blocks output by the up-sampling device 13 .

Specifically, the popularity prediction device 14 evaluates the popularity information of each pixel in the heat prediction map by using the probability that each received feature image block is consistent or inconsistent with the true value.

Considering that the convolutional neural network needs to evaluate the final input heat map y, we need to define a loss function for the output y. The loss function can be output and ground truth The sum of squares between. For example, the loss function is The loss function may also be set based on cross entropy. For example, the loss function is

In practical applications, since the true value can only take a value of 0 or 1, in this case the loss function takes cross entropy to better express the closeness of the output to the true value. An optional way is that the heat prediction device 14 performs heat evaluation on each pixel of the heat prediction map corresponding to each received characteristic image block one by one according to a preset loss function, and based on the preset heat evaluation and Corresponding relationship of heat color, drawing the heat prediction map. Wherein, the loss function is used to represent the probability that each feature image block where the pixel point in the heat prediction map is consistent with or inconsistent with the true value. For example, if the loss function is the mean square error between the feature image block and the true value, then the heat prediction device 14 sequentially evaluates the heat of each pixel in the heat prediction map as follows:

The feature image blocks y _i , y _i+1 , ..., and y _i+t all contain the pixel point O(a,b) in the heat prediction map, then the heat prediction device 14 uses the formula Calculate the heat of pixel O(a,b).

It should be noted that the above loss function is only an example rather than a limitation of the present invention. In fact, the loss function can also be cross entropy or other loss functions.

In order to more accurately predict the heat of each stacked image, the heat prediction device 14 corresponds to each pixel of the heat prediction map based on the loss function obtained through pre-learning, which includes the true value weight and the false value weight. The image blocks are evaluated one by one, and the heat prediction map is drawn based on the obtained heat evaluation.

Wherein, the loss function obtained through pre-machine learning may be an improved function based on the above mean square error formula. The preferred loss function in this embodiment is a cross-entropy function obtained based on pre-learning and including true value weights and false value weights:

$\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; log</span>}<span\; class="notranslate">\; (</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; b</span>}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; c</span>}\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; log</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; w</span>}\mathrm{<span\; class="notranslate">\; h</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span>$

Among them, black_ratio represents the true value weight determined by machine learning in advance, and white_ratio represents the false value weight determined by machine learning in advance.

Using the above loss function, the heat prediction device 14 evaluates the heat of each pixel in the heat prediction map one by one, and then draws the heat prediction map according to the preset correspondence between heat evaluation and color.

The balanced cross-entropy can well balance the true value of the black background and the true value of the white background, so that it is easier for the neural network to converge towards the global optimal solution at the beginning of training.

Embodiment two

Based on the optional solutions in the first embodiment above, this embodiment provides a convolutional neural network-based lung tumor recognition system. Wherein, the lung tumor identification system based on the convolutional neural network is applied in a CT image scanning instrument of a hospital, or an image processing device (such as a server) connected with the CT image scanning instrument. For a CT image scanning instrument, at least a CT scanning device is included. The CT scanning device scans the parts to be examined (such as the lungs, the brain, or the whole body) of the living body in a cross-sectional manner along the spatial axis, and obtains corresponding multiple CT stacked images. Correspondingly, the convolutional neural network-based lung tumor identification system 3 includes: an image library 31 , an image heat prediction system 32 combined with any optional solution in the first embodiment above, and a control device 33 . As shown in Figure 4.

In this embodiment, the image heat prediction system 32 of the lung tumor recognition system based on the convolutional neural network is used as an example for illustration. The lung tumor recognition system 3 based on the convolutional neural network is used to identify the tumor.

Wherein, the image heat prediction system 32 and the CT scanning device 2 share an image library 31 . The CT scanning device 2 stores the scanned CT stack images in the image library 31 in a spatial order. Under the control of the control device 33, the image heat prediction system 32 reads multiple CT stacked images therein, and performs heat prediction of multiple CT stacked images according to any scheme described in Embodiment 1, and According to the doctor's operation, the obtained heat prediction map is provided to the doctor.

Wherein, the control device 33 is used to judge whether the image heat prediction system 32 has extracted all the stored CT stack images, and if not, control the image heat prediction system 32 to extract multiple CT stack images in sequence according to the space. images until it is determined that the image extraction device has extracted all the stored CT stack images.

For example, the control device 33 provides a plurality of stacked images to the image heat prediction system 32 in stages by taking 5 CT stacked images as a group according to the sequence of the stacked images in the image library 31 . When the image heat prediction system 32 outputs the heat prediction map according to the received five CT stack images, the control device 33 judges whether all the stored CT stack images have been extracted, if so, exit the loop, otherwise continue to extract The next group of CT stacked images is input to the image temperature prediction system 32 until all CT stacked images are extracted.

Wherein, the heat prediction map output by the image heat prediction system 32 can be regarded as the heat prediction map shared by the group of CT stacked images. Preferably, the heat prediction map output by the image heat prediction system 32 is regarded as the heat prediction map corresponding to the middle CT stack image in the group of CT stack images.

In order to improve the heat prediction accuracy of CT stacked images, in an optional solution, when it is judged that the traversal of CT stacked images has not been completed, the control device 33 extracts different groups of CT stacked images based on the span of one image, so that Get the heat prediction map of all CT stack images except the first and last two CT stack images.

For example, the control device 33 first sends {x ₁ , x ₂ , x ₃ , x ₄ , x ₅ } five stacked CT images to the image heat prediction system 32 to predict the heat of image x ₃ . When judging that x ₅ is not the last one of the CT stack images in the image library 31, the control device 33 continues to select {x ₂ , x ₃ , x ₄ , x ₅ , x ₆ } 5 CT stack images to send to the image heat Prediction system 32 performs heat prediction for image _x4 . This is repeated until it is judged that the extracted group of CT stack images contains the last one, and the loop is exited.

As shown in the flow process in Figure 7, the working process of the convolutional neural network-based lung tumor recognition system 3 is exemplified as follows:

After the CT image scanning apparatus acquires multiple CT stacked images of the patient, the patient data and the multiple CT stacked images are correspondingly stored in the image database 31 . When the control device 33 completes the scan based on the instruction of the CT image scanning instrument, or the doctor sends an instruction to obtain the heat prediction map to the device where the control device 33 is located through the operation terminal, the control device 33 finds the corresponding scan according to the patient data. Store multiple CT stacked images sequentially, and inform the image extraction device in the image heat prediction system 32 to extract the 1st to 5th CT stacked images.

The image extraction device extracts corresponding multiple CT stacked images from the image library 31 and transmits them to the convolutional network-based down-sampling device using the receiving channel corresponding to each CT stacked image. The filter Filter11 in the first-level structure of the down-sampling device based on the convolutional network combines and convolves the image blocks at the same window position in each CT stack image according to the uniform window size and window movement rule, thus obtaining Feature image patches corresponding to each window position. At least one filter (Filter12, . . . , Filter1n) cascaded with Filter11 is also included in the first-level structure. In the first-level structure, the size of the convolution kernel of each filter can be consistent with the above-mentioned window size, so as to reduce the amount of calculation and optimize the structure of the convolutional neural network. The down-sampling unit in the first-level structure is located after the filter Filter1n, and it down-samples the feature image blocks corresponding to each window position by grouping the feature image blocks into groups and down-sampling. Then the down-sampled feature image blocks are sent to at least one filter (Filter21, . . . , Filter2n) in the second-level structure. Each filter in the second-level structure performs full convolution processing on the received feature image blocks to obtain further feature image blocks corresponding to the positions of each window, wherein the size of the convolution kernel in the second-level structure smaller than the size of the received feature image block. Then, the size of each feature image block is further reduced through the down-sampling unit in the second-level structure. Through the m-times cascade structure, the feature image blocks corresponding to each window position are greatly reduced.

Next, the up-sampling device recovers the reduced characteristic image blocks by means of secondary interpolation, so as to obtain thermal image blocks corresponding to the positions of the windows.

Next, the heat prediction device utilizes the feature that each heat image block has an overlapping area, and assigns the heat evaluation of each pixel in the heat prediction map to the heat image block that covers the pixel for determination. Specifically, the heat prediction device adopts a loss function based on pre-learned weights including true weights and false weights, and each pixel of the heat prediction map corresponds to each received feature image block (that is, heat image block) Carry out heat evaluation one by one, and draw the heat prediction map based on the obtained heat evaluation, the drawn heat prediction map is the heat prediction map corresponding to the middle CT stack image among the 5 CT stack images.

When the control device determines that the image heat prediction system 32 completes the heat prediction map for the middle CT stack image among the 5 controlled CT stack images, it is judged whether the completed 5 CT stack images already contain the same patient If it is the last one, exit the loop, if not, hand over the 2nd to 6th CT stack images to the image receiving device, and re-execute the image heat prediction system 32 to obtain the heat corresponding to the 4th CT stack image Forecast graph. With such an iterative cycle, the convolutional neural network-based lung tumor recognition system can obtain the heat prediction maps of the other CT stacked images except for the first and last two CT stacked images.

Embodiment Three

As shown in FIG. 5 , the present invention provides a flowchart of an image heat prediction method. The image heat prediction method is mainly performed by a prediction system. Wherein the prediction system includes software and hardware installed in computer equipment. Wherein, the hardware in the computer device includes: an input unit, a processing unit, a storage unit, a cache, and a display unit, etc., wherein the processing unit may include a chip or an integrated circuit dedicated to a convolutional neural network and include A computer program for the convolutional neural network algorithm. The processing unit allocates the operation of each hardware through the time sequence set by the program, so as to execute the functions of the following devices. Wherein, the computer equipment includes, but is not limited to: a single server, a server cluster in which multiple servers cooperate to operate, and the like.

The image heat prediction method realizes the prediction of image heat by performing the following steps.

Step S110, receiving multiple stacked images collected along preset dimensions. Here, the prediction system may comprise a processing unit, a cache, and an interface to an image library storing stacked images. According to the timing instruction of the program, the prediction system reads multiple stacked images pre-collected along the preset dimensions from the image library through the interface. Wherein, the plurality of stacked images are read from a set of images taken along a preset time dimension or space dimension and which can be overlapped together to reflect the correlation of features of the taken dimensions.

For example, in the CT image library, the prediction system reads a plurality of consecutive CT stacked images from the CT image library, where the number of the CT stacked images is an odd number. In order to facilitate subsequent heat prediction on the received multiple stacked images, the multiple stacked images have the same size.

In a specific example, the multiple stacked images are, for example, a lung CT image with a size of h×512×512, where h is the number of CT scan layers.

Step S120, the convolutional network-based down-sampling device in the prediction system divides each of the stacked images into blocks according to the same window and the same movement rule, and divides the image blocks in each stacked image corresponding to the same window position Combined convolution is performed to obtain feature image blocks, and each feature image block is down-sampled.

The convolutional network in the convolutional network-based down-sampling device is a convolutional neural network (Convolutional Neural Network, CNN for short), and the convolutional neural network includes a convolutional layer (alternating convolutional layer) and a pooling layer (pooling layer). Wherein, the convolutional layer can be regarded as filters that will be described in detail later, and the pooling layer can be regarded as a downsampling unit that will be described in detail later.

The convolutional network in the convolutional network-based down-sampling device is formed by stacking several layers, and generally the output of the next layer is the input of a higher layer. The picture is input from the bottom layer, and the output of the top layer is the final result. The structure of a more special neural network can be a directed graph, which is used to complete some special tasks. The input of each filter layer is a three-dimensional array h, w, d, where h and w are the dimensions of the input, and d represents the special feature map or channel. More specifically, for the input image, h and w are the height and width of the image, and d is the number of color channels of the input image (the number of color channels of the regular RGB image is 3, and the number of grayscale images is 1).

Here, the convolutional network-based down-sampling device includes: a processing unit capable of processing convolutional network algorithms, and a matching cache. The processing unit may be shared with the processing unit in the image receiving device, or may refer to a unit including a chip or an integrated circuit dedicated to convolutional neural networks. If the convolutional network-based down-sampling device includes the above-mentioned chip or integrated circuit. In order to improve the image processing speed, the chip or the integrated circuit is connected with the image receiving device through a hardware image receiving channel. Each receiving channel receives each stacked image provided by the image receiving device in parallel.

Specifically, the convolutional network-based down-sampling device utilizes the above-mentioned hardware to form a structure including a filter and a down-sampling unit. Wherein, the number of the filter can be one or more. Each filter and downsampling unit may be integrated in the chip/integrated circuit. The image receiving device is connected to at least one of the filters via receiving channels. Wherein, the filter includes at least one of the following: a color-based filter, a line-based filter, and a grayscale-based filter. Wherein, the window size of the window is smaller than the image size of the stacked image. The moving rule of the window in each stacked image is uniform. The processing unit in the down-sampling device based on the convolutional network first starts from the same starting pixel point (such as [0,0]) in each stacked image, and follows the same moving rule (such as a moving rule with a span of 1) , divide each stacked image into blocks.

For example, the processing unit divides the stacked image A into image blocks a11, a12, ..., a1n, a21, a22, ..., and amn; divides the stacked image B into image blocks b11, b12, ..., b1n, b21, b22, ... , and bmn; divide the stacked image C into image blocks c11, c12, ..., c1n, c21, c22, ..., and cmn; divide the stacked image D into image blocks d11, d12, ..., d1n, d21, b22, ..., and dmn; Divide the stacked image E into image blocks e11, e12, ..., e1n, e21, e22, ..., and emn. Among them, m is the number of rows of the image block, and n is the number of columns of the image block.

It should be noted that, according to the actual heat prediction requirements, the window can perform carpet-like block processing on each stacked image according to the movement rule with a span of 1 pixel row/pixel column, or it can target the preset image area Perform block processing. In existing or future technical solutions, if there is a block method based on the inspiration of this embodiment, it falls within the scope of the present invention.

Next, the processing unit sends the image blocks with the same superscript (ie superscript [m, n]) in each stacked image to the same filter for merging and convolution. Wherein, the combined convolution refers to convolving each image block with the convolution kernel in the filter and merging into a feature image block, so that the feature information reflected by each pixel in the filtered feature image block It is possible to comprehensively represent the features of each input image block.

For example, as shown in Figure 2, the filter receives image blocks with the same subtitle as [m,n] in each stacked image, and convolves each image block with the convolution kernel in the filter, and each The results of pixel convolution are summed to obtain pixel feature information that can comprehensively reflect the features of each image block.

Alternatively, the blocking process may be part of a filter. The convolutional network in the convolutional network-based down-sampling device 12 is formed by stacking several layers, and generally the output of the next layer is the input of a higher layer. The picture is input from the bottom layer, and the output of the top layer is the final result. The structure of a more special neural network can be a directed graph, which is used to complete some special tasks. The input of each filter layer is a three-dimensional array h, w, d, where h and w are the dimensions of the input, and d represents the special feature map or channel. More specifically, for the input image, h and w are the height and width of the image, and d is the number of color channels of the input image (the number of color channels of the regular RGB image is 3, and the number of grayscale images is 1). When each stacked image enters the filter according to the input and output paths of the signal, the size of the convolution kernel in the filter is the window size. In this embodiment, according to the full convolution filtering method, each stacked image is input into the filter, and the filter performs merge convolution on the divided image blocks corresponding to the same window position according to the window moving rule.

In an optional solution, the convolutional network-based down-sampling device includes: a normalization unit, configured to perform normalization processing on the multiple stacked images, and then follow the same window and the same moving rule , dividing the normalized multiple stacked images into blocks.

Specifically, the normalization unit normalizes the pixel values of each stacked image to be between [0,1]. The normalized stacked images are then fed into the filter.

In another option, the filter is used to Combine and convolute the image blocks corresponding to the same window position in each stacked image; where W _k is the kth convolution kernel, and b _k is the offset added to each channel corresponding to the kth convolution kernel , c is the number of channels of the stacked image, (u, v) is the size of W _k , (i, j) is the pixel position in the image block.

In order to better extract feature information that can comprehensively represent each image block, there are multiple filters in the convolutional network-based down-sampling device, and each filter is cascaded. As shown in Figure 3.

For example, Filter11, Filter12 and Filter13 are used to extract the color, line and grayscale features of the image block respectively. And the filter Filter11 is connected with the image receiving device to receive the laminated images of each receiving channel. Filters Filter12 and Filter13 are sequentially cascaded after Filter11, and perform correlation filtering for lines and grayscales on the feature image blocks output by Filter11.

Next, the down-sampling unit down-samples each feature image block output by the filter in a manner such as selecting an extreme value or an average value from a preset pixel point group. Among them, the size of the convolution kernel of the filter in each level structure is preferably not larger than the size of the pixel group in the downsampling unit, that is, 0≤u≤s; 0≤v≤s. Among them, (u, v) is the size of the convolution kernel, and (s, s) is the size of the pixel grouping in the downsampling unit.

It can be found that after passing through a downsampling layer with a window size of s, the number of channels of the data remains unchanged, and the length and width are respectively reduced to the original 1/s, so that the total data size is reduced to the original 1/s ² . Another advantage of the downsampling layer is that if the downsampling layer is not added, the convolutional neural network will be particularly sensitive to the position of the object, and even a small movement can cause great deviations in the intermediate results of the network. After adding the down-sampling layer, the convolutional neural network can better adapt to the impact of displacement, and keep the intermediate results from large deviations. After the data size is continuously reduced, due to the reduction of computational overhead, we can appropriately increase the number of feature maps in the deep layer of the network, so that the convolutional neural network can learn more image features.

In addition, in order to improve computing efficiency, each filter can use multiple CPUs (or GPUs) in the server to perform combined convolution on the image blocks at each window position in parallel.

For example, the down-sampling unit is grouped according to 4×4 pixel points, averages the feature values of every 4×4 pixel points in the received feature image block, and assigns the average value to the down-sampled image block corresponding pixels.

In order to improve the processing speed of subsequent upsampling processing, an optional solution is that the convolutional network-based downsampling device further concentrates the feature information of each image block.

Specifically, the convolutional network-based down-sampling device includes: multiple sets of structures composed of filters and down-sampling units, and each set of structures is cascaded to each other.

Wherein, the filter connected to the image receiving device has a receiving channel consistent with the number of stacked images, and the receiving channel transmits a stacked image; the filter is used to convert each image block corresponding to the same window position in each stacked image Combined convolution is performed to obtain the feature image block corresponding to the window position. The receiving channels of the filters in the other groups are cascaded with the output channels of the previous filter to perform full convolution on the received feature image blocks. The down-sampling unit is provided on the output channel of each stage filter, which is used to down-sample the received feature image block, and send the down-sampled feature image block to the next-stage filter or up-sampling device the receiving channel.

For example, the convolutional network-based down-sampling device includes four levels of the aforementioned structures Structure1, Structure2, Structure3 and Structure4. Among them, each level structure contains multiple filters and a downsampling unit, and the filters in each structure are also connected in cascade, and the downsampling unit receives the feature image block output by the last filter in the group and performs downsampling deal with. The filter connected to the image receiving device in Structure 1 performs combined convolution on the image blocks corresponding to the same window position in each stacked image, and sends each feature image block after the combined convolution to the next filter for feature Extract until the last filter in Structure1 finishes filtering each feature image block, and then send each feature image block to the down-sampling unit for down-sampling. The structure Structure1 sends the downsampled feature image blocks to the filter in the structure Structure2 in order to extract features one by one, and after the last filter in the structure Structure2 has filtered each feature image block, the downsampling in the structure The unit down-samples each feature image block again.

After each stacked image is extracted and down-sampled by the convolutional network-based down-sampling device, the size of each feature image block is much smaller than the size of the original image block, but the feature information of the original image block is still retained.

Step S130, each characteristic image block is sent to the up-sampling device in the prediction system. The up-sampling device performs up-sampling processing on the down-sampled characteristic image blocks, so that at least the up-sampled characteristic image blocks corresponding to adjacent window positions have pixel overlapping.

Specifically, according to the size difference between the received characteristic image block and the original image block, according to the area in the original image block corresponding to each pixel in each characteristic image block, each pixel point The value of is copied to the corresponding area.

For example, the upsampling device utilizes for upsampling. Wherein, the size of the region is (s, s). It can be seen that each input pixel is copied into a rectangle of size s*s in the output, and such upsampling maintains the spatial structure of the original data while enlarging the data scale.

In order to make the final heat prediction map more accurately describe the common features in each stacked image, an optional solution is that the up-sampling device performs a quadratic based on the received feature image block according to the window size. Upsampling for interpolation.

Specifically, the step S130 includes: S131, S132 (both not shown).

Step S131 , based on each pixel value in the received feature image block, fill in a preset expanded window to obtain a feature image block that has been upsampled once.

Specifically, using the formula: Repeatedly filling each pixel point in the received feature image block into the corresponding area of the preset expansion window. Wherein, the size of the expanded window is smaller than the size of the aforementioned window used to divide the image into blocks. Considering that i/s and j/s may be real numbers, this example adopts quadratic interpolation of the nearest four pixels to approximate the value of each pixel.

Step S132 , based on a preset convolution kernel consistent with the size of the window, perform convolution on the once-upsampled feature image block to obtain a second-upsampled feature image block corresponding to the window size.

The upsampling method adopted in steps S131 and S132 is intuitively superior to directly upsampling the image block, but the effect of the actual experiment is almost the same as the former. The reason behind it is that if the upsampling layer is followed by a convolutional layer and the filter core size of the convolutional layer is larger than s, then the above-mentioned secondary interpolation can be obtained by learning the filter parameters after this filter. Therefore, the above-mentioned secondary interpolation is in a sense included in the latter convolutional layer, although it is not completely equivalent. The receptive fields of the above filter layers are obviously interleaved, and the parameters of each filter are shared within the layer, so the GPU-based forward propagation and back propagation calculations will become very efficient.

Here, only convolution kernels with the same size as the window may be included, or processing such as weighted average may be performed on each feature image block on the basis of convolution. Each feature image block processed by the smoothing processing unit may have 1 span according to the window position sequence.

Here, the feature image block after the upsampling process can be regarded as reflecting the heat of the image block at the corresponding window position of each stacked image.

Next, step S140, use the probability that at least one feature image block after upsampling is true or false to draw the color of each pixel in the heat prediction map; wherein, each pixel in the heat prediction map belongs to the upsampling After at least one feature image block.

Specifically, the prediction system evaluates the heat information of each pixel in the heat prediction map by using the probability that each received feature image block is consistent or inconsistent with the true value.

Considering that the convolutional neural network needs to evaluate the final input heat map y, we need to define a loss function for the output y. The loss function can be output and ground truth The sum of squares between. For example, the loss function is The loss function may also be set based on cross entropy. For example, the loss function is

In practical applications, since the true value can only take a value of 0 or 1, in this case the loss function takes cross entropy to better express the closeness of the output to the true value.

An optional way is that, according to a preset loss function, the prediction system evaluates the heat of each pixel of the heat prediction map corresponding to each received feature image block one by one, and based on the preset heat evaluation and heat color Corresponding relationship, draw the heat prediction map. Wherein, the loss function is used to represent the probability that each feature image block where the pixel point in the heat prediction map is consistent with or inconsistent with the true value. For example, if the loss function is the mean square error between the feature image block and the true value, then the heat prediction device evaluates the heat of each pixel in the heat prediction map sequentially as follows:

The feature image blocks y _i , y _i+1 , ..., and y _i+t all contain the pixel point O(a,b) in the heat prediction map, then the prediction system uses the formula Calculate the heat of pixel O(a,b).

It should be noted that the above loss function is only an example rather than a limitation of the present invention. In fact, the loss function can also be cross entropy or other loss functions.

In order to more accurately predict the heat of each stacked image, the prediction system is based on a loss function obtained through pre-learning that includes true value weights and false value weights, and each pixel of the heat prediction map corresponds to each received feature image block Perform heat evaluation one by one, and draw the heat prediction map based on the obtained heat evaluation.

Wherein, the loss function obtained through pre-machine learning may be an improved function based on the above mean square error formula. The preferred loss function in this embodiment is a cross-entropy function obtained based on pre-learning and including true value weights and false value weights:

$\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}\mathrm{<span\; class="notranslate">\; log</span>}<span\; class="notranslate">\; (</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; b</span>}\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; c</span>}\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; log</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\stackrel{<span\; class="notranslate">\; ^</span>}{{\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; )</span>\mathrm{<span\; class="notranslate">\; w</span>}\mathrm{<span\; class="notranslate">\; h</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; \_</span>\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; t</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; o</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; .</span>$

Among them, black_ratio represents the true value weight determined by machine learning in advance, and white_ratio represents the false value weight determined by machine learning in advance.

Using the above loss function, the prediction system evaluates the heat of each pixel in the heat prediction map one by one, and then draws the heat prediction map according to the preset correspondence between heat evaluation and color.

The balanced cross-entropy can well balance the true value of the black background and the true value of the white background, so that it is easier for the neural network to converge towards the global optimal solution at the beginning of training.

Embodiment Four

On the basis of the alternative solutions in the third embodiment above, as shown in FIG. 6 , this embodiment provides a method for identifying lung tumors based on a convolutional neural network. Wherein, the convolutional neural network-based lung tumor identification method is mainly performed by a convolutional neural network-based lung tumor identification system. The convolutional neural network-based lung tumor identification system is applied to a CT image scanning instrument in a hospital, or an image processing device (such as a server) connected to a CT image scanning instrument. For a CT image scanning instrument, at least a CT scanning device is included. The CT scanning device scans the parts to be examined (such as the lungs, the brain, or the whole body) of the living body in a cross-sectional manner along the spatial axis, and obtains corresponding multiple CT stacked images.

In this embodiment, lung tumors are identified using the method for identifying lung tumors based on convolutional neural networks. The image heat prediction method described in each option in Embodiment 3 is used in the lung tumor identification method based on convolutional neural network. That is to say, the convolutional neural network-based lung tumor recognition system includes an image heat prediction system.

Wherein, the image heat prediction system and the CT scanning device share an image library.

Specifically, the CT scanning device stores the scanned CT stack images in an image library in a spatial order.

Next, the convolutional neural network-based lung tumor identification system starts the image heat prediction system to perform step S220 according to the control rules: read multiple CT stacked images, and follow any of the steps described in the third embodiment One solution is to predict the heat of multiple CT stacked images, and provide the obtained heat prediction map to the doctor according to the doctor's operation. The specific process is shown in Figure 7. Wherein, FIG. 8 is a schematic diagram of a grayscale image of a lung CT stack image, a manually recognized location of a lung tumor, and a location of a lung tumor identified by using this embodiment.

Next, the convolutional neural network-based lung tumor identification system executes step S230 according to the control rules: judging whether the image heat prediction system has extracted all the stored CT stack images, if not, then controlling the The image heat prediction system extracts a plurality of CT stack images in sequence according to space until it is determined that the image extraction device has extracted all the stored CT stack images.

For example, the convolutional neural network-based lung tumor recognition system uses five CT stacked images as a group according to the order of each stacked image in the image database, and starts from the first CT stacked image to predict the heat of the grouped images. The system provides CT stacked images. When the image heat prediction system outputs the heat prediction map according to the received 5 CT stack images, the control device judges whether all the stored CT stack images have been extracted, if so, exit the loop, otherwise continue to extract the next Group CT stacked images and input them into the image heat prediction system until all CT stacked images are extracted.

Wherein, the heat prediction map output by the image heat prediction system can be regarded as the heat prediction map shared by the group of CT stacked images. Preferably, the heat prediction map output by the image heat prediction system is regarded as the heat prediction map corresponding to the middle CT stack image in the group of CT stack images.

In order to improve the heat prediction accuracy of CT stacked images, in an optional solution, in the case of judging that the CT stacked images have not been traversed, the convolutional neural network-based lung tumor recognition system uses the span of one image in advance. The CT stack images are grouped, so that the heat prediction map of all CT stack images except the first and last CT stack images can be obtained.

For example, five stacked CT images of {x ₁ , x ₂ , x ₃ , x ₄ , x ₅ } are sent to the image heat prediction system to predict the heat of image x ₃ . When judging that x ₅ is not the last CT stacked image in the image library, the control device continues to select {x ₂ , x ₃ , x ₄ , x ₅ , x ₆ }5 CT stacked images to send to the image heat prediction system Make heat predictions for image x ₄ . This is repeated until it is judged that the extracted group of CT stack images contains the last one, and the loop is exited.

As shown in Figure 7, the working process of the lung tumor recognition system based on the convolutional neural network is exemplified as follows:

After the CT image scanning apparatus acquires multiple CT stacked images of the patient, the patient data and the multiple CT stacked images are correspondingly stored in the image database. When the convolutional neural network-based lung tumor recognition system is based on the instruction of the CT image scanning instrument to complete the scan, or the doctor sends an instruction to obtain the heat prediction map to the device where the control device is located through the operation terminal, the control device follows Corresponding to the patient data, multiple CT stacked images stored in the scanning order are found, and the image extraction device in the image heat prediction system is informed to extract the first to fifth CT stacked images.

The image extraction device in the image heat prediction system extracts corresponding multiple CT stacked images from the image library and transmits them to the convolutional network-based down-sampling device using the receiving channel corresponding to each CT stacked image. The filter Filter11 in the first-level structure of the down-sampling device based on the convolutional network combines and convolves the image blocks at the same window position in each CT stack image according to the uniform window size and window movement rule, thus obtaining Feature image patches corresponding to each window position. At least one filter (Filter12, . . . , Filter1n) cascaded with Filter11 is also included in the first-level structure. In the first-level structure, the size of the convolution kernel of each filter can be consistent with the above-mentioned window size, so as to reduce the amount of calculation and optimize the structure of the convolutional neural network. The down-sampling unit in the first-level structure is located after the filter Filter1n, and it down-samples the feature image blocks corresponding to each window position by grouping the feature image blocks into groups and down-sampling. Then the down-sampled feature image blocks are sent to at least one filter (Filter21, . . . , Filter2n) in the second-level structure. Each filter in the second-level structure performs full convolution processing on the received feature image blocks to obtain further feature image blocks corresponding to the positions of each window, wherein the size of the convolution kernel in the second-level structure smaller than the size of the received feature image patch. Then, the size of each feature image block is further reduced through the down-sampling unit in the second-level structure. Through multiple cascading structures, the feature image blocks corresponding to each window position are greatly reduced.

Next, the up-sampling device in the image heat prediction system recovers the reduced characteristic image blocks by means of secondary interpolation, so as to obtain heat image blocks corresponding to the positions of the windows.

Next, the heat prediction device in the image heat prediction system makes use of the feature that each heat image block has an overlapping area, and assigns the heat evaluation of each pixel in the heat prediction map to the heat image block covering the pixel for determination. Specifically, the heat prediction device adopts a loss function based on pre-learned weights including true weights and false weights, and each pixel of the heat prediction map corresponds to each received feature image block (that is, heat image block) Carry out heat evaluation one by one, and draw the heat prediction map based on the obtained heat evaluation, the drawn heat prediction map is the heat prediction map corresponding to the middle CT stack image among the 5 CT stack images. Fig. 8 is, from left to right, a diagram showing the comparison between the grayscale image of the stacked lung CT image, the artificially marked tumor image, and the heat prediction map of the lung tumor identified by the image heat prediction system.

When the control device determines that the image heat prediction system has completed the heat prediction map for the middle CT stack image among the 5 controlled CT stack images, it is judged whether the completed 5 CT stack images already contain the image corresponding to the same patient. The last one, if yes, exit the loop, if not, hand over the 2nd to 6th CT stacked images to the image receiving device, and predict the heat prediction map of the 4th CT stacked image. With such an iterative cycle, the convolutional neural network-based lung tumor recognition system can obtain the heat prediction maps of the other CT stacked images except for the first and last two CT stacked images.

In summary, each embodiment of the present invention uses a convolutional neural network to extract each feature image block that can represent the common features of multiple stacked images, thus obtaining the feature information of each image including the plane and the dimension outside the plane, and then By upsampling each feature image block and predicting the heat map, the recognition rate for the local area of the stacked image with spatial correlation is effectively improved. At the same time, setting the down-sampling unit in the convolutional neural network is beneficial to reduce the calculation amount of the convolutional neural network and effectively improve the efficiency of tumor recognition.

In addition, a secondary interpolation method is adopted during upsampling, which can smooth the color transition of each image block and improve recognition accuracy.

In addition, the use of an odd number of consecutive stacked images not only helps to preserve the feature information of the spatial dimension, but also ensures the balanced evaluation of the front and back images on the middle image.

In addition, because the loss function can only reflect whether the model training is benign, it cannot be used as the final accurate result indicator. After the experiment, calculate and compare the position of each tumor and calculate False Positive and False Negative. The present invention has dropped to close to 5% on False Negative and 20% on False Positive. It can be seen that, compared with the prior art, the present invention has achieved a breakthrough.

Therefore, the present invention effectively overcomes various shortcomings in the prior art and has high industrial application value.

The above-mentioned embodiments only illustrate the principles and effects of the present invention, but are not intended to limit the present invention. Anyone skilled in the art can modify or change the above-mentioned embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those skilled in the art without departing from the spirit and technical ideas disclosed in the present invention should still be covered by the claims of the present invention.

Claims (24)

1. An image heat prediction system, characterized in that, comprising: An image receiving device, configured to receive multiple stacked images collected along preset dimensions; The down-sampling device based on the convolutional network is used to block each of the stacked images according to the same window and the same moving rule, and merge and convolve the image blocks corresponding to the same window position in each stacked image to obtain the feature Image blocks, and downsampling each feature image block obtained; An upsampling device, configured to upsample each feature image block after downsampling; The heat prediction device is used to use the probability that at least one feature image block after upsampling is true or false to draw the color of each pixel in the heat prediction map; wherein, each pixel in the heat prediction map belongs to the above At least one feature image block after sampling. 2. image heat prediction system according to claim 1, is characterized in that, described down-sampling device based on convolution network also comprises: The normalization unit is configured to perform normalization processing on the multiple stacked images, and then divide the normalized multiple stacked images into blocks according to the same window and the same moving rule. 3. The image heat prediction system according to claim 1 or 2, wherein the down-sampling device based on the convolutional network comprises: at least one group of structures composed of filters and down-sampling units; wherein the The number of filters in the structure is at least one. When the number of filters in the structure is multiple, the filters in the same group of structures are cascaded; when the number of the structures is multiple, each group of structures cascade; Wherein, the filter connected to the image receiving device has a receiving channel consistent with the number of stacked images, and the receiving channel transmits a stacked image; the filter is used to convert each image block corresponding to the same window position in each stacked image Perform merged convolution to obtain the feature image block corresponding to the window position; The receiving channel of the filter in the other group is cascaded with the output channel of the previous filter to perform full convolution on the received feature image block; The down-sampling unit is provided on the output channel of each stage filter, which is used to down-sample the received feature image block, and send the down-sampled feature image block to the next-stage filter or up-sampling device the receiving channel. 4. The image heat prediction system according to claim 3, wherein the filter connected to the image receiving device is used to Combine and convolute the image blocks corresponding to the same window position in each stacked image; where W _k is the kth convolution kernel, and b _k is the offset added to each channel corresponding to the kth convolution kernel , c is the number of channels of the stacked image, (u, v) is the size of W _k , (i, j) is the pixel position in the image block. 5 . The image heat prediction system according to claim 3 , wherein the filters in each structure include at least one of the following: a color-based filter, a line-based filter, and a grayscale-based filter. 6 . The image heat prediction system according to claim 1 , wherein the upsampling device is configured to perform upsampling based on quadratic interpolation on the received feature image blocks according to the window size. 7. The image heat prediction system according to claim 6, wherein the upsampling device comprises: An upsampling unit, configured to fill in a preset expansion window based on each pixel value in the received feature image block, to obtain a feature image block that is upsampled once; The smoothing processing unit is configured to perform convolution on the once upsampled feature image block based on a preset convolution kernel consistent with the window size to obtain a second upsampled feature image block corresponding to the window size. 8. The image heat prediction system according to claim 1, wherein the heat prediction device is used for each of the heat prediction maps based on a loss function obtained through pre-learning that includes true value weights and false value weights. The pixels are evaluated one by one corresponding to each characteristic image block received, and the heat prediction map is drawn based on the obtained heat evaluation. 9. The image popularity prediction system according to claim 8, wherein the loss function is a cross-entropy function obtained based on pre-learning and including true value weights and false value weights. 10. The image heat prediction system according to claim 1, wherein the plurality of stacked image data are consecutive odd-numbered images. 11. A lung tumor recognition system based on convolutional neural network, characterized in that, comprising: An image library for storing multiple stacked lung CT images in a spatial order; The image heat prediction system according to any one of claims 1-10; The image heat prediction system outputs a heat prediction map corresponding to the multiple lung CT stack images according to the received multiple lung CT stack images; A control device for judging whether the image heat prediction system has extracted all the stored lung CT stack images, and if not, controlling the image heat prediction system to extract multiple lung CT stack images in order of space, until it is determined that the image extraction device has extracted all the stored lung CT stack images. 12 . The convolutional neural network-based lung tumor identification system according to claim 11 , wherein the control device extracts multiple CT stacked images in stages based on the span of one image. 13 . 13. An image heat prediction method, characterized in that, comprising: Receive multiple stacked images collected along preset dimensions; Sending the Dolphin stacked image into a down-sampling device based on a convolutional network, by which the down-sampling device based on a convolutional network blocks each of the stacked images according to the same window and the same moving rule, Merging and convolving image blocks corresponding to the same window position in each stacked image to obtain feature image blocks, and downsampling each feature image block; Perform upsampling processing on each feature image block after downsampling; Use the probability that at least one feature image block after upsampling is true or false to draw the color of each pixel in the heat prediction map; wherein, each pixel in the heat prediction map belongs to at least one feature after upsampling Image blocks. 14. The image heat prediction method according to claim 13, further comprising: after receiving multiple stacked images collected along preset dimensions: performing normalization processing on the plurality of stacked images; According to the same window and the same movement rule, the normalized multiple stacked images are divided into blocks. 15. The image heat prediction method according to claim 13 or 14, wherein the downsampling device based on a convolutional network comprises: at least one set of structures consisting of a filter and a downsampling unit; wherein the The number of filters in the structure is at least one. When the number of filters in the structure is multiple, the filters in the same group of structures are cascaded; when the number of the structures is multiple, each group of structures cascade; Wherein, the filter connected to the image receiving device has a receiving channel consistent with the number of stacked images, and the receiving channel transmits a stacked image; the filter performs a process on each image block corresponding to the same window position in each stacked image Combine convolution to obtain the feature image block corresponding to the window position; The receiving channel of the filter in the other group is cascaded with the output channel of the previous filter, and the received feature image block is fully convoluted; The down-sampling unit on the output channel of each filter stage down-samples the received feature image blocks, and sends the down-sampled feature image blocks to the next-stage filter or performs an up-sampling operation. 16. The image heat prediction method according to claim 15, wherein the filter connected to the image receiving device merges and convolves each image block corresponding to the same window position in each stacked image to obtain the corresponding window position. location of feature image blocks, comprising: the filter according to Combine and convolute the image blocks corresponding to the same window position in each stacked image; where W _k is the kth convolution kernel, and b _k is the offset added to each channel corresponding to the kth convolution kernel , c is the number of channels of the stacked image, (u, v) is the size of W _k , (i, j) is the pixel position in the image block. 17. The image heat prediction method according to claim 15, wherein the filters in each structure include at least one of the following: color-based filters or/and line-based filters and grayscale-based filters . 18. The image heat prediction method according to claim 13, wherein said performing up-sampling processing on each of the down-sampled feature image blocks comprises: performing up-sampling processing on the received feature image blocks according to the window size Upsampling based on quadratic interpolation. 19. The image heat prediction method according to claim 18, characterized in that, according to the window size, performing up-sampling based on secondary interpolation to the received feature image block comprises: Based on each pixel value in the received feature image block, fill in a preset expansion window to obtain a feature image block that is upsampled once; Based on a preset convolution kernel consistent with the size of the window, the once upsampled feature image block is convolved to obtain a second upsampled feature image block corresponding to the window size. 20. The image heat prediction method according to claim 13, characterized in that, the probability that the at least one feature image block after the upsampling is true or false is used to draw the color of each pixel in the heat prediction map, Including: based on the pre-learned loss function including true value weights and false value weights, each pixel in the heat prediction map corresponds to each upsampled feature image block to perform heat evaluation one by one, and based on the obtained The heat evaluation draws the heat prediction map. 21. The image popularity prediction method according to claim 20, wherein the loss function is a cross-entropy function obtained based on pre-learning and including true value weights and false value weights. 22. The image heat prediction method according to claim 13, characterized in that, the plurality of stacked image data are consecutive odd-numbered images. 23. A method for identifying lung tumors based on a convolutional neural network, comprising: Store multiple lung CT stack images in spatial order in advance; According to the image heat prediction method described in any one of claims 13-22, outputting a heat prediction map corresponding to the multiple lung CT stack images according to the received multiple lung CT stack images; Judging whether the image heat prediction system has extracted all the stored lung CT stack images, if not, controlling the image heat prediction system to extract multiple lung CT stack images in order of space until the image is determined The extracting device extracts all the stored stacked CT images of the lungs. 24. The convolutional neural network-based lung tumor identification method according to claim 23, wherein the control device extracts multiple CT stacked images in stages based on the span of one image.