TWI873056B - Method and computer system for analyzing gastric endoscopic image using image segmentation technology - Google Patents

Abstract

This disclosure is related to a method for analyzing a gastric endoscopy image using image segmentation technology. The method includes: cutting the gastric endoscopy images into multiple patches; inputting the patches into an encoder to obtain an encoded feature map; inputting the encoded feature map into a decoder to obtain a decoded feature map; and inputting the decoded feature map into a mask focal modulation decoder. This mask focal modulation decoder inputs a first prototype feature into a focal modulation function to obtain a focal modulation feature, input the decoded feature map into a scaling function then into a projection function to obtain a scaled feature, and generate a second prototype feature based on a mask, the scaled feature, and the focal modulation feature. A predictive mask is generated based on the second prototype feature for segmenting gastric intestinal metaplasia area.

Description

Gastroendoscopic image analysis method and computer system using image segmentation technology

The present disclosure relates to a method for analyzing gastric endoscopic images using deep learning technology, and in particular, for segmenting gastric mucosal intestinal metaplasia areas in the images.

Gastric cancer ranks sixth in the incidence of cancer and fourth in the cause of cancer death in the world. Gastric cancer is caused by Helicobacter pylori infection, followed by gastric inflammation, gastric atrophy, and gastric mucosal intestinal metaplasia. Since early diagnosis can improve the survival rate of the disease, it is very important to regularly follow up patients who have precancerous lesions such as gastric mucosal intestinal metaplasia. How to correctly diagnose and grade gastric inflammation and gastric mucosal intestinal metaplasia during gastroscopy is an important clinical issue. Currently, the diagnosis of gastric inflammation and gastric mucosal intestinal metaplasia requires five to six biopsy sections in the stomach during gastroscopy, which are then interpreted by pathologists. The advantage is that it can be correctly diagnosed through pathological interpretation, but the disadvantage is that it is time-consuming, has risks such as bleeding, and is not suitable for large-scale screening. Therefore, the development of technology that does not require invasive biopsy and can quickly analyze gastric mucosal intestinal metaplasia is an important topic in precision health that still needs new technology development.

The disclosed embodiment proposes a method for analyzing gastric endoscopic images using image segmentation technology, which is applicable to a computer system. The method includes: obtaining a gastric endoscopic image and cutting the gastric endoscopic image into multiple blocks; inputting one of the blocks into an encoder to obtain a coded feature map; inputting the coded feature map into a decoder to obtain a decoded feature map; and inputting the decoded feature map into a mask focus modulation decoder. The mask focus modulation decoder includes a plurality of mask focus modulation stages, one of which is used to receive a decoded feature map and a first template feature, input the first template feature into a focus modulation function to obtain a focus modulation feature, input the decoded feature map into a scaling function and then substitute it into a projection function to obtain a scaling feature, and generate a second template feature according to a mask, the scaling feature and the focus modulation feature. The above method also includes: generating a predicted mask according to the second template feature, the predicted mask including a plurality of pixels, for segmenting a gastric mucosal intestinal metaplasia area.

In some embodiments, the step of generating a predicted mask based on the second template feature includes: inputting the second template feature corresponding to the last level in the mask focus modulation level into multiple neural networks to obtain a mask feature and a class feature; performing element-wise multiplication of the mask feature and the decoded feature map to obtain a binary mask; and performing element-wise multiplication of the binary mask and the class feature to obtain a predicted mask.

In some embodiments, the step of generating a second template feature based on the mask, the scaling feature, and the focus modulation feature includes: performing element-wise multiplication on the mask, the scaling feature, and the focus modulation feature to generate the second template feature.

In some embodiments, the mask includes multiple values, and the analysis method further includes: determining whether each value is less than a critical value to generate a next-level mask.

In some embodiments, the encoder includes a plurality of focus modulation blocks, and the decoder is a pixel decoder.

From another perspective, the embodiment of the present disclosure provides a computer system, including a memory and a processor. The memory is used to store multiple instructions, and the processor is communicatively connected to the memory to execute the instructions to complete multiple steps, including: obtaining a gastric endoscopic image and cutting the gastric endoscopic image into multiple blocks; inputting one of the blocks into an encoder to obtain an encoded feature map; inputting the encoded feature map into a decoder to obtain a decoded feature map; and inputting the decoded feature map into a mask focus modulation decoder. The mask focus modulation decoder includes a plurality of mask focus modulation stages, one of which is used to receive a decoded feature map and a first template feature, input the first template feature into a focus modulation function to obtain a focus modulation feature, input the decoded feature map into a scaling function and then substitute it into a projection function to obtain a scaling feature, and generate a second template feature according to a mask, the scaling feature and the focus modulation feature. The above steps also include: generating a predicted mask according to the second template feature, the predicted mask including a plurality of pixels, for segmenting a gastric mucosal intestinal metaplasia area.

In some embodiments, the step of generating a predicted mask based on the second template feature includes: inputting the second template feature corresponding to the last level in the mask focus modulation level into multiple neural networks to obtain a mask feature and a class feature; performing element-wise multiplication of the mask feature and the decoded feature map to obtain a binary mask; and performing element-wise multiplication of the binary mask and the class feature to obtain a predicted mask.

In some embodiments, the step of generating a second template feature based on the mask, the scaling feature, and the focus modulation feature includes: performing element-wise multiplication on the mask, the scaling feature, and the focus modulation feature to generate the second template feature.

In some embodiments, the mask includes multiple values, and the processor is used to determine whether each value is less than a threshold value to generate a next-level mask.

In some embodiments, the encoder includes a plurality of focus modulation blocks, and the decoder is a pixel decoder.

The terms “first,” “second,” etc. used herein do not particularly refer to order or sequence, but are only used to distinguish elements or operations described with the same technical term.

FIG1 is a schematic diagram of a computer system according to an embodiment. Referring to FIG1 , a computer system 120 includes a processor 121 and a memory 122, which are used to process a gastric endoscopic image 110 to generate a prediction mask 130, and the prediction mask 130 is used to segment the location of gastric mucosal intestinal metaplasia. From another perspective, the prediction mask 130 can also be called the segmentation result of the gastric endoscopic image 110, and what is to be segmented is the gastric mucosal intestinal metaplasia area. Here, the gastric endoscopic image 110 can be a gastric endoscopic image of the antrum, the body, or the heart. The present disclosure does not limit the location, resolution, and angle of the gastric endoscopic image 110.

The computer system 120 may be a personal computer, a server, medical-related equipment, or various electronic devices with computing capabilities, but the present invention is not limited thereto. The processor 121 may be a central processing unit, a graphics processor, a microprocessor, a microcontroller, a tensor processing unit, a special application integrated circuit, etc. The memory may be a random access memory, a read-only memory, a flash memory, a floppy disk, a hard disk, an optical disk, a flash drive, a tape, or a database accessible via a network, in which a plurality of instructions are stored. The processor executes these instructions to complete the gastric endoscopic image analysis method, which will be described below.

Here, a deep learning network is used to generate a prediction mask. FIG2 is a schematic diagram of a deep learning network according to an embodiment. Referring to FIG2, first in step 201, the gastric endoscopic image 110 is divided into a plurality of blocks. For example, the resolution of the gastric endoscopic image 110 is , here we cut 4 blocks on both the horizontal and vertical axes. After cutting, the size of each block is , then a trainable linear layer (e.g. a fully connected layer) can be used to transform the block into The above-mentioned H, W, and C are positive integers, such as H=224, H=224, and C=96, but the present disclosure is not limited to these values.

The feature map converted from the block is input to the encoder 210. The encoder 210 is, for example, a focus modulation network, which includes a plurality of focus modulation stages 211 to 214. For details, please refer to the first paper YANG, Jianwei, et al. Focal modulation networks. Advances in Neural Information Processing Systems, 2022, 35: 4203-4217. In this embodiment, the focus modulation stage 211 includes 2 focus modulation blocks. The size of the feature map output by the focus modulation stage 211 is The focus modulation stage 212 includes two focus modulation blocks. The size of the feature map output by the focus modulation stage 212 is The focus modulation stage 213 includes 6 focus modulation blocks. The size of the feature map output by the focus modulation stage 213 is The size of the feature map output by the focus modulation stage 214 is The feature map output by the encoder 210 is also called a coded feature map. Through focus modulation, the entire network can learn visual features from a small field of view to a large field of view.

Next, the encoded feature map is input to the decoder 220, which includes multiple stages, each of which generates a feature map of different resolutions. The decoder 220 is, for example, a pixel decoder. For details, please refer to the second paper CHENG, Bowen, et al. Masked-attention mask transformer for universal image segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022. p. 1290-1299, which is not described in detail here. The feature map generated by the decoder 220 is called the decoded feature map 221, which is also represented as follows , where s represents the level of pixel decoder, and the output resolution of the last level is equivalent to one quarter of the original resolution, that is, .

Next, the decoded feature map The mask focus modulation decoder 230 includes a plurality of mask focus modulation stages 232-234. Each of the mask focus modulation stages 232-234 receives a decoded feature map. And contains 3 mask focus modulation blocks. In addition, the mask focus modulation stage 232 also receives a trainable template feature 231, whose size is , where N is a positive integer (for example, 256), is expressed as , where b represents the number of mask focus modulation blocks. In each mask focus modulation level 232-234, the sample feature in the bth mask focus modulation block is and decoded feature maps It will be used to generate the template features in the b+1th mask focus modulation block. .

FIG3 is a schematic diagram showing the structure of a mask focus modulation block according to an embodiment. Referring to FIG3 , the mask focus modulation block 300 includes mask focus modulation layers 311~312, residual normalization layers 321~323 and a feedforward layer 331. The mask 301 is input to the mask focus modulation layers 311~312. The template feature 302 is input to the mask focus modulation layer 311 and the residual normalization layer 321. The decoded feature map 221 is input to the mask focus modulation layers 311~312. Each of the residual normalization layers 321~323 includes a residual connection and a layer normalization. The mask focus modulation block 300 generates template features 340 for the next mask focus modulation block. The mask 301 provides segmentation results, which are used to guide the model to focus on specific areas during training. At the same time, the mask focus modulation provides semantic texture features at different scales.

Here, the operation of the mask focus modulation layer 311-312 is described. FIG4 is a schematic diagram of the mask focus modulation layer according to an embodiment. Referring to FIG4, the operation of the mask focus modulation layer can be expressed as the following mathematical formula 1. [Mathematical formula 1]

in represents the overall operation of the mask focus modulation layer. Z() is the focus modulation function, which is represented as focus modulation function 410 in FIG. 4. Also known as focus modulation feature. R() is the scaling function. F() is the projection function, which can be implemented by linear transformation, for example. is the mask 301 in the b-th mask focus modulation block. The template feature 302 in FIG. 4 is the formula 1 in , the decoded feature graph 221 is the mathematical formula 1 In addition, the operator This is element-wise multiplication.

The template feature 302 is linearly transformed (for example, by a fully connected layer) and then provided to a plurality of convolutional layers 411-414 and a plurality of gates 421-424. The focus modulation function 410 can be referred to in the first paper mentioned above and will not be described in detail here. Similarly, the decoded feature map 221 is scaled and then linearly transformed (i.e., projected) to obtain the scaled feature. .

Here, the mask is a binary mask that masks in the bth mask focus modulation box is generated from the b-1th mask focus modulation block. Specifically, the mask of the previous level contains multiple values, which are represented as , where x and y are coordinates. Next, we determine whether each of these values is less than a critical value (for example, 0.5) to generate the next level mask. , as shown in the following mathematical formula 2. [Mathematical formula 2]

Masking can force the model to focus on pixels that are difficult to classify. From another perspective, the above mathematical formula 1 converts the sample features Input into a focus modulation function Z() to obtain the focus modulation characteristic Z( ), the decoded feature map Input to the scaling function R() and then substitute into the projection function F() to get the scaling feature , and then apply the mask , Zoom Features and focus modulation characteristics Multiply elements together to generate template features for subsequent networks .

2, after the mask focus modulation decoder 230, the template feature (also called the second template feature) generated by the final mask focus modulation stage 234 can be used to generate the prediction mask 130. Specifically, the second template feature is input to a plurality of neural networks 241-242, which are, for example, multilayer perceptrons (MLP). The neural network 241 outputs a mask feature 251, and the neural network 242 outputs a class feature 252, wherein the size of the mask feature 251 is , the size of class feature 252 is , where K is a positive integer. Next, the mask feature 251 is element-wise multiplied with the decoded feature map 221 of the maximum resolution generated by the decoder 220 to obtain a binary mask 261, whose size is . Then, the binary mask 261 is element-wise multiplied with the class feature 252 to obtain the predicted mask 130.

The loss function used in the present disclosure is shown in the following mathematical formula 3. [Mathematical formula 3]

in , and They represent binary cross-entropy loss, dice loss, and classification loss respectively. , , is the weight corresponding to the loss. For details, please refer to the second paper mentioned above.

The dimension of the prediction mask 130 is , which includes a plurality of pixels, for segmenting the gastric mucosal intestinal metaplasia region, for example, the gastric mucosal intestinal metaplasia category is set to the value "1", and the background is set to the value "0", but the present disclosure is not limited to such an example. FIG. 5 is a schematic diagram illustrating experimental results according to an embodiment. FIG. 5 illustrates test data 501-505, which are generated by superimposing the predicted mask on the original gastric endoscopic image. For example, the test data 501 segmented the gastric mucosal intestinal metaplasia region 510, and the others are analogous.

In this disclosure, a mask focus modulation network is proposed to analyze gastric endoscopic images, from which the area where gastric mucosal intestinal metaplasia occurs can be segmented. In the network proposed in this disclosure, due to the cooperation between the encoder and the decoder, the mask focus modulation decoder will focus on the overall features generated by different fields of view, which makes the network focus on the learnable mask and provides better segmentation results.

FIG6 is a flow chart illustrating a method for analyzing gastric endoscopic images using image segmentation technology according to an embodiment. Referring to FIG6, in step 601, a gastric endoscopic image is obtained and cut into a plurality of blocks. In step 602, the blocks are input into an encoder to obtain a coded feature map. In step 603, the coded feature map is input into a decoder to obtain a decoded feature map. In step 604, the decoded feature map is input into a mask focus modulation decoder. This mask focus modulation decoder includes a plurality of mask focus modulation stages, one of which is used to receive a decoded feature map and a first template feature, input the first template feature into a focus modulation function to obtain a focus modulation feature, input the decoded feature map into a scaling function and substitute it into a projection function to obtain a scaling feature, and generate a second template feature based on a mask, scaling feature and focus modulation feature. In step 605, a predicted mask is generated based on the second template feature, and the predicted mask includes a plurality of pixels for segmenting a gastric mucosal intestinal metaplasia area. Each step in FIG. 6 has been described in detail above and will not be repeated here. It is worth noting that each step in FIG. 6 can be implemented as a plurality of program codes or circuits, and the present invention is not limited thereto. In addition, the method of FIG. 6 can be used in conjunction with the above embodiments or can be used alone. In other words, other steps can be added between the steps of FIG. 6 .

Although the present invention has been disclosed as above by the embodiments, they are not intended to limit the present invention. Any person with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be defined by the scope of the attached patent application.

110: gastroendoscopic image 120: computer system 121: processor 122: memory 130: prediction mask 201: step 210: encoder 211~214: focus modulation level 220: decoder 221: decoded feature map 230: mask focus modulation decoder 231: template feature 232~234: mask focus modulation level 241~242: neural network 251: mask feature 252: class feature 261: binary mask 300: mask focus modulation block 301: mask 302: template feature 311~312: mask focus modulation layer 321~323: Residual normalization layer 331: Feedforward layer 340: Sample characteristics 410: Focus modulation function 411~414: Convolution layer 421~424: Gate 501~505: Test data 510: Region 601～605: Steps

In order to make the above features and advantages of the present invention more clear and easy to understand, the following examples are specifically cited and detailed with the attached figures. FIG. 1 is a schematic diagram of a computer system according to an embodiment. FIG. 2 is a schematic diagram of a deep learning network according to an embodiment. FIG. 3 is a schematic diagram of the structure of a mask focus modulation block according to an embodiment. FIG. 4 is a schematic diagram of a mask focus modulation layer according to an embodiment. FIG. 5 is a schematic diagram of experimental results according to an embodiment. FIG. 6 is a flow chart of a method for analyzing gastric endoscopic images using image segmentation technology according to an embodiment.

110: Gastroendoscopic images

130: Prediction mask

201: Steps

210: Encoder

211~214: Focus modulation level

220:Decoder

221: Decoding feature map

230:Mask Focus Modulation Decoder

231: Sample features

232~234: Mask focus modulation level

241~242: Neural network

251:Mask feature

252:Classification characteristics

261: Binarization mask

Claims (10)

A method for analyzing gastric endoscopic images using image segmentation technology is applicable to a computer system, and the method comprises: Obtaining a gastric endoscopic image and cutting the gastric endoscopic image into a plurality of blocks; Inputting one of the blocks into an encoder to obtain a coded feature map; Inputting the coded feature map into a decoder to obtain a decoded feature map; The decoded feature map is input into a mask focus modulation decoder, wherein the mask focus modulation decoder includes a plurality of mask focus modulation stages, one of which is used to receive the decoded feature map and a first template feature, the first template feature is input into a focus modulation function to obtain a focus modulation feature, the decoded feature map is input into a scaling function and then substituted into a projection function to obtain a scaling feature, and a second template feature is generated according to a mask, the scaling feature and the focus modulation feature; and a predicted mask is generated according to the second template feature, the predicted mask includes a plurality of pixels, and is used to segment a gastric mucosal intestinal metaplasia area. The analysis method as described in claim 1, wherein the step of generating the predicted mask according to the second template feature comprises: Inputting the second template feature corresponding to the last level of the mask focus modulation levels into multiple neural networks to obtain a mask feature and a class feature; Performing element-wise multiplication on the mask feature and the decoded feature map to obtain a binary mask; and Performing element-wise multiplication on the binary mask and the class feature to obtain the predicted mask. The analysis method as described in claim 1, wherein the step of generating the second template feature according to the mask, the scaling feature and the focus modulation feature comprises: Element-wise multiplication of the mask, the scaling feature and the focus modulation feature to generate the second template feature. The analysis method as described in claim 1, wherein the mask includes multiple values, and the analysis method further includes: Determining whether each of the values is less than a critical value to generate a next-level mask. An analysis method as described in claim 1, wherein the encoder comprises a plurality of focus modulation blocks and the decoder is a pixel decoder. A computer system includes: A memory storing a plurality of instructions; and A processor communicating with the memory for executing the instructions to complete a plurality of steps: Obtaining a gastric endoscopic image and cutting the gastric endoscopic image into a plurality of blocks; Inputting one of the blocks into an encoder to obtain a coded feature map; Inputting the coded feature map into a decoder to obtain a decoded feature map; The decoded feature map is input into a mask focus modulation decoder, wherein the mask focus modulation decoder includes a plurality of mask focus modulation stages, one of which is used to receive the decoded feature map and a first template feature, the first template feature is input into a focus modulation function to obtain a focus modulation feature, the decoded feature map is input into a scaling function and then substituted into a projection function to obtain a scaling feature, and a second template feature is generated according to a mask, the scaling feature and the focus modulation feature; and a predicted mask is generated according to the second template feature, the predicted mask includes a plurality of pixels, and is used to segment a gastric mucosal intestinal metaplasia area. A computer system as described in claim 6, wherein the step of generating the predicted mask according to the second template feature comprises: Inputting the second template feature corresponding to the last level of the mask focus modulation levels into multiple neural networks to obtain a mask feature and a class feature; Performing element-wise multiplication on the mask feature and the decoded feature map to obtain a binary mask; and Performing element-wise multiplication on the binary mask and the class feature to obtain the predicted mask. A computer system as described in claim 6, wherein the step of generating the second template feature based on the mask, the scaling feature and the focus modulation feature comprises: Element-wise multiplication of the mask, the scaling feature and the focus modulation feature to generate the second template feature. A computer system as described in claim 6, wherein the mask comprises a plurality of numerical values, and the processor is further used to determine whether each of the numerical values is less than a critical value to generate a next-level mask. A computer system as described in claim 6, wherein the encoder comprises multiple focus modulation blocks and the decoder is a pixel decoder.