CN113159051B - A Lightweight Semantic Segmentation Method for Remote Sensing Images Based on Edge Decoupling - Google Patents

Abstract

The invention discloses a remote sensing image lightweight semantic segmentation method based on edge decoupling, belongs to the field of computer vision, and can be used for intelligent interpretation in the field of remote sensing images. On one hand, the number of model parameters is reduced and network calculation overhead is reduced through a Ghost bottleneck module and deep separable convolution, the efficiency of semantic segmentation of the remote sensing image is effectively improved, and the lightweight of the proposed semantic segmentation network is realized; on the other hand, the precision of semantic segmentation is improved through the multi-scale feature pyramid, the global context module and the edge decoupling module, so that the proposed lightweight semantic segmentation network can accurately and efficiently realize the semantic segmentation of the remote sensing image, and the edge details of the remote sensing image are further refined.

Description

A Lightweight Semantic Segmentation Method for Remote Sensing Images Based on Edge Decoupling

technical field

The invention belongs to the field of computer vision, and in particular relates to a lightweight semantic segmentation method for remote sensing images based on edge decoupling, which can be used for intelligent interpretation in the field of remote sensing images.

Background technique

High-resolution remote sensing images contain detailed information such as color and texture features of targets such as roads and buildings. The intelligent interpretation of this information is of great significance in many fields such as military, agricultural, and environmental sciences. In order to complete the parsing and classification tasks of remote sensing images, each pixel in the image should be assigned a label associated with its category, which is consistent with the purpose of image semantic segmentation.

Inspired by deep learning, this task has a better development direction, especially the proposal of full convolutional network, which makes the image semantic segmentation method based on deep learning go to the mainstream. There have been methods such as UNet, SegNet, PSPNet, Deeplab series, etc., which have more advantages than traditional remote sensing image segmentation algorithms. Although these algorithms can guarantee excellent segmentation results when applied to the semantic segmentation of high-resolution remote sensing images, the training speed and segmentation efficiency are often low due to the large image size and complex network structure. Furthermore, due to the rich diversity of remote sensing images, the unbalanced distribution of target categories, and the easy overlapping of edges between different target categories, it is impossible to achieve fine segmentation of remote sensing images.

Contents of the invention

The purpose of the present invention is to provide a lightweight semantic segmentation method for remote sensing images based on edge decoupling, so as to solve the problem of inference speed caused by the large amount of parameters and large computational overhead of existing semantic segmentation methods when facing high-resolution remote sensing images Slow segmentation is inefficient, and the edges of different types of objects are easy to overlap, which leads to unsatisfactory edge segmentation results.

In order to achieve the above purpose, the present invention proposes a remote sensing image lightweight semantic segmentation method based on edge decoupling. The specific technical scheme is as follows:

A lightweight semantic segmentation method for remote sensing images based on edge decoupling, including the construction, training and testing of the semantic segmentation network. The semantic segmentation network is a lightweight codec network with a dual-branch structure, and the semantic segmentation network is completed based on training samples After training, input the remote sensing image to be tested into the semantic segmentation network, and output the final semantic segmentation result of the remote sensing image;

Include the following steps, and follow the steps in order:

Step 1. Obtain a remote sensing image dataset and prepare training and testing samples;

Step 2. Build a lightweight codec semantic segmentation network with a dual-branch structure;

Step 3. Input the training sample into the encoder, perform feature encoding through feature extraction, and obtain the encoded feature map F _E ;

Step 4. Input the encoded feature map F _E into the decoder, perform edge feature refinement processing and up-sampling operations, and obtain the decoded feature map F _D ;

Step 5. Input the decoded feature map into the classifier for pixel-level classification prediction, output the segmentation result, and conduct supervised training on the semantic segmentation network through the supervision mechanism;

Step 6, using the training sample to carry out the training of the semantic segmentation network according to steps (3-5) to the semantic segmentation network built in step 2;

Step 7. Input the sample to be tested into the trained semantic segmentation network, output the final semantic segmentation result of the remote sensing image, and complete the test of the semantic segmentation network.

Further, the step 2 builds a lightweight encoding and decoding semantic segmentation network with a double-branch structure, and the network includes three parts: an encoder, a decoder, and a classifier;

The encoder is a dual-branch structure, including a global downsampling block, a lightweight dual-branch sub-network, and a global feature fusion module;

The decoder is composed of a lightweight edge decoupling module and an upsampling module;

The classifier consists of regular convolutional layers and SoftMax layers.

Further, obtaining the encoded feature map F _E in the step 3 includes the following steps:

Step 3.1. Input the training samples into the global downsampling block of the encoder to obtain low-level feature maps;

Step 3.2. Input the low-level feature map into the lightweight dual-branch sub-network in the encoder to obtain the spatial detail feature map and the abstract semantic feature map;

Step 3.3. The acquired spatial detail feature map and abstract semantic feature map are used for multi-level feature fusion through the global feature fusion block of the encoder, and an encoded feature map F _E is output.

Further, the global downsampling block in step 3.1 is composed of 3 parts, one of which is a conventional convolution, the other is a Ghost bottleneck module, and the third is a global context module;

After the input samples pass through the global downsampling block, a low-level feature map with an output resolution of 1/4 of the original input will be generated as the input for the subsequent process.

Further, the step 3.2 lightweight dual-branch sub-network includes two branches, which are the backbone depth branch for obtaining abstract semantic features and the space-preserving branch for obtaining spatial detail features, and the two branches share the low-level feature map output by the global downsampling block;

The backbone depth branch is constructed based on the GhostNet feature extraction network, which includes two structures, one of which is a branch main structure composed of 16 Ghost bottleneck modules, which is used to perform 4 down-sampling processes to realize the extraction of deep features; the other is Lightweight feature pyramid, the structure is composed of four parts: depth separable convolution, upsampling block, lightweight hollow space pooling pyramid module and element fusion, using four deep feature maps of different scales formed by the main body as input, and finally Output abstract semantic features with increased receptive field and multi-scale information;

The space-preserving branch is composed of three depth-separable convolutions, which implements one downsampling for the input low-level features, and the resolution of the output spatial detail feature map is 1/2 of the input.

Further, the step 3.3 global feature fusion module includes 3 parts, one of which is two parallel convolution kernels with a depth separable convolution of 1*1; the second is element fusion; the third is the global context module;

The input abstract semantic features and spatial detail features are dimensionally adjusted through two parallel convolutions, and then a feature map with rich spatial details and abstract semantic information is output through element fusion. Finally, lightweight context construction is carried out through the global context module. Modulus, so that the final form of the encoded feature map F _E can better integrate the global information.

Further, the step 4 obtains the decoded feature map F _D , first inputs the encoded feature map F _E into the lightweight edge decoupling module of the decoder, performs edge feature refinement processing, and generates a fine feature map with refined edges; Then input the fine feature map into the up-sampling module of the decoder, perform an up-sampling operation, restore the fine feature map to the size of the original input remote sensing image, and use it as the decoded feature map F _D output by the decoder.

Further, the lightweight edge decoupling module consists of three parts, which are the lightweight hollow space pooling pyramid, the main feature generator and the edge retainer; the encoded features are first generated through the lightweight hollow space pooling pyramid with multi-scale information And the feature map F _aspp with a larger receptive field, and then generate a more consistent feature representation for the pixels inside the same object through the subject generator, and then form the subject feature map F _body of the target object; input F _body , F _aspp and F _E To the edge preserver, after explicit subtraction, channel stacking and fusion, and 1*1 conventional convolution dimensionality reduction, the feature map F _edge of the refined edge is output, and finally the main feature map and the refined edge feature map are fused, and the output is used for The fine output feature map restored by upsampling is denoted as F _final ; the whole process can be expressed by the following formula:

边缘保持函数；In the formula, f _dsaspp represents the lightweight empty space pooling pyramid function, φ represents the main feature generation function,

edge preserving function;

The up-sampling module includes two steps, which are 1*1 conventional convolution operation and up-sampling operation, and the fine feature map F _final will be restored to a feature map with the size of the original input remote sensing image after being output by this module, that is, the decoder The output feature map F _D of .

Further, in the supervision mechanism in step 5, the decoded feature map F _D is processed by the classifier to complete the pixel-level classification prediction, and the output is the result of semantic segmentation. Perform supervised training to make the semantic segmentation network achieve the best segmentation performance.

Further, the supervision mechanism in step 5 is an edge-based supervision method, which is realized by a designed loss function, and the total loss function is denoted as L, and its formula is shown in the following formula:

In the formula, L _body , L _edge , L _final , and L _G represent the main body feature loss, edge feature loss, fine feature loss, and global coding loss respectively. The inputs of the four loss functions are: main body feature map, refined edge feature map, The fine output feature map and the coded feature map respectively undergo upsampling restoration and SoftMax layer to form the segmentation results and their corresponding real labels;

The loss function L _edge is a comprehensive loss function based on the edge prediction part to obtain the boundary edge prior, which includes two aspects: one is the binary cross entropy loss L _bce for boundary pixel classification, and the other is the edge part of the scene The cross-entropy loss L _bce , λ ₁ , λ ₂ , λ ₃ , λ ₄ , λ ₅ , and λ ₆ represent hyperparameters used to control the weight between several losses.

The method of the present invention has the following advantages: the method can fully take into account the total parameter amount of the semantic segmentation network, the total amount of calculation, and the impact of a large number of redundant features on the segmentation efficiency and accuracy, and fully consider the relationship between the target ontology and the edge. The role of refined segmentation results;

First, the present invention combines the idea of feature sharing, and designs a global downsampling block based on the global context module and the Ghost bottleneck module. As the first part of the encoder in the semantic segmentation network proposed by this method, it effectively reduces the early extraction of low-level features in the network. The parameter scale reduces computational overhead and better integrates global context information in low-level features.

Second, the present invention combines the dual-branch structure and the global feature fusion method based on the global context module. First, a lightweight dual-branch sub-network is built based on the Ghost bottleneck module and depth-separable convolution, which significantly reduces the feature extraction stage. The parameter scale reduces the computational complexity, so that the final output encoding features contain rich spatial details and abstract semantic information. Secondly, the output features of the dual branches are fused through the global feature fusion method based on the global context module, so that the final output coding features can deepen the understanding of global information and reduce the loss of weak feature information in the network.

Third, the present invention uses depth-separable convolution to build a lightweight edge decoupling module, and introduces the connection between the object body and its edge by modeling the target object body and edge; effectively improving the existing remote sensing The image semantic segmentation algorithm is not finely segmented at the edge, which improves the segmentation effect of edge details in remote sensing images.

Description of drawings

Fig. 1 is the flowchart of the method of the present invention.

Fig. 2 is a schematic diagram of the structure of the semantic segmentation network built by the method of the present invention.

Fig. 3 is a schematic structural diagram of the Ghost bottleneck module.

Fig. 4 is a schematic diagram of the structure of the global context module.

Figure 5 is a schematic diagram of the lightweight feature pyramid structure in the main feature extraction branch.

Fig. 6 is a schematic structural diagram of a lightweight edge decoupling module.

Figure 7 is an example diagram of remote sensing images and corresponding semantic labels in the dataset.

Fig. 8 is a comparison diagram of semantic segmentation results in the method embodiment of the present invention. (where (a), (b) are input samples and corresponding labels, and (c)-(g) are the semantic segmentation results of Fast-SCNN, Sem-FPN, the method of the present invention, UNet, and PSPNet in turn).

Detailed ways

In order to better understand the purpose, structure and function of the present invention, a method for lightweight semantic segmentation of remote sensing images based on edge decoupling in the present invention will be further described in detail in conjunction with the accompanying drawings.

As shown in Figure 1, the present invention designs a remote sensing image lightweight semantic segmentation method based on edge decoupling and applies it to high-resolution remote sensing images, so that the edge segmentation effect can be refined while ensuring accuracy, and the segmentation efficiency can be greatly improved.

As shown in Figure 1, the present invention relates to a remote sensing image lightweight semantic segmentation method based on edge decoupling, which specifically includes the following steps:

Step 1. Obtain a remote sensing image dataset and prepare training and testing samples;

First, obtain a high-resolution remote sensing image dataset with semantic annotations, and then cut the labels and data correspondingly. The cutting method is a fixed window with a resolution of 512*512, and the sliding window is cut with a sliding step size of 384 according to a coverage rate of 0.75. . Correspond the cropped new data with the new label, and use methods such as rotation and color enhancement to augment the data. Sufficient samples can effectively reduce the impact of overfitting. Finally, the samples of the training set and the test set are divided according to the ratio of 4:1;

Step 2. Build a lightweight codec semantic segmentation network with a dual-branch structure;

The structure of the semantic segmentation network built by the present invention is shown in FIG. 2 . The network is a lightweight codec semantic segmentation network with a dual-branch structure, which includes three parts: encoder, decoder, and classifier. The encoder is a dual-branch structure, which contains a global downsampling block, a lightweight dual-branch subnetwork, and a global feature fusion module; the decoder consists of a lightweight edge decoupling module and an upsampling module. The classifier consists of regular convolutional layers and SoftMax layers;

Step 3. Input the training samples into the encoder, perform feature encoding through feature extraction, and obtain the encoded feature map F _E ; specifically, it will go through the following three sub-steps:

Step 3.1. Input the training samples into the global downsampling block of the encoder to obtain low-level feature maps;

Input the obtained training samples into the encoder of the network, the scale of the input samples is 512*512, and firstly pass through the global downsampling block in the encoder. The module consists of three parts, one is conventional convolution, the other is a Ghost bottleneck module, and the third is the global context module. The low-level feature map output by the global downsampling module better integrates the global context information, and also contains rich spatial detail information;

The conventional convolution is a convolution block with a convolution kernel of 3*3, a step size of 2, and a batch normalization layer and a ReLU activation layer. After the training sample is down-sampled by this part, the output resolution is 256*256. feature map;

The Ghost bottleneck module is a lightweight module derived from the GhostNet network. It is composed of Ghost modules and can generate feature maps with deeper dimensions with fewer parameters. Its structure is shown in Figure 3. The structure of the module is different according to the step size. When the step size is 1, the structure contains two layers of Ghost modules with a step size of 1. When the step size is 2, the two layers of Ghost modules are mixed with a channel-by-channel volume with a step size of 2. product. In the global downsampling block, the step size of this module is 2, and the second downsampling is implemented to further reduce the resolution of the feature map. The resolution of the output feature map after this module is 128*128;

The global context module, whose structure is shown in Figure 4, includes three processes: one is the global attention concentration mechanism for context modeling; the self-attention of the input feature map is obtained by using 1*1 conventional convolution and SoftMax layer Weight, and then focus on the input feature map to obtain the global background feature map; the second is feature conversion to obtain channel dependence; this part consists of two 1*1 convolutional layers, and the batch normalization layer is passed between them It is connected with the ReLu activation function; the third is element fusion; through element fusion, the original input feature map is fused with the feature map after obtaining channel dependencies, and the global context features are aggregated to the features of each position. After this module, the size of the output feature map is consistent with the size of the input feature map, so the scale of the final low-level feature map is 128*128;

Step 3.2 Input the low-level feature map into the lightweight dual-branch sub-network in the encoder to obtain the spatial detail feature map and the abstract semantic feature map;

The dual-branch sub-network consists of two branches, namely the backbone depth branch for obtaining abstract semantic features and the space-preserving branch for obtaining spatial detail features. The two branches share the low-level features output by the global downsampling block, which reduces the input path compared to the traditional dual-branch network, and reduces the parameter scale and computational overhead when extracting low-level features in the early stage of the network;

The backbone depth branch is built based on the GhostNet network. The main part of the network contains 16 Ghost bottleneck modules, which are used to perform 4 down-sampling processes to realize the extraction of deep features. This method retains the 16 Ghost bottleneck modules in GhostNet and becomes a fully convolutional network as the main body of the backbone depth branch. The input low-level feature map should be processed by this branch, and finally four depth feature maps with scales of 64*64, 32*32, 16*16, and 8*8 will be generated. The 4 scales represent 4 stages. The number of Ghost bottleneck modules corresponding to each stage are: [3, 2, 6, 5], and the corresponding convolution kernel sizes are: [3, 5, 3, 5]. Since 4 times of downsampling is implemented, each stage has a Ghost bottleneck module with a step size of 2.

At the same time, in order to obtain rich abstract semantic features, this method combines depthwise separable convolution, upsampling block and lightweight empty space pooling pyramid module, and uses 4 feature maps to build a lightweight feature pyramid. Its structure is shown in Figure 5 . The newly generated 4 levels are closely connected, the receptive field is enlarged and the feature map with multi-scale information is sampled to the 64*64 scale, and the final abstract semantic feature group is formed after element fusion as the output of the main depth branch;

The space-preserving branch is composed of three depth-separable convolutions. The convolution kernel size of the three is 3*3, and the step size is [1, 2, 1]. The input low-level feature map can be down-sampled once, and the resolution of the output spatial detail feature map is 64*64; this branch preserves the spatial scale of the input image with fewer parameters and less computational overhead. Capable of encoding rich spatial information;

Step 3.3 performs multi-level feature fusion on the obtained spatial detail feature map and abstract semantic feature map through the global feature fusion block of the encoder, and outputs the encoded feature map F _E ;

Among them, the global feature fusion module consists of three parts, one is the depth separable convolution with two parallel convolution kernels of 1*1; the other is element fusion; the third is the global context module; from the dual-branch sub-network Abstract semantic features and spatial detail features are dimensionally adjusted through two parallel 1*1 convolutions, and after element fusion, a feature map with rich spatial details and abstract semantic information is output. Finally, through the global context module, lightweight context modeling is performed, so that the final encoded feature map F _E can better integrate global information. Since the downsampling process is not included, the output encoded feature map remains the same size as the input;

After the above steps, the encoder finally generates a coded feature map with a scale of 64*64, which is used as input for the subsequent process;

Step 4. Input the encoded feature map F _E into the decoder, perform edge feature refinement processing and up-sampling operations, and obtain the decoded feature map F _D ;

The decoder consists of two modules, the edge decoupling module and the upsampling module. The lightweight edge decoupling module consists of three parts, the structure of which is shown in Figure 6. They are the lightweight empty space pooling pyramid, the main feature generator, and the edge retainer; the specific process of obtaining the decoded feature map is as follows: first, the encoded feature map F _E is input into the lightweight edge decoupling module of the decoder, and the edge Feature refinement processing to generate a fine feature map with refined edges; then input the fine feature map to the upsampling module of the decoder, perform an upsampling operation, restore the fine feature map to the size of the original input remote sensing image, and use it as a decoder The output decoded feature map F _D ;

The main body feature generator includes two processes of flow field generation and feature deformation. The flow field generation is composed of a micro codec structure including one downsampling and one upsampling and a conventional convolution with a convolution kernel of 3*3. It is used to generate a flow field feature representation with prominent features in the center of the target object. Feature deformation is to obtain the significant main feature representation of the target object by performing deformation operations on the flow field features; therefore, the main feature generator is responsible for generating a more consistent feature representation for the pixels in the same object, and extracts the main features of the target object;

The edge preserver consists of two steps. The first is the subtractor, which is used to subtract the encoded feature map after the receptive field expansion from the subject feature map to obtain a rough edge feature map; the second is the edge feature refiner, which uses Edge features are complemented by low-level feature maps of fine details. Specifically, the rough edge feature maps generated from the low-level feature map pairs from the encoder are combined for feature fusion through channel stacking, thereby supplementing high-frequency information. After a 1*1 conventional convolution for dimensionality reduction, the feature map of the refined edge is output;

Input the encoded feature F _E , first generate a feature map F _aspp with multi-scale information and a larger receptive field through a lightweight empty space pooling pyramid, and then form the subject feature map F _body of the target through the subject generator; F _body , F _Aspp and F _E are input into the edge retainer. First, the F _body and F _E are explicitly subtracted to generate a preliminary edge feature map, and then the preliminary edge feature map is channel-stacked and fused with F _E , and then after 1* 1 Conventional convolution operation dimensionality reduction processing outputs an edge feature map F _edge with refined edges. Finally, elements of F _body and F _edge are fused to obtain a fine output feature map F _final for upsampling recovery; the whole process can be expressed by the following formula:

边缘保持函数；In the formula, f _dsaspp represents the lightweight empty space pooling pyramid function, φ represents the main feature generation function,

edge preserving function;

In order to obtain the final decoded feature map F _D , input F _final into the upsampling module including 1*1 conventional convolution and upsampling operations for processing, and restore it to the size of the original input image, and the final generated feature map is the decoding Feature map F _D , whose scale is 512*512;

Step 5. Input the decoded feature map into the classifier for pixel-level classification prediction, output the segmentation result, and conduct supervised training on the semantic segmentation network through the supervision mechanism;

The main body of the classifier is a SoftMax layer, and the decoded feature map F _D is processed by the SoftMax layer to complete the pixel-level classification prediction and obtain the result of semantic segmentation. The training of the network is supervised by the supervision mechanism formed by the segmentation result and the real label, so that the semantic segmentation network can achieve the best segmentation performance;

The supervision mechanism does not only supervise the final segmentation result, but jointly supervises the four parts of F _body , F _edge , F _final , and F _E . This mechanism is realized through the designed loss function, and the total loss function is denoted as L, and its formula is as follows:

代表真实语义标签，

代表由

生成的二进制掩码，b代表边界预测结果，s_body、s_final和s_E代表从F_body、F_edge、F_E中获取的分割图预测结果；In the formula, L _body , L _edge , L _final , and L _G represent body feature loss, edge feature loss, fine feature loss, and global coding loss, respectively. Among them, L _final and L _G use the common cross-entropy loss function in semantic segmentation tasks. The L _body adopts the boundary relaxation loss, which enables the F _body to relax the classification of boundary pixels during the training process, allowing the segmentation network to predict the boundary pixel points into multiple categories. L _edge is a comprehensive loss function based on the edge prediction part to obtain the boundary edge prior, which includes two aspects: one is the binary cross entropy loss L _bce for boundary pixel classification. The second is the cross-entropy loss L _ce of the edge part in the scene. λ ₁ , λ ₂ , λ ₃ , λ ₄ , λ ₅ , and λ ₆ represent hyperparameters used to control the weight between several losses. The default value of the first three is 1, and the last three are 0.4, 20, and 1 respectively.

represents the real semantic label,

represented by

The generated binary mask, b represents the boundary prediction result, s _body , s _final and s _E represent the segmentation map prediction results obtained from F _body , F _edge , and F _E ;

Step 6, using the training sample to carry out the training of the semantic segmentation network according to steps (3-5) to the semantic segmentation network built in step 2;

According to the above process, after building the semantic segmentation network, continuously input training samples to train the network according to steps (3-5); before training, it is necessary to set up related training such as network input scale, input sample batch, learning rate, etc. parameter.

Step 7. Input the sample to be tested into the trained semantic segmentation network, output the final remote sensing image semantic segmentation result, and complete the test of the semantic segmentation network;

The experimental results of specific examples are given below, which are not intended to limit the use of the method of the present invention, but are only an example with better performance for analysis.

The experiment uses the Vaihingen dataset provided by ISPRS, which contains three channels of IRRG images, DSM images and NDSM images. 16 remote sensing images of size 6000*6000 and corresponding labels. The corresponding visualization results are shown in Figure 7. The label contains 6 types of targets, and the corresponding semantic annotations of each type are determined by the RGB value, as shown in Table 1 below:

Table 1 Semantic annotation information table

In this implementation example, the above data set is preprocessed according to the sliding window clipping method and data augmentation described in step 1, and the obtained data are all multi-channel images of 512*512*3. Then it is divided into training samples and test samples according to the ratio of 4:1;

Then build the semantic segmentation network of this method, and set the relevant parameters before training. The input scale of the network is 512*512, the input batch is set to 10 (depending on the video memory), the optimizer uses the SGD optimizer, the initial learning rate is set to 0.001, the minimum learning rate is set to 0.00001, and the momentum is set to 0.9 , and the weight decay coefficient is set to 0.0005.

In this implementation example, the selected evaluation indicators for semantic segmentation are average pixel intersection-over-union ratio (mIoU), average pixel accuracy (mAcc), GFLOPs (floating-point calculation amount), parameter amount, and segmentation reasoning time of a single image. Four semantic segmentation methods were selected to compare this method in terms of segmentation accuracy and efficiency. The four methods are: UNet, PSPNet, Fast_SCNN, and Sem-FPN. Utilize mIoU and mAcc as the standard to measure the accuracy of semantic segmentation. The higher the two, the closer the segmentation result is to the real label, and the higher the semantic segmentation accuracy. GFLOPs, parameter amount, and single image segmentation inference time are used as the criteria for semantic segmentation efficiency. The smaller the GFLOPs, parameter amount, and inference time, the higher the segmentation efficiency. The experimental results of different semantic segmentation methods are shown in Table 2:

Table 2 Comparison between this method and existing methods

method mIoU(%) mAcc(%) GFLOPs Parameter amount (M) Split inference time (s) UNet 86.19 91.16 203.04 29.06 0.067 PSPNet 86.40 92.19 178.48 48.98 0.066 Fast-SCNN 76.23 83.83 0.91 1.21 0.015 Sem-FPN 83.57 90.91 45.48 28.50 0.029 This method 85.33 90.98 6.63 4.17 0.031

From the results in Table 2, it can be seen that this method achieves 89.42% mIoU and 93.15% mAcc, GFlOPs is 6.9, the number of parameters is 4.1M, and the single image segmentation inference speed is 0.031s. Compared with Fast-SCNN, although it has the least amount of parameters, the least amount of floating-point calculations, and the shortest inference time, its accuracy is much lower than this method; compared with Sem-FPN, although this method is slightly inferior in inference time, but in mIoU and mAcc are higher than Sem-FPN, and Sem-FPN's parameters and GFLOPs are much higher than this method. Both UNet and PSPNet are classic semantic segmentation networks. In comparison, this method is not as accurate as the two. However, UNet and PSPNet are several times faster than this method in terms of parameter quantity, GFLOPs and inference time. Therefore, considering the accuracy and efficiency of semantic segmentation, the semantic segmentation network proposed by this method is superior to other semantic segmentation networks. And from the perspective of the amount of parameters, GFLOPs and inference speed, it is verified that the present invention is a lightweight semantic segmentation method for remote sensing images;

Figure 8 shows the visual semantic segmentation results obtained after the test samples are input. Compared with the semantic segmentation results obtained by the Fast-SCNN and Sem-FPN methods, this method is more accurate in pixel classification, and effectively improves the segmentation error caused by misclassification; and this method can handle edge details It is more accurate and closer to the real result of semantic labeling. Compared with UNet and PSPNet, although the overall segmentation accuracy is insufficient, this method is closer to the semantic label in the segmentation of edge details.

It can be understood that the present invention is described through some embodiments, and those skilled in the art know that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. In addition, the features and examples may be modified to adapt a particular situation and material to the teachings of the invention without departing from the spirit and scope of the invention. Therefore, the present invention is not limited by the specific embodiments disclosed here, and all embodiments falling within the scope of the claims of the present application belong to the protection scope of the present invention.

Claims (7)

1. A remote sensing image lightweight semantic segmentation method based on edge decoupling is characterized by comprising the steps of building, training and testing a semantic segmentation network, wherein the semantic segmentation network is a lightweight coding and decoding network with a double-branch structure, after training of the semantic segmentation network is completed based on a training sample, a remote sensing image to be tested is input into the semantic segmentation network, and a final remote sensing image semantic segmentation result is output;

the method comprises the following steps in sequence:

step 1, acquiring a remote sensing image data set, and preparing a training and testing sample;

step 2, building a lightweight coding and decoding semantic segmentation network with a double-branch structure;

step 2, building a lightweight coding and decoding semantic segmentation network with a double-branch structure, wherein the network comprises an encoder, a decoder and a classifier;

the encoder is of a double-branch structure and comprises a global downsampling block, a lightweight double-branch sub-network and a global feature fusion module;

the decoder consists of a lightweight edge decoupling module and an up-sampling module;

the classifier is composed of a conventional convolutional layer and a SoftMax layer;

step 3, inputting the training samples into an encoder, performing feature encoding through feature extraction, and obtaining an encoding feature map F _E ；

Obtaining a coding feature map F in the step 3 _E The method comprises the following steps:

step 3.1, inputting the training sample into a global downsampling block of an encoder to obtain a low-level feature map;

the global downsampling block in the step 3.1 consists of 3 parts, wherein one part is 1 conventional convolution, the other part is 1 Ghost bottleneck module, and the third part is 1 global context module;

after an input sample passes through a global downsampling block, a low-level feature map with the output resolution being 1/4 of the original input is generated and used as the input of the subsequent process;

step 3.2, inputting the low-level feature map into a lightweight double-branch sub-network in an encoder to obtain a space detail feature map and an abstract semantic feature map;

step 3.3, the obtained space detail feature map and the abstract semantic feature map are subjected to multi-level feature fusion through a global feature fusion block of the encoder, and an encoding feature map F is output _E ；

Step 4, encoding feature map F _E Inputting the data into a decoder, performing edge feature refinement processing and up-sampling operation to obtain a decoded feature map F _D ；

Step 5, inputting the decoding characteristic graph into a classifier to perform pixel-level classification prediction, outputting a segmentation result, and performing supervised training on the semantic segmentation network through a supervision mechanism;

step 6, training the semantic segmentation network built in the step 2 by using the training sample according to the step (3-5);

and 7, inputting the sample to be tested into the trained semantic segmentation network, outputting a final remote sensing image semantic segmentation result, and completing the test of the semantic segmentation network.

2. The remote sensing image light-weight semantic segmentation method based on edge decoupling as claimed in claim 1, wherein the light-weight double-branch sub-network in step 3.2 comprises two branches, namely a trunk depth branch for obtaining abstract semantic features and a space preservation branch for obtaining space detail features, and the two branches share a low-level feature map output by a global downsampling block;

the trunk deep branch is constructed based on a GhostNet feature extraction network and comprises two structures, wherein one structure is a branch main body structure consisting of 16 GhostNet bottleneck modules and is used for carrying out a downsampling process for 4 times to realize the extraction of deep features; the structure comprises four parts of depth separable convolution, an up-sampling block, a lightweight cavity space pooling pyramid module and element fusion, 4 deep feature maps with different scales formed by a main body are used as input, and finally, abstract semantic features with enlarged receptive field and multi-scale information are output;

the spatial preservation branch is composed of 3 depth separable convolutions, down-sampling is performed 1 time for the input low-level feature map, and the resolution of the output spatial detail feature map is 1/2 of the input.

3. The remote sensing image light-weight semantic segmentation method based on edge decoupling as claimed in claim 1, wherein the step 3.3 global feature fusion module comprises 3 parts, one of which is a depth separable convolution with two parallel convolution kernels of 1 x 1; second, element fusion; the third is 1 global context module;

performing dimension adjustment on the input abstract semantic features and space detail features through two parallel convolutions, outputting feature graphs with rich space details and abstract semantic information through element fusion, and finally performing lightweight context modeling through a global context module to finally form a coding feature graph F _E The global information can be better fused.

4. The remote sensing image light-weight semantic segmentation method based on edge decoupling as claimed in claim 1, wherein the step 4 obtains a decoding feature map F _D Firstly, the coding feature map F _E Inputting the data into a lightweight edge decoupling module of a decoder, and performing edge feature refinement processing to generate a fine feature map with refined edges; inputting the fine characteristic diagram into an up-sampling module of a decoder, performing up-sampling operation, restoring the fine characteristic diagram to the size of the original input remote sensing image, and using the fine characteristic diagram as a decoding characteristic diagram F output by the decoder _D 。

5. The remote sensing image light-weight semantic segmentation method based on the edge decoupling as claimed in claim 1, wherein the light-weight edge decoupling module is composed of 3 parts, namely a light-weight cavity space pooling pyramid, a main body feature generator and an edge retainer; firstly, the coding characteristics are subjected to light-weight cavity space pooling pyramid to generate a characteristic diagram F with multi-scale information and a larger receptive field _aspp Then, more consistent feature representation is generated for pixels in the same object through a main body generator, and further a main body feature graph F of the target object is formed _body (ii) a F is to be _body 、F _aspp And F _E Inputting the data into an edge holder, and outputting a feature map F of a refined edge through explicit subtraction operation, channel stack fusion and 1 x 1 conventional convolution dimensionality reduction _edge Finally, the main body characteristic diagram and the refined edge characteristic diagram are fused, and a refined output characteristic diagram F for carrying out up-sampling recovery is output _final (ii) a The whole process can be represented by the following formula:

in the formula, f _dsaspp Representing a lightweight void space pooling pyramid function, phi representing a principal feature generating function,

an edge preservation function;

the up-sampling module comprises two steps of 1 × 1 conventional convolution operation and up-sampling operation, and a fine feature map F _final After being output by the module, the characteristic diagram is restored to have the size of the original input remote sensing image, namely the output characteristic diagram F of the decoder _D 。

6. The remote sensing image light-weight semantic segmentation method based on edge decoupling as claimed in claim 1, wherein the supervision mechanism in step 5 is decoding feature map F _D After the classification of the pixel level is completed by the classifier, the output is the result of semantic segmentation, and the network is supervised and trained by a supervision mechanism formed by the result of semantic segmentation and a real label, so that the semantic segmentation network achieves the best segmentation performance.

7. The remote sensing image light-weight semantic segmentation method based on edge decoupling as claimed in claim 1, wherein the supervision mechanism in step 5 is an edge-based supervision mode, the mechanism is realized by a designed loss function, the total loss function is denoted as L, and the formula is shown as follows:

in the formula L _body 、L _edge 、L _final 、L _G Respectively representing the main feature loss, the edge feature loss, the fine feature loss and the global coding loss, and the input of the 4 loss functions is respectively: respectively forming segmentation results and corresponding real labels of the segmentation results after the main body characteristic diagram, the refined edge characteristic diagram, the refined output characteristic diagram and the coding characteristic diagram are subjected to up-sampling recovery and a SoftMax layer;

wherein the loss function L _edge The method is based on the edge prediction part to obtain the comprehensive loss function of the boundary edge prior, and comprises two aspects: one is binary cross-entropy loss for boundary pixel classificationL _bce And secondly, the cross entropy loss L of the edge part in the scene _bce ，λ ₁ 、λ ₂ 、λ ₃ 、λ ₄ 、λ ₅ 、λ ₆ The representative hyperparameter is used to control the weighting between several losses.

Images