CN110097541A - A kind of image of no reference removes rain QA system - Google Patents

Abstract

The invention proposes a reference-free image rain removal quality evaluation system, which includes a multi-scale feature forward extraction path, a multi-scale feature backward extraction path, a gated fusion module and a score prediction module; the image to be evaluated is input to the multi-scale feature forward After the path is extracted, the multi-scale features are output to the gated fusion module through the multi-scale feature extraction path, and the gated fusion module performs weighted fusion of each scale feature and outputs it to the score prediction module to obtain the score. The bidirectional gated fusion network first obtains high-dimensional features through the forward path, then obtains more local content and context information of the image through the backward path, and finally obtains the multi-scale features of the image through gated fusion. The present invention is used to predict the human perception quality of different rain removal results, which is helpful for evaluation and development in a real rain scene.

Description

A Reference-Free Image Removal Quality Evaluation System

technical field

The present invention relates to the technique of removing rain in image processing.

Background technique

Deraining Quality Assessment (DQA) plays an important role in evaluating and guiding the design of image deraining algorithms. Since no rain-free images exist in actual rainy weather, existing rain-removing algorithms simulate very limited types of rain, which are far from sufficient to evaluate the practicality of rain-removing algorithms.

In the field of rain removal quality evaluation, traditional methods rely on real images without rain and full reference image quality evaluation (such as PSNR SSIM), and all existing image rain removal algorithms can only obtain evaluation results on artificial data. Compared with various real rain scenes, the computer synthesized rainfall pictures are far from guaranteeing the practicability of the rain removal algorithm. At the same time, the quality evaluation of real rainfall images cannot verify the statistical significance of each rain removal algorithm. To the best of our knowledge, there is currently no Blind Image Quality Assessment (BIQA) model specifically designed for derained images.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to propose a bidirectional gated fusion network (B-GFN) quality evaluation system for rain removal that integrates the multi-scale features of the image for the anisotropy and content variability of the rain removal image.

The technical solution adopted by the present invention to solve the above technical problems is a reference-free image rain removal quality evaluation system, including a multi-scale feature forward extraction path, a multi-scale feature backward extraction path, a gated fusion module and a score prediction module; after the evaluation image is input to the multi-scale feature forward extraction path, the multi-scale feature is output to the gated fusion module through the multi-scale feature backward extraction path, and the gated fusion module performs weighted fusion of each scale feature and outputs it to score prediction Modules get points.

The present invention proposes a blind image quality assessment model to accurately predict the quality of rain removal. Because, we propose a bidirectional gated fusion network (B-GFN), which first obtains high-dimensional features through the forward path, and then obtains more image local content and context information through the backward path, and finally passes through the gated fusion. Get multi-scale features of the image.

The beneficial effect of the present invention is that a bidirectional gated fusion network (B-GFN) is proposed to predict the human perception quality of different rain removal results, which is helpful for evaluation and development in real rain scenes.

Description of drawings

FIG. 1 is a network structure diagram of an embodiment system.

Detailed ways

As shown in Figure 1, the architecture of the proposed DQA model, the Bidirectional Gated Fusion Network (B-GFN), is presented. The multi-scale feature forward extraction path includes a convolution and ReLU block, a maximum pooling module, and a 4-level feature forward extraction unit from large to small; the multi-scale feature backward extraction path includes a scale from small to large. 4-level feature backward extraction unit, 3-level upsampling module, 3-level addition operation module, and 4-level spatial pyramid pooling module.

In the forward extraction path of multi-scale features, the image to be evaluated is input into the convolution with a kernel of 7*7 and the ReLU block CB (the number of channels is 96 and the image spectrum with a scale of 160*160) is passed through the maximum pooling module maxPool (obtained). The number of channels is 96 and the image spectrum with a scale of 80*80) is input to the feature forward extraction unit in series, and the feature forward extraction unit at each level except the fourth level is output to the next level feature forward In the extraction unit and the feature extraction module of the same scale in the multi-scale feature backward extraction path, the fourth-level feature is only output to the extraction unit to the first-level feature backward extraction unit in the multi-scale feature backward extraction path; The feature forward extraction unit obtains an image spectrum with 192 channels and a scale of 40*40. The second-level feature forward extraction unit obtains 384 channels and an image spectrum with a scale of 20*20. After the third-level feature forward extraction The unit obtains an image spectrum with a channel number of 1056 and a scale of 10*10, and the fourth-level feature forward extraction unit obtains an image spectrum with a channel number of 2208 and a scale of 5*5. The feature forward extraction unit includes a dense connection block DB and a conversion layer TL; the input of the dense connection block is the input of the feature forward extraction unit, the output of the dense connection block is connected with the input of the conversion layer, and the output of the conversion layer is the feature forward extraction. The output of the unit; the densely connected block has the same structure as densenet161.

In the multi-scale feature backward extraction path, the first-level feature backward extraction unit outputs to the input of the next-level upsampling module and the first-level spatial pyramid pooling module, and the second-level and third-level scale feature backward The extraction unit is output to the addition operation module of this level, and the addition operation module of this level adds the output of the feature backward extraction unit of this level and the output of the upsampling module of this level, and then outputs it to the spatial pyramid pooling module of this level and the upper level of the next level. Sampling module; the fourth-level feature backward extraction unit is output to the current-level addition operation module, and the current-level addition operation module adds the output of the current-level feature backward extraction unit and the output of the current-level up-sampling module and outputs it to the current-level space Pyramid pooling module. The feature backward extraction unit of the first-level scale outputs 256 image spectra with a scale of 5*5, and the feature backward extraction unit of the second-level scale outputs 256 image spectra with a scale of 10*10. The level-scale feature backward extraction unit outputs 256 image spectra with a scale of 20*20, and the fourth-level feature backward extraction unit outputs 256 image spectra with a scale of 40*40. The feature backward extraction unit is convolution and ReLU block CB.

The multi-scale feature backward extraction path bridges the gap between multi-layer feature maps between different databases. X = {x _i } _1≤i≤4 represents the "forward path" features output from the four DBs, now, we reuse these feature maps in a "backward path", which propagates the top-level features back to the previous layers. Represents features resulting from the "reverse path". We calculate with the following equation each element of where f _1×1 ( ) represents the 1×1 convolution and ReLU operations, and ↑2 represents the upsampling operation by a factor of 2.

The gated fusion module includes a feature vector stacking module, a convolution and ReLU block, a Sigmod function, and a dot product operation module; the feature vector stacking module receives the output from the N-level spatial pyramid pooling module and superimposes the received feature vector. The point product operation module and the convolution and ReLU blocks, the convolution and ReLU blocks are output to the Sigmod function, the weighting matrix is elementalized by the Sigmod function, and the Sigmod function outputs the weighted and fused feature vector to the point product operation module. The fused feature vector and the output of the feature vector superposition module are pixel-multiplied and output to the score prediction module.

The weights assigned to each scale are adaptively determined by a gated fusion module. We first use the spatial pyramid pooling SPP (the pooling method proposed by Kaiming He in 2014) to convert the All elements of are superimposed on equal-length eigenvectors, and then they are superimposed to a 4×5120 eigenmatrix Y=[y ₁ ; y ₂ ; y ₃ ; y ₄ ]. The weight matrix W is given by: W=s[f _1×1 (Y)] where s(·) is the sigmoid function. We represent the weights by an elemental product operation and use a 1×1 convolution to generate the fused feature vector, ie: z = f _1×1 (W⊙Y), where ⊙ denotes the pixel multiplication operation.

The score prediction module includes three fully connected layers FC and a Sigmod function. The gated fusion module outputs three fully connected layers, and the three fully connected layers output to the Sigmod function. The Sigmod function maps the input to obtain the quality score Q _p . Q _gt represents the true quality score.

Our learning objective is to minimize the following loss function:

where N represents the number of all training samples, and Q _p (j) and Q _gt (j) represent the jth predicted and true quality scores, respectively.

Claims (5)

1. A reference-free image deraining quality evaluation system, characterized in that it comprises a multi-scale feature forward extraction path, a multi-scale feature backward extraction path, a gated fusion module and a score prediction module; the image to be evaluated is input to a multi-scale feature. After the feature forward extraction path, the multi-scale feature is output to the gated fusion module through the multi-scale feature backward extraction path, and the gated fusion module performs weighted fusion of each scale feature and outputs it to the score prediction module to obtain the score; The multi-scale feature forward extraction path includes a convolution and ReLU block, a maximum pooling module, and an N-level feature forward extraction unit from large to small; the multi-scale feature backward extraction path includes a scale from small to large. N-level feature backward extraction unit, N-1-level upsampling module, N-1-level addition operation module, and N-level spatial pyramid pooling module; In the multi-scale feature forward extraction path, the image to be evaluated is input to the convolution and ReLU block, and then input to the feature forward extraction unit in series in series after the maximum pooling module. The forward extraction unit outputs to the next-level feature forward extraction unit and the feature extraction module of the same scale in the multi-scale feature backward extraction path, and the Nth level feature is only output to the extraction unit in the multi-scale feature backward extraction path. Level 1 feature backward extraction unit; In the multi-scale feature backward extraction path, the first-level feature backward extraction unit outputs to the input of the next-level upsampling module and the first-level spatial pyramid pooling module. The feature backward extraction unit of the level scale is output to the addition operation module of this level, and the addition operation module of this level adds the output of the feature backward extraction unit of this level and the output of the upsampling module of this level and outputs it to the spatial pyramid pooling of this level. module and the next-level upsampling module; the Nth-level feature backward extraction unit is output to the current-level addition operation module, and the current-level addition operation module adds the output of the current-level feature backward extraction unit and the output of the current-level upsampling module. Then output to the spatial pyramid pooling module of this level. 2. system as claimed in claim 1 is characterized in that, gated fusion module comprises feature vector superposition module, convolution and ReLU block, Sigmod function, point multiplication operation module; Feature vector superposition module receives from N-level space pyramid pool The output of the conversion module and the received eigenvectors are superimposed, and then output to the dot product operation module and the convolution and ReLU blocks. The convolution and ReLU blocks are output to the Sigmod function. The weighting matrix is elementalized by the Sigmod function. The feature vector is sent to the point multiplication operation module, and the point multiplication operation module performs pixel multiplication on the weighted and fused feature vector and the output of the feature vector superposition module, and then outputs it to the score prediction module. 3. system as claimed in claim 1 is characterized in that, score prediction module comprises three fully connected layers and Sigmod function, gated fusion module is output to three fully connected layers, and three fully connected layers are output to Sigmod function, Sigmod The function maps the input to get a quality score. 4. The system according to claim 1, wherein the feature forward extraction unit comprises a dense connection block and a conversion layer; the input of the dense connection block is the input of the feature forward extraction unit, and the output of the dense connection block is the same as that of the conversion layer. The input is connected, and the output of the conversion layer is the output of the feature forward extraction unit. 5. The system of claim 1, wherein the feature backward extraction unit is a convolution and ReLU block.