CN116703735A - Image processing method, model training method and device, equipment and storage medium - Google Patents

Abstract

The present disclosure relates to an image processing method, an image processing model training method and apparatus, an electronic device, and a storage medium, where the image processing method may include: performing multi-scale feature coding on the first image to obtain N feature images with different sizes; wherein N is a positive integer equal to or greater than 2; performing local attention mechanism processing on the nth decoding graph, and then performing upsampling to obtain an nth upsampling graph; wherein N is a positive integer less than N; the 0 th decoding diagram is determined according to the N-1 th characteristic diagram; the N-1 th feature map is the last generated feature map in the N feature maps with different sizes; the N-N feature map and the N up-sampling map are fused based on a global attention mechanism to obtain an n+1 decoding map; and obtaining a second image based on the N-1 decoding graph, wherein the image content of the second image is the same as that of the first image, and the image quality of the second image is higher than that of the first image.

Description

Image processing method, model training method and device, equipment and storage medium

technical field

The present disclosure relates to the field of information technology, and in particular to an image processing method, an image processing model training method and device, electronic equipment, and a storage medium.

Background technique

The improvement of the performance of the display screen has brought people an excellent visual experience, but poor quality images or video sources will affect the final display effect. In actual production and life, affected by the scene conditions and the transmission process, some of the captured images not only contain large noise, but also the problems of overexposure and underexposure lead to the loss of image details and color deviation, and the final generated image quality is low . Therefore, image quality enhancement technology has emerged as the times require, that is, through certain technical means, images that are noise-free, correctly exposed, and have no color cast can be recovered as much as possible.

In related technologies, image quality enhancement algorithms use different methods to solve problems such as exposure, color shift, and noise caused by imaging devices in different environments. The problems targeted by this type of method are relatively simple, the robustness is not high, and it cannot meet the complex and changeable environmental conditions and production requirements in real scenarios.

In related technologies, the image quality enhancement algorithm based on deep learning can further improve the visual effect and robustness of the image quality enhancement algorithm compared with the traditional image quality enhancement algorithm, but the generality is not strong, or only It is possible to improve the image quality of a single aspect of image comparison.

Contents of the invention

The disclosure provides an image processing method, an image processing model training method and device, electronic equipment, and a storage medium.

The first aspect of the embodiments of the present disclosure provides an image processing method, wherein the method includes:

Performing multi-scale feature encoding on the first image to obtain N feature maps of different sizes; wherein, the N is a positive integer equal to or greater than 2;

After performing local attention mechanism processing on the nth decoded image, perform upsampling to obtain the nth upsampled image; wherein, the n is a positive integer smaller than the N; the 0th decoded image is determined according to the N-1th feature map The N-1th feature map is the last feature map generated in N feature maps of different sizes;

Based on the global attention mechanism, the N-n feature map and the nth upsampled image are fused to obtain the n+1th decoded image;

A second image is obtained based on the N-1th decoded image, wherein the image content of the second image is the same as that of the first image, and the image quality of the second image is higher than that of the first image Image Quality.

In some embodiments, the multi-scale feature encoding is performed on the first image to obtain N feature maps of different sizes, including:

Convolving the first image to obtain the 0th feature map;

Performing multi-scale feature extraction on the mth feature map to obtain the m+1th feature map, wherein the m is a positive integer less than or equal to the N-1 or 0.

In some embodiments, performing multi-scale feature extraction on the mth feature map to obtain the m+1th feature map includes:

Convolving the mth feature map using convolution kernels of different scales to obtain multiple first-type intermediate feature maps of the same size;

Fusing multiple intermediate feature maps of the first type to obtain the m+1th feature map.

In some embodiments, the nth decoded image is processed based on the local attention mechanism and then upsampled to obtain the nth upsampled image, including:

generating a first weight of the local attention mechanism based on the nth decoded image;

performing weighting processing on the nth decoded image based on the first weight to obtain a second type of intermediate feature map;

Perform upsampling processing on the intermediate feature map of the second type to obtain the nth upsampling map.

In some embodiments, the global attention mechanism is used to fuse the N-n feature map and the nth upsampled image to obtain the n+1th decoded image, including:

generating a second weight of the global attention mechanism according to the nth decoding map;

performing weighting processing on the N-nth feature map according to the second weight to obtain a third type of intermediate feature map;

fusing the third type of intermediate feature map with the nth upsampled image to obtain the n+1th decoded image.

The second aspect of the embodiments of the present disclosure provides an image processing model training method, which is characterized in that it includes:

An image processing model for performing the image processing method of the first aspect is trained.

In some embodiments, the training of the image processing model that executes the image processing method of the first aspect includes:

inputting the sample image into the first model to obtain the predicted image output by the first model;

The predicted image is input to the second model to obtain M first feature maps output by the second model; wherein, the M is a positive integer greater than or equal to 2;

Inputting the target image corresponding to the sample image into the second model, and obtaining M second feature maps output by the second model;

According to the difference between the pth first feature map and the pth second feature map, the first type of loss value is obtained, wherein the p is a positive integer less than or equal to the M;

Adjusting model parameters of the first model based on the first type of loss value.

In some embodiments, the training of the image processing model that executes the image processing method of the first aspect further includes:

Obtaining a second type of loss value according to the difference between the predicted image and the target image;

The adjusting the model parameters of the first model based on the first type of loss value includes:

Adjust model parameters of the first model according to the first type of loss value and the second type of loss value.

In some embodiments, the training of the image processing model that executes the image processing method of the first aspect further includes:

According to the pixel difference between the sth pixel in the predicted image and the adjacent pixels of the sth pixel, the third type of loss value is obtained, wherein the s is a positive integer less than or equal to S; Said S is the total number of pixels included in the predicted image;

The adjusting the model parameters of the first model according to the first type loss value and the second type loss value includes:

Obtaining an image loss value according to the first type of loss value, the second type of loss value, and the third type of loss value;

Based on the image loss value, a model parameter of the first model is adjusted.

In some embodiments, the image loss value obtained according to the first type of loss value, the second type of loss value and the third type of loss value includes:

For the first type of loss value, the weight of the first type of loss value, the second type of loss value, the weight of the second type of loss value, the third type of loss value, the third type The weight of the loss value is weighted and summed to obtain the image loss value;

Wherein, the weight of the first type of loss value is less than the weight of the second type of loss value;

The weight of the third type of loss value is smaller than the weight of the first type of loss value.

A third aspect of the present disclosure provides an image processing device, the device comprising:

An encoding module, configured to perform multi-scale feature encoding on the first image to obtain N feature maps of different sizes; wherein, the N is a positive integer equal to or greater than 2;

The conversion module is used to perform upsampling on the nth decoded image after local attention mechanism processing to obtain the nth upsampled image; wherein, the n is a positive integer smaller than the N; the 0th decoded image is based on the Nth The -1 feature map is determined; the N-1th feature map is the last feature map generated in the N feature maps of different sizes;

The fusion module is used to fuse the N-n feature map and the nth upsampled image based on the global attention mechanism to obtain the n+1th decoded image;

An obtaining module, configured to obtain a second image based on the N-1th decoded image, wherein the image content of the second image is the same as that of the first image, and the image quality of the second image is higher than the Describe the image quality of the first image.

In some embodiments, the encoding module is specifically configured to perform convolution on the first image to obtain the 0th feature map; perform multi-scale feature extraction on the mth feature map to obtain the m+1th feature map, wherein, Said m is a positive integer less than or equal to said N-1 or 0.

In some embodiments, the encoding module is specifically configured to use convolution kernels of different scales to convolve the mth feature map to obtain multiple first-type intermediate feature maps of the same size; A class of intermediate feature maps to obtain the m+1th feature map.

In some embodiments, the conversion module is specifically configured to generate a first weight of the local attention mechanism based on the nth decoded image; perform weighting processing on the nth decoded image based on the first weight , to obtain the second type of intermediate feature map; performing upsampling processing on the second type of intermediate feature map to obtain the nth upsampled map.

In some embodiments, the fusion module is specifically configured to generate a second weight of the global attention mechanism according to the nth decoded image; perform weighting processing on the N-nth feature map according to the second weight , to obtain a third type of intermediate feature map; fusing the third type of intermediate feature map with the nth upsampling image to obtain the n+1th decoding image.

The fourth aspect of the embodiments of the present disclosure provides an image processing model training device, including:

The training module is used for training the image processing model for executing the image processing method of the first aspect.

In some embodiments, the training module includes:

a prediction unit, configured to input the sample image into the first model, and obtain a predicted image output by the first model;

The first input unit is configured to input the predicted image to the second model to obtain M first feature maps output by the second model; wherein, the M is a positive integer greater than or equal to 2;

The second input unit is configured to input the target image corresponding to the sample image into the second model, and obtain M second feature maps output by the second model;

The first loss unit is configured to obtain a first-type loss value according to the difference between the p-th first feature map and the p-th second feature map, wherein the p is less than or equal to the a positive integer of M;

An adjustment unit, configured to adjust model parameters of the first model based on the first type of loss value.

In some embodiments, the training module further includes:

a second loss unit, configured to obtain a second type of loss value according to the difference between the predicted image and the target image;

The adjusting unit is specifically configured to adjust model parameters of the first model according to the first type of loss value and the second type of loss value.

In some embodiments, the training module further includes:

The third loss unit is configured to obtain a third type of loss value according to the pixel difference between the sth pixel in the predicted image and the adjacent pixel of the sth pixel, wherein the s is less than or A positive integer equal to S; the S is the total number of pixels contained in the predicted image;

The adjustment unit is configured to obtain an image loss value according to the first type loss value, the second type loss value, and the third type loss value; adjust the first model based on the image loss value model parameters.

In some embodiments, the fourth loss unit is configured to calculate the first type of loss value, the weight of the first type of loss value, the second type of loss value, and the weight of the second type of loss value The weight, the third type of loss value, and the weight of the third type of loss value are weighted and summed to obtain the image loss value;

Wherein, the weight of the first type of loss value is smaller than the weight of the second type of loss value;

The weight of the third type of loss value is smaller than the weight of the first type of loss value.

A fifth aspect of an embodiment of the present disclosure provides an electronic device, including:

memory for storing processor-executable instructions;

a processor connected to the memory;

Wherein, the processor is configured to execute the method provided in any technical solution of the first aspect or the second aspect.

The sixth aspect of the embodiments of the present disclosure provides a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the computer, the computer can execute any technical solution according to the first aspect or the second aspect. provided law.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

Through multi-scale feature encoding, in the process of optimizing the image quality of the first image, more image information such as image details is retained; through the combination of local attention mechanism and global attention mechanism, the global image can be more completely preserved Information, local information, and image texture recovery and reconstruction, so as to realize denoising, color correction, and exposure adjustment of the first image, and obtain a second image with optimized image quality.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure.

Fig. 1 is a schematic flowchart of an image processing method according to an exemplary embodiment;

Fig. 2 is a schematic diagram showing an image processing model performing image processing according to an exemplary embodiment;

Fig. 3 is a schematic diagram of a multi-scale feature extraction according to an exemplary embodiment;

Fig. 4 is a schematic diagram of image processing based on a global attention mechanism according to an exemplary embodiment;

Fig. 5 is a schematic diagram showing the generation of the first weight of a local attention mechanism and the second weight of the global attention mechanism, and image fusion according to an exemplary embodiment;

Fig. 6 is a schematic diagram of an image processing model training method according to an exemplary embodiment;

Fig. 7 is a schematic diagram of determining the first type of loss value according to an exemplary embodiment;

Fig. 8 is a schematic diagram showing another image processing model training method according to an exemplary embodiment;

Fig. 9 is a schematic diagram of an image processing method according to an exemplary embodiment;

Fig. 10A is a schematic diagram showing comparison of image processing effects according to an exemplary embodiment;

Fig. 10B is a schematic diagram showing comparison of image processing effects according to an exemplary embodiment;

Fig. 10C is a schematic diagram showing comparison of image processing effects according to an exemplary embodiment;

Fig. 11 is a schematic structural diagram of an image processing device according to an exemplary embodiment;

Fig. 12 is a schematic structural diagram of an image processing training device according to an exemplary embodiment;

Fig. 13 is a schematic structural diagram of a terminal device according to an exemplary embodiment;

Fig. 14 is a schematic structural diagram of a server according to an exemplary embodiment.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices consistent with aspects of the present disclosure as recited in the appended claims.

As shown in FIG. 1, an embodiment of the present disclosure provides an image processing method, including:

S110: Perform multi-scale feature encoding on the first image to obtain N feature maps of different sizes; wherein, N is a positive integer equal to or greater than 2;

S120: Perform upsampling on the nth decoded image after local attention mechanism processing to obtain the nth upsampled image; wherein, the n is a positive integer smaller than the N; the 0th decoded image is based on the N-1th The feature map is determined; the N-1th feature map is the last feature map generated in the N feature maps of different sizes;

S130: Fusion processing the N-n feature map and the nth upsampled image based on the global attention mechanism to obtain the n+1th decoded image;

S140: Obtain a second image based on the N-1th decoded image, wherein the image content of the second image is the same as that of the first image, and the image quality of the second image is higher than that of the first image The image quality of the image.

Executing devices of the image processing method include but are not limited to electronic devices such as terminal devices and servers.

The terminal devices include, but are not limited to: mobile phones, tablet computers, wearable devices, smart home devices, or smart office devices.

The server may include various application servers or image servers.

In an embodiment of the present disclosure, the first image may be an original image to be optimized.

The image processing method provided by the embodiments of the present disclosure may be executed based on a deep learning model. The deep learning model is used for image processing, so it can be called an image processing model.

As shown in Figure 2, the image processing module may include:

The coding layer is used to perform feature extraction on the first image to obtain a feature map; exemplary, the image processing model has at least N coding layers for generating N feature maps; exemplary, the coding layer may correspond to FIG. 2 Produce the network layer of F ₀ to F _n in;

The decoding layer is used to further decode the feature map to obtain an optimized image; for example, the image processing model has at least N decoding layers. Exemplarily, the decoding layer may be the network layer generating R ₀ to R _m in FIG. 2 .

There can be a jump connection between the encoding layer and the decoding layer, which can realize the jump transmission of feature maps of different sizes between different encoding layers and different decoding layers.

The multi-scale encoding of the first image is performed using convolution kernels of different sizes to obtain feature maps containing different image information. The image sizes of these feature maps are different, so that the feature maps of different scales can retain the image information of different levels of the first image, so as to facilitate the subsequent second image with high restoration. The N feature maps may be the 0th feature map to the N−1th feature map, respectively. Among them, the image size of the 0th feature map to the N-1th feature map decreases sequentially.

Referring to Figure 2, F ₀ can be the 0th feature map, which is the convolution operation on the original image I _ori , F ₁ can be the first feature map obtained after multi-scale feature extraction of F ₀ , and F ₂ can be F ₁ The second feature map is obtained after multi-scale feature extraction; F _n can be F _n-1 and the n-th feature map is obtained after multi-scale feature extraction. The features included by F ₀ to F _n may be low-level features as shown in FIG. 2 .

The encoding layer of the image processing model shown in FIG. 2 can generate feature maps R ₀ to R _m . The features comprised by R ₀ to R _m may be high-level features as shown in FIG. 2 .

After the N-1th feature map is processed, the input image of the 0th decoding layer will be obtained.

In an embodiment, the 0th feature map may directly be the N-1th feature map.

In another embodiment, a global feature extraction module (GFEB) is used to perform global feature extraction on the N-1th feature map to obtain the 0th decoded map. Exemplarily, the N-1th feature map is convoluted through cascaded convolutional layers to obtain the 0th decoded map.

After the nth decoded image is processed by the local attention mechanism, the processed image is obtained. The local attention mechanism is used to process the nth decoded image, which may include: after weighting the nth decoded image and weights, suppressing noise, realizing color cast correction, underexposure exposure enhancement, and reducing exposure value after overexposure and other feature maps.

The image processed by the local attention mechanism on the nth decoded image is obtained to be up-sampled, so that an up-sampled image with a larger image size can be obtained. The image size of the upsampled image will be equal to the image size of the N-nth feature map.

The N-nth feature map can be transmitted to the decoding layer that processes the nth decoded image through the jump connection of the image processing model.

The nth decoding layer uses a global attention mechanism to weight the N-nth feature map and each pixel of the upsampled image to more completely preserve the image information of the N-nth feature map.

When the upsampled image is fused with the N-nth feature map in S130, after the pixels of the upsampled image and the N-nth feature map are aligned, pixel-by-pixel fusion is implemented based on the weight corresponding to the global attention mechanism. That is, according to the weight corresponding to the global attention mechanism, the pixel in the xth row and the yth column in the upsampled image is fused with the xth row and the yth column in the N-nth feature map. After fusion, the nth +1 for the decoded diagram.

The n+1th decoded picture may be an input picture of the nth decoded layer.

The N-1th decoded picture may be a decoded picture of the last decoding layer. Wherein, the image size of the 0th decoded picture to the N-1th decoded picture increases sequentially.

Exemplarily, S140 may include: performing convolution on the N-1th decoded image to obtain the second image. Exemplarily, a convolution kernel is used to perform a convolution operation on the N-1th decoded image, so that the convolution operation obtains pixel values after convolution to form the second image. For example, the image size of the N-1th decoded image may be larger than the image size of the first image. After the convolution operation of the convolution kernel with a convolution step size greater than 1 and the N-1th decoded image, the image size is equal to The second image of the first image.

Through multi-scale feature encoding, in the process of optimizing the image quality of the first image, more image information such as image details is retained; through the local attention mechanism and the global attention mechanism, more global information and local information of the image can be retained. Information and image texture recovery and reconstruction, so as to realize the denoising, color correction, and exposure adjustment of the first image, and obtain the second image with optimized image quality.

In some embodiments, the S110 may include:

Convolving the first image to obtain the 0th feature map;

Performing multi-scale feature extraction on the mth feature map to obtain the m+1th feature map, wherein the m is a positive integer less than or equal to the N-1 or 0.

There are two adjacent coding layers, and the output image of the previous coding layer is the input image of the next coding layer. Exemplarily, the output image of the 0th coding layer is the input image of the 1st coding layer.

The 1st programming to the N-1th encoding layer respectively perform multi-scale feature extraction on the input image to obtain the output image of the current encoding layer.

Multi-scale feature extraction can use different convolutions to convolve the input image, and then fuse the feature maps after different size convolution kernel feature extraction to obtain the output image of the current coding layer.

Through multi-scale feature extraction, different levels of image information can be extracted, thereby generating a second image with a higher restoration degree of image content and higher image quality.

In some embodiments, performing multi-scale feature extraction on the mth feature map to obtain the m+1th feature map includes:

Convolving the mth feature map using convolution kernels of different scales to obtain multiple first-type intermediate feature maps of the same size;

Fusing multiple intermediate feature maps of the first type to obtain the m+1th feature map.

Convolution kernels of different sizes can convolve the m-th image with the same convolution step size (step size for short) to obtain a convolved image. Referring to FIG. 3 , two convolution kernels are used to perform convolution processing on the mth feature map respectively to obtain a convolved image.

Since the sizes of the two convolution kernels are different, this is the case, but using the same convolution step size to perform convolution on the mth feature map, the convolution image of the same size will be obtained, and then these convolution images will be fused. Get the m+1th feature map.

The two convolution kernels shown in Figure 3, one is a 3*3 convolution kernel, the other is a 1*1 convolution kernel, and the convolution step is 2. The image obtained after convolution of the 3*3 convolution kernel The feature map of 3*3 pixels is like this: the size of the image processed by the 3*3 convolution kernel and the image after the convolution kernel of 1*1 are reduced relative to the number of pixels of the basic image processed, and the two The convolved image is added to the phase, and the m+1th feature map can be obtained.

In this way, more image information such as local color and/or texture of the first image is preserved by using a larger-sized convolution kernel, and content information of the first image is preserved by using a smaller-sized convolution kernel. And after the feature map processed by the multi-scale convolution kernel is fused, the feature map that retains more image information can be obtained, and the refined processing of the image content can be realized.

The multi-scale feature extraction of the mth feature map to obtain the m+1th feature map includes:

Convolving the mth feature map using convolution kernels of different scales to obtain multiple first-type intermediate feature maps of the same size;

Fusing multiple intermediate feature maps of the first type to obtain the m+1th feature map.

Convolution kernels of multiple scales use the same convolution step to process the m-th feature map, and the obtained image after convolution is called the first type of intermediate feature map. The image size of multiple first-class feature maps is the same.

Fusing multiple intermediate feature maps of the first type includes: after pixel alignment of multiple intermediate feature maps of the first type, adding feature values or arithmetically averaging feature values to obtain a final m+1th feature map.

In some embodiments, the nth decoded image is processed based on the local attention mechanism and then upsampled to obtain the nth upsampled image, including:

generating a first weight of the local attention mechanism based on the nth decoded image;

performing weighting processing on the nth decoded image based on the first weight to obtain a second type of intermediate feature map;

Perform upsampling processing on the intermediate feature map of the second type to obtain the nth upsampling map.

For example, the generating the first weight of the local attention mechanism based on the nth decoded image includes:

Perform global pooling on the nth decoded image to obtain the feature vector;

Performing full connection processing on the feature vectors to obtain a first weight vector, where the first weight vector includes: a plurality of first weights, and the plurality of first weights are the same or different. Exemplarily, when performing full-connection processing, one or more convolutions may be performed to obtain an nth upsampled image having the same image size as the nth decoded image.

The weighting process here may include: multiplying the eigenvalue (or pixel value) of the nth decoded image by the corresponding weight to obtain a new eigenvalue, thereby generating the second type of intermediate eigenmap.

By upsampling the intermediate feature map of the second type, an nth upsampled map with an increased number of pixels will be obtained.

Referring to FIG. 5 , R _mk-1 can be obtained by performing global pooling to contain 2C vectors, and 2C first weights can be obtained through a fully connected operation involving convolution operations.

Performing upsampling processing on the second type of intermediate feature map to obtain the nth upsampling map may include:

performing bilinear interpolation processing on the second type of intermediate image to obtain an interpolated image;

Perform convolution processing on the interpolated image to obtain the nth upsampled image.

In this way, the local content of the first image is extracted through the processing of the local attention mechanism.

Referring to Figure 5, 2C feature maps with an image size of h*w are obtained through bilinear interpolation to obtain 2C intermediate feature maps with an image size of 2h*2w, and then convolution is performed through a 3*3 convolution kernel, etc. Product operation to obtain C feature maps with image size of 2h*2w, and the C feature maps with image size of 2h*2w are the aforementioned nth upsampled maps.

The different feature maps of the 2C feature maps with image size h*w shown in FIG. 5 may be feature maps of different channels of the same first image processed by the same decoding layer. For example, for an RGB image, there are feature maps of the R channel, the G channel, and the B channel respectively.

R _mk shown in FIG. 5 is the input image of the next decoding layer after the fusion of the aforementioned nth upsampled image and the Nnth feature map.

As shown in Figure 4, the said global attention mechanism is based on the fusion processing of the N-n feature map and the nth upsampled image to obtain the n+1th decoded image, including:

S131: Generate a second weight of the global attention mechanism according to the nth decoded image;

S132: Perform weighting processing on the N-nth feature map according to the second weight to obtain a third type of intermediate feature map;

S133: Fuse the third type of intermediate feature map and the nth upsampled image to obtain the n+1th decoded image.

In some embodiments, generating the second weight of the global attention mechanism according to the nth decoded image may include:

Perform global pooling on the nth decoded image to obtain the feature vector;

Performing full connection processing on the feature vectors to obtain a second weight vector, where the second weight vector includes: a plurality of second weights. Exemplarily, weight values of multiple second weights may be the same or different.

Referring to FIG. 5 , R _mk-1 can be obtained by global pooling to include 2C vectors, and C second weights can be obtained through a fully connected operation involving convolution operations.

The number of feature maps represented by C or 2C in Figure 5. 2h or h in Figure 5 represents the number of rows corresponding to the feature map. w or 2w represents the number of columns of the corresponding feature map.

The N-nth feature map is weighted according to the second weight to obtain a third type of intermediate feature map, including:

The second weight is multiplied by the feature value in the N-nth feature map, and the product is multiplied to the feature value of the third type of intermediate feature map.

The fusing of the third type of intermediate feature map and the nth upsampling image to obtain the n+1th decoding image includes:

The eigenvalues of the third type of intermediate feature maps in the same row and the same column are fused with the eigenvalues of the nth upsampled image to obtain the eigenvalues of the corresponding pixels in the n+1th decoded image.

In this way, when processing the first image, the global attention mechanism and the local attention mechanism are integrated, and the two attention mechanisms are mixed, that is, the second image is encoded and decoded using the mixed attention mechanism. In this way, the scale feature extraction makes the image information acquired by the network more abundant, and the mixed attention module fuses global attention and local attention at the same time during the network upsampling process, so that the global information, local information and detailed texture information of the image are obtained. Various degrees of restoration and reconstruction. HAU in Figure 2 represents a hybrid attention mechanism that integrates local attention mechanisms and global attention mechanisms at each decoding layer.

An embodiment of the present disclosure also provides a method for training an image processing model, including: training an image processing model that executes the image processing method provided by any of the foregoing technical solutions.

That is, using the sample data and the labels of the sample data, the trained image processing model can be the image processing model shown in 2.

As shown in Figure 6, the image processing model provided by the embodiment of the present disclosure may include:

S210: Input the sample image into the first model, and obtain the predicted image output by the first model;

S220: Input the predicted image into the second model to obtain M first feature maps output by the second model; wherein, the M is a positive integer greater than or equal to 2;

S230: Input the target image corresponding to the sample image into the second model, and obtain M second feature maps output by the second model;

S240: Obtain a first-type loss value according to the difference between the p-th first feature map and the p-th second feature map, where the p is a positive integer less than or equal to the M;

S250: Adjust model parameters of the first model based on the first type of loss value.

The first model here is the image processing model to be trained, that is, a model that can execute the image processing method provided by any of the foregoing embodiments.

The sample image to be optimized is input to the first model, and after the first model undergoes a series of optimization processes, a predicted image output based on the current model parameters will be obtained.

In the embodiment of the present disclosure, the predicted image output by the first model and the target image to be optimized corresponding to the sample image are respectively input into the second model. The second model includes one or more feature extraction layers, and these feature extraction layers obtain feature maps through one or more convolutions.

In the embodiment of the present disclosure, after the predicted image is input to the second model, M first feature maps are obtained, and then M second feature maps of the target image showing the optimized target effect are obtained.

The first feature map and the second feature map are sorted and compared according to the feature extraction layer, and the difference between the pth first feature map and the pth second feature map is obtained, and the first type of loss value is obtained.

According to the difference between the first first feature map and the first second feature map, a loss item will be obtained. In this way, M first feature maps and M second feature maps will get a total of M loss items. After adding the M loss items, the first type of loss value will be obtained.

The reference figure shows that the second model can be a visual geometry group network (Visual Gemetry Group network, VCG) VGG, and one or more rectified linear units (Rectified linear unit, ReLU) can be used for image feature extraction.

Exemplarily, the second model shown in FIG. 7 may include 5 layers, namely relu1_2, relu2_2, relu3_3, relu4_3, relu5_3 layers, and then obtain the first feature map and the second feature respectively output by the 5 layers, and then Compare the first feature map and the second feature map of the corresponding layer to obtain the difference between the first feature map and the second feature map of the corresponding layer, and combine the first feature map and the second feature map respectively output by the five layers. The difference between , get the first type of loss value. I _gt shown in FIG. 7 may be a target image; I _gen may be a predicted image output by the first model. f _R ¹ to f _R ⁵ are the first feature maps when M is equal to 5. f _G ¹ to f _G ⁵ are the second feature maps when M is equal to 5. The a _i shown in Figure 7 is the weight for calculating the loss of each layer of network output. This set of weights scales the error of each layer output to ensure that different important feature maps get different degrees of pixel difference calculation.

If the model parameters of the first model can achieve the image optimization of the target effect, the first type of loss value will be small enough. If the first type of loss value is too large, it is necessary to adjust the model parameters, and then through the training of sample images and model The update of the parameters implements the update of the model parameters of the first model. The model parameters of the first model may include: weights and/or thresholds of each node in the first model.

Exemplarily, the model parameters of the first model are updated in a reverse update manner according to the loss value of the first model.

In some embodiments, as shown in FIG. 8, the training of the image processing model further includes:

S241: Obtain a second type of loss value according to the difference between the predicted image and the target image;

The S250 may include: adjusting model parameters of the first model according to the first type of loss value and the second type of loss value.

In the embodiment of the present disclosure, the predicted image can be directly compared with the target image to obtain the difference between the target image and the predicted image, thereby obtaining the second type of loss value. Exemplarily, the mean absolute error (MAE) or mean square error (MSE) between the predicted image and the target image is used as the second type of loss value.

After the second type of loss value is calculated, the arithmetic mean or weighted average of the first type of loss value and the second type of loss value will be calculated to obtain a loss value, and the model parameters will be updated based on the loss value by means of backpropagation, etc. wait.

If the weighted average of the first type loss value and the second type loss value is used as the loss value for updating model parameters, the weight value of the first type loss value can be smaller than the weight value of the second type loss value, thus paying more attention to The second type of loss value This global loss value to speed up the training of the first model.

In some embodiments, the training of the image processing model also includes:

S242: Obtain a third type of loss value according to the pixel difference between the sth pixel in the predicted image and the adjacent pixels of the sth pixel, where the s is a positive integer less than or equal to S ; The S is the total number of pixels included in the predicted image;

As shown in Figure 8, the S250 may include:

S251: Obtain an image loss value according to the first type of loss value, the second type of loss value, and the third type of loss value;

S252: Adjust model parameters of the first model based on the image loss value.

Through the introduction of the loss of the third type of loss value, the smoothness between the optimized image pixels can be considered.

For example, the sum of the first type loss value, the second type loss value, and the third type loss value is calculated to obtain the image loss value. Alternatively, calculate the arithmetic mean of the first type of loss value, the second type of loss value and the third type of loss value to obtain the image loss value. Alternatively, a weighted average of the first type loss value, the second type loss value, and the third type loss value is calculated to obtain the image loss value. When calculating the weighted average of the first type of loss value, the second type of loss value and the third type of loss value, the weight of the first type of loss value is less than the weight of the second type of loss value, and the weight of the third type of loss value can be Weights less than or equal to the first class loss value.

Therefore, in the embodiment of the present disclosure, the first type loss value to the third type loss value can be calculated simultaneously to obtain the final image loss value, and then finally update the model parameters of the first model in combination with the image loss value.

When the calculated loss value is less than the loss threshold, or the minimum image loss value is obtained, it can be determined that the first model has completed the model training, the current model parameters of the first model are the target parameters after training, and the first model that collects the target parameters is is an image processing model capable of any of the aforementioned image processing methods.

In actual production and life, the image transmission and acquisition system and transmission process are relatively complex, and the imaging device is easily affected by the surrounding environment during the image acquisition process, such as the overall light distribution caused by the lighting environment or the reflection of the object surface. Uniform and mechanical equipment will automatically collect noise and so on. In addition, image compression coding, image storage and communication in the process of image transmission have brought a certain degree of damage to the image quality, making the image on the final display device unable to represent the natural scene under real conditions, losing brightness and contrast , color and other information, not only does not meet the visual perception effect of the human eye, but also does not meet the actual production needs.

For example, when image acquisition is performed in an outdoor environment, too strong sunlight will have a great impact on the final imaging, resulting in image loss of contrast and exposure problems.

In the monitoring system, the collected video images often contain a lot of noise, and the image illumination is low, which is not convenient for fast and direct tracking and monitoring of criminal behavior at night.

In the professional field of image design, the image transmission system will produce some color deviation, so that the final displayed image does not meet the creator's intention.

In addition, the problem of poor image quality caused by various environmental and equipment reasons will also directly affect the performance of military reconnaissance, automatic driving, remote sensing imaging and other systems.

Aiming at the above-mentioned quality problems that may arise during imaging and transmission, this disclosure proposes an image quality enhancement method based on deep learning, given an input image with noise, color cast and exposure problems, and finally restores a noise-free image , images with high contrast and correct color representation.

The image quality enhancement algorithm based on deep learning has made significant progress in both accuracy and speed, but there are still some problems in the current image quality enhancement algorithm, as follows:

First: Most networks design different network structures according to different scenarios, and lack a unified and general network framework to solve image quality enhancement problems such as image exposure adjustment, color correction, and image denoising.

Second: The image cannot be reconstructed with high resolution due to the loss of information due to the network structure of upsampling and downsampling.

Third: The neural network lacks a complete feature extraction module in the process of image processing.

Based on the network structure of encoding and decoding, the embodiment of the present disclosure proposes an image enhancement algorithm using a multi-scale downsampling module and a mixed attention strategy. Multi-scale feature extraction makes the image information acquired by the network richer, and the mixed attention module In the process of network upsampling, the global attention and local attention are fused simultaneously, so that the global information, local information and detailed texture information of the image are restored and reconstructed to varying degrees.

Based on the codec structure, an end-to-end image quality enhancement network (image processing model) is built, and a multi-scale feature extraction and mixed attention strategy is proposed to improve the network structure.

In view of the incompleteness of the deep learning network in the feature extraction process, a global feature enhancement sub-network is added to the basic network, and convolution kernels of different sizes are used for feature extraction, and features of various scales are fused for transmission, and according to different sizes The feature extraction capability of the convolution kernel determines the final convolution kernel size.

Aiming at the lack of obvious layers and details and halo problems in the process of image restoration, the hybrid attention strategy proposed by the embodiments of the present disclosure is adopted. Incorporate global attention into the skip-connection structure, and use high-level features to generate weights to give different attention to low-level channels, so that the final reconstructed image has rich details and obvious layers. Incorporate local attention when upsampling images, and generate a set of self-attention weights for high-level features in the decoding process to guide its own detail recovery and reconstruction.

Considering that the L2 norm function as a network loss function leads to the problem that the final generated image is too smooth, the embodiment of the present disclosure jointly uses the perceptual loss function based on the VGG network on the basis of the L2 loss function, and performs image processing in a higher-dimensional feature space. The semantic information of the network is constrained to ensure that the network output image has a better visual effect.

As shown in Figure 2, the input of the entire network is an RGB image. First, a convolutional layer is used to extract basic features, and then five multi-scale feature extraction modules are used to perform multi-scale feature extraction and cross-scale feature fusion.

When the size of the feature map is small, the global feature extraction module aggregates the global context information on the basis of the input feature map and transmits it backwards. The encoded feature map is decoded by five mixed attention upsampling networks, and the feature map is finally Return to the original size of the image, complete the reconstruction of the image, and then output a three-channel image through a convolutional layer, which is the final image.

The multi-scale feature extraction module consists of two steps:

Multi-scale feature extraction and cross-scale feature fusion.

In the process of multi-scale feature extraction, the feature map in the same dimension is subjected to feature extraction and downsampling processing through convolution kernels of two sizes at the same time, and the convolution kernel of 3×3 size is used to extract local color, texture and other information of the image. , the 1×1 convolution kernel is used to maintain the content information of the image, and achieve more refined processing in the process of downsampling to ensure that each pixel of the image can be uniquely and accurately mapped.

Different from other multi-scale strategies, the embodiment of the present disclosure adopts pixel-by-pixel addition instead of cascading in cross-scale feature fusion. This operation can avoid changes in the number of channels, reduce the amount of calculation, and accelerate network convergence. .

The attention mechanism rescales the features of each channel according to the importance of each channel information by modeling the dependencies and correlations between feature channels. This mechanism allows the network model to focus more on effective channel information and improve Performance of the network on different tasks. The mixed attention upsampling module in this disclosure is implemented based on the local attention module and the global attention module. The use of the mixed attention enables the network to take into account the recovery of both global image information and local information during the upsampling process.

The implementation process of the mixed attention upsampling module is shown in Figure 3. The upper part of the module is the process of high-level features generating attention to guide the transmission of low-level features, and the lower part represents the process of high-level features generating attention and guiding itself to perform upsampling reconstruction. process, and finally the two parts of features are added pixel by pixel for fusion, and the output of higher-resolution advanced features continues to be passed backwards. As shown in Figure 2, the advanced features first undergo a global pooling to generate 2C feature maps of 1×1 size, which represent the richness of various information of the advanced features, and then generate global attention and local attention respectively.

First pass through a fully connected layer to get a set of weights, which are multiplied by channels with the low-level features passed through a convolutional layer to generate a feature map F′ _k with global attention. At the same time, 2C feature maps of 1×1 size pass through a fully connected layer to the right to obtain another set of self-attention weights. This set of features is multiplied by advanced features to generate a feature map R′ _mk-1 with local attention. So far, the global attention and local attention operations are all over. Finally, the high-level feature R′ _mk-1 with local attention is upsampled, and then the low-level feature map with global attention is fused to generate a higher-level feature R′ _mk with mixed attention.

Ordinary image quality enhancement networks mostly use the mean absolute error (MAE) or mean square error (MSE) between two images as a loss function to optimize network parameters.

MAE sums the absolute value of each pixel difference between two images, while MSE sums the square of each pixel difference.

Compared with MAE, MSE will amplify the gap between larger errors and smaller errors due to the operation of the square of the pixel difference. The loss function is more punitive for larger errors and more tolerant of smaller errors.

In the embodiment of the present disclosure, the L2 loss function, that is, MSE, is used as the content loss function of the network, and the L2 loss function is more inclined to constrain the global information distribution of the image in the process of reverse transmission, and the final solution is the global optimal solution. As a result, the resulting image becomes blurred. Therefore, on the basis of the L2 loss function, the perceptual loss function based on the VGG pre-training model is jointly used. The high-dimensional perceptual loss function can produce high-quality images. The calculation process of the perceptual loss function can be shown in Figure 7.

The calculation of perceptual loss is based on the output of each Relu layer of the VGG feature extraction network. In this way, the difference between the generated image and the real image is calculated in each dimension of the feature map, and the feature information of different levels is constrained to be reconstructed. The calculation process of the perceptual loss function is as follows:

Pass the generated image f _i and the true value image g ⁱ through the trained VGG16 network respectively, and obtain the feature map Φ _i (i=1, 2.3, 4.5) of each layer of the model, that is, corresponding to relu1_2, reluu2_2, relu3_3, relu4_3, relu5_3 The output of the layer, the average absolute error between pixels is calculated for this output.

a _i is the weight for calculating the loss of each layer of network output. This set of weights scales the error of each layer output to ensure that different important feature maps get different degrees of pixel difference calculation.

Since the feature information of the middle layer covers both texture and content information when the VGG network performs feature extraction, it has a great influence on the optimization direction of the entire network. Therefore, when conducting experiments, we scale down the weights of the middle two layers to determine the final weight ratio.

The entire image quality enhancement network process is shown in Figure 9. The input image is sequentially passed through the feature extraction module and the feature reconstruction module to generate an output image;

Then calculate the content loss, perceptual loss and total variation loss between the output image and the real value image respectively, and then optimize the parameters of the network feature extraction module and feature reconstruction module by reducing these three losses, so that the next network iteration The final output is a generated image that is closer to the ground truth map.

The ultimate optimization goal of the network is to make the generated image as consistent as possible with the ground truth image. Here, the content loss may be one of the aforementioned second type of loss values; the perceptual loss may be one of the aforementioned first type of loss values; the total variation loss may be the aforementioned third type of loss values.

The image quality enhancement network in the embodiment of the disclosure is built based on the Pytorch deep learning framework, and a GFORCE RTX 2080Ti GPU is used in the experiment for model training and testing.

In the above hardware and software environments, three image quality enhancement tasks, such as image exposure adjustment, color correction, and image denoising, were tested. The number of training sessions was set to 200 epochs, and the batch size (batchsize) during the training process was set to 4. The test process batch size (batchsize) is set to 1.

In the training process, the L2 norm loss function and the perceptual loss function are used alternately. The first 100 rounds use the L2 norm loss and the perceptual loss to optimize the network parameters, and the last 100 rounds use the L2 loss function alone to fine-tune the network parameters. That is, in the training of the aforementioned image processing model, the training of one stage may include: the first part of the training in the N trainings simultaneously combines the first type loss value and the second type loss value to update the model parameters of the first model, and after N training times Another part of the training can only update the model parameters of the first model according to the second type of loss value, thereby reducing the overhead of model training and improving the efficiency of model training.

The Adam optimizer is used in the network learning process, the learning rate is set to 0.00001, and the parameters of the network are iteratively optimized with a smaller step size to ensure that the network can achieve finer image detail reconstruction. Among them, the optimization steps of the Adam algorithm are shown in Table 2.

Table 2

The embodiments of the disclosure design ablation experiments for the multi-scale feature extraction module and the mixed attention upsampling module respectively. The experimental results show that the multi-scale feature extraction module enables the entire network feature extraction part to obtain relatively complete feature information, 1×1 The joint use of the convolution kernel and the 3×3 convolution kernel also makes the final reconstruction result more refined and the image definition is higher. The attention weight generated by the high-level features in the mixed attention upsampling network "purposely" guides the transmission of low-level features and the resolution recovery of high-level features themselves, so that the region of interest with a large amount of information in the image is enhanced and redundant Information is suppressed. At the same time, the addition of perceptual loss constrains the image features in a higher-dimensional space, making the network output an image that is more in line with the characteristics of the human eye.

The comparison of image processing effects can be shown in FIG. 10A to FIG. 10C .

The pictures (a) in Figure 10A to Figure 10C are all original images that need to be processed by the model; the picture (b) is the target image corresponding to the target effect; the pictures (c) to (g) are all other models after the original image is processed The obtained image; Picture (h) is the second image obtained after processing by the image processing model provided by the embodiment of the present disclosure.

Pictures (a) to (h) in FIG. 10A are picture effects after denoising.

Pictures (a) to (h) in FIG. 10B are pictures with image details processed.

Pictures (a) to (h) in FIG. 10C are picture effects of illuminance adjustment.

As shown in FIG. 11 , an embodiment of the present disclosure provides an image processing device, and the device includes:

The encoding module 110 is configured to perform multi-scale feature encoding on the first image to obtain N feature maps of different sizes; wherein, the N is a positive integer equal to or greater than 2;

The conversion module 120 is configured to perform upsampling on the nth decoded image after local attention mechanism processing to obtain the nth upsampled image; wherein, the n is a positive integer smaller than the N; the 0th decoded image is based on the The N-1 feature map is determined; the N-1th feature map is the last feature map generated in the N feature maps of different sizes;

The fusion module 130 is used to fuse the N-n feature map and the nth upsampled image based on the global attention mechanism to obtain the n+1th decoded image;

Obtaining module 140, configured to obtain a second image based on the N-1th decoded image, wherein the image content of the second image is the same as that of the first image, and the image quality of the second image is higher than The image quality of the first image.

In some embodiments, the encoding module 110 , the conversion module 120 , the fusion module 130 and the obtaining module 140 may be program modules; after the program modules are executed by the processor, the above functions can be realized.

In some other embodiments, the encoding module 110, the conversion module 120, the fusion module 130 and the obtaining module 140 can all be soft and hard combination modules; the soft and hard combination modules include but are not limited to various programmable arrays; The aforementioned programmable arrays include, but are not limited to: Field Programmable Arrays and/or Complex Programmable Arrays.

In some other embodiments, the encoding module 110 , the conversion module 120 , the fusion module 130 and the obtaining module 140 can all be pure hardware modules; the pure hardware modules include but are not limited to: application specific integrated circuits.

In some embodiments, the encoding module 110 is specifically configured to perform convolution on the first image to obtain the 0th feature map; perform multi-scale feature extraction on the mth feature map to obtain the m+1th feature map, wherein , the m is a positive integer less than or equal to the N-1 or 0.

In some embodiments, the encoding module 110 is specifically configured to use convolution kernels of different scales to convolve the mth feature map to obtain multiple first-type intermediate feature maps of the same size; The first type of intermediate feature map is used to obtain the m+1th feature map.

In some embodiments, the conversion module 120 is specifically configured to generate a first weight of the local attention mechanism based on the nth decoded image; and weight the nth decoded image based on the first weight processing to obtain the second type of intermediate feature map; performing upsampling processing on the second type of intermediate feature map to obtain the nth upsampling map.

In some embodiments, the fusion module 130 is specifically configured to generate a second weight of the global attention mechanism according to the nth decoded image; and weight the N-nth feature map according to the second weight processing to obtain a third type of intermediate feature map; fusing the third type of intermediate feature map with the nth upsampling image to obtain the n+1th decoding image.

An embodiment of the present disclosure provides an image processing model training device, including:

The training module is used to train an image processing model that executes the image processing method provided by any of the aforementioned technical solutions.

In some embodiments, the training module may be a program module; after the program module is executed by the processor, it can realize the training function of the image processing model.

In some other embodiments, the training module can be a combination of hardware and software; the combination of hardware and software includes, but is not limited to, various programmable arrays; the programmable arrays include, but are not limited to: field programmable arrays and/or complex programmable arrays.

In some other embodiments, the training module may be a pure hardware module; the pure hardware module includes but not limited to: an application specific integrated circuit.

In some embodiments, as shown in Figure 12, the training module includes:

A prediction unit 210, configured to input the sample image into the first model, and obtain a predicted image output by the first model;

The first input unit 220 is configured to input the predicted image to the second model to obtain M first feature maps output by the second model; wherein, the M is a positive integer greater than or equal to 2;

The second input unit 230 is configured to input the target image corresponding to the sample image into the second model, and obtain M second feature maps output by the second model;

The first loss unit 240 is configured to obtain a first-type loss value according to the difference between the p-th first feature map and the p-th second feature map, wherein the p is less than or equal to the A positive integer of M;

The adjustment unit 250 is configured to adjust model parameters of the first model based on the first type of loss value.

In some embodiments, the training module further includes:

a second loss unit, configured to obtain a second type of loss value according to the difference between the predicted image and the target image;

The adjusting unit 250 is specifically configured to adjust model parameters of the first model according to the first type of loss value and the second type of loss value.

In some embodiments, the training module further includes:

The third loss unit is configured to obtain a third type of loss value according to the pixel difference between the sth pixel in the predicted image and the adjacent pixel of the sth pixel, wherein the s is less than or A positive integer equal to S; the S is the total number of pixels contained in the predicted image;

The adjustment unit 250 is configured to obtain an image loss value according to the first type loss value, the second type loss value, and the third type loss value; based on the image loss value, adjust the first The model parameters for the model.

In some embodiments, the fourth loss unit is configured to calculate the first type of loss value, the weight of the first type of loss value, the second type of loss value, and the weight of the second type of loss value The weight, the third type of loss value, and the weight of the third type of loss value are weighted and summed to obtain the image loss value;

Wherein, the weight of the first type of loss value is less than the weight of the second type of loss value;

The weight of the third type of loss value is smaller than the weight of the first type of loss value.

An embodiment of the present disclosure provides an electronic device, including:

memory for storing processor-executable instructions;

a processor connected to the memory;

Wherein, the processor is configured to execute the image processing method or the image processing model training method provided by any of the foregoing technical solutions.

The processor may include various types of storage media, which are non-transitory computer storage media, and can continue to memorize and store information thereon after the communication device is powered off.

The processor can be connected to the memory through a bus, etc., and is used to read the executable program stored on the memory, for example, it can execute the image processing method and image processing model training shown in any one of Figures 1, 4, 6 to 8 at least one of the methods.

The electronic device may be the aforementioned terminal device and/or server.

Fig. 13 is a block diagram showing a terminal device 800 according to an exemplary embodiment. For example, the terminal device 800 may be a terminal device such as a mobile phone or a mobile computer, or a smart home device or a smart office device.

Referring to Fig. 13, the terminal device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, a multimedia data component 810, an input/output (I/O) interface 812, a sensor component 814, and a communication component 816.

The processing component 802 generally controls the overall operations of the terminal device 800, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802 .

The memory 804 is configured to store various types of data to support operations at the device 800 . Examples of such data include instructions for any application or method operating on the terminal device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

The power supply component 806 provides power to various components of the terminal device 800 . Power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for end device 800 .

The multimedia component 808 includes a screen providing an output interface between the terminal device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may not only sense a boundary of a touch or a swipe action, but also detect duration and pressure associated with the touch or swipe operation. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. When the device 800 is in an operating state, such as a shooting state or a video state, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capability.

The multimedia data component 810 is configured to output and/or input multimedia data signals. For example, the multimedia data component 810 includes a microphone (MIC), which is configured to receive external multimedia data signals when the terminal device 800 is in an operating state, such as a calling state, a recording state, and a voice recognition state. The received multimedia data signals may be further stored in memory 804 or transmitted via communication component 816 . In some embodiments, the multimedia data component 810 further includes a speaker for outputting multimedia data signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: a home button, volume buttons, start button, and lock button.

The sensor component 814 includes one or more sensors for providing status assessment of various aspects of the terminal device 800 . For example, the sensor component 814 can detect the open/closed state of the device 800, the relative positioning of components, for example, the components are the display and the keypad of the terminal device 800, and the sensor component 814 can also detect the position of the terminal device 800 or a component of the terminal device 800 changes, the presence or absence of user contact with the terminal device 800 , the orientation or acceleration/deceleration of the terminal device 800 and the temperature change of the terminal device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 814 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communications between the terminal device 800 and other devices. The terminal device 800 can access a wireless network based on communication standards, such as Wi-Fi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, apparatus 800 may be programmed by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable A gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation for performing the methods described above.

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, which can be executed by the processor 820 of the device 800 to implement the above method. For example, the non-transitory computer readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

As shown in FIG. 14 , an embodiment of the present disclosure shows a structure of a server. Referring to FIG. 14 , server 900 includes processing component 922 , which further includes one or more processors, and a memory resource represented by memory 932 for storing instructions executable by processing component 922 , such as application programs. The application program stored in memory 932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute instructions to perform the aforementioned method and the aforementioned image processing method, for example, the image processing method and/or image processing model training shown in any one of Figures 1, 4, 6 to 8 method.

Server 900 may also include a power component 926 configured to perform power management of server 900 , a wired or wireless network interface 950 configured to connect server 900 to a network, and an input-output (I/O) interface 958 . The server 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™ or the like.

An embodiment of the present disclosure provides a computer storage medium, which may be a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by the processor of the server or the terminal, the power supply device can perform any of the aforementioned implementations. The image processing method and/or image processing model training method provided by the example can execute at least one of the methods shown in any one of Fig. 1 to Fig. 4 .

After the instructions stored in the computer storage medium are executed, the UE can implement an image processing method and/or an image processing model training method.

Image processing methods may include:

Image processing method, is characterized in that, described method comprises:

Performing multi-scale feature encoding on the first image to obtain N feature maps of different sizes; wherein, the N is a positive integer equal to or greater than 2;

After performing local attention mechanism processing on the nth decoded image, perform upsampling to obtain the nth upsampled image; wherein, the n is a positive integer smaller than the N; the 0th decoded image is determined according to the N-1th feature map The N-1th feature map is the last feature map generated in N feature maps of different sizes;

Based on the global attention mechanism, the N-n feature map and the nth upsampled image are fused to obtain the n+1th decoded image;

A second image is obtained based on the N-1th decoded image, wherein the image content of the second image is the same as that of the first image, and the image quality of the second image is higher than that of the first image Image Quality.

Understandably, the multi-scale feature encoding is performed on the first image to obtain N feature maps of different sizes, including:

Convolving the first image to obtain the 0th feature map;

Performing multi-scale feature extraction on the mth feature map to obtain the m+1th feature map, wherein the m is a positive integer less than or equal to the N-1 or 0.

In some embodiments, performing multi-scale feature extraction on the mth feature map to obtain the m+1th feature map includes:

Convolving the mth feature map using convolution kernels of different scales to obtain multiple first-type intermediate feature maps of the same size;

Fusing multiple intermediate feature maps of the first type to obtain the m+1th feature map.

In some embodiments, the nth decoded image is processed based on the local attention mechanism and then upsampled to obtain the nth upsampled image, including:

generating a first weight of the local attention mechanism based on the nth decoded map;

performing weighting processing on the nth decoded image based on the first weight to obtain a second type of intermediate feature map;

Perform upsampling processing on the intermediate feature map of the second type to obtain the nth upsampling map.

In some embodiments, the global attention mechanism is used to fuse the N-n feature map and the nth upsampled image to obtain the n+1th decoded image, including:

generating a second weight of the global attention mechanism according to the nth decoding map;

performing weighting processing on the N-nth feature map according to the second weight to obtain a third type of intermediate feature map;

fusing the third type of intermediate feature map with the nth upsampled image to obtain the n+1th decoded image.

The image processing model training method may include: training the image processing model of the image processing method provided in any of the foregoing embodiments.

Specifically, the image processing model training method may include:

inputting the sample image into the first model to obtain the predicted image output by the first model;

The predicted image is input to the second model to obtain M first feature maps output by the second model; wherein, the M is a positive integer greater than or equal to 2;

Inputting the target image corresponding to the sample image into the second model, and obtaining M second feature maps output by the second model;

According to the difference between the pth first feature map and the pth second feature map, the first type of loss value is obtained, wherein the p is a positive integer less than or equal to the M;

Adjusting model parameters of the first model based on the first type of loss value.

Understandably, the image processing model training method also includes:

Obtaining a second type of loss value according to the difference between the predicted image and the target image;

The adjusting the model parameters of the first model based on the first type of loss value includes:

Adjust model parameters of the first model according to the first type of loss value and the second type of loss value.

The image processing model training method also includes:

According to the pixel difference between the sth pixel in the predicted image and the adjacent pixels of the sth pixel, the third type of loss value is obtained, wherein the s is a positive integer less than or equal to S; Said S is the total number of pixels included in the predicted image;

The adjusting the model parameters of the first model according to the first type loss value and the second type loss value includes:

Obtaining an image loss value according to the first type of loss value, the second type of loss value, and the third type of loss value;

Based on the image loss value, a model parameter of the first model is adjusted.

It can be understood that the image loss value obtained according to the first type loss value, the second type loss value and the third type loss value includes:

For the first type of loss value, the weight of the first type of loss value, the second type of loss value, the weight of the second type of loss value, the third type of loss value, the third type The weight of the loss value is weighted and summed to obtain the image loss value;

Wherein, the weight of the first type of loss value is less than the weight of the second type of loss value;

The weight of the third type of loss value is smaller than the weight of the first type of loss value.

Other embodiments of the present disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. The present disclosure is intended to cover any modification, use or adaptation of the present disclosure. These modifications, uses or adaptations follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure. . The specification and examples are to be considered exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It should be understood that the present disclosure is not limited to the precise constructions which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (22)

1. An image processing method, the method comprising:

performing multi-scale feature coding on the first image to obtain N feature images with different sizes; wherein N is a positive integer equal to or greater than 2;

performing local attention mechanism processing on the nth decoding graph, and then performing upsampling to obtain an nth upsampling graph; wherein N is a positive integer less than N; decoding diagram 0 is determined according to the N-1 characteristic diagram; the N-1 th feature map is the last generated feature map in N feature maps with different sizes;

the N-N characteristic diagram and the N up-sampling diagram are fused based on a global attention mechanism to obtain an n+1 decoding diagram;

and obtaining a second image based on the N-1 decoding graph, wherein the image content of the second image is the same as that of the first image, and the image quality of the second image is higher than that of the first image.

2. The method of claim 1, wherein the multi-scale feature encoding of the first image results in N feature maps of different sizes, comprising:

convolving the first image to obtain a 0 th feature map;

and carrying out multi-scale feature extraction on the m feature map to obtain an m+1 feature map, wherein m is a positive integer less than or equal to N-1 or 0.

3. The method of claim 2, wherein the multi-scale feature extraction of the mth feature map to obtain the mth+1 feature map comprises:

convolving the mth feature map by using convolution cores of different scales to obtain a plurality of first class intermediate feature maps of the same size;

and fusing a plurality of the first class intermediate feature graphs to obtain the m+1th feature graph.

4. A method according to any one of claims 1 to 3, wherein upsampling the nth decoding map after processing based on the local attention mechanism to obtain an nth upsampled map comprises:

generating a first weight of the local attention mechanism based on the nth decoding graph;

weighting the nth decoding graph based on the first weight to obtain a second class intermediate feature graph;

and carrying out up-sampling processing on the second class intermediate feature map to obtain the nth up-sampling map.

5. A method according to any one of claims 1 to 3, wherein the fusing the N-N feature map and the N up-sample map based on the global attention mechanism to obtain an n+1 decoding map comprises:

generating a second weight of the global attention mechanism according to the nth decoding diagram;

Weighting the N-N feature map according to the second weight to obtain a third class of intermediate feature map;

and fusing the third class intermediate feature map and the nth sampling map to obtain the (n+1) th decoding map.

6. An image processing model training method, comprising:

training an image processing model performing the method of any of claims 1 to 5.

7. The method according to claim 6, wherein the training performs the image processing model of the method of any one of claims 1 to 5, comprising:

inputting a sample image into a first model to obtain a predicted image output by the first model;

inputting the predicted image into a second model to obtain M first feature images output by the second model; wherein M is a positive integer greater than or equal to 2;

inputting a target image corresponding to the sample image into the second model to obtain M second feature images output by the second model;

obtaining a first type of loss value according to the difference between the p-th first characteristic diagram and the p-th second characteristic diagram, wherein p is a positive integer smaller than or equal to M;

and adjusting model parameters of the first model based on the first class loss values.

8. The method of claim 7, wherein the training performs the image processing model of the method of any one of claims 1 to 5, further comprising:

obtaining a second class of loss values according to the difference between the predicted image and the target image;

the adjusting the model parameters of the first model based on the first class of loss values includes:

and adjusting model parameters of the first model according to the first type of loss values and the second type of loss values.

9. The method of claim 8, wherein the training performs the image processing model of the method of any one of claims 1 to 5, further comprising:

obtaining a third type of loss value according to the pixel difference between the S-th pixel and the adjacent pixel of the S-th pixel in the predicted image, wherein S is a positive integer less than or equal to S; the S is the total number of pixels contained in the predicted image;

the adjusting the model parameters of the first model according to the first class of loss values and the second class of loss values comprises:

obtaining an image loss value according to the first type loss value, the second type loss value and the third type loss value;

And adjusting model parameters of the first model based on the image loss value.

10. The method of claim 9, wherein obtaining the image loss value based on the first type of loss value, the second type of loss value, and the third type of loss value comprises:

the first class loss value, the weight of the first class loss value, the second class loss value, the weight of the second class loss value, the third class loss value and the weight of the third class loss value are weighted and summed to obtain the image loss value;

wherein the weight of the first class of loss values is less than the weight of the second class of loss values;

the third class of penalty values have a weight that is less than the weight of the first class of penalty values.

11. An image processing apparatus, characterized in that the apparatus comprises:

the coding module is used for carrying out multi-scale feature coding on the first image to obtain N feature images with different sizes; wherein N is a positive integer equal to or greater than 2;

the conversion module is used for carrying out local attention mechanism processing on the nth decoding graph and then carrying out up-sampling to obtain an nth up-sampling graph; wherein N is a positive integer less than N; the 0 th decoding diagram is determined according to the N-1 th characteristic diagram; the N-1 th feature map is the last generated feature map in N feature maps with different sizes;

The fusion module is used for carrying out fusion processing on the N-N characteristic diagram and the N up-sampling diagram based on a global attention mechanism to obtain an n+1 decoding diagram;

the obtaining module is used for obtaining a second image based on the N-1 decoding graph, wherein the image content of the second image is the same as the image content of the first image, and the image quality of the second image is higher than that of the first image.

12. The apparatus of claim 11, wherein the encoding module is specifically configured to convolve the first image to obtain a 0 th feature map; and carrying out multi-scale feature extraction on the m feature map to obtain an m+1 feature map, wherein m is a positive integer less than or equal to N-1 or 0.

13. The apparatus according to claim 12, wherein the encoding module is specifically configured to use convolution kernels of different scales to convolve the mth feature map to obtain a plurality of first class intermediate feature maps of the same size; and fusing a plurality of the first class intermediate feature graphs to obtain the m+1th feature graph.

14. The apparatus according to any one of claims 11 to 13, wherein the conversion module is configured to generate a first weight of the local attention mechanism, in particular based on the nth decoding graph; weighting the nth decoding graph based on the first weight to obtain a second class intermediate feature graph; and carrying out up-sampling processing on the second class intermediate feature map to obtain the nth up-sampling map.

15. The apparatus according to any one of claims 11 to 13, wherein the fusion module is configured to generate a second weight of the global attention mechanism, in particular according to the nth decoding graph; weighting the N-N feature map according to the second weight to obtain a third class of intermediate feature map; and fusing the third class intermediate feature map and the nth sampling map to obtain the (n+1) th decoding map.

16. An image processing model training apparatus, comprising:

training module for training an image processing model for performing the method of any of claims 1 to 5.

17. The apparatus of claim 16, wherein the training module comprises:

the prediction unit is used for inputting the sample image into the first model to obtain a predicted image output by the first model;

the first input unit is used for inputting the predicted image into the second model to obtain M first feature images output by the second model; wherein M is a positive integer greater than or equal to 2;

the second input unit is used for inputting the target image corresponding to the sample image into the second model to obtain M second feature images output by the second model;

The first loss unit is used for obtaining a first type of loss value according to the difference between the p-th first characteristic diagram and the p-th second characteristic diagram, wherein p is a positive integer smaller than or equal to M;

and the adjusting unit is used for adjusting the model parameters of the first model based on the first type loss value.

18. The apparatus of claim 17, wherein the training module further comprises:

the second loss unit is used for obtaining a second class loss value according to the difference between the predicted image and the target image;

the adjusting unit is specifically configured to adjust model parameters of the first model according to the first type of loss values and the second type of loss values.

19. The apparatus of claim 18, wherein the training module further comprises:

a third loss unit, configured to obtain a third type of loss value according to a pixel difference between an S-th pixel and a pixel adjacent to the S-th pixel in the predicted image, where S is a positive integer less than or equal to S; the S is the total number of pixels contained in the predicted image;

the adjusting unit is configured to obtain an image loss value according to the first type loss value, the second type loss value, and the third type loss value; and adjusting model parameters of the first model based on the image loss value.

20. The apparatus according to claim 19, wherein the fourth loss unit is configured to perform weighted summation on the first type of loss value, the weight of the first type of loss value, the second type of loss value, the weight of the second type of loss value, the third type of loss value, and the weight of the third type of loss value to obtain the image loss value;

wherein the weight of the first class of loss values is less than the weight of the second class of loss values;

the third class of penalty values have a weight that is less than the weight of the first class of penalty values.

21. An electronic device, comprising:

a memory for storing processor-executable instructions;

a processor connected to the memory;

wherein the processor is configured to perform the method as provided in any one of claims 1 to 5 or 6 to 10.

22. A non-transitory computer readable storage medium, which when executed by a processor of a computer, enables the computer to perform the method provided in any one of claims 1 to 5 or 6 to 10.