CN118365523A - Method, system, electronic device and storage medium for representing images of any scale - Google Patents

Abstract

The application discloses a method, a system, electronic equipment and a storage medium for representing an image with any scale, which comprise the following steps: acquiring a low-resolution image, a multiple and modulation lookup table and super-resolution multiple parameters; encoding the low-resolution image by an encoder to obtain a low-resolution high-dimensional feature map; and inputting the low-resolution high-dimensional feature map into an implicit multi-layer perceptron to obtain hidden modulation and compression hidden codes of each feature vector. And according to the super-resolution multiple parameters, the compressed hidden codes, the hidden modulation and multiple and modulation lookup tables, performing rendering treatment on the low-resolution image by adopting a target rendering multi-layer perceptron to obtain the super-resolution image. The embodiment of the application decouples the decoding process from a high-resolution high-dimensional space to a low-resolution high-dimensional hidden space and a high-resolution low-dimensional rendering space based on hidden modulation, combines a target rendering multi-layer perceptron to perform rendering processing, has the advantages of low calculation cost and high image processing speed, and can be widely applied to the technical field of image processing.

Description

Method, system, electronic device and storage medium for representing images of any scale

Technical Field

The present application relates to the field of image processing technology, and in particular to a method, system, electronic device and storage medium for representing images of any scale.

Background technique

Implicit neural representations (INR) can be used for signal reconstruction, image and video compression, and image generation. INR is based on neural radiance fields (NeRFs) and learns a multi-layer perceptron (MLP) to infer the pixel value (RGB) and density of any given coordinates and viewing angles in a 3D scene. However, a major challenge facing INR is the need for an extensive training process to fit the signal from scratch. To address this issue, some people use multiple small MLPs to independently learn different areas of the signal. Another solution is the hypernetwork-base network design. During testing, the pre-trained hypernetwork can directly generate INR for any input signal. Another challenge facing INR is that the MLP needs to predict each pixel independently during rendering, which will cost a lot of computation and require a long running time.

Arbitrary-scale image super-resolution methods aim to use a single network to reconstruct resolution at any magnification, which is practical and convenient. Based on INR, a local implicit image function (LIIF) is proposed, which uses MLP to receive coordinates and adjacent two-dimensional feature inputs to infer the pixel value at the coordinate. Although arbitrary-scale super-resolution methods based on INR can provide stable performance in super-resolution up to 30 times, they follow the paradigm of decoding in high-resolution-high-bit space, resulting in high computational cost and long running time with quadratic increase in magnification, and their computational cost also increases rapidly with the increase in scaling.

Summary of the invention

The main purpose of the embodiments of the present application is to propose a method, system, electronic device and storage medium for representing images of any scale, aiming to reduce the computational cost of representing images of any scale and improve the image processing speed.

To achieve the above purpose, an embodiment of the present application provides a method for representing an image of any scale, the method comprising:

Obtain low-resolution images, magnification and modulation lookup tables, and super-resolution magnification parameters;

The low-resolution image is encoded by an encoder to obtain a low-resolution high-dimensional feature map;

Inputting the low-resolution high-dimensional feature map into an implicit multi-layer perceptron to obtain an implicit modulation and compression code for each feature vector;

According to the super-resolution multiple parameter, the compression hidden code, the hidden modulation and the multiple and modulation lookup table, a target rendering multi-layer perceptron is used to render the low-resolution image to obtain a super-resolution image.

In some embodiments, in the step of acquiring the low-resolution image, the magnification and modulation lookup table and the super-resolution magnification parameter, the step of acquiring the magnification and modulation lookup table includes:

Sampling common magnifications to obtain a minimum modulation mean value and a maximum modulation mean value of the magnifications;

A multiplier and modulation lookup table is generated according to the minimum modulation mean value and the maximum modulation mean value.

In some embodiments, the step of inputting the low-resolution high-dimensional feature map into an implicit multi-layer perceptron to obtain an implicit modulation and compression code of each feature vector includes:

Performing feature vector extraction processing on the feature map to obtain a high-dimensional hidden code;

Performing perceptual processing on the high-dimensional latent code through an implicit multi-layer perceptron to obtain a first processing result;

Performing channel-wise segmentation processing on the first processing result to obtain a compressed hidden code;

Hidden modulation is generated according to the high-dimensional hidden code.

In some embodiments, the implicit modulation is composed of several characteristic linear modulation layers; wherein each characteristic linear modulation layer includes a scaling modulation and an offset modulation.

In some embodiments, the method of rendering the low-resolution image using a target rendering multi-layer perceptron according to the super-resolution multiple parameter, the compression hidden code, the hidden modulation, and the multiple and modulation lookup table to obtain a super-resolution image includes:

Initializing a first rendering multiple and a first rendering image, and calculating an offset modulation mean value of each of the compression hidden codes;

Update the first rendering multiple according to the multiple and the next multiple in the modulation lookup table;

Using bilinear interpolation to sample the first rendered image to a resolution corresponding to the first rendering multiple;

Querying the multiple and modulation lookup table to determine the modulation mean range of the first rendering multiple;

Inputting the compressed hidden code whose offset modulation mean value falls within the modulation mean value range into a target rendering multi-layer perceptron for rendering, to obtain a rendering pixel value of the first rendering multiple;

Return to execute the step of updating the first rendering multiple according to the multiple and the next multiple of the modulation lookup table, until all the compression codes have been decoded or the first rendering multiple is the final required super-resolution multiple, to obtain the target super-resolution image.

In some embodiments, the method further comprises:

The implicit modulation is used to modulate the initial rendering multi-layer perceptron to obtain a target rendering multi-layer perceptron.

In some embodiments, the target rendering multi-layer perceptron performs rendering at each layer as follows:

in, represents the hidden features of the kth layer; represents the hidden features of the k+1th layer; ⊙ represents the channel direction dot product; σ represents the activation function; Represents the rendering process of the k+1th layer of the target rendering multi-layer perceptron.

To achieve the above purpose, another aspect of the present application provides a representation system for an image of any scale, the system comprising:

The first module is used to obtain low-resolution images, magnification and modulation lookup tables, and super-resolution magnification parameters;

The second module is used to encode the low-resolution image through an encoder to obtain a low-resolution high-dimensional feature map;

The third module is used to input the low-resolution high-dimensional feature map into an implicit multi-layer perceptron to obtain an implicit modulation and compression code of each feature vector;

The fourth module is used to render the low-resolution image using a target rendering multi-layer perceptron according to the super-resolution multiple parameter, the compression hidden code, the hidden modulation and the multiple and modulation lookup table to obtain a super-resolution image.

To achieve the above objective, another aspect of an embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method when executing the computer program.

To achieve the above objective, another aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described above is implemented.

The embodiments of the present application include at least the following beneficial effects: by inputting a low-resolution high-dimensional feature map into an implicit multi-layer perceptron, the implicit modulation and compression hidden code of each feature vector are obtained, and the decoding process can be decoupled from the high-resolution high-dimensional space to the low-resolution high-dimensional hidden space and the high-resolution low-dimensional rendering space, which is beneficial to reducing the computational cost. In combination with super-resolution multiple parameters, compression hidden codes, hidden modulation, and multiple and modulation lookup tables, a target rendering multi-layer perceptron is used to render low-resolution images, which can achieve rendering with different accuracy and speed, while naturally adapting to the image content. The embodiments of the present application have the advantages of low computational cost and fast image processing speed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to provide further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

FIG1 is a step diagram of a method for representing an image of any scale provided in an embodiment of the present application;

FIG2 is an overall framework diagram of an arbitrary scale image representation method provided by an embodiment of the present application;

FIG3 is a data processing flow chart of a method for representing an arbitrary primary image provided in an embodiment of the present application;

FIG4 is a schematic diagram of a normalized mean value of offset modulation under 4 super-resolutions provided in an embodiment of the present application;

FIG5 is a schematic diagram of the normalized residual between the LM-LIIF predicted image and the bilinear upsampled image provided in an embodiment of the present application;

FIG6 is a schematic diagram of the structure of a system for representing images of any scale provided in an embodiment of the present application;

FIG. 7 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

Although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first/S100", "second/S200", etc. in the specification, claims and the above drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

In the related art, the research on arbitrary-scale image representation based on implicit neural representation (INR) uses a multi-layer perceptron (MLP) to decode low-resolution features for arbitrary-scale super-resolution reconstruction. However, these continuous image representations are usually decoded in a high-resolution-high-dimensional space, which causes the computational cost to increase quadratically with the increase in the image magnification, which is not conducive to the practical application of arbitrary-scale super-resolution.

In view of this, the present application provides a method, system, electronic device and storage medium for representing images of any scale. The scheme proposes a latent modulation function (LMF). By combining the latent modulation function with the LMF for representing images of any scale, the high-resolution-high-dimensional space decoding process can be decoupled into a shared latent decoding process in a low-resolution-high-dimensional space and an independent rendering process in a high-resolution-low-dimensional space, thereby realizing the first computationally optimal paradigm for continuous image representation. Furthermore, the present application proposes to use a high-dimensional MLP in the latent space to generate a corresponding latent space modulation for each low-resolution feature vector, which enables the low-dimensional MLP with multiple modulation in the rendering space to quickly adapt to any input feature vector and render at any resolution. The present application further proposes a controllable multi-scale rendering method (CMSR) using the positive correlation between the modulation intensity and the complexity of the input image. By combining this method for image rendering, flexible decoding efficiency adjustment can be provided based on rendering accuracy.

The representation method of an image of any scale provided in the embodiment of the present application relates to the field of image processing technology. The representation method of an image of any scale provided in the embodiment of the present application can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or it can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured to provide cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms and other basic cloud computing services. The cloud server, the server can also be a node server in a blockchain network; the software can be an application that implements the representation method of an image of any scale, etc., but is not limited to the above forms.

The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

FIG. 1 is an optional step diagram of a method for representing an image of any scale provided in an embodiment of the present application. The method in FIG. 1 may include but is not limited to steps S100 to S400 .

Step S100, obtaining a low-resolution image, a magnification and modulation lookup table, and a super-resolution magnification parameter.

Step S200: encoding the low-resolution image through an encoder to obtain a low-resolution high-dimensional feature map.

Step S300, inputting the low-resolution high-dimensional feature map into an implicit multi-layer perceptron to obtain an implicit modulation and compression code for each feature vector.

Step S400, according to the super-resolution multiple parameter, the compression hidden code, the hidden modulation and the multiple and modulation lookup table, a target rendering multi-layer perceptron is used to render the low-resolution image to obtain a super-resolution image.

In steps S100 to S400 shown in the embodiment of the present application, by compressing the high-dimensional hidden code obtained from the low-resolution high-dimensional feature map to obtain a low-dimensional compressed hidden code, and calculating the hidden modulation, the low-dimensional rendering multi-layer perceptron can quickly adapt to any input hidden code and achieve efficient rendering at any resolution.

In step S100 of some embodiments, the low-resolution image and the super-resolution multiple parameter may be obtained by input, and the step of obtaining the multiple and modulation lookup table may include but is not limited to steps S110 to S120:

Step S110, sampling common amplification factors to obtain the minimum modulation mean value and the maximum modulation mean value of the amplification factors;

Step S120: Generate a multiple and modulation lookup table according to the minimum modulation mean value and the maximum modulation mean value.

In step S200 of some embodiments, the encoder may be any feature extractor in existing super-resolution methods, and the essence of continuous image representation is a decoding function that can arbitrarily upsample low-resolution high-dimensional features to high-resolution low-dimensional pixel values.

In some embodiments, step S300 may include but is not limited to the following steps S310 to S340:

Step S310, extracting feature vectors from the feature graph to obtain a high-dimensional hidden code;

Step S320, performing perception processing on the high-dimensional latent code by using an implicit multi-layer perceptron to obtain a first processing result;

Step S330, performing channel direction segmentation processing on the first processing result to obtain a compressed hidden code;

Step S340: generating hidden modulation according to the high-dimensional hidden code.

In some embodiments, the implicit modulation is composed of several characteristic linear modulation layers; wherein each characteristic linear modulation layer includes a scaling modulation and an offset modulation.

In some embodiments, step S400 includes but is not limited to the following steps S410 to S460:

Step S410, initializing a first rendering multiple and a first rendering image, and calculating an offset modulation mean value of each of the compression hidden codes;

Step S420, updating the first rendering multiple according to the multiple and the next multiple in the modulation lookup table;

Step S430, sampling the first rendered image to a resolution corresponding to the first rendering multiple by using bilinear interpolation;

Step S440, querying the multiple and modulation lookup table to determine the modulation mean range of the first rendering multiple;

Step S450, inputting the compressed hidden code whose offset modulation mean value falls within the modulation mean value range into a target rendering multi-layer perceptron for rendering, to obtain a rendering pixel value of the first rendering multiple;

Step S460, returns to execute the step of updating the first rendering multiple according to the multiple and the next multiple of the modulation lookup table, until all the compressed hidden codes have been decoded or the first rendering multiple is the final required super-resolution multiple, to obtain the target super-resolution image.

In some embodiments, the method for representing an image of any scale further includes the following steps S500:

Step S500, using the implicit modulation to modulate the initial rendering multi-layer perceptron to obtain a target rendering multi-layer perceptron.

In some embodiments, the target rendering multi-layer perceptron performs rendering at each layer as follows:

in, represents the hidden features of the kth layer; represents the hidden features of the k+1th layer; ⊙ represents the channel direction dot product; σ represents the activation function; Represents the rendering process of the k+1th layer of the target rendering multi-layer perceptron.

Below, the solution of the embodiment of the present application is described in detail and explained in conjunction with an application example of a specific image rendering representation scene:

In an embodiment of the present application, a representation method for an image of any scale is provided, which can be applied to rendering an image of any scale at any resolution. In the continuous image representation, the input image is represented as a feature map in a low-resolution high-dimensional latent space. A hidden code represents a feature vector extracted from the feature map, which is used to decode all coordinates closest to it in the continuous space. Since the encoder can be any feature extractor in the existing super-resolution method, the essence of the continuous image representation is a decoding function that can arbitrarily upsample low-resolution high-dimensional features to high-resolution low-dimensional pixel values. A simple and intuitive paradigm is to learn a decoding function parameterized as an MLP in a high-resolution high-dimensional space, and independently decode the pixel value of each high-resolution coordinate. However, this basic paradigm has a lot of computational redundancy. Based on this, an embodiment of the present application proposes a representation method for an image of any scale, which decomposes the high-resolution high-dimensional decoding process into shared decoding in the latent space and independent rendering in the rendering space through an implicit modulation function, and can also perform controllable multi-scale rendering adapted to the content of the input image, and finally obtain an image of any scale.

As shown in Figure 2, Figure 2 illustrates the framework structure of the arbitrary scale image representation method of the embodiment of the present application. The encoder maps the input image to a feature map of the latent space as its implicit representation, and then a high-dimensional MLP generates hidden modulations for these hidden codes in the latent space. Each hidden modulation will be used as part of a low-dimensional MLP parameter, greatly improving the accuracy of the low-dimensional MLP in rendering high-resolution pixel values in the corresponding area. Given the resolution of the output image, the embodiment of the present application further uses the proposed controllable multi-scale rendering method, and uses the mean of the hidden modulation to calculate the actual rendering multiple required for each hidden code under a given rendering accuracy, thereby reducing the computational cost to the actual level required by the content of the input image, and realizing flexible adjustment of computational efficiency and rendering accuracy. In the training stage, the pixel-level loss can be calculated using the predicted pixel values and the pixel values of the real image for optimization. The encoder and decoding functions are jointly trained in the self-supervised super-resolution task, and the learned network parameters are shared for all images.

The implicit modulation function provided by the embodiment of the present application is introduced below:

In order to eliminate computational redundancy in high-resolution high-dimensional decoding, a two-stage decoding process consisting of an implicit MLP (θ _l ) and a rendering MLP (θ _r ) is first provided. The implicit MLP handles the shared decoding process inside the high-resolution region corresponding to each hidden code z ^* , while the rendering MLP independently predicts each high-resolution coordinate The low-resolution, high-dimensional output from the implicit MLP is used to assist the rendering MLP in rendering, and its rendering expression is:

In formula (1), Representing coordinates The rendered pixel value, represents implicit MLP processing, Represents the rendering MLP process.

However, if the output of the implicit MLP is directly used as the input of the rendering MLP, it is similar to directly using the hidden code itself as input, and it will not bring about an improvement in computational efficiency. In addition, in order to accurately render pixels of arbitrary resolution, the rendering MLP needs to be calculated in a high-resolution space. Therefore, the most feasible way to minimize the amount of rendering calculations is to reduce the dimension or depth of the rendering MLP. However, directly inputting the output of the implicit MLP into the rendering MLP means that a high-dimensional rendering MLP is required to fully receive and process the input, but this application uses a low-dimensional rendering MLP to reduce complexity.

To overcome these limitations, the embodiments of the present application use modulation to connect the latent space and the rendering space. The current modulation is always shared by the input image and works at the image level, resulting in poor performance and difficult training. Therefore, the present application proposes an implicit modulation m ^* generated by an implicit MLP using a hidden code z ^* . The implicit modulation consists of K FiLM layers (feature linear modulation layers), each of which includes a scaling modulation and an offset modulation. The expression of the implicit modulation is:

In formula (2), represents the scaling modulation of the first layer, represents the offset modulation of the first layer, represents the scaling modulation of the last layer, represents the offset modulation of the last layer.

The latent modulation m ^* modulates the rendering MLP only when rendering the high-resolution coordinates closest to its latent code z ^* , and is applied to the hidden features after each hidden linear layer:

In formula (3), represents the hidden features of the kth layer, represents the hidden features of the k+1th layer, ⊙ represents the channel direction dot product, σ represents the activation function, Represents the process of rendering the k+1th layer of the MLP.

Importantly, implicit modulation effectively resolves the incompatibility between low-resolution high-dimensional hidden codes and high-resolution low-dimensional rendering, and significantly decouples computation from parameters. For example, for a potential modulation with 192 dimensions, an embodiment of the present application can accurately render an image using a 7-layer rendering MLP with only 16 dimensions. This extremely small rendering MLP only learns general rendering rules for natural images, while the implicit modulation is responsible for optimizing the rendering process of the rendering MLP based on the characteristics of each input hidden code. Therefore, implicit modulation enables the low-dimensional rendering MLP to quickly adapt to any input hidden code and achieve efficient rendering at arbitrary resolution. However, a remaining problem is that the hidden codes in existing INR-based arbitrary-scale super-resolution methods are typically high-dimensional, which can result in unnecessary computation when directly input into the rendering MLP. Therefore, the present application further uses an implicit MLP to compress the high-dimensional hidden code z ^* into a low-dimensional hidden code In general, based on implicit modulation, this application proposes an implicit modulation function for continuous image representation:

In formula (4), Split represents channel direction segmentation. Represents the rendered MLP modulated by the implicit modulation m ^* .

The LMF proposed in this application is a general method for efficient arbitrary-scale super-resolution, which is fully compatible with existing INR-based methods. By easily transferring time-consuming operations and large MLPs to the latent space, LMF allows a minimal rendering MLP to perform the same level of super-resolution as the original arbitrary-scale super-resolution method. In addition, LMF also effectively decouples computationally expensive techniques such as feature expansion and local integration, which has positive beneficial effects.

The processing flow of the implicit modulation function of this application is as follows:

1. Input: feature map of the low-resolution image after being encoded by the encoder, and super-resolution multiple parameters.

2. Hidden decoding: Input the low-resolution feature map into the implicit MLP to obtain the implicit modulation and compression hidden code of each feature vector.

3. Generate high-resolution coordinates: Generate a high-resolution coordinate grid based on the super-resolution multiples.

4. Rendering the image: Each high-resolution coordinate and its compressed hidden code are input into the rendering MLP modulated by hidden modulation in turn to calculate the pixel value of the coordinate.

5. Output: super-resolution image.

The controllable multi-scale rendering method of the embodiment of the present application is introduced below:

The high-resolution regions of the two hidden codes typically have different signal complexities, allowing different decoding functions to be used. However, the challenge lies in obtaining the signal complexity of each hidden code. The modulation strategy in formula (3) shows that the strength of the hidden modulation is proportional to the signal complexity of its hidden code. A lower intensity means that the signal can be adequately rendered by the rendering MLP itself. Figure 4 shows the normalized mean of the offset modulation. Figure 5 shows the normalized residual of the super-resolution result of the LMF-based LIIF and the bilinear upsampling result. It is obvious that the mean of the modulation is closely related to the signal complexity of the predicted image. Regions with higher modulation means always correspond to more complex textures and edges. Based on this finding, the embodiment of the present application uses the mean of the offset modulation to indicate the strength of the hidden modulation.

In LMF, the signal complexity of a hidden code z ^* is defined as the minimum magnification s ^* required to fully represent its signal. In simple terms, any rendering with a magnification s, s>s ^* will produce the same signal complexity as rendering with a magnification of s ^* and then using Therefore, when upsampling the image by a factor of s, it is only necessary to render the hidden z ^* by min(s,s ^* ) times.

In order to obtain the minimum rendering multiple corresponding to each modulation mean, the embodiment of the present application creates a multiple-modulation lookup table by sampling common magnifications and recording their minimum and maximum modulation means. Based on the multiple-modulation lookup table, the present application proposes a controllable multi-scale rendering (CMSR) method to decode hidden codes with minimum rendering multiples. For each sampled multiple s _i ∈ [s ₁ ,s ₂ ,…,s], first use bilinear interpolation to upsample the result of s _i-1 to a resolution corresponding to s _i ; then query the multiple-modulation lookup table to obtain the modulation mean range of s _i For s _i , the rendering MLP only renders the modulation mean falling within The above process is repeated until the final super-resolution result is obtained or all hidden codes have been decoded. For controllable precision rendering, when creating the multiplication-modulation lookup table, an MSE threshold t is set and the signal complexity of the hidden code is defined as the minimum magnification required to render its signal within this error t. By easily adjusting the MSE threshold during testing, CMSR can render with different accuracy and speed while naturally adapting to the content of the image.

The processing flow of controllable multi-scale rendering of this application is as follows:

1. Input: low-resolution image, compression code, hidden modulation, multiplier-modulation lookup table, and super-resolution multiplier parameters.

2. Initialization: The initial rendering factor is 1, and the initial rendering image is a low-resolution image. Calculate the offset modulation mean of each implicit modulation.

3. Update rendering multiple: The rendering multiple is updated to multiple - the next multiple of the modulation lookup table.

4. Interpolation upsampling: For the current rendering multiple, first use bilinear interpolation to upsample the image of the previous multiple to the resolution corresponding to the current multiple.

5. Lookup table: Query the multiple-modulation lookup table to obtain its modulation mean range.

6. Rendering: For the current multiple, the hidden code whose modulation mean falls within the query range is input into the rendering MLP for rendering to obtain the rendered pixel value of the current multiple.

7. Iteration: Repeat the process of (2)-(5) until the final super-resolution result is obtained or all hidden codes have been decoded.

8. Output: super-resolution image.

Based on the above implicit modulation function and controllable multi-scale rendering, referring to FIG. 5 , the processing flow of a method for representing an image of any scale in an embodiment of the present application is as follows:

1. Input: low-resolution image, multiplication-modulation lookup table, and super-resolution multiplication parameters.

2. Encoding: Input the low-resolution image into the encoder to obtain a low-resolution high-dimensional feature map.

3. Hidden decoding: Input the low-resolution high-dimensional feature map into the implicit MLP to obtain the implicit modulation and compression hidden code of each feature vector.

4. Controllable multi-scale rendering:

(1) Initialization: The initial rendering factor is 1, and the initial rendered image is a low-resolution image. Calculate the offset modulation mean of each implicit modulation.

(2) Update the rendering multiple: The rendering multiple is updated to multiple - the next multiple in the modulation lookup table.

(3) Interpolation upsampling: For the current rendering multiple, bilinear interpolation is first used to upsample the image of the previous multiple to the resolution corresponding to the current multiple.

(4) Lookup: Look up the multiplier-modulation lookup table to obtain its modulation mean range.

(5) Rendering: For the current multiple, the hidden code whose modulation mean falls within the query range is input into the rendering MLP for rendering to obtain the rendered pixel value of the current multiple.

(6) Iteration: Repeat the process of (3)-(6) until the final super-resolution result is obtained (that is, the current rendering magnification is the magnification required by the final super-resolution result) or all hidden codes have been decoded.

5. Output: super-resolution image.

In summary, the embodiments of the present application have at least the following beneficial effects:

The present application proposes a novel arbitrary-scale image representation method and system based on implicit modulation function to achieve optimal calculation of continuous image representation. By utilizing implicit modulation, LMF successfully decouples the decoding process from high-resolution high-dimensional space to latent space (low-resolution high-dimensional) and rendering space (high-resolution low-dimensional). LMF first uses implicit MLP to generate implicit modulation for a given hidden code. When querying coordinates near the hidden code, implicit modulation is applied to each hidden linear layer in the rendering MLP. This modulation enables LMF to minimize the dimensions and parameters of the rendering MLP without sacrificing performance. In addition, the present application provides a controllable multi-scale rendering algorithm based on the positive correlation between the implicit modulation intensity and the complexity of the input image. CMSR not only allows the computational cost of rendering to be proportional to the complexity of the input image rather than the output resolution, but also provides the flexibility to balance accuracy and efficiency during testing. Combined with implicit modulation functions and controllable multi-scale rendering, the present application can calculate the optimal and fast generation of arbitrary-scale image super-resolution representations.

Experiments show that converting existing INR-based arbitrary-scale super-resolution methods to LMF-based methods can reduce the computational cost by 90.4% to 99.9%, increase the inference speed by 2.4 times to 56.9 times, and save 45.1% to 76.0% of parameters, while maintaining competitive PSNR performance.

Referring to FIG. 6 , the embodiment of the present application further provides a system 100 for representing an image of any scale, which can implement the above-mentioned method for representing an image of any scale. The system includes:

The first module 101 is used to obtain a low-resolution image, a magnification and modulation lookup table, and a super-resolution magnification parameter;

The second module 102 is used to encode the low-resolution image through an encoder to obtain a low-resolution high-dimensional feature map;

The third module 103 is used to input the low-resolution high-dimensional feature map into an implicit multi-layer perceptron to obtain an implicit modulation and compression code of each feature vector;

The fourth module 104 is used to render the low-resolution image using a target rendering multi-layer perceptron according to the super-resolution multiple parameter, the compression hidden code, the hidden modulation and the multiple and modulation lookup table to obtain a super-resolution image.

It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned method for representing an image of any scale when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

Please refer to FIG. 7 , which schematically shows the hardware structure of an electronic device according to another embodiment. The electronic device includes:

The processor 201 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

The memory 202 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 202 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 202, and the processor 201 calls and executes the method for representing an image of any scale in the embodiment of this application;

Input/output interface 203, used to implement information input and output;

The communication interface 204 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

Bus 205 , which transmits information between various components of the device (e.g., processor 201 , memory 202 , input/output interface 203 , and communication interface 204 );

The processor 201 , the memory 202 , the input/output interface 203 and the communication interface 204 are connected to each other in communication within the device via the bus 205 .

An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned method for representing an image of any scale is implemented.

It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules/units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And/or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and/or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character "/" generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims (10)

1. A method for representing an image of any scale, characterized by comprising: Obtain low-resolution images, magnification and modulation lookup tables, and super-resolution magnification parameters; The low-resolution image is encoded by an encoder to obtain a low-resolution high-dimensional feature map; The low-resolution high-dimensional feature map is input into an implicit multi-layer perceptron to obtain an implicit modulation and compression code for each feature vector. According to the super-resolution multiple parameter, the compression hidden code, the hidden modulation and the multiple and modulation lookup table, a target rendering multi-layer perceptron is used to render the low-resolution image to obtain a super-resolution image. 2. The method according to claim 1, characterized in that in the step of obtaining the low-resolution image, the multiple and modulation lookup table and the super-resolution multiple parameter, the step of obtaining the multiple and modulation lookup table comprises: Sampling common magnifications to obtain a minimum modulation mean value and a maximum modulation mean value of the magnifications; A multiplier and modulation lookup table is generated according to the minimum modulation mean value and the maximum modulation mean value. 3. The method according to claim 1, characterized in that the step of inputting the low-resolution high-dimensional feature map into an implicit multi-layer perceptron to obtain an implicit modulation and compression code of each feature vector comprises: Performing feature vector extraction processing on the feature map to obtain a high-dimensional hidden code; Performing perceptual processing on the high-dimensional latent code through an implicit multi-layer perceptron to obtain a first processing result; Performing channel-wise segmentation processing on the first processing result to obtain a compressed hidden code; Hidden modulation is generated according to the high-dimensional hidden code. 4. The method according to claim 1 or 3 is characterized in that the implicit modulation is composed of several characteristic linear modulation layers; wherein each layer of the characteristic linear modulation layer includes a scaling modulation and an offset modulation. 5. The method according to claim 1, characterized in that the step of rendering the low-resolution image using a target rendering multi-layer perceptron according to the super-resolution multiple parameter, the compression hidden code, the hidden modulation and the multiple and modulation lookup table to obtain a super-resolution image comprises: Initializing a first rendering multiple and a first rendering image, and calculating an offset modulation mean value of each of the compression hidden codes; Update the first rendering multiple according to the multiple and the next multiple in the modulation lookup table; Using bilinear interpolation to sample the first rendered image to a resolution corresponding to the first rendering multiple; Querying the multiple and modulation lookup table to determine the modulation mean range of the first rendering multiple; Inputting the compressed hidden code whose offset modulation mean value falls within the modulation mean value range into a target rendering multi-layer perceptron for rendering, to obtain a rendering pixel value of the first rendering multiple; Return to execute the step of updating the first rendering multiple according to the multiple and the next multiple of the modulation lookup table, until all the compression codes have been decoded or the first rendering multiple is the final required super-resolution multiple, to obtain the target super-resolution image. 6. The method according to claim 1, characterized in that the method further comprises: The implicit modulation is used to modulate the initial rendering multi-layer perceptron to obtain a target rendering multi-layer perceptron. 7. The method according to claim 1, characterized in that the expression for rendering by the target rendering multi-layer perceptron at each layer is: in, represents the hidden features of the kth layer; represents the hidden features of the k+1th layer; ⊙ represents the channel direction dot product; σ represents the activation function; Represents the rendering process of the k+1th layer of the target rendering multi-layer perceptron. 8. A system for representing images of any scale, characterized by comprising: The first module is used to obtain low-resolution images, magnification and modulation lookup tables, and super-resolution magnification parameters; The second module is used to encode the low-resolution image through an encoder to obtain a low-resolution high-dimensional feature map; The third module is used to input the low-resolution high-dimensional feature map into an implicit multi-layer perceptron to obtain an implicit modulation and compression code of each feature vector; The fourth module is used to render the low-resolution image using a target rendering multi-layer perceptron according to the super-resolution multiple parameter, the compression hidden code, the hidden modulation and the multiple and modulation lookup table to obtain a super-resolution image. 9. An electronic device, comprising a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7. 10. A computer storage medium storing a program executable by a processor, wherein the program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.