HK40092635A - Image processing method, apparatus, device, and computer-readable storage medium - Google Patents

Description

Image processing methods, apparatus, devices and computer-readable storage media

Technical Field

This application relates to the field of computer technology, and more specifically to an image processing method, apparatus, device, and computer-readable storage medium.

Background Technology

Artificial intelligence (AI) has been applied to a wide range of fields, and the technologies involved include computer vision, speech processing, and natural language processing. Among these, computer vision technology has profound significance in the application of image processing. For example, computer vision technology can be used to complete different types of image processing tasks.

To accomplish image adjustment tasks, related techniques are generally based on existing diffusion models, combined with descriptive information about image adjustment, to adjust the target image and obtain the adjusted image.

In the process of researching and practicing related technologies, the inventors of this application discovered that when performing image adjustment, the related technologies adaptively adjust the descriptive information while keeping the network parameters of the existing diffusion model unchanged. This adjustment of the target image based on the adjusted descriptive information easily overlooks many image details, reduces the accuracy of image adjustment, and makes the adjusted image effect inconsistent with the actual needs, which is not conducive to the development of subsequent business.

Summary of the Invention

This application provides an image processing method, apparatus, device, and computer-readable storage medium, which can solve the problem that the adjusted image effect does not match the actual needs and improve the accuracy of image adjustment.

This application provides an image processing method, including:

Obtain a reference image, and then obtain similar images that are similar to the reference image;

Determine the difference information between the reference image and the similar image;

Determine the target mask image in the reference image for the difference information;

Based on the aforementioned difference information, the difference description text is obtained by expanding upon it.

Based on the target mask image, the difference description text, and the reference image, the similar image is locally adjusted to obtain the adjusted target image.

Accordingly, embodiments of this application provide an image processing apparatus, including:

An acquisition unit is used to acquire a reference image and acquire similar images that are similar to the reference image;

The first determining unit is used to determine the difference information between the reference image and the similar image;

The second determining unit is used to determine the target mask image in the reference image for the difference information;

An expansion unit is used to expand the information based on the differences to obtain a difference description text.

The adjustment unit is used to perform local adjustments on the similar image based on the target mask image, the difference description text, and the reference image to obtain the adjusted target image.

In some embodiments, the adjustment unit is further configured to:

The similar images are subjected to noise processing, and a first similar noise map and a second similar noise map at adjacent time steps are obtained in the noise processing.

Based on the target mask image and the difference description text, the first similarity noise image is denoised to obtain the first image;

Based on the target mask image and the reference image, the second similarity noise image is denoised to obtain the second image;

The first image and the second image are fused together to obtain the adjusted target image.

In some embodiments, the adjustment unit is further configured to:

The first similar noise map is masked according to the target mask map to obtain the first mask noise map;

Obtain the difference text vector corresponding to the difference description text, and perform noise reduction processing on the first similarity noise map based on the difference text vector to obtain the first feature map;

The first feature map is decoded to obtain the first image.

In some embodiments, the adjustment unit is further configured to:

Invert the target mask image to obtain the target inverse mask image corresponding to the target mask image;

The second similar noise map is masked according to the target inverse mask map to obtain the second masked noise map;

The second mask noise map is masked according to the feature map corresponding to the reference image to obtain the second feature map;

The second feature map is decoded to obtain the second image.

In some embodiments, the adjustment unit is further configured to:

Based on the target mask image and the difference description text, the similar images are locally fine-tuned to obtain the first image;

Based on the target mask image and the reference image, the similar image is locally fine-tuned to obtain the second image;

The first image and the second image are fused together to obtain the adjusted target image.

In some embodiments, the adjustment unit is further configured to:

The similar image is masked according to the target mask image to obtain a first similar mask image;

The first similarity mask image is subjected to noise processing to obtain a first similarity noise image;

Obtain the difference text vector corresponding to the difference description text;

The first similarity noise map is denoised based on the difference text vector to obtain the first feature map;

The first feature map is decoded to obtain the first image.

In some embodiments, the adjustment unit is further configured to:

The similar image is masked according to the target mask image to obtain a first similar mask image;

The first similarity mask image is subjected to noise processing to obtain a first similarity noise image;

Obtain the difference text vector corresponding to the difference description text;

The first similarity noise map is denoised based on the difference text vector to obtain the first feature map;

The first feature map is decoded to obtain the first image.

In some embodiments, the adjustment unit is further configured to:

The first similar noise map is subjected to multiple consecutive noise reduction processes, and the difference text vector is incorporated into each noise reduction process through an attention mechanism to obtain the first feature map.

In some embodiments, the adjustment unit is further configured to:

The first similar mask image is encoded to obtain an encoded feature map;

The encoded feature map is subjected to noise processing to obtain a first similar noise map.

In some embodiments, the adjustment unit is further configured to:

The first image is generated by locally fine-tuning the similar image based on the target mask image and the difference description text using a first neural network model.

The image processing device further includes a training unit, used for:

Obtain the sample reference image and sample similar images, as well as the first sample target image;

Based on the difference information between the sample reference image and the sample similar image, a sample target mask image and sample difference description text are generated for the sample reference image;

Based on the sample target mask image and the sample difference description text, the sample similar image is locally fine-tuned using a preset model to generate a first predicted image.

The prediction loss is determined based on the first sample target image and the first prediction image;

The preset model is iteratively trained based on the prediction loss until the preset conditions are met, thus obtaining the first neural network model.

In some embodiments, the adjustment unit is further configured to:

Invert the target mask image to obtain the target inverse mask image corresponding to the target mask image;

The similar image is masked according to the target inverse mask image to obtain a second similar mask image;

The second similar mask image is subjected to noise processing to obtain a second similar noise map, wherein the time step of the second similar noise map is adjacent to the time step of the first similar noise map.

The second similarity noise map is denoised based on the feature map corresponding to the reference image to obtain the second feature map;

The second feature map is decoded to obtain the second image.

In some embodiments, the expansion unit is further configured to:

Based on the difference information, determine the difference objects between the reference image and the similar image;

Determine the object relationship information between the differing object and other objects in the reference image;

Obtain the global description text of the reference image and the target object description text of the difference object;

Based on the global description text, the target object description text, and the object relationship information, the text is expanded to obtain the difference description text.

In some embodiments, the acquiring unit is further configured to:

Determine the reference cluster center to which the reference image belongs;

Determine the feature category distance between each pre-constructed image cluster center in the preset database and the reference cluster center;

Based on the feature category distance, similar images are selected for the reference image.

In some implementations, the first determining unit is further configured to:

Obtain the first descriptive text corresponding to the reference image;

Obtain the second descriptive text corresponding to the similar image;

Based on the differences between the first description text and the second description text, difference information is generated.

In some implementations, the first determining unit is further configured to:

The reference image is globally described using a first preset model, generating a global description text for the reference image;

The pixel region where each object is located in the reference image is processed by the second preset model to obtain the object description text corresponding to each object in the reference image;

The first description text corresponding to the reference image is determined based on the global description text of the reference image and the object description text corresponding to each object.

In some embodiments, the second determining unit is further configured to:

Based on the difference information, determine the difference objects in the reference image relative to the similar image;

A target mask image is generated based on the differences in the reference image.

Furthermore, embodiments of this application also provide a computer device, including a processor and a memory, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in any of the image processing methods provided in embodiments of this application.

Furthermore, embodiments of this application also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the image processing methods provided in embodiments of this application.

Furthermore, embodiments of this application also provide a computer program product, including computer instructions, which, when executed, implement the steps in any of the image processing methods provided in embodiments of this application.

This application embodiment first obtains similar images to a reference image from existing data. Then, based on the difference information between the reference image and the similar image, a target mask image is generated. The difference information is then expanded to enrich the difference description text representing it. Finally, the existing similar image is locally fine-tuned using the target mask, the difference description text, and the reference image to obtain the fine-tuned target image. In this way, the differences between images can be expanded in description, and the expanded description text and the reference image can be used as constraints to locally adjust similar images, improving the accuracy of image adjustment and ensuring that the adjusted image effect meets actual requirements, thus facilitating the development of other subsequent services.

Attached Figure Description

To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

Figure 1 is a scene diagram of the image processing system provided in an embodiment of this application;

Figure 2 is a flowchart illustrating the steps of the image processing method provided in an embodiment of this application.

Figure 3 is a schematic diagram of the global description text generation scenario provided in the embodiments of this application;

Figure 4 is a schematic diagram of the scene of generating object description text in an image provided in an embodiment of this application;

Figure 5 is a schematic diagram of a mask segmentation scenario of the mask segmentation model provided in the embodiments of this application;

Figure 6 is a schematic diagram of the hidden space diffusion model provided in an embodiment of this application;

Figure 7 is a schematic diagram of the structure of the noise reduction network layer in the back diffusion provided in an embodiment of this application;

Figure 8 is a schematic flowchart of another step of the image processing method provided in the embodiment of this application;

Figure 9 is a schematic diagram of the framework structure of the image processing system provided in the application embodiment;

Figure 10 is a schematic diagram of the residual network layer provided in an embodiment of this application;

Figure 11 is a schematic diagram of a scenario where difference information is summarized to generate difference description text, as provided in an embodiment of this application.

Figure 12 is a schematic diagram of the image fine-tuning process provided in an embodiment of this application;

Figure 13 is a schematic diagram of the image processing apparatus provided in an embodiment of this application;

Figure 14 is a schematic diagram of the structure of the computer device provided in an embodiment of this application.

Detailed Implementation

The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

In some processes described in the specification, claims, and the foregoing drawings, multiple steps are included in a specific order. However, it should be clearly understood that these steps may be performed in any order or in parallel. The step numbers are merely used to distinguish different steps and do not represent any execution order. Furthermore, descriptions such as "first" and "second" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

This application provides an image processing method, apparatus, device, and computer-readable storage medium. Specifically, this application will describe the image processing apparatus from the perspective of an image processing device, which can be integrated into a computer device, such as a server or a user terminal. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The user terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, smart home appliance, vehicle terminal, smart voice interaction device, aircraft, etc., but is not limited to these.

It is understood that in the specific implementation of this application, data related to user information, user usage records, and user status are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

It should be noted that the image processing method provided in this application can be applied to any image adjustment scenario. These scenarios are not limited to those implemented through cloud services, big data, artificial intelligence, or a combination thereof. The following embodiments illustrate this further:

The image processing method provided in this application relates to Artificial Intelligence (AI) technology. AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in tasks such as target recognition, tracking, and measurement, and further performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

However, the embodiments of this application can achieve image processing through technologies such as image processing, image recognition, and image semantic understanding in computer vision technology to complete the image processing task. This will be specifically illustrated through the following embodiments:

It should be noted that in this image processing scenario, it is mainly implemented through Artificial Neural Networks (ANNs) models, hereinafter referred to as "models". The image processing process can include a training phase (A) and an application phase (B) of the model. The training phase and the application phase can be implemented through a combination of one or more devices in the image processing system.

For example, referring to Figure 1, which is a schematic diagram of an image processing system provided in an embodiment of this application, the system may include a server and/or a terminal; when the system includes only a server or a terminal, the server or terminal includes a target database, a model training device and a model application device; when the system is a combination of a terminal and a server, the server may include a target database, a model training device and a model application device.

The target database can store a large amount of data, which is not limited to image data, but can be used as sample similar images during the model training phase.

(A) Model training phase:

During the model training phase, the model training device can train a pre-defined model based on the acquired training data, which serves as samples. Specifically, this model training phase may include preparing training data and model training.

The process of preparing training data involves the following steps: First, a reference image and a target image to be adjusted are defined. Then, similar images of the reference image are retrieved from a target database. Next, based on the difference information between the reference image and the similar images, a target mask image of the reference image and descriptive text describing the differences are generated. This completes the acquisition of training data.

The training of the model can be understood as comparative learning training between the model's output image and the sample target image. Since the image adjustment process in this embodiment may include: fine-tuning similar images using descriptive text of the difference information as guiding conditions, and simultaneously fine-tuning similar images using a reference image as a constraint condition, which can be achieved separately using two models for image fine-tuning, the model training process may include training different models. In this case, the sample target image includes a first sample target image and a second sample target image.

Referring to Figure 1, the training process of a model using difference text as a guiding condition is taken as an example. Specifically, the training process of this model is as follows: Based on the sample target mask image and the sample difference description text, the preset model performs local fine-tuning on the sample similar images to obtain the first predicted image. Then, the first sample target image is compared with the first predicted image, and when there is a difference between the two, a prediction loss is constructed based on the difference between the first sample target image and the first predicted image. The preset model is trained based on this prediction loss. The model is iteratively trained in the above manner until the preset conditions are met, such as the first predicted image output by the preset model being the same as the first sample target image, or the number of iterations reaching a certain number, or the first predicted image output by the preset model no longer changing, etc., to obtain the trained first neural network model.

Similarly, referring to Figure 1, let's take the training process of a model using a reference image as a constraint as an example. Specifically, in preparing training data, in addition to obtaining the sample reference image, sample target mask image, and sample similar images, it is also necessary to obtain the sample target inverse mask image associated with the sample target mask image. Then, based on the sample target inverse mask image and the sample reference image, the preset model performs local fine-tuning on the sample similar images to obtain the second predicted image. Then, the second sample target image is compared with the second predicted image, and when there is a difference between the two, a prediction loss is constructed based on the difference between the second sample target image and the second predicted image. The preset model is iteratively trained based on this prediction loss until the preset conditions are met, resulting in the trained second neural network model.

At this point, the training process based on the model training device is complete, yielding a first neural network model and a second neural network model. These trained models can be used in the image processing process of this application. It should be noted that a conditional latent space diffusion (SD) model can be used for training during the model training phase, resulting in the trained first and second neural network models belonging to the SD model category.

Furthermore, fine-tuning similar images can be achieved using descriptive text about differences as guiding conditions, and simultaneously, fine-tuning similar images can be performed using reference images as constraints. Alternatively, a single model can be used to fine-tune images, completing the image processing. During the model training phase, a pre-defined model can be used to locally fine-tune sample-like images based on sample target masks, sample difference descriptive text, and sample reference images to obtain predicted images. Then, the sample target images are compared with the predicted images, and when differences exist, a prediction loss is constructed based on these differences. The pre-defined model is then iteratively trained based on this prediction loss until the pre-defined conditions are met, resulting in the trained target model.

(B) Application phase of the model:

In the application phase of the model, the trained first and second neural network models can be uploaded or installed into the model application device. This allows the device to run these models during image processing to facilitate the completion of the relevant image processing workflow. Specifically, the image processing workflow includes: acquiring a reference image and acquiring similar images; determining the difference information between the reference image and the similar images; determining a target mask image in the reference image that addresses the difference information; expanding the target mask image based on the difference information to obtain a difference description text; and performing local adjustments on the similar images based on the target mask image, the difference description text, and the reference image to obtain the adjusted target image.

It should be noted that the first and second neural network models are mainly used in the process of local fine-tuning of similar images. Specifically, the first neural network model can perform local fine-tuning of similar images based on the target mask image and the difference description text to generate a first image; and the second neural network model can perform local fine-tuning of similar images based on the target inverse mask image and the reference image to generate a second image; the first image and the second image are then fused to obtain the adjusted target image.

Furthermore, if a model is used to fine-tune an image, the trained target model can be uploaded or installed into a model application device. This allows the application device to run the target model during image processing to complete the relevant image processing workflow. This workflow includes: acquiring a reference image and acquiring similar images; determining the difference information between the reference image and the similar images; determining a target mask image in the reference image that addresses the difference information; expanding the target mask image based on the difference information to obtain a difference description text; and performing local adjustments on the similar images based on the target mask image, the difference description text, and the reference image to obtain the adjusted target image.

The image processing method of this application can be realized through the training and application scenarios of the above model.

For example, assuming a server or terminal includes a target database, a model training device, and a model application device, the server or terminal can prepare training data based on sample image data in the target database, train a preset model using the training data through the model training device, and then transmit the trained first and second neural network models to the model application device for execution. In this case, the terminal or server can perform the following: acquire a reference image and acquire similar images; determine the difference information between the reference image and the similar images; determine a target mask image in the reference image that addresses the difference information; expand the target image based on the difference information to obtain difference description text; and perform local adjustments on the similar images based on the target mask image, the difference description text, and the reference image to obtain the adjusted target image.

For example, consider a system combining a terminal and a server, where a communication connection is established between the terminal and the server. The server can be a distributed service system composed of multiple physical service providers, containing at least a target database, a model training device, and a model application device. After training the model on the server, the trained first and second neural network models, or the trained target model, can be run on the server to implement the image processing workflow. Specifically, in the application phase, a reference image can be sent to the server via a client on the terminal. After obtaining the reference image, the server can retrieve similar images from the target database; determine the difference information between the reference image and the similar images; determine the target mask image in the reference image based on the difference information; expand the image based on the difference information to obtain difference description text; and perform local adjustments on the similar images according to the target mask image, difference description text, and the reference image to obtain the adjusted target image. The server can then return the adjusted target image to the terminal.

For example, referring to Figure 1, assuming an image processing application (client) is installed on the terminal, the user can select an image search task on the image processing application to execute the image processing process. Specifically, the image processing process is as follows: First, the user can select an image search task on the client page on the terminal and set a reference image to be searched for the image search task; then, the client transmits the reference image to the server. Then, after obtaining the reference image, the server can retrieve similar images from the target database; determine the difference information between the reference image and the similar images; determine the target mask image in the reference image based on the difference information; expand based on the difference information to obtain difference description text; and perform local adjustments on the similar images according to the target mask image, difference description text, and reference image to obtain the adjusted target image. Finally, the server returns the target image with local fine-tuning of the similar images to the client, so that the images provided by the image search service can better match the actual needs (reference image), thus facilitating the development of the image search service.

It should be noted that the above are just examples and can be applied to other image services, which will not be elaborated here.

For ease of understanding, each step of the image processing method will be described in detail below. It should be noted that the order of the following embodiments is not intended to limit the preferred order of the embodiments.

In this embodiment, the description will focus on the image processing device, which can be integrated into a computer device such as a terminal or server. Referring to Figure 2, which is a flowchart illustrating the steps of an image processing method provided in this embodiment, this embodiment takes the image processing device integrated into a server as an example. When the processor on the server executes the program instructions corresponding to the image processing method, the specific process is as follows:

101. Obtain a reference image, and then obtain similar images that are similar to the reference image.

In this embodiment of the application, after obtaining the reference image, in order to obtain an image that meets the actual needs of the reference image, it is generally possible to search for the image most similar to the reference image from the existing image data so that the searched similar image can be further processed. The image processing process can be image adjustment, such as large-scale adjustment or local adjustment, so that the information contained in the adjusted target image is more in line with the reference image. For example, the obtained image is more similar to the reference image in terms of style and content, and has reliability.

The reference image can be an image containing any content information, such as an image containing one or more types of content information, including fruits, tableware, animals, people, and animations. It can also be an image containing other forms of content information, which are not listed here. It should be noted that the reference image can be used as the basis for image adjustment in the image processing process, that is, other images can be adjusted based on the reference image.

The similar image can be the image in the database that is most similar to the reference image. It can be understood as the image in the existing data that is most similar to the reference image in terms of image content or style. It should be noted that this embodiment uses the similar image as the basis for image adjustment; that is, image adjustments are performed based on the similar image.

To facilitate understanding of reference images and similar images, we will introduce these two types of images with examples. For instance, taking an image search service as an example, a customer sends an example image to the image search platform through a client. The example image contains information about two cats lying down; this example image can be considered a reference image. Then, after receiving the example image from the customer, the image search platform can search its database for the most similar image to that example image. This allows the platform to further adjust the similar image based on relevant information from the reference image, in order to best meet the customer's image search needs.

In some implementations, to find similar images to a reference image from an existing database, the similarity between any two images can be determined by using feature distance, or the similarity between two images can be measured by feature distance. For example, "obtaining similar images to the reference image" in step 101 may include: determining the reference cluster center to which the reference image belongs; determining the feature category distance between each pre-constructed image cluster center in a preset database and the reference cluster center; and selecting similar images to the reference image based on the feature category distance.

It should be noted that different images contain different information. Images can be classified according to their content. For example, images under the animal category can be categorized by animal species, such as cats, dogs, tigers, horses, pigeons, eagles, and other animals. For one or more images belonging to the same category, cluster centers for that category can be calculated using one or more images.

The reference cluster center can be a feature cluster center constructed based on one or more reference images, representing the feature category center point among these reference images, which can be understood as the feature mean point. For example, when there is one reference image, the reference image can be converted into a pixel matrix, and the pixel matrix of the reference image can be regarded as the reference cluster center; when there are multiple reference images, the pixel matrix of each reference image can be determined separately, and the reference cluster center of these multiple reference images can be calculated by combining each pixel matrix, such as using the mean of multiple pixel matrices as the reference cluster center. The above are merely examples and are not intended to limit the specific implementation of this application.

The image cluster center can be the cluster center of the image set corresponding to each category in the existing database. Each image cluster center can change according to the update of the image set of each category in the database, that is, the image cluster center can be constructed in real time. For example, the preset database may contain images of subjects such as food, animals, plants, vehicles, and ornaments. Each subject may contain one or more image categories, and each image category corresponds to an image cluster center. Taking animals as an example, assuming that there is an image set containing the cat category, each image under the cat category is represented by a matrix of each pixel point, and the mean of all pixel point matrices under the cat category is calculated as the image cluster center corresponding to the cat category. Assuming that the animal subject also includes a dog category, the calculation method for the image cluster center of the dog category is the same as the calculation method for the image cluster center of the cat category. It should be noted that the calculation method for the image cluster center of any category under other subjects is the same as the above, and will not be elaborated here.

To select similar images from a pre-defined database, after determining the reference cluster center corresponding to the reference image, the feature class distance between the reference cluster center and each image cluster center in the database can be calculated. Further, similar images can be selected based on the feature class distance. Specifically, the magnitude of the feature class distance can be used to determine which image cluster center in the database is more similar to the cluster center of the reference image, thus identifying the image class of that similar target image cluster center as the image class of the reference cluster center. Then, the feature distance between each image in the image class of the reference image and the target image cluster center is calculated. It should be noted that the magnitude of the feature distance can reflect the similarity between any two images; therefore, similar images can be selected based on the magnitude of the feature distance, for example, selecting the image with the smallest feature distance to the reference image as the similar image.

Using the above method, after obtaining the reference image, we can search for the most similar image from the existing image data. This allows us to make further image adjustments to the searched image to obtain an image that better meets the actual needs of the reference image, thus ensuring reliability.

102. Determine the differences between the reference image and similar images.

In this embodiment of the application, in order to adjust similar images more accurately, the differences between the reference image and the similar image can be determined so that the differences between the two images can be used as the basis for adjustment in the image processing process, and the similar image can be adjusted to improve the accuracy of image adjustment.

The difference information can represent the feature differences between a reference image and a similar image, and is not limited to differences in the number of objects (things) that differ in the image, the location of the objects, and/or the posture of the objects. For example, if the reference image contains two orange cats, the first orange cat is lying on the grass, and the second orange cat is running around the first orange cat, and the similar image contains two orange cats, one of which is lying on the grass, and the other is running at a distance from the lying orange cat, then the difference information between the reference image and the similar image could be the positional difference between the two orange cats, i.e., their positional relationship. As another example, if the reference image contains two orange cats and one blue cat, and the similar image contains two orange cats, then the difference information between the reference image and the similar image includes a different object (the blue cat). It should be noted that this difference information can be generated by combining the difference object and the positional relationship of the objects. The above are merely examples and are not intended to limit the specific implementation of this application.

In some implementations, to obtain difference information between a reference image and similar images, the differences between the images can be determined based on the descriptive text of the reference image and the descriptive text of the similar images, thereby generating difference information. For example, step 102 may include:

(102.1) Obtain the first descriptive text corresponding to the reference image;

(102.2) Obtain the second descriptive text corresponding to similar images;

(102.3) Generate difference information based on the difference between the first description text and the second description text.

The first descriptive text can be image content description text generated based on the content information in the reference image. Since the content information of the reference image can include object information and the environment in which the object is located, the first descriptive text is not limited to including global descriptive text for the overall content of the reference image and object descriptive text for objects in the reference image. For example, assuming the reference image contains two orange cats and one blue cat, the global descriptive text can be "There are two orange cats and one blue cat on the grass. The second orange cat is located in the grass area to the upper right of the first orange cat, and the blue cat is located in the grass area to the upper left of the first orange cat." The object descriptive text for the first orange cat is "The orange cat is lying on the grass." The object descriptive text for the second orange cat is "The orange cat is running on the grass." The object descriptive text for the blue cat is "The blue cat is rolling on the grass." The above global descriptive text and object descriptive text are only examples and are not intended as specific limitations for implementing this application. Any of the above descriptive texts can be described in more detail or more concisely according to the actual situation.

The second descriptive text can be image content description text generated based on content information in similar images. Similarly, the second descriptive text is not limited to global descriptive text for the overall content of similar images and object descriptive text for objects in similar images. Specific examples can be found in the description of the first descriptive text, which is not limited here.

Specifically, in order to obtain the difference information between the reference image and similar images, after obtaining the first descriptive text of the reference image and the second descriptive text of the similar images, the differences in content information between the reference image and the similar images can be learned based on the descriptive differences between the first and second descriptive texts. For example, it can be determined whether there is a difference in the number of objects between the reference image and the similar images, whether there is a difference in the distribution of object positions between the reference image and the similar images, whether there is a difference in the shape of objects between the reference image and the similar images, etc. It should be noted that the differences between the reference image and the similar images may include one or more of the above situations, and may also include other differences. Furthermore, based on the differences determined above, difference information is obtained.

For example, suppose the first descriptive text for a reference image is "There are two orange cats on the grass. The first orange cat is lying on the grass, and the second orange cat is located in the upper right area of the first orange cat. The second orange cat is running on the grass." Suppose the second descriptive text for a similar image is "There are two orange cats on the grass. The first orange cat is lying on the grass, and the second orange cat is located in the upper left area of the first orange cat. The second orange cat is running on the grass." Thus, the difference between the first and second descriptive texts is the difference in the positional distribution of the second orange cat (object) in the image. Therefore, difference information can be generated based on these two differences. This difference information can be information about the second orange cat (difference object) in the reference image, such as the position, shape, and size information of the second orange cat. In addition, since the upper left area of the first orange cat in the reference image is grass, and the upper left area of the first orange cat in the similar image contains the second orange cat, the difference information can also include information about the grass (difference object) in the upper left area of the first orange cat in the reference image.

For example, suppose the first descriptive text for a reference image is "There are two orange cats and one blue cat on the grass. The first orange cat is lying on the grass, the second orange cat is located to the upper right of the first orange cat and is running on the grass, and the blue cat is located to the upper left of the first orange cat and is rolling on the grass." Suppose the second descriptive text for a similar image is "There are two orange cats on the grass. The first orange cat is lying on the grass, the second orange cat is located to the upper right of the first orange cat and is running on the grass." Thus, the difference between the first and second descriptive texts is the difference in the number of cats (objects) and the difference in the distribution of the cats in the image. Therefore, difference information can be generated based on these two differences. This difference information can be information about the blue cat (difference object), such as the object description text of the blue cat and its position information in the reference image.

In some implementations, since both the reference image and the similar image are image data, in order to accurately obtain the text information describing the reference image and the similar image, a first descriptive text for the reference image and a second descriptive text for the similar image can be obtained through image-to-text conversion. For example, to obtain the first descriptive text, step (102.1) may include:

(102.1.1) The reference image is globally described using the first preset model, and a global description text of the reference image is generated;

(102.1.2) The pixel region where each object is located in the reference image is processed by the second preset model to obtain the object description text corresponding to each object in the reference image;

(102.1.3) Determine the first description text corresponding to the reference image based on the global description text of the reference image and the object description text corresponding to each object.

The global description text can be understood as a comprehensive description and summary of the content information contained in the image. This global description text allows for a quick understanding of the image based on text. For example, taking a reference image as an example, if the reference image contains grass, a frisbee, and a dog jumping towards the frisbee, the global description text could be "The dog is jumping on the grass playing with a frisbee." Or, if the reference image contains grass, a reclining orange cat, and a running blue cat, the global description text could be "The orange cat is resting on the grass, and the blue cat is running and playing on the grass." These are just a few examples.

The object description text can be text used to describe the characteristics of the corresponding object in the image, and may include descriptions of the object's color, shape, posture, position, and other aspects. For example, taking a reference image as an example, the object description text can describe the position, action, posture, shape, and other information of the corresponding object in the image. It should be noted that the object description text may include an object category label and object description information. The object category label is used to indicate the category or name to which the corresponding object belongs, and the object description information is used to specifically describe the object's action, position, shape, posture, etc. For example, assuming that the content information contained in an image is "a brown dog playing frisbee on the grass," then the object category label can be "dog" or "brown dog," and the object description information can be "a brown dog playing frisbee on the grass." The above are merely examples and are not intended to limit the specific implementation of this application.

The first preset model can be a model for global textual description of the overall situation of the image. For example, the first preset model can be a pre-trained frozen image encoder and large language model (BLIP2), which introduces large language models (LLM).

Specifically, as shown in Figure 3, the first preset model can be structurally divided into two parts: Vision-and-Language Representation Learning and Vision-to-Language Generative Learning. The Vision-and-Language Representation Learning includes an Image Encoder and a lightweight Querying Transformer (Q-Former), while the Vision-to-Language Generative Learning includes Large Language Models (LLM). Taking the global description of a reference image as an example, in the first preset model, the reference image is first input into an image encoder, which encodes the reference image to obtain an image encoding result. Then, the image encoding result is input into a lightweight query transformer, and the object category label in the reference image is determined. The object category label is then fused with the object category label in the lightweight query transformer to obtain a fused feature result. Finally, the fused feature result is input into a large-scale language model for language processing to output a global description text for the reference image.

The second preset model can be a model for describing the text of image regions containing objects in an image. For example, the second preset model can be a generative region-to-text transformer (GRIT). Specifically, as shown in Figure 4, the second preset model can structurally include a visual encoder, a foreground object extractor, and a text decoder. Taking the description of the pixel region where each object is located in the reference image as an example, firstly, each object in the reference image is identified to determine the pixel region where each object is located, and each identified object in the reference image is categorized. Then, the reference image with the determined pixel region and category of each object is input into the second preset model. In the second preset model, the region-to-language conversion processing is performed on the pixel region where each object is located, and the converted reference image is output. The converted reference image contains a bounding box for labeling each object, and the object description text of the object in each bounding box.

Furthermore, after obtaining the global description text and object description text of the reference image, the global description text and object description text can be merged to obtain the first description text for the reference image.

In the embodiments of this application, the method for obtaining the second descriptive text of similar images can refer to the above-described process for obtaining the first descriptive text of the reference image, and will not be repeated here.

By using the above methods, the differences between the reference image and similar images can be determined, so that the differences between the two images can be used as the basis for adjustment in the image processing process, so as to adjust the similar images more accurately.

103. Determine the target mask image for the difference information in the reference image.

In this embodiment of the application, after determining that similar images need to be adjusted, image processing can be performed by local adjustment. In order to achieve local adjustment of similar images, it is necessary to obtain a mask image of the difference information in the reference image, so as to use the mask image to participate in the local fine adjustment of similar images, so as to improve the accuracy of subsequent image adjustment.

The target mask map can be a pixel region mask map generated based on the difference information between the reference image and similar images. It is used to mask part of the pixel region in the similar image when fine-tuning the similar image, so that the masked pixel region of the similar image is blank (without content).

In some implementations, a target mask map may be constructed based on objects in a reference image that differ from those in similar images. For example, step 103 may include:

(103.1) Based on the difference information, identify the objects in the reference image that differ from similar images;

(103.2) Generate a target mask based on the difference objects in the reference image.

Specifically, to obtain a target mask map of the reference image based on the difference information, after obtaining the difference information, the difference objects contained in the reference image relative to similar images can be determined based on this difference information, and the target mask map can be generated based on the determined difference objects. It should be noted that when generating the target mask map, firstly, an initial mask map of the same size as the reference image can be constructed. Then, using the relevant information of the difference objects as indicator information, the target mask map is generated according to the indicator information, the initial mask map, and the reference image.

In some implementations, a target mask for the difference information in the reference image can be obtained through semantic segmentation. For example, step (103.2) may include: constructing an initial mask of the same size as the reference image; obtaining indication information for the difference object in the reference image, which is not limited to including the pixel region of the difference object's background, bounding boxes, and object description text; and generating a target mask based on the initial mask, indication information, and reference image using a semantic segmentation model.

The segment everything model is used to generate a mask for any image based on the indication information. As shown in Figure 5, the model structurally includes an image encoder, a convolutional module (conv), a fusion module, an indication information encoder, and a mask decoder. Specifically, a reference image is input into the semantic segmentation model. The image encoder encodes the reference image to obtain an image vector. Simultaneously, the convolutional module (conv) extracts features from the initial mask to obtain a mask vector. Then, the image vector and the mask vector are fused to obtain image fusion features. Furthermore, the indication information encoder (conv)... The `t encoder` encodes the indication information, which may include the pixel region (point), bounding box, and object description text of the difference object's background. This yields the encoded features of the indication information. The image fusion features and the encoded features of the indication information are then decoded by the mask decoder, outputting one or more predicted masks. It should be noted that when only one predicted mask is output, it is used as the target mask. When multiple predicted masks are output, each has a corresponding score, and the predicted mask with the highest score is selected as the target mask.

To facilitate understanding of the target mask generation principle, the following scenario example will be used to illustrate the target mask: For example, suppose a reference image contains two orange cats. The first orange cat is lying on the grass, and the second orange cat is running in the area above and to the right of the first orange cat. A similar image contains two orange cats, the first orange cat is lying on the grass, and the second orange cat is running in the area above and to the left of the first orange cat. The difference information includes information about the second orange cat in the reference image and information about the grass in the area above and to the left of the first orange cat, such as its location, size, and shape. Then, after constructing an initial mask of the same size as the reference image, a target mask can be generated based on the obtained difference information. This target mask is mainly used to mask the pixel area of the second orange cat in the similar image and the pixel area of the grass in the area above and to the right of the first orange cat in the similar image during image processing. In the target mask, these two pixel areas are represented by "0", and all pixel areas other than these two are represented by "1".

By using the above methods, a target mask image can be obtained that shows the difference between the reference image and similar images. This target mask image can then be used as a necessary element when adjusting similar images to participate in local fine-tuning of the similar images, thereby improving the accuracy of subsequent image adjustments.

104. Expand upon the difference information to obtain the difference description text.

In this embodiment of the application, in order to make local adjustments to similar images, in addition to obtaining the mask image of the difference information in the reference image, it is also necessary to obtain the relevant descriptive text of the difference information in the reference image, so that the relevant descriptive text of the difference information can be used as the guiding condition for subsequent image adjustment, thereby improving the accuracy of image adjustment.

The difference description text can be text describing the content differences between the reference image and similar images. It is mainly derived from textual expansion based on the difference information, which can enrich and accurately represent the differences between the reference image and similar images. This difference description text can serve as a guiding condition for adjusting similar images, participating in indicating local adjustments to similar images and improving the accuracy of image adjustments.

In some implementations, since difference information can reflect the differences between a reference image and similar images, in order to obtain descriptive text that fully describes the differences between two images, the description can be expanded based on the relationship between the differing object and other objects in the reference image, and the object description text of the differing object, to obtain descriptive text for the differences between the two images, i.e., difference description text. For example, step 104 may include:

(104.1) Based on the difference information, determine the objects of difference between the reference image and similar images;

(104.2) Determine the object relationship information between the differing object and other objects in the reference image;

(104.3) Obtain the global description text of the reference image and the target object description text of the difference object;

(104.4) Based on the global description text, the target object description text and the object relationship information, the text is expanded to obtain the difference description text.

The object relationship information can represent the relationship between the differing object and any other object. This object relationship is not limited to the positional distribution relationship (such as distance and direction) and category relationship (whether they belong to the same species) between the differing object and other objects. For example, the object elements of the reference image include grass, an orange cat, and a blue cat. The orange cat is lying in the grass area in the center of the reference image, and the blue cat is located in the grass area to the upper right of the orange cat. The object elements of the similar image also include grass, an orange cat, and a blue cat, but the difference is that the blue cat in the similar image is located in the grass area to the upper left of the orange cat. Therefore, the differing object of the reference image relative to the similar image can at least include the blue cat. Taking the blue cat as the differing object as an example, the object relationship information can include the positional distribution relationship between the blue cat and the orange cat in the reference image (such as the blue cat being located to the upper right of the orange cat) and the category relationship between the blue cat and the orange cat (the blue cat and the orange cat belong to the same species category).

Specifically, to provide a rich description of the differences between the reference image and similar images, firstly, the object corresponding to the difference in the reference image is identified, and the object relationship information between this object and any other object in the reference image is obtained, such as determining the positional relationship between the current object and other objects in the image, and whether they belong to the same object category (object class). Further, referring to the aforementioned acquisition of the "first descriptive text," the global descriptive text of the reference image and the object descriptive text associated with the object are obtained. Finally, based on the global descriptive text of the reference image, the descriptive text of the object, and the object relationship information between the object and other objects, the text is expanded to provide a comprehensive description of the differences between the reference image and similar images, resulting in the expanded difference descriptive text. This difference descriptive text richly expresses the relationship between the object and other objects in the reference image, as well as the object state information of the object. Thus, compared to the object descriptive text, this difference descriptive text can more comprehensively represent the relevant information of the object in the reference image, serving as a guiding condition when adjusting similar images, enabling accurate and reliable fine-tuning of local areas in similar images.

For example, when augmenting the text of the difference information, an existing large-scale language processing model can be used. Specifically, the object relationship information, global description text, and object description text of the difference objects can be transmitted to the large-scale language processing model. Based on the object relationship information, the large-scale language processing model can mine relevant information from the global description text and the object description text of the difference objects, and further augment the text to generate a text containing a rich description of the difference objects, i.e., difference description text, which can be used as data for subsequent image adjustment.

By using the above methods, rich descriptive text about the differences in the reference image can be obtained. This descriptive text can then be used as a guiding condition to indicate local adjustments to similar images, thereby improving the accuracy of image adjustments.

105. Based on the target mask image, the difference description text, and the reference image, perform local adjustments on the similar images to obtain the adjusted target image.

In this embodiment of the application, in order to obtain a target image that better matches the content information of the reference image, after obtaining a similar image that is similar to the reference image, the similar image can be used as the image to be adjusted. Then, the target mask image, the difference description text, and the reference image are used as guiding conditions for image processing. Based on the guiding conditions, the similar image is adjusted to make local fine adjustments to the similar image and generate a target image that better matches the content requirements of the reference image, which has reliability.

The target image can be an image not currently stored in the database; it is primarily obtained by fine-tuning a similar image. Specifically, after finding the most similar image to the reference image from the preset database, since this similar image may actually have some differences from the reference image, in order to obtain a more closely matching similar image, the similar image can be used as a base image. Based on the reference image and the target mask image and difference description text, the base image is further fine-tuned to obtain a target image that is more similar to the reference image.

It should be noted that when adjusting local regions of similar images, the primary guiding conditions are the target mask, the difference description text, and the reference image. The target mask primarily influences the rendering of pixels belonging to the target pixel region within the similar image. This target pixel region can be a pixel region in the reference image that differs from or does not differ from the similar image. The adjustment of similar images can include two parts: first, using the difference description text as a guiding condition, and combining it with the target mask to perform local adjustments; second, using the reference image as a guiding (constraint) condition, and combining it with the target mask to perform local adjustments. For ease of understanding, the adjustment of similar images will be described in detail below.

In this embodiment, when performing local fine-tuning on similar images, the similar images can first be masked, and then the masking result can be fine-tuned to obtain the adjusted target image. Alternatively, the image can be masked during the fine-tuning process. It should be noted that the above image fine-tuning process is not limited to noise processing; specific optional implementation methods are described below.

(A) First, perform masking on similar images, then fine-tune the masking results:

In some implementations, the difference description text and the reference image can be used as guiding conditions for similar images, and local adjustments can be made to the similar images respectively, so as to fuse the two adjustment results to obtain the target image. For example, step 105 may include:

(105.A.1) Based on the target mask image and the difference description text, the similar images are locally fine-tuned to obtain the first image;

(105.A.2) Based on the target mask image and the reference image, the similar image is locally fine-tuned to obtain the second image;

(105.A.3) The first image and the second image are fused to obtain the adjusted target image.

The first image can be obtained by locally adjusting a similar image according to the difference description text. The content of this first image differs from that of the similar image. Some areas of the first image contain blank pixel regions, i.e., regions without content. Specifically, the pixel regions corresponding to the difference information in the reference image are blank pixel regions. For example, suppose that both the reference image and the similar image contain a blue cat and an orange cat playing on the grass, but the difference between the two images is "the position of the blue cat in the image". Therefore, when making local fine-tuning of the similar image according to the difference description text, the pixel regions at the same position in the similar image can be adjusted according to the pixel region where the blue cat is located in the reference image. The adjusted pixel regions are defined as the difference pixel regions, resulting in the first image. The difference pixel regions in the first image will be replaced by blank pixel regions, i.e., without content. In addition, when fine-tuning the similar image, the pixel region where the original blue cat was located in the similar image can also be adjusted so that the pixel region where the original blue cat was located in the first image is replaced by blank pixel regions. For example, suppose that the reference image and the similar image both contain the content "a plate with food, cutlery, and utensils", but the difference between the reference image and the similar image is the placement of the cutlery. After making local adjustments to the similar image, the pixel regions in the similar image at the same positions are adjusted according to the pixel regions where the cutlery is located in the reference image. The adjusted pixel regions in the resulting first image are replaced with blank pixel regions.

The second image can be obtained by locally adjusting a similar image according to the guiding conditions of a reference image. Most of the area in this second image is blank pixel area, with only the pixel areas corresponding to the difference information in the reference image being non-blank pixel areas. The content presented by the pixel combinations in these non-blank pixel areas is the image content of the difference object corresponding to the difference information in the reference image. For example, suppose both the reference image and the similar image contain the content "a blue cat and an orange cat playing on the grass," but the difference between the two images is "the position of the blue cat in the image." Therefore, when locally fine-tuning the similar image according to the reference image, the pixel areas at the same position in the similar image can be adjusted according to the pixel area where the blue cat is located in the reference image. The resulting second image contains the pixel area where the blue cat is located in the reference image in its non-blank pixel areas, presenting the blue cat in the reference image along with its shape, posture, and actions. It can be understood that the second image contains the image content of the orange cat and the grass. For example, suppose that both the reference image and the similar image contain the content "a plate with food, cutlery", but the difference between the reference image and the similar image is the placement of the cutlery. After making local adjustments to the similar image, the resulting second image will only contain the cutlery content from the reference image.

To obtain a target image that is more similar to a reference image, the process can include two stages: image adjustment and image fusion. Specifically, the image adjustment stage can include two parts: difference description text guidance and reference image guidance. The first part can use the difference description text as a guiding condition for image adjustment, combined with a target mask image, to adjust the similar image to obtain a first image where the pixel regions corresponding to the difference information are blank pixel regions, containing no image content. The second part can use the reference image as a guiding condition for image adjustment, combined with a target mask image, to adjust the similar image to obtain a second image where the second image only contains image content in the pixel regions corresponding to the difference information. Furthermore, the obtained first and second images are superimposed and fused, so that the image content in the pixel regions corresponding to the difference information in the second image overlaps and fills the blank pixel regions corresponding to the difference information in the first image, achieving image content complementarity between the first and second images. This results in a target image that has undergone local fine-tuning, and compared to similar images, better meets the image content requirements of the reference image and is more similar to it.

It should be noted that when performing local fine-tuning on similar images, a conditional latent space diffusion (SD) model can be used. Specifically, similar images are input into the conditional latent space diffusion model, which performs noise diffusion processing on the similar images. During the noise diffusion process, guiding conditions are introduced to assist in instructing the image to make precise fine-tuning of relevant pixel regions, thereby improving the accuracy of image adjustment.

In some implementations, step (105.A.1) may include: using a first neural network model to locally fine-tune similar images based on a target mask and difference description text to generate a first image. Step (105.A.2) may include: using a second neural network model to locally fine-tune similar images based on a target mask and a reference image to generate the first image. Both the first and second neural network models are conditional latent space diffusion (SD) models.

In some implementations, in order to achieve local fine-tuning of similar images using a conditional latent space diffusion (SD) model, the conditional latent space diffusion model needs to be trained to obtain a first neural network model and a second neural network model for image fine-tuning, respectively.

For example, taking the training of the first neural network model as an example, before step (105.A.1), it may also include: acquiring a sample reference image and a sample similar image, as well as a first sample target image; generating a sample target mask map and sample difference description text of the sample reference image based on the difference information between the sample reference image and the sample similar image; performing local fine-tuning on the sample similar image based on the sample target mask map and the sample difference description text using a preset model to generate a first prediction image; determining the prediction loss based on the difference between the first sample target image and the first prediction image; and iteratively training the preset model based on the prediction loss until the preset conditions are met to obtain the first neural network model.

For example, before step (105.A.2), the method may further include: acquiring a sample reference image and a sample similar image, as well as a second sample target image; generating a sample target mask image of the sample reference image based on the difference information between the sample reference image and the sample similar image, and acquiring a sample target inverse mask image that is the opposite of the sample target mask image; performing local fine-tuning on the sample similar image based on the sample target inverse mask image and the reference image using a preset model to generate a second prediction image; determining the prediction loss based on the difference between the second sample target image and the second prediction image; and iteratively training the preset model based on the prediction loss until a preset condition is met to obtain a second neural network model.

In some implementations, local fine-tuning of similar images primarily involves introducing guiding conditions within the noise diffusion of the image to instruct local fine-tuning of target regions in the similar images, thereby improving the accuracy of image adjustment. For example, using the difference description text as the guiding condition for image adjustment, and adjusting similar images through a first neural network model, step (105.A.1) may include:

(105.A.1.1) Perform masking processing on similar images based on the target mask image to obtain the first similarity mask image;

(105.A.1.2) Add noise to the first similarity mask image to obtain the first similarity noise map;

(105.A.1.3) Obtain the difference text vector corresponding to the difference description text;

(105.A.1.4) Perform noise reduction processing on the first similarity noise map based on the difference text vector to obtain the first feature map;

(105.A.1.5) Decode the first feature map to obtain the first image.

It should be noted that the purpose of the target mask image is to mask the target pixel regions at the same locations in the regions where the difference between the target and reference images are located, thereby blocking the representation of pixels in the target pixel regions in the similar image. Therefore, before adjusting the similar image through the first neural network model, the similar image is first masked according to the target mask image. This masking process can be achieved by multiplying the target mask image and the similar image, so that each value in the target mask image is multiplied by the corresponding pixel in the similar image, resulting in a first similarity mask image. Then, this first similarity mask image is input into the first neural network model to perform local fine-tuning of the first similarity mask image. For ease of understanding, the image adjustment process of the first neural network model can be described with reference to Figure 6. Specifically, the first neural network model encodes the first similarity mask image and introduces the encoding result into the latent space. The encoding result is then forward diffused in the latent space. This forward diffusion can be understood as a noise addition process to obtain a first similarity noise map. Next, before backdiffusion of the first similarity noise map, the difference description text used as a guiding condition is text-encoded to obtain a difference text vector, thus introducing the difference description text into the latent space. Then, backdiffusion (noise reduction) is performed on the first similarity noise map in the latent space, and during backdiffusion, the difference text vector is integrated into the noise of the first similarity noise map for backdiffusion together to obtain the first feature map. Finally, the first feature map is decoded to recover the first feature map in the latent space, thus obtaining the first image.

It should be noted that when the first similar noise map is locally fine-tuned through the first neural network model, it mainly includes two processes: forward diffusion and backward diffusion. The forward diffusion is specifically a process of noise reduction of the image, which can be understood as a process of gradually adding noise. The backward diffusion is a process of noise reduction of the noise map, which can be understood as a process of gradually reducing noise.

In some implementations, in order to perform forward diffusion on similar images, it is necessary to convert the similar images into vectors, import them into the latent space, and perform forward diffusion processing in the latent space. For example, taking the forward diffusion of a model guided by the difference description text as an example, step (105.A.1.2) may include: encoding the first similarity mask image to obtain an encoded feature map; and processing the encoded feature map for noise to obtain a first similarity noise map.

Specifically, after the first similarity mask image is input into the first neural network model, the first neural network model performs graph feature encoding processing on the first similarity mask image in the pixel space to obtain an encoded feature map, which is the vector feature matrix of the first similarity mask image. Then, the encoded feature map is transmitted to the latent space, where it is forward diffused. This forward diffusion process is a noise addition process, which mainly involves gradually adding noise to the encoded feature map and performing noise addition processing at multiple time steps to achieve complete noise reduction of the first similarity mask image, resulting in a fully noise-reduced first similarity noise map.

In some implementations, after obtaining a first similar noise map through forward noise addition, a reverse noise reduction process is performed on the first similar noise map, and the feature vector corresponding to the guiding condition is combined during the noise reduction process to obtain a noise map after noise reduction. For example, taking the difference text vector corresponding to the difference description text as the guiding condition, step (105.A.1.4) may include: performing multiple consecutive noise reduction processes on the first similar noise map, and incorporating the difference text vector in each noise reduction process through an attention mechanism to obtain a first feature map.

Specifically, in order to perform backdiffusion processing on the first similar noise map that has been completely noisy, it is necessary to perform noise reduction processing on the first similar noise map for multiple rounds (time steps), and add the difference text vector corresponding to the difference description text in each noise reduction process to achieve accurate fine-tuning of similar images until the noise reduction has been performed for a preset number of time steps to obtain the first feature map.

For example, to facilitate understanding of the noise reduction process in each round, let's take the first round as an example. Since the first similar noise map is a completely noisy map, assuming it has undergone T time steps of noise addition, this first similar noise map can be represented as "Z _{_T} " or "X_{_T}". In the first round of noise reduction, the first similar noise map "Z_{_T}" is subjected to noise reduction, and the difference text vector is incorporated through an attention mechanism during the noise reduction process. Referring to Figure 6, the denoising process in each round is implemented in the Denoising U-Net layer. See Figure 7 for details. The Denoising U-Net layer includes a Residual Network layer (ResNet) and an attention module. The Residual Network layer (ResNet) is mainly used for feature extraction to realize the gradual noise reduction process, and the attention module is used to fuse the feature vectors corresponding to the guiding conditions (such as text difference vectors) with the noise feature map to realize the indication and guidance in image fine-tuning.

Specifically, the denoising U-Net consists of two residual network layers and two attention modules, structurally arranged as "residual network layer - attention module - residual network layer - attention module". The following describes the denoising process of one round, based on the specific results of the denoising network layer: First, the first residual network layer extracts features from the first similar noise map. The extracted features are then passed to the first attention module, along with the differing text vectors. The first attention module integrates the differing text vectors into the first feature result. For example, attention can be calculated on the differing text vectors using an attention mechanism, and the attention result is fused with the first feature result to obtain the first initial fusion result. Further, the second residual network layer extracts features from the first initial fusion result to obtain the second feature result. This second feature result is then passed to the second attention module, along with the differing text vectors. The second attention module integrates the differing text vectors into the second feature result to obtain the first fused noise map.

Following the example above, after multiple rounds ("T-1" times) of noise reduction processing, the final "T-1"th fused noise map is obtained, which is the first feature map.

Furthermore, after obtaining the first feature map, the decoding process of the first feature map can specifically be as follows: the first feature map in the latent space is decoded by the decoding module to restore the first feature map into a pixel matrix in the pixel space, so that the first neural network outputs the locally fine-tuned first image.

In some implementations, a reference image is used as a guiding condition for image adjustment, and similar images are adjusted using a second neural network model. For example, step (105.A.2) may include: inverting the target mask image to obtain a target inverse mask image corresponding to the target mask image; performing masking processing on similar images based on the target inverse mask image to obtain a second similar mask image; adding noise to the second similar mask image to obtain a second similar noise image, wherein the time step of the second similar noise image is adjacent to the time step of the first similar noise image; performing noise reduction processing on the second similar noise image based on the feature map corresponding to the reference image to obtain a second feature map; and decoding the second feature map to obtain a second image.

It should be noted that the purpose of using the reference image as a guiding condition to fine-tune the local area of the similar image is to obtain a second image containing representations of the differing objects after image fine-tuning. Since the target mask image is used to cover up the same target pixel regions in the similar image where the differing objects are located compared to the reference image, it is necessary to invert the target mask image. This inversion process involves replacing the original "0"s in the target mask image with "1"s, and simultaneously replacing the original "1"s with "0"s to obtain the target inverse mask image. The target inverse mask image allows the representation of pixels in the target pixel regions where the differing objects are located, while rejecting the representation of other pixels in non-target pixel regions.

Furthermore, after obtaining the target inverse mask image, firstly, the similar image is masked based on the target inverse mask image. This masking process can involve multiplying the target inverse mask image and the similar image so that each value in the target mask image is multiplied by the corresponding pixel in the similar image, resulting in a second similarity mask image. Then, this second similarity mask image is input into the second neural network model to perform local fine-tuning of the second similarity mask image. For ease of understanding, the image adjustment process of the second neural network model can be described with reference to Figure 6. Specifically, the second neural network model encodes the second similarity mask image and introduces the encoding result into the latent space. The encoding result is then forward-diffused in the latent space. This forward-diffusing can be understood as a noise-adding process to obtain a second similarity noise image. Next, before backdiffusion of the second similar noise map, image encoding is performed on the reference image used as a guiding condition to obtain the feature map (i.e., vector matrix) corresponding to the reference image, thus introducing the feature map corresponding to the reference image into the latent space. Then, backdiffusion is performed on the second similar noise map in the latent space, and during the backdiffusion process, the feature map corresponding to the reference image is integrated into the noise of the second similar noise map and backdiffused together to obtain the second feature map. Finally, the second feature map is decoded to restore the first feature map in the latent space to the pixel space for representation, so as to output the second image.

It should be noted that when using the feature map of the reference image as a constraint, the specific processing procedure for backdiffusion of the second similar noise map is the same as the steps in the above-mentioned "backdiffusion processing of the first similar noise map in the denoising U-Net layer". The only difference is the "feature map of the reference image" and the "difference text vector of the difference description text". The backdiffusion process can be understood in conjunction with Figures 6 and 7 and the aforementioned content, and will not be elaborated here.

(B) Masking the image during fine-tuning:

It should be noted that when performing local fine-tuning on similar images, a conditional latent space diffusion (SD) model can be used. Specifically, the similar image and the target mask image are input into the conditional latent space diffusion model, which performs noise diffusion processing on the similar image. During the noise diffusion process, guiding conditions (such as difference description text and reference images) are introduced to assist in instructing the image to make precise fine-tuning of relevant pixel regions, thereby improving the accuracy of image adjustment.

In some implementations, the similar images requiring fine-tuning are first subjected to noise processing at multiple consecutive time steps to obtain two noise images at adjacent time steps. The noise images are then fine-tuned based on the target mask image, difference description text, and a reference image, and the fine-tuning results are fused to obtain the target image. For example, step 105 may include:

(105.B.1) Noise is added to the similar images, and a first similar noise map and a second similar noise map at adjacent time steps are obtained in the noise addition process;

(105.B.2) Based on the target mask image and the difference description text, the first similarity noise image is denoised to obtain the first image;

(105.B.3) Based on the target mask image and the reference image, the second similarity noise image is denoised to obtain the second image;

(105.B.4) The first image and the second image are fused to obtain the adjusted target image.

Specifically, a conditional latent space diffusion (SD) model can be used to fine-tune similar images locally. The similar images and the target mask are fed to the diffusion model, which encodes the similar images to obtain feature maps. These feature maps are then imported into the latent space for forward diffusion. This forward diffusion process involves gradual noise addition over multiple time steps, with each time step considered as one noise addition operation, until a fully noisy image is obtained. Then, the fully noisy image and the noise image from the adjacent previous time step are taken. Specifically, the noise image from the adjacent previous time step can be used as the first similar noise image, and the fully noisy image as the second similar noise image. Further, based on the target mask image and the difference description text, the first similar noise image undergoes backdiffusion processing. This backdiffusion process involves multiple consecutive noise reduction operations to obtain the first image. Similarly, based on the target mask image and a reference image, the second similar noise image undergoes backdiffusion processing to obtain the second image. Finally, the first and second images are fused to obtain the adjusted target image.

For example, as shown in Figure 6, a similar image X, a target mask image, a difference description text, and a reference image are transmitted to a conditional latent space diffusion (SD) model. The diffusion model encodes the similar image X to import the encoding result (feature map Z) into the latent space. Through forward diffusion (noising) processing, assuming that through T time steps of progressive noise processing, a fully noisy noise map "Z_{_T}", which can also be represented as "X_{_T}", is obtained. Thus, the noise map "Z_{_T}" is taken as the second similar noise map, and the noise map "Z_{_T-1}" of the adjacent time step is taken as the first similar noise map. Furthermore, based on the target mask image and the difference description text, the first similarity noise image undergoes backdiffusion processing. This backdiffusion involves noise reduction processing over a corresponding number of time steps, such as "T-1" time steps, and is then decoded to obtain the first image. Similarly, the similarity noise image undergoes backdiffusion processing over a corresponding number of time steps, such as "T-1" time steps, to obtain the first feature map "Z", which is then decoded to obtain the second image. Finally, the first and second images are fused, such as through overlay or stitching, to obtain the adjusted target image.

It should be noted that the noise map at the time step adjacent to the first similar noise map is taken as the second similar noise map. A reference image is introduced in the noise reduction process to guide the noise reduction, so that the features of the reference image and the similar image can be consistent at adjacent time steps "T-1" and "T", such as size and dimensions, which has reliability.

In some implementations, the first image is fine-tuned using the difference description text as a guiding condition. For example, step (105.B.2) may include: masking the first similarity noise map according to the target mask map to obtain a first mask noise map; obtaining the difference text vector corresponding to the difference description text, and performing noise reduction processing on the first similarity noise map according to the difference text vector to obtain a first feature map; and decoding the first feature map to obtain the first image.

For example, the Conditional Latent Diffusion (SD) model, after undergoing multiple noise addition processes to obtain a first similarity noise map "Z _T-1 ", multiplies the target mask map with the first similarity noise map to obtain a first mask noise map. Simultaneously, using the difference description text as a guiding condition, the difference description text is text-encoded to obtain a difference text vector. Then, based on this difference text vector, the first mask noise map is progressively denoised. Specifically, during the denoising process at each time step, an attention mechanism is used to introduce the difference text vector into the noise at that time step. This denoising process continues for "T-1" consecutive time steps until complete denoising is achieved, resulting in a first feature map, as shown in "Z" in Figure 6. Finally, this first feature map is decoded to obtain the first image. It should be noted that the above is only an example; masking can also be performed on the first feature map after complete denoising. The specific timing of the masking process is not limited here.

In some implementations, the second image is obtained by fine-tuning a reference image as a guiding condition. Step (105.B.3) may include: inverting the target mask image to obtain a target inverse mask image corresponding to the target mask image; performing masking processing on the second similarity noise image based on the target inverse mask image to obtain a second mask noise image; performing masking processing on the second mask noise image based on the feature image corresponding to the reference image to obtain a second feature image; and decoding the second feature image to obtain the second image.

For example, after the Conditional Latent Diffusion (SD) model undergoes multiple noise addition processes to obtain the second similarity noise map "Z _T ", it needs to be masked. Since it is necessary to extract information about the differing objects in the reference image and mask information about other non-dissimilar objects in the reference and similar images, the target mask image needs to be inverted to obtain a target inverse mask image that is the opposite of the target mask image. Then, the target inverse mask image is multiplied with the second similarity noise map to obtain the second mask noise map. Simultaneously, using the reference image as a guiding condition, image encoding is performed on the reference image to obtain its feature map. Then, based on this feature map, the second mask noise map is progressively denoised. Specifically, during the denoising process at each time step, an attention mechanism is used to introduce the feature map of the reference image into the noise at that time step. This denoising is continued for "T" consecutive time steps until complete denoising is achieved, resulting in the second feature map, as shown in "Z" in Figure 6. Finally, the second feature map is decoded to obtain the second image. It should be noted that the above is just an example. The second feature map can also be masked after complete denoising. The timing of the masking process is not specified here.

In some implementations, the training process for a conditional latent space diffusion (SD) model is as follows: A sample reference image, sample similar images, and a sample target image are acquired. A sample difference description text and a sample target mask image are also acquired relative to the sample similar images. The sample reference image, sample similar images, sample difference description text, and sample target mask image are then transmitted to a pre-defined SD model. Using the sample difference description text and the sample reference image as guiding conditions, the sample similar images are locally fine-tuned in conjunction with the sample target mask image to obtain a predicted target image. The difference between the predicted target image and the sample target image is then acquired to construct a prediction loss. The pre-defined model is iteratively trained based on this prediction loss until the pre-defined conditions are met, resulting in the trained target model, i.e., the conditional latent space diffusion model.

As can be seen from the above, the embodiments of this application can first obtain similar images to the reference image from existing data, then generate a target mask image based on the difference information between the reference image and the similar image, and expand the difference information to enrich the difference description text representing the difference information. Finally, the existing similar image is locally fine-tuned by combining the target mask, the difference description text, and the reference image to obtain the fine-tuned target image. In this way, the difference description between images can be expanded, and the expanded description text of the difference and the reference image can be used as constraints to locally adjust the similar image, improving the accuracy of image adjustment and making the adjusted image effect conform to the actual needs, so as to facilitate the development of other subsequent businesses.

Based on the method described in the above embodiments, the following examples will provide further detailed explanations.

This application takes image processing as an example to further describe the image processing method provided in this application.

Figure 8 is a flowchart illustrating another step of the image processing method provided in this application embodiment; Figure 9 is a schematic diagram of the framework structure of the image processing system provided in this application embodiment; Figure 10 is a schematic diagram of the structure of the residual network layer provided in this application embodiment; Figure 11 is a schematic diagram of a scenario where difference information is summarized to generate difference description text provided in this application embodiment; and Figure 12 is a schematic diagram of a scenario where image fine-tuning is performed provided in this application embodiment. For ease of understanding, this application embodiment is described in conjunction with Figures 3-12.

In this embodiment, the description will focus on the image processing apparatus, which can be integrated into a computer device such as a server. For example, when the processor on the computer device executes the program corresponding to the image processing method, the specific flow of the image processing method is as follows:

201. Obtain a reference image, and then obtain similar images that are similar to the reference image.

In this embodiment of the application, in order to obtain a target image that is more similar to the reference image, a similar image that is most similar to the reference image can be searched from a preset database so that the searched similar image can be further processed, such as making local fine adjustments to the similar image to obtain a target image that fits the reference image better.

The reference image can be an image containing any content. For example, taking an image query service platform as an example, a customer sends one or more example images to the platform. These example images are reference images in the embodiments of this application. After receiving the example image, the platform can search for similar images similar to the example image from its existing database, so that it can subsequently use the similar image as a basis and combine the differences between the similar image and the reference image to make local adjustments to the similar image.

The similar image can be the image in the database that is most similar to the reference image. It can be understood as the image in the existing data that is most similar to the reference image in terms of image content or image style.

Specifically, to select similar images from a pre-defined database, the process of obtaining similar images can be as follows: First, the reference cluster center corresponding to the reference image can be determined, and the feature class distance between the reference cluster center and each image cluster center in the existing database can be calculated. Further, similar images can be selected based on the feature class distance. Specifically, the magnitude of the feature class distance can be used to determine which image cluster center in the database is more similar to the cluster center of the reference image, thus determining the image class of the similar target image cluster center as the image class of the reference cluster center. Then, the feature distance between each image in the image class of the reference image and the target image cluster center is calculated. It should be noted that the magnitude of the feature distance can reflect the similarity between any two images; therefore, similar images can be selected based on the magnitude of the feature distance, for example, selecting the image with the smallest feature distance to the reference image as the similar image.

202. Determine the differences between the reference image and similar images.

The difference information can be information representing the feature differences between the reference image and similar images, and is not limited to differences such as the number of objects (things) that differ in the image, the location of the objects, and/or the shape of the objects.

Specifically, in order to obtain the difference information between the reference image and similar images, the first descriptive text of the reference image can be obtained through image-to-text conversion, and the second descriptive text of the similar images can be obtained through image-to-text conversion; further, based on the difference between the first descriptive text and the second descriptive text, the difference information between the reference image and the similar images is generated.

To facilitate understanding of the first and second descriptive texts, we will take the first descriptive text as an example. Specifically, the first descriptive text may include global descriptive text for the overall image content in the reference image and object descriptive text for each object in the image.

The global descriptive text generation approach can be implemented based on a pre-trained frozen image encoder and large language models (BLIP2). Specifically, the model can be structurally divided into two parts: vision-and-language representation learning and vision-to-language generative learning. The vision-and-language representation learning includes an image encoder and a lightweight query transformer (Q-Former), while the vision-to-language generative learning includes large language models (LLM). The global descriptive text generation process is as follows: The reference image is input into an image encoder, which encodes the reference image to obtain an image encoding result; then, the image encoding result is input into a lightweight query transformer, and the object category labels in the reference image are determined and fused with the object category labels in the lightweight query transformer to obtain a fused feature result; finally, the fused feature result is input into a large-scale language model for language processing to output a global descriptive text for the reference image.

The object description text generation method can be implemented based on a Generative Region-to-Text Transformer (GRIT). This model structurally includes a Visual Encoder, a Foreground Object Extractor, and a Text Decoder. Specifically, the object description text generation process is as follows: Each object in the reference image is identified to determine its pixel region and class. The reference image with the identified pixel regions and classes is then input into the Generative Region-to-Text Transformer to perform region-to-language conversion. The output is a converted reference image containing bounding boxes for each object and object description text for each object within the bounding box.

This allows us to determine the differences between the reference image and similar images, so that we can use these differences as a basis for adjustments during image processing and improve the accuracy of image adjustments.

203. Determine the target mask image for the difference information in the reference image, and obtain the target inverse mask image that is the opposite of the target mask image.

In the embodiments of this application, the image processing of similar images is mainly carried out by local adjustment. Therefore, it is necessary to obtain a mask image of the difference information in the reference image so as to use the mask image to participate in the local fine adjustment of the similar image, so as to improve the accuracy of subsequent image adjustment.

It should be noted that this target mask image can be a pixel region masking image generated based on the difference information between the reference image and similar images. It is used to mask the target pixel regions at the same locations in the similar image where the differing objects are located, thereby blocking the representation of pixels in the target pixel region in the similar image, making the target pixel region in the similar image appear blank, i.e., containing content. This target mask image is mainly represented by "0" and "1", where the value in the target pixel region is "0", and the value in other regions is "1".

Furthermore, the target inverse mask is the opposite of the target mask, used to allow representation of pixels within the target pixel region where the difference object resides, while rejecting representation of pixels outside the target pixel region. Understandably, in this target inverse mask, the value within the target pixel region is "1", and the values in other regions are "0".

In this way, the target mask map and the target inverse mask map of the difference objects in the reference image can be obtained respectively, which can be used as necessary elements when adjusting similar images, and participate in the local fine-tuning of similar images to improve the accuracy of subsequent image adjustment.

204. Based on the difference information, the text is expanded to obtain the difference description text.

In this embodiment, relevant descriptive text for the difference information in the reference image can be obtained, so that the relevant descriptive text for the difference information can be used as a guiding condition for subsequent image adjustment, so as to improve the accuracy of image adjustment by making local adjustments to similar images.

The difference description text can be text describing the content differences between the reference image and similar images. It is mainly derived from textual expansion based on the difference information, which can enrich and accurately represent the differences between the reference image and similar images. This difference description text can serve as a guiding condition for adjusting similar images, participating in indicating local adjustments to similar images and improving the accuracy of image adjustments.

Specifically, to provide a rich description of the differences between the reference image and similar images, firstly, the object corresponding to the difference in the reference image is identified, and the object relationship information between this object and any other object in the reference image is obtained, such as determining the positional relationship between the current object and other objects in the image, and whether they belong to the same object category (object class). Further, referring to the aforementioned acquisition of the "first descriptive text," the global descriptive text of the reference image and the object description text associated with the object are obtained. Finally, based on the global descriptive text of the reference image, the descriptive text of the object, and the object relationship information between the object and other objects, the text is expanded to provide a more comprehensive description of the differences between the reference image and similar images, resulting in the expanded difference description text. It should be noted that this difference description text richly expresses the relationship between the object and other objects in the reference image, as well as the object state information of the object. Thus, compared to the object description text, this difference description text can more comprehensively represent the relevant information of the object in the reference image.

By using the above methods, rich descriptive text about the differences in the reference image can be obtained. This descriptive text can then be used as a guiding condition to indicate local adjustments to similar images, thereby improving the accuracy of image adjustments.

It should be noted that when adjusting local regions of similar images, the adjustment process includes two parts: First, using the difference description text as a guiding condition, the similar images are locally adjusted in conjunction with the target mask image to obtain the first image, as detailed in step 205; Second, using the reference image as a guiding (constraint) condition, the similar images are locally adjusted in conjunction with the target inverse mask image to obtain the second image, as detailed in step 206.

205. Based on the target mask image and the difference description text, the first neural network model performs local fine-tuning on similar images to generate the first image.

In this embodiment of the application, when performing local fine-tuning on similar images, the first neural network model can be a conditional latent space diffusion (SD) model. Specifically, the similar images are input into the conditional latent space diffusion model, and noise diffusion processing is performed on the similar images through the conditional latent space diffusion model. During the noise diffusion process, guiding conditions are introduced to assist in instructing the image to perform precise fine-tuning on relevant pixel regions, thereby improving the accuracy of image adjustment.

Specifically, before adjusting the similar images using the latent space diffusion model, the similar images are first masked based on the target mask image. This masking process involves multiplying the target mask image and the similar image so that each value in the target mask image is multiplied by the corresponding pixel in the similar image, resulting in a first similarity mask image. Then, this first similarity mask image is input into the latent space diffusion model for local fine-tuning. For ease of understanding, the image adjustment process of this first neural network model can be described in conjunction with Figures 6 and 7. Specifically, the latent space diffusion model encodes the first similarity mask image and introduces the encoding result (first encoded feature map) into the latent space. Forward diffusion is then performed on the encoding result in the latent space. This forward diffusion can be understood as a process of gradually adding noise over multiple time steps to obtain a fully noisy first similarity noise map. Next, before backdiffusion of the first similar noise map, the differential description text, which serves as the guiding condition, is text-encoded to obtain a differential text vector, thus introducing the differential description text into the latent space. Then, in the latent space, a denoising U-Net layer is used to perform backdiffusion on the first similar noise map. This backdiffusion process can be understood as performing multiple denoising processes (multiple time steps). In each denoising process, a ResNet layer is used to extract features from the noise map at the current time step, and the differential text vector is incorporated through an attention mechanism. After multiple time steps of denoising, the first feature map is obtained. Finally, the first feature map is decoded to restore the first feature map in the latent space to the pixel space for pixel representation, thus obtaining the first image.

The first image obtained after local adjustment according to the difference description text is different from the similar image. Specifically, the pixel area corresponding to the difference object in the first image is a blank pixel area, that is, there is no image content.

206. Based on the target inverse mask image and the reference image, the second neural network model performs local fine-tuning on similar images to generate a second image.

Specifically, firstly, the target inverse mask image is multiplied with a similar image so that each value in the target mask image is multiplied with the corresponding pixel in the similar image, resulting in a second similarity mask image. Then, this second similarity mask image is input into a second neural network model, where it is encoded in pixel space to obtain a second encoded feature map. Next, the second encoded feature map is introduced into the latent space, where it undergoes progressive noise addition at multiple time steps to obtain a fully noisy second similarity noise map. Then, the reference image, used as a guiding condition, is image encoded to obtain its corresponding feature map (i.e., a vector matrix). The second similarity noise map is then subjected to noise reduction at multiple time steps in the latent space, where an attention mechanism is used to repeatedly incorporate the reference image's feature map during each noise reduction step. After multiple noise reduction steps, the second feature map is obtained. Finally, the second feature map is decoded, specifically by restoring it from the latent space to pixel space for pixel representation to obtain the second image.

The second image obtained by making local adjustments to the reference image differs from the similar image. Specifically, in the second image, the pixel regions corresponding to non-difference information in the similar image are blank pixel regions, while the pixel regions corresponding to difference information are not blank pixel regions. That is, the second image only contains the image content of the pixel regions corresponding to difference information.

It should be noted that the training process of the first neural network model and the second neural network model can be referred to the description in the foregoing embodiments, and will not be repeated here.

207. Fuse the first image with the second image to obtain the adjusted target image.

In this embodiment, after obtaining a first image adjusted using the difference description text as a guide condition and a second image adjusted using a reference image as a guide condition, the first and second images can be superimposed and fused so that the image content of the pixel area corresponding to the difference information in the second image is superimposed and filled with the blank pixel area corresponding to the difference information in the first image, thereby achieving image content complementarity between the first and second images to obtain a target image after local fine-tuning. This target image, compared to similar images, better meets the image content requirements of the reference image, is more similar to the reference image, and has reliability.

In some implementations, steps 205-206 can also be achieved through the following process: Specifically, a conditional latent space diffusion (SD) model can be used to fine-tune the locality of similar images. The similar images and the target mask are transmitted to the diffusion model. The diffusion model encodes the similar images to obtain feature maps and imports these feature maps into the latent space for forward diffusion. This forward diffusion process involves a gradual noise-adding process over multiple time steps, with each time step considered as one noise-adding process, until a fully noise-generated noise map is obtained. Then, the fully noise-generated noise map and the noise map from the adjacent previous time step can be taken. Specifically, the noise map from the adjacent previous time step can be used as the first similar noise map, and the fully noise-generated noise map can be used as the second similar noise map. Further, based on the target mask map and the difference description text, the first similar noise map is subjected to backdiffusion processing. This backdiffusion processing involves multiple consecutive noise reduction processes to obtain the first image. Based on the target mask map and the reference image, the second similar noise map is subjected to backdiffusion processing to obtain the second image. Finally, the first image and the second image are fused to obtain the adjusted target image. For a description of this implementation method, please refer to the preceding embodiment (B), which will not be repeated here.

To facilitate understanding of the embodiments of this application, specific application scenario examples will be used to describe the embodiments of this application. Specifically, the application scenario example will be described by performing the above steps 201-207 and referring to Figures 3-12.

Specifically, this image processing method is mainly used for scenarios involving local fine-tuning of images. Examples of such scenarios are as follows:

As shown in Figure 9, the image processing system framework includes: a training set (preset database), a feature extraction layer of a residual network (ResNet50), a semantic segmentation layer (Segmenteverything, GRIT, and BLip2), a large-scale language processing model, and a fine-tuned latent space diffusion model (SD).

To facilitate understanding, the image processing process is summarized below with reference to the processing layers in Figure 9:

The process involves extracting similar images from the training set (the currently stored set of images). These similar images are highly similar to the example image but may differ from or be missing key local information (a specific object that is different from the example image). It should be noted that this process of extracting similar images can be implemented using the feature extraction layer of a residual network (ResNet50).

As shown in Figure 10, the ResNet50 serves as the backbone network (base network). Structurally, it includes an encoder, which is a Convolutional Neural Network (CNN). The feature extraction module of this CNN consists of 3 convolutional layers and 6 ResBlocks. For the input image (e.g., the reference image), after passing through three convolutional layers, the width (w) and height (h) of the image are reduced to 1/4 of their original values, and the number of channels increases from 3 to 128, forming a feature map of w/4 * h/4 * 128. This feature map then passes through a sub-network composed of 6 ResBlocks to generate a new high-level semantic feature map. Each ResBlock consists of two convolutional layers and one identity layer. It should be noted that after passing through 6 ResBlocks, the resulting high-level semantic feature map is w*h*c (e.g., w/64 * h/64 * 1024).

Specifically, firstly, the example image provided by the client is input into the feature extraction layer of the residual network (ResNet50) to obtain the high-level semantic feature map of the reference image, and the feature mean of the reference image is obtained based on the high-level semantic information. Similarly, the image data in the training set is traversed in the same way to obtain the high-level semantic feature map of each image in the training set, and its feature mean is obtained. Then, the feature dimension of the high-level semantic feature map of the reference image and the high-level semantic feature map of each image in the training set is reduced, for example, by global max pooling, so that the feature map dimension is 128 dimensions.

Furthermore, the cluster centers Ac of each category in the training set are calculated, as well as the cluster center Bc of the example image. The similarity between the example image and each category in the training set is then calculated using these cluster centers, as shown in the following formula:

Finally, for each example image, we determine whether its similarity "" with the cluster center Ac of each category of images in the training set is greater than the similarity "" between the customer's example image cluster center and the cluster centers of each category of images in the training set. Images with similarity greater than this are identified as similar images for subsequent adjustments.

(2) Obtain semantic information of example image (reference image) and similar image respectively to determine the difference information between them.

Specifically, the global descriptive text of the reference image is obtained through a pre-trained frozen image encoder and a large language model (BLIP2). Similarly, the global descriptive text of similar images is obtained.

Specifically, a Generative Region-to-Text Transformer (GRIT) is used to obtain the object description text for each object in the reference image. Similarly, the object description text for each object in similar images is obtained.

Furthermore, the difference information between the reference image and similar images can be determined based on the differences in global description text and object description text between the reference image and similar images.

In addition, a target mask image for objects corresponding to the difference information is obtained through a mask segmentation model (segmenteverything).

(3) Referring to Figure 11, the difference information is passed to the large-scale language processing model for aggregation and processing, and the model is guided to generate key descriptions of the difference information (i.e., difference description text). Specifically, the large-scale language processing model is allowed to infer the relationships between objects in the image and the information of the objects to obtain high-quality text for the difference objects, i.e., difference description text. It should be noted that the large-scale language processing model can be any type of language processing model, such as "Chat-Gpt", and is not limited here.

(4) Local fine-tuning of similar images mainly includes two parts. Specifically, first, using the difference description text as a prompt, local fine-tuning of similar images is performed; second, using a reference image as a prompt, local fine-tuning of similar images is performed. A detailed introduction to this image fine-tuning is as follows:

Figure 12 illustrates the scene of the image fine-tuning process. Specifically, the purpose of this image fine-tuning is to use a similar image as the input image for the upstream process, and based on the example image provided by the downstream customer, adjust the similar image to a target image that is more similar to the example image. As shown in the figure, the target image does not contain a person's hand, and the knife and fork are placed on a plate, which is more similar to the example image.

First, similar images are input into the upper path and subjected to noise diffusion processing. During diffusion (e.g., denoising or noise reduction), descriptive text describing the differences is introduced as a prompt, which can be understood as a guiding condition. It should be noted that during noise diffusion, a noise diffusion map at a random time step (defined as the first noise map) can be selected and combined with the target mask image to obtain a preliminary image related to the example image through reverse diffusion, i.e., the first image. The upper path diffusion process can be represented as follows:

Then, the lower path performs noise diffusion on similar images in the same manner as the upper path, and takes the noise diffusion map at the next time step relative to the noise diffusion map of the upper path (defined as the second noise map). In the denoising diffusion process, using the example image (reference image) as a constraint, the second noise map is combined with the inverse mask image of the target mask. After denoising, the second image is obtained. It should be noted that two noise maps at adjacent time steps can be selected as the first noise map and the second noise map, respectively. This ensures that the features of the first image obtained from the upper path and the second image obtained from the lower path are consistent, such as having the same image size, so that the first image and the second image can be accurately fused subsequently. The lower path diffusion process can be represented as follows:

Finally, the first image and the second image are fused to obtain the adjusted target image. This image fusion process can be represented as follows:

By executing the scenario steps (1) to (4) above, the following can be achieved: find images in the training set that are relatively similar to the customer data, then input the customer example images into multiple large models (segmenteverything, blip2, grit, etc.) to output corresponding prompts (image descriptions), then use chatgpt to describe these descriptive words, and focus on guiding chatgpt to add details for several key features identified by the user, so as to obtain the final descriptive words; finally, use the SD model to focus on smoothing the differences between similar images and the example images provided by the customer, so that the image is regenerated locally to obtain the target image.

Through the above application scenario examples, the following effects can be achieved: using the differences between the example image and similar images as guiding conditions, and making local fine-tuning to a small number of images provided by the customer, the accuracy of image fine-tuning is improved, so as to obtain a target image that is more consistent with the example image.

As can be seen from the above, the embodiments of this application can first obtain similar images to the reference image from existing data, then generate a target mask image based on the difference information between the reference image and the similar image, and expand the difference information to enrich the difference description text representing the difference information. Finally, the existing similar image is locally fine-tuned by combining the target mask, the difference description text, and the reference image to obtain the fine-tuned target image. In this way, the difference description between images can be expanded, and the expanded description text of the difference and the reference image can be used as constraints to locally adjust the similar image, improving the accuracy of image adjustment and making the adjusted image effect conform to the actual needs, so as to facilitate the development of other subsequent businesses.

To better implement the above methods, embodiments of this application also provide an image processing apparatus. For example, as shown in FIG13, the image processing apparatus may include an acquisition unit 401, a first determination unit 402, a second determination unit 403, an expansion unit 404, and an adjustment unit 405.

The acquisition unit 401 is used to acquire a reference image and acquire similar images that are similar to the reference image;

The first determining unit 402 is used to determine the difference information between the reference image and similar images;

The second determining unit 403 is used to determine the target mask map for the difference information in the reference image;

The expansion unit 404 is used to expand based on the difference information to obtain the difference description text;

The adjustment unit 405 is used to perform local adjustments on similar images based on the target mask image, the difference description text, and the reference image to obtain the adjusted target image.

In some embodiments, the adjustment unit 405 is further configured to: add noise to similar images and obtain a first similar noise map and a second similar noise map at adjacent time steps in the noise addition process; perform denoising processing on the first similar noise map according to the target mask map and the difference description text to obtain a first image; perform denoising processing on the second similar noise map according to the target mask map and the reference image to obtain a second image; and fuse the first image and the second image to obtain the adjusted target image.

In some embodiments, the adjustment unit 405 is further configured to: perform masking processing on the first similar noise map according to the target mask map to obtain a first mask noise map; obtain the difference text vector corresponding to the difference description text, and perform noise reduction processing on the first similar noise map according to the difference text vector to obtain a first feature map; and perform decoding processing on the first feature map to obtain a first image.

In some embodiments, the adjustment unit 405 is further configured to: invert the target mask image to obtain a target inverse mask image corresponding to the target mask image; perform masking processing on the second similar noise image according to the target inverse mask image to obtain a second mask noise image; perform masking processing on the second mask noise image according to the feature image corresponding to the reference image to obtain a second feature image; and perform decoding processing on the second feature image to obtain a second image.

In some embodiments, the adjustment unit 405 is further configured to: perform local fine-tuning on similar images based on the target mask image and the difference description text to obtain a first image; perform local fine-tuning on similar images based on the target mask image and the reference image to obtain a second image; and fuse the first image and the second image to obtain an adjusted target image.

In some embodiments, the adjustment unit 405 is further configured to: perform masking processing on the similar image according to the target mask image to obtain a first similar mask image; perform noise processing on the first similar mask image to obtain a first similar noise image; obtain the difference text vector corresponding to the difference description text; perform noise reduction processing on the first similar noise image according to the difference text vector to obtain a first feature image; and perform decoding processing on the first feature image to obtain a first image.

In some implementations, the adjustment unit 405 is further configured to: perform multiple consecutive noise reduction processes on the first similar noise map, and incorporate the difference text vectors in each noise reduction process through an attention mechanism to obtain the first feature map.

In some embodiments, the adjustment unit 405 is further configured to: encode the first similar mask image to obtain an encoded feature map; and perform noise processing on the encoded feature map to obtain a first similar noise map.

In some embodiments, the adjustment unit 405 is further configured to: perform local fine-tuning on similar images based on the target mask image and the difference description text using a first neural network model to generate a first image;

The image processing device further includes a training unit, used for: acquiring a sample reference image and a sample similar image, as well as a first sample target image; generating a sample target mask image and sample difference description text of the sample reference image based on the difference information between the sample reference image and the sample similar image; performing local fine-tuning on the sample similar image based on the sample target mask image and the sample difference description text using a preset model to generate a first predicted image; determining the prediction loss based on the first sample target image and the first predicted image; and iteratively training the preset model based on the prediction loss until a preset condition is met to obtain a first neural network model.

In some embodiments, the adjustment unit 405 is further configured to: invert the target mask image to obtain a target inverse mask image corresponding to the target mask image; perform masking processing on similar images based on the target inverse mask image to obtain a second similar mask image; perform noise processing on the second similar mask image to obtain a second similar noise image; perform noise reduction processing on the second similar noise image based on the feature map corresponding to the reference image to obtain a second feature map; and perform decoding processing on the second feature map to obtain a second image.

In some embodiments, the expansion unit 404 is further configured to: determine the difference objects of the reference image relative to similar images based on the difference information; determine the object relationship information between the difference objects and other objects in the reference image; obtain the global description text of the reference image and the target object description text of the difference objects; and perform text expansion based on the global description text, the target object description text, and the object relationship information to obtain the difference description text.

In some implementations, the acquisition unit 401 is further configured to: determine the reference cluster center to which the reference image belongs; determine the feature category distance between each image cluster center pre-constructed in the preset database and the reference cluster center; and select similar images for the reference image based on the feature category distance.

In some implementations, the first determining unit 402 is further configured to: obtain a first descriptive text corresponding to a reference image; obtain a second descriptive text corresponding to a similar image; and generate difference information based on the difference between the first descriptive text and the second descriptive text.

In some embodiments, the first determining unit 402 is further configured to: perform a global description of the reference image using a first preset model to generate a global description text for the reference image; process the pixel region where each object in the reference image is located using a second preset model to obtain an object description text corresponding to each object in the reference image; and determine a first description text corresponding to the reference image based on the global description text of the reference image and the object description text corresponding to each object.

In some implementations, the second determining unit 403 is further configured to: determine, based on the difference information, a difference object in the reference image relative to a similar image; and generate a target mask image based on the difference object in the reference image.

As can be seen from the above, the embodiments of this application can obtain similar images to the reference image from existing data. Then, based on the difference information between the reference image and the similar image, a target mask image is generated. Furthermore, the difference information is expanded to enrich the difference description text representing the difference information. Finally, the existing similar image is locally fine-tuned by combining the target mask, the difference description text, and the reference image to obtain the fine-tuned target image. In this way, the difference description between images can be expanded, and the expanded description text and the reference image can be used as constraints to locally adjust the similar image, improving the accuracy of image adjustment and making the adjusted image effect conform to the actual needs, which is conducive to the development of other subsequent services.

This application also provides a computer device, as shown in FIG14, which illustrates a schematic diagram of the structure of the computer device involved in this application embodiment. Specifically:

The computer device may include components such as a processor 501 with one or more processing cores, a memory 502 with one or more computer-readable storage media, a power supply 503, and an input unit 504. Those skilled in the art will understand that the computer device structure shown in FIG14 does not constitute a limitation on the computer device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

The processor 501 is the control center of the computer device, connecting various parts of the computer device through various interfaces and lines. It performs various functions and processes data by running or executing software programs and/or modules stored in the memory 502, and by calling data stored in the memory 502. Optionally, the processor 501 may include one or more processing cores; preferably, the processor 501 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 501.

The memory 502 can be used to store software programs and modules. The processor 501 executes various functional applications and image processing processes by running the software programs and modules stored in the memory 502. The memory 502 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 502 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 502 may also include a memory controller to provide the processor 501 with access to the memory 502.

The computer equipment also includes a power supply 503 that supplies power to the various components. Preferably, the power supply 503 can be logically connected to the processor 501 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 503 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

The computer device may also include an input unit 504, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

Although not shown, the computer device may also include a display unit, etc., which will not be described in detail here. Specifically, in the embodiments of this application, the processor 501 in the computer device loads the executable files corresponding to the processes of one or more application programs into the memory 502 according to the following instructions, and the processor 501 runs the application programs stored in the memory 502 to realize various functions, as follows:

Acquire a reference image and similar images; determine the difference information between the reference image and the similar images; determine the target mask image in the reference image based on the difference information; expand based on the difference information to obtain the difference description text; perform local adjustments on the similar images according to the target mask image, the difference description text, and the reference image to obtain the adjusted target image.

For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

Therefore, this solution can obtain similar images from existing data that are similar to the reference image. Then, based on the difference information between the reference image and the similar image, a target mask image is generated. Furthermore, the difference information is expanded to enrich the difference description text representing the difference information. Finally, the existing similar images are locally fine-tuned by combining the target mask, the difference description text, and the reference image to obtain the fine-tuned target image. In this way, the difference description between images can be expanded, and the expanded difference description text and the reference image can be used as constraints to locally adjust similar images, improving the accuracy of image adjustment and ensuring that the adjusted image effect meets the actual requirements, thus facilitating the development of other subsequent businesses.

Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the image processing methods provided in embodiments of this application. For example, the instructions can execute the following steps:

Acquire a reference image and similar images; determine the difference information between the reference image and the similar images; determine the target mask image in the reference image based on the difference information; expand based on the difference information to obtain the difference description text; perform local adjustments on the similar images according to the target mask image, the difference description text, and the reference image to obtain the adjusted target image.

For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

Since the instructions stored in the computer-readable storage medium can execute the steps of any of the image processing methods provided in the embodiments of this application, the beneficial effects that any of the image processing methods provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations of the above embodiments.

The foregoing has provided a detailed description of an image processing method, apparatus, device, and computer-readable storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims (19)

1. An image processing method, characterized in that it comprises: Obtain a reference image, and then obtain similar images that are similar to the reference image; Determine the difference information between the reference image and the similar image; Determine the target mask image in the reference image for the difference information; Based on the aforementioned difference information, the difference description text is obtained by expanding upon it. Based on the target mask image, the difference description text, and the reference image, the similar image is locally adjusted to obtain the adjusted target image. 2. The method according to claim 1, characterized in that, the step of locally adjusting the similar image based on the target mask image, the difference description text, and the reference image to obtain the adjusted target image includes: The similar images are subjected to noise processing, and a first similar noise map and a second similar noise map at adjacent time steps are obtained in the noise processing. Based on the target mask image and the difference description text, the first similarity noise image is denoised to obtain the first image; Based on the target mask image and the reference image, the second similarity noise image is denoised to obtain the second image; The first image and the second image are fused together to obtain the adjusted target image. 3. The method according to claim 2, characterized in that, the step of performing denoising processing on the first similarity noise map based on the target mask map and the difference description text to obtain the first image includes: The first similar noise map is masked according to the target mask map to obtain the first mask noise map; Obtain the difference text vector corresponding to the difference description text, and perform noise reduction processing on the first similarity noise map based on the difference text vector to obtain the first feature map; The first feature map is decoded to obtain the first image. 4. The method according to claim 2, characterized in that, the step of performing denoising processing on the second similarity noise map based on the target mask map and the reference image to obtain the second image includes: Invert the target mask image to obtain the target inverse mask image corresponding to the target mask image; The second similar noise map is masked according to the target inverse mask map to obtain the second masked noise map; The second mask noise map is masked according to the feature map corresponding to the reference image to obtain the second feature map; The second feature map is decoded to obtain the second image. 5. The method according to claim 1, characterized in that, based on the target mask image, the difference description text, and the reference image, local adjustments are made to the similar image to obtain the adjusted target image, comprising: Based on the target mask image and the difference description text, the similar images are locally fine-tuned to obtain the first image; Based on the target mask image and the reference image, the similar image is locally fine-tuned to obtain the second image; The first image and the second image are fused together to obtain the adjusted target image. 6. The method according to claim 5, characterized in that, the step of locally fine-tuning the similar image based on the target mask image and the difference description text to obtain the first image includes: The similar image is masked according to the target mask image to obtain a first similar mask image; The first similarity mask image is subjected to noise processing to obtain a first similarity noise image; Obtain the difference text vector corresponding to the difference description text; The first similarity noise map is denoised based on the difference text vector to obtain the first feature map; The first feature map is decoded to obtain the first image. 7. The method according to claim 6, wherein the step of performing noise reduction processing on the first similarity noise map based on the difference text vector to obtain the first feature map includes: The first similar noise map is subjected to multiple consecutive noise reduction processes, and the difference text vector is incorporated into each noise reduction process through an attention mechanism to obtain the first feature map. 8. The method according to claim 6, wherein adding noise to the first similar mask image to obtain a first similar noise map comprises: The first similar mask image is encoded to obtain an encoded feature map; The encoded feature map is subjected to noise processing to obtain a first similar noise map. 9. The method according to claim 5, characterized in that, the step of locally fine-tuning the similar image based on the target mask image and the difference description text to obtain the first image includes: The first image is generated by locally fine-tuning the similar image based on the target mask image and the difference description text using a first neural network model. Before generating the first image by locally fine-tuning the similar image based on the target mask image and the difference description text using a first neural network model, the method further includes: Obtain the sample reference image and sample similar images, as well as the first sample target image; Based on the difference information between the sample reference image and the sample similar image, a sample target mask image and sample difference description text are generated for the sample reference image; Based on the sample target mask image and the sample difference description text, the sample similar image is locally fine-tuned using a preset model to generate a first predicted image. Based on the difference between the first sample target image and the first predicted image, a prediction loss is constructed; The preset model is iteratively trained based on the prediction loss until the preset conditions are met, resulting in the first trained neural network model. 10. The method according to claim 6, wherein the step of locally fine-tuning the similar image based on the target mask image and the reference image to obtain a second image comprises: Invert the target mask image to obtain the target inverse mask image corresponding to the target mask image; The similar image is masked according to the target inverse mask image to obtain a second similar mask image; The second similar mask image is subjected to noise processing to obtain a second similar noise map, wherein the time step of the second similar noise map is adjacent to the time step of the first similar noise map. The second similarity noise map is denoised based on the feature map corresponding to the reference image to obtain the second feature map; The second feature map is decoded to obtain the second image. 11. The method according to claim 10, wherein the step of performing noise reduction processing on the second similarity noise map based on the feature map corresponding to the reference image to obtain the second feature map includes: The second similar noise map is subjected to multiple consecutive noise reduction processes, and the feature map corresponding to the reference image is incorporated into each noise reduction process through an attention mechanism to obtain the second feature map. 12. The method according to claim 1, wherein the step of expanding based on the difference information to obtain the difference description text includes: Based on the difference information, determine the difference objects between the reference image and the similar image; Determine the object relationship information between the differing object and other objects in the reference image; Obtain the global description text of the reference image and the target object description text of the difference object; Based on the global description text, the target object description text, and the object relationship information, a textual description is performed to obtain the difference description text. 13. The method according to claim 1, wherein obtaining a similar image to the reference image comprises: Determine the reference cluster center to which the reference image belongs; Determine the feature category distance between each pre-constructed image cluster center in the preset database and the reference cluster center; Based on the feature category distance, similar images are selected for the reference image. 14. The method according to claim 1, wherein determining the difference information between the reference image and the similar image comprises: Obtain the first descriptive text corresponding to the reference image; Obtain the second descriptive text corresponding to the similar image; Based on the differences between the first description text and the second description text, difference information is generated. 15. The method according to claim 14, wherein obtaining the first descriptive text corresponding to the reference image comprises: The reference image is globally described using a first preset model, generating a global description text for the reference image; The pixel region where each object is located in the reference image is processed by the second preset model to obtain the object description text corresponding to each object in the reference image; The first description text corresponding to the reference image is determined based on the global description text of the reference image and the object description text corresponding to each object. 16. The method according to claim 1, wherein determining the target mask map in the reference image for the difference information comprises: Based on the difference information, determine the difference objects in the reference image relative to the similar image; A target mask image is generated based on the differences in the reference image. 17. An image processing apparatus, characterized in that it comprises: An acquisition unit is used to acquire a reference image and acquire similar images that are similar to the reference image; The first determining unit is used to determine the difference information between the reference image and the similar image; The second determining unit is used to determine the target mask image in the reference image for the difference information; An expansion unit is used to represent the difference information in text to obtain a difference description text. The adjustment unit is used to perform local adjustments on the similar image based on the target mask image, the difference description text, and the reference image to obtain the adjusted target image. 18. A computer device, characterized in that it comprises a processor and a memory, the memory storing a computer program, the processor being configured to run the computer program in the memory to implement the steps of the image processing method according to any one of claims 1 to 16. 19. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the steps of the image processing method according to any one of claims 1 to 16.