CN112102303A - Semantic image analogy method for generating countermeasure network based on single image - Google Patents

Abstract

The invention discloses a semantic image analogy method for generating an anti-network based on a single image, and the technical scheme provided by the invention can be seen that a generation model special for a given image can be trained under the condition of giving any image and a semantic segmentation image thereof, the model can recombine a source image according to different expected semantic layouts to generate an image conforming to a target semantic layout, and the semantic image analogy effect is achieved. The visual quality and the conformity accuracy of the result generated by the method are optimal.

Description

A Semantic Image Analogy Method Based on Single Image Generative Adversarial Network

technical field

The invention relates to the technical field of image processing, in particular to a semantic image analogy method based on a single-image generation confrontation network.

Background technique

Generative models such as Variational Auto-Encoder (VAE) and Generative Adversarial Network (GAN) have made great strides in modeling natural image layouts in a generative manner. By taking additional signals such as class labels, text, edges or segmentation maps as input, conditional generative models can generate photo-realistic samples in a controllable manner, which is useful in many multimedia applications such as interaction design and artistic style transfer. Very useful.

Specifically, segmentation maps provide dense pixel-level guidance for generative models and enable users to spatially control expected instances, which is much more flexible than image-level guidance like class labels or styles.

The Pix2Pix model proposed by Isola et al. shows the ability of conditional GANs to generate controllable images given dense conditional signals, including sketches and segmentation maps (Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image- to-Image Translation with Conditional Adversarial Networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).5967–5976). Wang et al. extended the above framework with a coarse-to-fine generator and a multi-scale discriminator to generate images with high-resolution details (Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, JanKautz, and Bryan Catanzaro. 2018. High-Resolution Image Synthesis and Semantic Manipulation With Conditional GANs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8798–8807). Park et al. proposed a spatially adaptive normalization technique (SPADE) that uses semantic maps to predict affine transformation parameters to modulate activations in normalization layers (Taesung Park, Ming-Yu Liu, Ting - Chun Wang, and Jun-Yan Zhu. 2019. Semantic Image Synthesis With Spatially-AdaptiveNormalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2337–2346). Typically, these methods require a large training dataset to map segmentation class labels to patch appearances across the dataset. However, the occurrence of a label instance in the generated images is limited to the appearance of that label in the training dataset, thus limiting the generalization ability of these models on random natural images.

On the other hand, recent work on single-image GANs has shown that it is possible to learn generative models from the internal patch layout of a single image. InGAN defines resizing transformations and trains a generative model to capture internal patch statistics for retargeting (Assaf Shocher, Shai Bagon, Phillip Isola, and Michal Irani. 2019. InGAN: Capturing and Retargeting the "DNA" of a Natural Image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV. 4491–4500). SinGAN utilizes a multi-stage training scheme to generate unconditional images that can generate images of arbitrary size from noise (Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. 2019. SinGAN: Learning a Generative Model From a Single Natural Image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 4569–4579). KernelGAN uses a deep linear generator and constrains it to learn image-specific degradation kernels for blind super-resolution (Sefi Bell-Kligler, Assaf Shocher, and Michal Irani. 2019. Blind Super-Resolution Kernel Estimation using an Internal- GAN. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems (NeurIPS). 284–293). Although these image-specific GANs are dataset-independent and yield promising results, the semantic meaning of patches within a single image remains poorly explored.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a semantic image analogy method based on a single-image generative confrontation network, and the resultant visual quality and coincidence accuracy are optimized.

The purpose of this invention is to realize through the following technical solutions:

A semantic image analogy method based on a single-image generative confrontation network is implemented by a network model composed of an encoder, a generator, an auxiliary classifier and a discriminator; wherein:

Training phase: During each training iteration, for a given source image and the corresponding source semantic segmentation image, perform the same random expansion operation to obtain the corresponding enhanced image and enhanced semantic segmentation image; for the source semantic segmentation image and the enhanced semantic segmentation image The segmented images extract their respective feature tensors through the same encoder, and then use the semantic feature conversion module in the generator to predict the transformation parameters in the image domain based on the two feature tensors, so as to combine the source images under the guidance of the transformation parameters to generate the target. image; the target image will be input to the discriminator and the auxiliary classifier, respectively, to predict the score map of the target image and the enhanced image and the target semantic segmentation image corresponding to the target image; using the appearance similarity loss between the target image and the source image, based on The target image and the enhanced image feature matching loss obtained from the score map, and the semantic alignment loss between the target semantic segmentation image and the enhanced semantic segmentation image construct a total loss function for training;

Inference stage: input the source image, the corresponding source semantic segmentation image, and the specified semantic segmentation image to the semantic image analogy network, and output an image with the same semantic layout as the specified semantic segmentation image.

It can be seen from the above technical solutions provided by the present invention that, given any image and its semantic segmentation map, a generative model dedicated to a given image can be trained, and the model can perform a generation model on the source image according to the desired semantic layout. Recombine to generate images that conform to the target semantic layout, and achieve the effect of semantic image analogy.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

FIG. 1 is a conceptual diagram of a semantic image analogy provided by an embodiment of the present invention;

2 is a schematic diagram of a semantic image analogy method based on a single-image generative adversarial network provided by an embodiment of the present invention;

Fig. 3 is the calculation flow chart of the SFT module provided for the embodiment of the present invention;

Fig. 4 is the visual effect comparison of the image generation effect of the present invention and the existing image analogy method provided for the embodiment of the present invention;

5 is a comparison between the image generation effect of the present invention and the visual effect of the existing single-image GAN method provided for the embodiment of the present invention;

6 is a visual effect comparison between the image generation effect of the present invention and the existing semantic image translation method provided for the embodiment of the present invention;

7 is a visual effect of the present invention on a semantic image analogy task provided for an embodiment of the present invention;

8 is a visual effect of the present invention on an image object removal task provided for an embodiment of the present invention;

9 is a visual effect of the present invention on a face editing task provided for an embodiment of the present invention;

FIG. 10 is a visual effect of the present invention on an edge-to-image translation task provided for an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present invention.

Embodiments of the present invention provide a semantic image analogy method based on a single-image generative adversarial network. Through a conditional single-image GAN, a generative model is trained, and the model generates semantics by segmenting labels in the source image itself rather than in an external data set. Controllable images. Name this task "Semantic Image Analogies", as "Image Analogies" (Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David Salesin. 2001. Image analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques .327–340) and is defined below.

Given a source image I and its corresponding semantic segmentation map P, and some other semantic segmentation maps P', synthesize a new target image I' such that:

In the above formula, :: represents the category relationship.

As shown in Figure 1, the target image I' (the four images in the dashed box) should match both the appearance of the source image I and the layout of the target partition P' (the four segmented images in the dashed box). The task set aims to find a similar transition from I to I' in the same way as from P to P'. Furthermore, two metrics are used to evaluate the quality of images generated from the semantic image analogy model: image patch-level distance and semantic alignment score. The former restricts the original image I to be the only source of patches for the generated image I', while the latter enforces that the generated image I' must have a semantic layout aligned with the target segmentation map P'.

In practice, one can edit the source segmentation map P or provide another image with similar context to obtain the target segmentation map P'. The generator can then generate a semantically aligned target image I' from the source image I, similar to how P' is obtained from P. Comparison with existing methods shows that the method provided by the present invention has advantages in both quantitative and qualitative evaluation. Due to the flexible task setup, the proposed method can be easily extended to various applications including object removal, face editing and sketch-to-image generation of natural images.

As shown in Figure 2, the present invention provides a main principle of a semantic image analogy method based on a single-image generative adversarial network, which is realized by a network model composed of an encoder, a generator, an auxiliary classifier and a discriminator; wherein:

Training Phase: We design a self-supervised framework for training conditional GANs from a single image. During each training iteration, for a given source image I _source and the corresponding source semantic segmentation image P _source , the same random expansion operation is performed to obtain the corresponding enhanced image I _aug and the enhanced semantic segmentation image P _aug ; for the source semantic segmentation image P aug ; The segmentation image and the enhanced semantic segmentation image extract their respective feature tensors through the same encoder, and then predict the transformation parameters in the image domain based on the two feature tensors through the semantic feature transformation module in the generator, so as to guide the transformation parameters. The target image I _target is generated in conjunction with the source image below; the target image will be input to the discriminator and the auxiliary classifier respectively, and the target semantic segmentation image corresponding to the score map of the target image and the enhanced image and the target image is predicted separately; Utilize the difference between the target image and the source image The appearance similarity loss between the two, the feature matching loss between the target image and the enhanced image based on the score map, and the semantic alignment loss between the target semantic segmentation image and the enhanced semantic segmentation image construct a total loss function for training;

During training, gradually increase the randomness of the augmentation. Since the generator is homomorphic, the source image can be reconstructed well when P _target is the same as P _source . The P _target here is a general representation, and actually P _target =P _aug in training, so the two can be mixed in the following training process.

In the embodiment of the present invention, two modes of sampling and reconstruction are used for alternate training. In the sampling mode, which is the method described above, the generator is guided by the enhanced semantic segmentation image to generate a target image with the same appearance as the enhanced semantic segmentation image I _aug and the same semantic layout as the enhanced semantic segmentation image P _aug . The working process of the reconstruction mode is the same as that of the sampling mode. The input is the given source image and the corresponding source semantically segmented image, and the source image is reconstructed by using the source semantically segmented image.

Inference stage: input the source image, the corresponding source semantic segmentation image, and the specified semantic segmentation image to the semantic image analogy network, and output an image with the same semantic layout as the specified semantic segmentation image.

After training, the network model can generate a target image that matches the given semantic segmentation map given a semantic segmentation map of arbitrary shape layout, which not only retains the content information of the source image, but also matches the target semantic layout. As shown in Figure 1, the trained network model can change the shape of the source image horse according to the given shape.

For ease of understanding, the principles and processes of the present invention are described in detail below.

The technical principle of the present invention is a single-image generation adversarial network. For a single image, train a generative adversarial network (ie, a generative model) conditioned on its semantic segmentation map, which mainly includes the generator, auxiliary classifier and discriminator mentioned above, and adopts a series of novel designs to establish semantic segmentation. Semantic associations between graphs and image pixels, and then use this association to recombine images through semantic segmentation graphs.

The semantic image analogy task is transformed into a patch-level layout matching problem, and the transformation is guided in the semantic segmentation domain. To do this, three main challenges need to be addressed: paired data sources to train generative models from a single image, conditional schemes that provide guidance from the segmentation domain to the image domain, and proper supervision of the generated samples (i.e., the generator's output). To accomplish this task, a novel approach is proposed that integrates the following three essential parts:

1) A self-supervised training framework with a progressive data augmentation strategy is designed. A conditional GAN is successfully trained from a single image by alternating optimization with augmented and raw segmentation, which generalizes well for unseen transformations.

2) A semantic feature transformation module is designed, which transforms the transformation parameters from the segmentation domain to the image domain.

3) A semantic-aware patch consistency loss is designed, which encourages the transformed image to contain only patches from the source image. Together with semantic alignment constraints, it enables our generator to generate realistic images with the target semantic layout.

As shown in Figure 1, the main steps in the training phase include:

Step 1. Given the source image I _source and the corresponding source semantic segmentation image P _source , first perform random expansion to obtain the enhanced image I _aug and the enhanced semantic segmentation image P _aug , and then the source semantic segmentation image P _source and the enhanced semantic segmentation image P aug . _Paug is input to the same encoder E (ie, E _seg in Fig. 1 ) to extract features separately.

In this embodiment of the present invention, the random expansion operation includes a combination of one or more of the following operations: random flip, size adjustment, rotation, and cropping. As the training step linearly increases the randomness of these operations, this progressive strategy can help the encoder learn the appearance of the source images in early iterations of training.

Step 2. A Semantic Feature Translation (SFT) module is designed to predict the transformation parameters in the image domain from the feature tensor.

The transformation parameters are explicitly transformed from the segmentation domain to the image domain by the SFT module, as shown in Figure 3. The transformation from the source semantically segmented image P _source to the augmented semantically segmented image P _aug is modeled as a feature-level linear transformation. Therefore, the feature tensor F _source of the source semantic segmentation image and the feature tensor F _aug of the enhanced semantic segmentation image are compared and compared element by element, and the obtained feature scaling tensor F _scale and feature shift tensor F _shift are In subsequent downsampling stages, for the l-th downsampling stage, calculate:

分别为第l个下采样阶段中从特征张量F_aug、特征张量F_source提取出的特征张量；例如，下采样次数为K，则两个特征张量被划分为K个部分，每一个下采样阶段取出相应部分带入上述进行计算。in,

are the feature tensors extracted from the feature tensor F _aug and the feature tensor F _source in the l-th downsampling stage respectively; for example, if the number of downsampling is K, the two feature tensors are divided into K parts, each A downsampling stage takes out the corresponding part and brings it into the above calculation.

和特征移位张量

来近似作为分割图变换的缩放因子

和移位因子，如图3所示，可以使用两个SFT单元对从分割域到图像域的转换过程进行建模，分别处理

和

得到图像域的缩放因子和移位因子

SFT单元的参数通过训练过程学习得到。Scale tensors using features

and feature shift tensor

to approximate the scaling factor as the split map transform

and the shift factor, as shown in Figure 3, can be used to model the transformation process from segmentation domain to image domain using two SFT units, which are processed separately

and

Get the scale factor and shift factor of the image domain

The parameters of the SFT unit are learned through the training process.

的指导下映射到目标图像。Step 3, from the SFT module, the scaling factor and shift factor (γ _img , β _img ) of the image domain are obtained, and the encoder-decoder part of the generator G is in

map to the target image under the guidance of .

For each downsampling stage l+1 in the generator, the output feature tensor is given by:

Among them, DS represents the downsampling module (ie encoder), and mean and std represent the mean and standard deviation, respectively;

The upsampling module of the generator (ie the decoder) then maps the image feature tensor output from the downsampling stage to the image domain to generate the target image. In this embodiment of the present invention, the generator is an encoder-decoder structure with K downsampling blocks and K upsampling blocks; each block includes a 3×3 convolutional layer with a stride of 3, and a stride of 3 A 4×4 convolutional layer of 2 or a transposed convolutional layer for downsampling or upsampling, and each block also uses spectral normalization, batch normalization, and LeakyReLU activation operations. Exemplarily, the starting number of channels is 32, which is doubled during downsampling.

然后按上式输出

而上采样块的输入是

输出是目标图像I_target。Exemplarily, K=3 can be set, and each stage of the three downsampling blocks will receive

Then press the above formula to output

while the input to the upsampling block is

The output is the target image I _target .

Step 4. The discriminator D takes the enhanced image I _aug as a real sample, and takes the generated target image I _target as a pseudo sample. At the same time, the generated image is also input into an auxiliary classifier S to predict its segmentation map.

In the embodiment of the present invention, the discriminator is a fully convolutional PatchGAN (Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. In Proceedings of the IEEE/ CVF Conference on Computer Vision and Pattern Recognition (CVPR).5967–5976), which predicts score maps to distinguish real and fake samples.

In the embodiment of the present invention, the DeepLabV3 architecture (Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. 2018. Encoder-Decoder with Atrous Separable Convolution for Semantic Image) is used in the auxiliary classifier (Segmentation Network in FIG. 2 ). A simplified version of Segmentation.In Proceedings of the European Conference on Computer Vision (ECCV), Vol.11211.833–851) for semantic segmentation.

Step 5. Construct the self-supervised network designed for loss function training.

According to the task setting of semantic image analogy, the generated images should meet the following requirements: 1) Consistent with the source image content; 2) The semantic layout is aligned with the target segmentation map. Therefore, Patch Cohenrence Loss is proposed to measure the appearance similarity between the generated image and the source image. And propose Semantic Alignment Loss, which is measured by the consistency between the target semantic segmentation image predicted from the target image by the auxiliary classifier and the source semantic segmentation image P _source . Specifically:

1) The appearance similarity between the generated image and the source image is measured by the patch coherence loss, this constraint will adversely affect the generator G if the generator generates a corresponding patch that cannot be found in the source image , defined as the average of the lower bounds of the patch distances between the source and destination images:

Among them, N _target is the number of image blocks in the target image I _target , I _source represents the source image, G(I _source )=I _target ; U _class and V _class represent the segmentation labels of the image blocks UP and VQ, and d( ) is distance metric function. This loss relaxes the position dependence of pixel distance. Instead, we treat images as bags of visual features. For each patch in the target image, a nearest neighbor search is run to find the most similar patch from the source image with the same class label, and their distances are averaged. We have empirically found that although other feature descriptors are also suitable, from a pretrained VGG network (Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Net-works for Large-Scale Image Recognition. In Proceedings of the 3rd International Conference on Learning Representations (ICLR)) will yield good results.

2) Use an auxiliary classifier to predict the segmentation map of the target image (ie, the target semantic segmentation image). Then, the cross-entropy (CE) loss between the predicted segmentation map and the enhanced segmentation map is calculated. The semantic alignment loss of the generator is defined as:

Among them, CE represents cross entropy loss, where P _predict =S(G(I _source ))=S(I _target ), and P _predict is the target semantic segmentation image (the output of the auxiliary classifier S).

作为对抗约束，并从鉴别器中获取特征以计算增强图像与生成的图像之间的特征匹配损失

图像。3) Use the least squares GAN loss

as an adversarial constraint and take features from the discriminator to compute the feature matching loss between the augmented image and the generated image

image.

The total loss function is:

表示外观相似度损失，

表示语义对齐损失；λ_seg、λ_GAN和λ_fm均为超参数，实验中均设置为1.0。in,

represents the appearance similarity loss,

represents the semantic alignment loss; λ _seg , λ _GAN and λ _fm are hyperparameters, which are all set to 1.0 in the experiments.

In the embodiment of the present invention, two modes of sampling and reconstruction are used for alternate training;

The sampling mode is also the steps 1 to 5 introduced above, the generator is guided by the enhanced semantic segmentation image to generate a target image with the same appearance as the enhanced semantic segmentation image I _aug and the same semantic layout as the enhanced semantic segmentation image P _aug .

外观相似度损失被替换为输出的重构图像和源图像之间的L1重建损失，特征匹配损失及语义对齐损失各自为目标图像与源图像之间、目标语义分割图像与源语义分割图像之间的损失。The working process of the reconstruction mode is the same as that of the sampling mode, the difference is that there is no need to perform the random operation in step 1, the input is a given source image and the corresponding source semantic segmentation image, and the source semantic segmentation image is used to reconstruct the source image; There is a slight difference in the loss function, that is, there is less appearance similarity loss

The appearance similarity loss is replaced by the L1 reconstruction loss between the output reconstructed image and the source image, and the feature matching loss and semantic alignment loss are respectively between the target image and the source image, and between the target semantic segmentation image and the source semantic segmentation image. Loss.

In the inference stage, the parameters of the network model have been fixed. At this time, the source image I _source and the corresponding source semantic segmentation image P _source are given, as well as the specified semantic segmentation image (which can be obtained by editing P _source , or by other methods); then , input the two semantically segmented images to the encoder E, and then perform steps 2 to 3, the obtained images are consistent with the specified semantically segmented images, and the content information of the source image is retained.

In order to verify the effectiveness of the present invention, the performance of the above method of the present invention is evaluated in terms of quantitative indicators and visual effects, respectively.

Apply the semantic image analogy task to images from different datasets, including COCO-Stuff, ADE20K, CelebAMask-HQ, and the network (i.e., randomly selected natural pictures in the network). The results of the above method of the present invention and the comparison method are evaluated in the following two aspects: 1) the appearance similarity between the source image and the target image; 2) the semantic consistency of the target image and the target segmentation map.

In order to evaluate the appearance similarity of the generated image to the source image, a user study was conducted in the following way. 10 pairs of images with the same class label are randomly selected from the COCO-Stuff dataset. For each pair of images, with one image as the source image and the other image used to provide the target layout, we use image analogies (Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David Salesin. 2001. Image analogies. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. 327–340, IA) and Deep Image Analogy (Jing Liao, Yuan Yao, Lu Yuan, Gang Hua, and Sing Bing Kang. 2017. Visual attribute transfer through deep imageanalogy. ACM Trans .Graph.36, 4 (2017), 120:1–120:15, DIA) method to transfer a source image into the layout of another image. IA and DIA are the two works most relevant to the above method of the present invention. DIA requires a pair of images as source and target, while the above method and IA of the present invention only require one source image and two segmentation maps. Results were presented in random order and 20 users were asked to rank appearance similarity, with the source image as a reference. Then, calculate the average ranking (Avg.User Ranking) of each method among all images and users. Table 1 shows the superiority of the above method of the present invention (Our) over two competitors.

Table 1. Performance and user subjective evaluation performance under the semantic alignment index

To evaluate the semantic consistency of the generated image with the target segmentation map, the panoptic segmentation model of Detectron2 is used to predict the segmentation map of the generated image, and then the pixel-wise accuracy (Pixel Accuracy) and the mean intersection ratio (mIOU) of the target segmentation are calculated. The images used for evaluation are the same as those in the user study. As shown in Table 1, this method achieves the highest accuracy.

In Figure 4, Figure 5 and Figure 6, with the current optimal image analogy algorithms IA and DIA, single-image generative adversarial network model SinGAN (Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. 2019. SinGAN: Learninga Generative Model From a Single Natural Image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 4569–4579) and a segmentation map-to-image translation model SPADE (Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-YanZhu .2019. Visual quality comparison by Semantic Image Synthesis With Spatially-Adaptive Normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2337–2346), the above method of the present invention will generate consistent content and aligned semantic distribution As a result, when the source and target images are semantically dissimilar, IA tends to be filled with duplicate textures, and DIA produces unrealistic results. Without considering the semantic structure, SinGAN editing often changes unedited regions and produces undesired textures, or just blurs pasted objects, which leads to very similar versions of the edited results. While SPADE generates results that are semantically consistent with the target layout, their content is limited to the training dataset and loses the appearance of the source images. The images produced by our method are faithful to the source image in appearance and semantically consistent with the target layout.

The above method of the present invention can perform semantic processing on the image through the segmentation map of the image. Instances can be moved, resized or deleted in the source semantic segmentation map to obtain the target layout. As shown in Fig. 7, the above method of the present invention produces high-quality results with arbitrary semantic changes, while well preserving the local appearance of the changed instances.

Our flexible semantic image analogy task setup enables a variety of applications. Due to the dense conditional input, patches in the image can be regrouped using pixel-level control. In Fig. 8, Fig. 9 and Fig. 10, three applications of the above-mentioned method of the present invention are shown, including 1) object removal, in which it can be easily removed by modifying the class label in the semantic segmentation map to the background class , 2) face editing, where the face image can be edited by changing the shape of the face in the segmentation map, and 3) edge-to-image generation, where other spatial conditions such as edge maps can be used as conditional inputs.

From the description of the above embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software or by means of software plus a necessary general hardware platform. Based on this understanding, the technical solutions of the above embodiments may be embodied in the form of software products, and the software products may be stored in a non-volatile storage medium (which may be CD-ROM, U disk, mobile hard disk, etc.), including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

The above description is only a preferred embodiment of the present invention, but the protection scope of the present invention is not limited to this. Substitutions should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (8)

1. A semantic image analogy method for generating an anti-collision network based on a single image is characterized by being realized by a network model consisting of an encoder, a generator, an auxiliary classifier and a discriminator; wherein:

a training stage: during each training iteration, carrying out the same random expansion operation on a given source image and a corresponding source semantic segmentation image to obtain a corresponding enhanced image and an enhanced semantic segmentation image; extracting respective feature tensors of the source semantic segmentation image and the enhanced semantic segmentation image through the same encoder, and predicting transformation parameters in an image domain based on the two feature tensors through a semantic feature conversion module in the generator so as to generate a target image by combining the source image under the guidance of the transformation parameters; the target image is respectively input into the discriminator and the auxiliary classifier, and the score map of the target image and the enhanced image and the target semantic segmentation image corresponding to the target image are respectively predicted; constructing a total loss function by using the appearance similarity loss between a target image and a source image, the feature matching loss between the target image and an enhanced image obtained based on a score map, and the semantic alignment loss between the target semantic segmentation image and the enhanced semantic segmentation image for training;

and (3) an inference stage: and inputting the source image, the corresponding source semantic segmentation image and the appointed semantic segmentation image into a semantic image analogy network, and outputting an image with the same semantic layout as the appointed semantic segmentation image.

2. The method of claim 1, wherein the stochastic augmentation operation comprises one or more of the following operations: random turning, size adjustment, rotation and cutting.

3. The method according to claim 1, wherein the predicting transformation parameters in the image domain based on two feature tensors by the semantic feature transformation module in the generator comprises: feature tensor F for source semantic segmentation image_sourceAnd enhancing the feature tensor F of the semantically segmented image_augPerforming element-by-element comparison and subtraction to obtain a feature scaling tensor F_scaleAnd an eigenshift tensor F_shiftFor a subsequent down-sampling stage; for the l-th downsampling stage, calculate:

wherein,

respectively the feature tensor F in the l-th down-sampling stage_augFeature tensor F_sourceExtracting a feature tensor;

using feature scaling tensors

And the characteristic shift tensor

As a scaling factor for the segmentation map transformation

And a shifting factor, modeling the conversion process from the segmentation domain to the image domain by using two semantic feature conversion modules, and respectively processing

And

obtaining a scaling factor and a shifting factor of an image field

4. The method for generating semantic image analogy based on single image antagonizing network as claimed in claim 3, wherein the output feature tensor of the l +1 th down-sampling stage in the generator is obtained by the following formula:

wherein DS represents the down-sampling module, mean and std represent the mean and standard deviation, respectively

An up-sampling module of the generator maps the image characteristic tensor output in the down-sampling stage to an image domain, so that a target image is generated;

the generator is an encoder-decoder structure having K downsample blocks and K upsample blocks; each block contains a 3 x 3 convolutional layer of step size 3 and a 4 x 4 convolutional or transposed convolutional layer of step size 2 for down-sampling or up-sampling, and each block also uses spectral normalization, batch normalization and leakage ReLU activation operations.

5. The method of claim 1, wherein the semantic image analogy method for generating the countermeasure network based on the single image,

measuring appearance similarity between a generated image and a source image through image block coherence loss, wherein the appearance similarity is defined as an average value of image block distance lower limits between the source image and a target image:

wherein N is_targetIs a target image I_targetNumber of image blocks in, I_sourceRepresenting a source image, G (I)_source)＝I_target；U_classAnd V_classThe segmentation labels, d (-) is a distance metric function, representing the image blocks UP and VQ.

6. The method of claim 1 or 3, wherein the semantic image analogy method for generating the countermeasure network based on the single image is characterized in that the semantic alignment loss is expressed as:

where CE represents cross entropy loss; i is_sourceRepresenting a source image, P_augRepresenting an enhanced semantically segmented image.

7. The method of claim 1, wherein the overall loss function is expressed as:

wherein,

indicating a loss of the degree of similarity of the appearance,

indicating that the loss of semantic alignment is present,

a loss of the matching of the features is indicated,

representing least squares GAN loss as an antagonistic constraint; lambda [ alpha ]_seg、λ_GANAnd λ_fmAre all hyper-parameters.

8. The semantic image analogy method for generating the countermeasure network based on the single image according to claim 1, characterized in that two modes of sampling and reconstruction are adopted for alternate training;

under the sampling mode, the generator takes the enhancement semantic segmentation image as a guide to generate the appearance and enhancement image I_augIdentical and semantically laid out and enhanced semantically segmented image P_augThe same target image;

the working process of the reconstruction mode and the sampling mode is the same, a given source image and a corresponding source semantic segmentation image are directly input, and the source image is reconstructed by utilizing the source semantic segmentation image; the loss of appearance similarity in the total loss function is replaced by the L1 reconstruction loss between the output reconstructed image and the source image, and the loss of feature matching and the loss of semantic alignment are respectively the loss between the target image and the source image and the loss between the target semantic segmentation image and the source semantic segmentation image.

Images