CN106203269A - A kind of based on can the human face super-resolution processing method of deformation localized mass and system - Google Patents

Abstract

The invention discloses a face super-resolution processing method and system based on deformable local blocks. This method mainly focuses on the face super-resolution method based on variable local models. The SIFT Flow feature deforms the sample image blocks of the library, expands the image block mode of the sample library, enhances the expressive ability of the existing image blocks, makes the reconstruction result more accurate, and further mines the face image blocks in the sample library and the input person. The relationship between the face image blocks optimizes the result of the face super-resolution algorithm; the invention can significantly improve the visual experience of the restored image, and is especially suitable for the restoration of the face image in a low-quality monitoring environment.

Description

A face super-resolution processing method and system based on deformable local blocks

technical field

The invention relates to the technical field of image processing and image restoration, in particular to a face super-resolution processing method and system based on deformable local blocks.

Background technique

Face super-resolution technology is to learn the corresponding relationship between high and low resolution through the auxiliary training library, and then achieve the purpose of estimating high-resolution face images from existing low-resolution face images. Face super-resolution is now widely used in many fields, one of the most representative fields is face image enhancement in surveillance video. With the widespread popularization of surveillance systems, surveillance video is playing an increasingly important role in the process of criminal evidence collection and criminal investigation. Face images, as one of the direct evidence, occupy an important position in case analysis and court evidence collection. However, due to the existing conditions, the distance between the target suspect and the camera is relatively far away, and the available pixels of the captured surveillance face are very few. Too dark, uneven light and shade), devices and other factors cause serious damage to the captured image (for example: severe blur and noise), image restoration, amplification and identification are often seriously disturbed. This requires the use of face super-resolution technology to improve image resolution and restore from low-resolution images to high-resolution images.

Face super-resolution technology is a technology to solve this problem. It can reconstruct a high-resolution clear face image from one or more low-resolution face images. Since the technology was first proposed by Baker et al. in 2000, this field has received extensive attention from scholars and produced a series of excellent research results, among which the learning-based face super-resolution algorithm has attracted the most attention from scholars. In recent years, manifold learning has gradually become the mainstream method for face super-resolution. The core idea of this type of method is to describe the manifold spatial relationship of the low-resolution image, find out the local properties around each low-resolution image data point, and then map the manifold of the low-resolution image nonlinearly to the high-resolution image. In the manifold space of high-resolution images, projection is made on the high-resolution corresponding space to synthesize high-resolution images. There are several representative methods: In 2004, Chang ^[1] et al first introduced the manifold learning method into image super-resolution reconstruction, and proposed a neighborhood-embedded image super-resolution reconstruction method. Sung Won Park ^[2] proposed an adaptive manifold learning method based on local projection, which analyzed the intrinsic features of the face from the local sub-manifold, and reconstructed the high-frequency components missing from the low-resolution image. In 2005, Wang ^[3] proposed a method based on PCA (Principal component analysis, principal component analysis) decomposition, which represented the low-resolution image to be processed by a linear combination of the principal components of the low-resolution space, and projected the coefficients to the corresponding High-resolution principal component space to obtain the final result. This method is more robust to the morning, but there are still ghosting and aliasing at the edges of the resulting image. In 2010, Lan ^[5] proposed a face super-resolution method based on shape constraints to address the problem of serious image pixel damage caused by severe blur and noise in the monitoring environment, adding shape constraints to the traditional PCA architecture as similarity The metric criterion uses the robustness of the human eye to recognize shapes to interfere with artificially adding shape feature points as constraints to optimize the reconstruction results of low-quality images. In 2014, Dong ^[4] proposed a method based on local feature transformation to further solve this problem.

However, if the learning-based face super-resolution technology wants to get better restoration results, it must use a larger sample library to cover more image block patterns, and solve the problem that the training samples in the manifold space are not dense enough (limited The training library is compared with the problem of representing the high-dimensional manifold space of facial feature information). However, establishing a face training sample library is a heavy and complicated project. In addition, the more samples in the training library, the higher the computational complexity of face super-resolution reconstruction. Therefore, how to enhance the expressive ability of the face image in the existing training sample library so that it can give an accurate expression when fitting low-resolution image blocks has become an urgent problem to be solved in the current face super-resolution technology research. question.

To sum up, in view of the above problems, this paper proposes a face super-resolution algorithm based on a locally deformable model. On the basis of the face position prior, by introducing the SIFT Flow ^[6] feature, the sample image block in the library is processed. Deformation, expand the available image block mode, enhance the expressive ability of the existing image block, make the reconstruction result have higher accuracy, and further mine the relationship between the face in the sample database and the input face image, so as to realize the optimization of face super Target for resolution algorithm results. In the CAS-PEAL-R1 face database, the experiments on the test samples under the condition of four times downsampling and noise prove that the objective evaluation index and subjective reconstruction results of the proposed algorithm are better than the current best algorithms, and the Noise is robust.

To sum up, by introducing SIFT Flow features to deform the image blocks of the sample library, the expression ability of existing image blocks is enhanced, and more image block modes are covered, which further optimizes the face super-resolution algorithm.

The following references are involved in the article:

[1] H. Chang, D.-Y. Yeung, and Y. Xiong, “Super-resolution through neighbor embedding,” in Proc.IEEE Conf.Comput.Vis.Pattern Recog.,Jul.2004,pp.275–282 .

[2]Sung Won Park, Savvides, M."Breaking the Limitation of Manifold Analysis for Super-Resolution of Facial Images", ICASSP, pp:573-576, 2007.

[3] Xiaogang Wang and Xiaoou Tang, "Hallucinating face byeigen transformation," Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, vol.35, no.3, pp.425–434, 2005.

[4] Dong Xiaohui, Gao Ge, Chen Liang, etc. 2014. Noisy face illusion based on data-driven local feature transformation [J]. Computer Applications, 34(12): 3576-3579.

[5] C Lan, R Hu, Z Han, A face super-resolution approach using shapesemantic mode regularization. IEEE International Conference on Image Processing (ICIP), 2021–2024, 26-29Sept.2010.

[6] Ce Liu, Jenny Yuen, Antonio Torralba, et al. 2011. SIFT Flow: Dense Correspondence across Scenes and its Applications [J], IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5): 978-994.

Contents of the invention

Aiming at the problems existing in the prior art, the present invention provides a face super-resolution processing method and system based on deformable local blocks, especially suitable for restoration of face images in low-quality surveillance videos.

In order to solve the above technical problems, the present invention adopts the following technical solutions;

A face super-resolution processing method based on deformable local blocks, comprising the following steps:

S1: Construct a training library containing a high-resolution face image library and its corresponding low-resolution face image library;

S2: Divide the low-resolution face image to be processed and the images in the training library into image blocks with overlapping parts in the same block-dividing manner, and the image blocks are square image blocks with a side length of psize;

S3: For each block of the low-resolution face image to be processed, search for its neighbor blocks in the low-resolution training block set at the corresponding position;

S4: Calculate the low-resolution face image block to be processed, and the deformation field matrix from each block to the neighbor;

S5: According to the deformation field matrix, calculate each adjacent block to the deformation block corresponding to the low-resolution face image block to be processed;

S6: Calculate the weight coefficient between the low-resolution face image block to be processed and its adjacent deformation block;

S7: Project the weights onto the high-resolution space and restore the image blocks according to the reconstruction coefficients Obtain its corresponding high-resolution face image block

S8: Stitching high-resolution face image blocks high-resolution face images.

Further, in said S1; aligning the positions of the high-resolution face images in the high-resolution face image library, and performing degrading processing to obtain the corresponding low-resolution face image library, and the high-resolution face image library and the low-resolution face image library to form a training library;

Before S2, the size of the low-resolution face image to be processed is the same as that of the image in the training library, and the positions are aligned.

Further, the position alignment adopts the affine transformation method to perform position alignment; the specific five positions include: two corners of the eyes, one tip of the nose, and two corners of the mouth;

The affine transformation method is to add all the face images in the high-resolution face image library and divide them by the number of samples to obtain the average face; let (x′ _i ,y′ _i ) be the i-th feature point on the average face Coordinates, (x _i , y _i ) are the coordinates of the i-th feature point corresponding to the high-resolution face image to be aligned; set the affine matrix Where a, b, c, d, e, f are affine transformation coefficients, Represents the relationship between the average face and the i-th feature point coordinates (x′ _i , y′ _i ) and ( _xi , y _i ) on the high-resolution face image to be aligned, and uses the direct linear transformation method to solve the affine transformation Matrix M; the coordinates obtained by multiplying all coordinate points of the high-resolution face image to be aligned with the affine matrix M are the coordinates of the high-resolution face image after alignment.

Further, in the above S3, for the low-resolution image x _in to be processed, it is assumed that the block at position i is The low-resolution image library is set to X, and the image block at position i on X is recorded as X ⁱ ; The K ^nearest neighbor blocks on Xi, by Compared with the absolute value of each image block difference of X ⁱ one by one, the K low-resolution image blocks with the smallest absolute value of the difference are obtained as neighbors of

Further, in the S4, each image block The deformation field matrix [u, v] of the image to be processed can be and It is calculated by the following method;

Let p=(x,y) be the coordinates of pixel p on the image, let w(p)=(u(p),v(p)) be the flow vector of pixel p, only u(p) and v( p) is the set of integers; let s ₁ and s ₂ denote and SIFT features; W(p) is the flow vector at the pixel point of p, where u(p) and v(p) represent the displacement field of the pixel point in the horizontal and vertical directions respectively; the energy function E(w ):

$\begin{array}{c}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; p</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\\ \underset{\mathrm{<span\; class="notranslate">\; p</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; \&eta;</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\\ \underset{\mathrm{<span\; class="notranslate">\; p</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span>\end{array}$

Among them, the first line is a data item, which means that the SIFT feature is matched along the flow vector w(p) of the pixel point p; the second item is a displacement item, which means that when no other information is available, let the flow direction The amount w(p) is as small as possible; the third item is a smoothing item, which makes adjacent pixels similar; in this objective function, the L1 criterion is used in the first row of data items and the third row of smoothing items to calculate the matching Outliers and flow vectors are discontinuous, t and d are used as thresholds respectively; η and α are balance parameters, which are assigned according to experience;

By minimizing the energy function E(w), the deformation field matrix [u,v] is obtained.

Further, in said S5, according to the deformation field matrix, calculate each adjacent block to the deformation block corresponding to the low-resolution face image block to be processed, the specific process is:

By introducing [u,v], let Match low-resolution image patches get to the low-resolution image patch in the sample library The mapping relationship is defined as the feature operator sf( ), through the feature operator sf( ), the image block matched to the sample library is deformed to be similar to the input image block, and the deformation field [u,v ] expression:

$<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>$

According to the similar consistency of high and low resolution image blocks in the manifold space, the deformed image block R ^k (i, j) is obtained through the deformation matrix [u, v], and Deform( ) is defined to represent deformation process, then

${\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; D.</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; )</span>$

For low-resolution image patches in the training sample library The deformed image block is expressed as Similarly, for the high-resolution image block corresponding to the low-resolution image block, the deformed high-resolution image block is expressed as

According to the introduction of the SIFT Flow feature operator sf( ), the deformation matrix [u, v] is obtained; the deformation matrix [u, v] is used to deform the image block to obtain a deformed image block that is more similar to the input image block, so that the sample library image block The expression is more accurate, and the expression ability of the samples in the sample library is enhanced.

Further, in said S6, the weight coefficient between the image block to be processed and the adjacent deformation block is calculated, and the specific process is as follows:

First introduce the Euclidean distance d _k (i,j):

${\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}$

That is, the input low-resolution image block with the low-resolution image patches in the training samples The Euclidean distance; then select K high-resolution image blocks with the closest Euclidean distance for deformation, so the deformed image block can be expressed as:

${\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; D.</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; K</span>}<span\; class="notranslate">\; .</span>$

Input low-resolution face image patch From the sample training set K nearest low-resolution deformed image blocks The optimal representation weight w ^* (i,j) expressed is:

${\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; argmin</span>}}<span\; class="notranslate">\; \{</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&tau;</span>}\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; \&lsqb;</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&rsqb;</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; .</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; .</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}$

in, is to use the low-resolution warped image patch from the sample library Linearly represents the weight coefficient of the input low-resolution image block, w _i is the element The vector; τ is a balance parameter, assigned according to experience.

Further, in said S7, for the reconstructed target high-resolution image block It can be deformed by K nearest neighbor high-resolution image patches with its optimal weight synthesis:

${\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; K</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}$

in, for Corresponding high-resolution samples.

A face super-resolution processing system based on deformable local blocks, including:

The training library construction model is used to build a training library comprising a high-resolution face image library and its corresponding low-resolution face image library;

The block module is used to divide the low-resolution face image to be processed and the image in the training library into image blocks with overlapping parts in the same block mode, and the image blocks are square image blocks with a side length of psize ;

The neighbor acquisition module is used to search for each block of the low-resolution face image to be processed in the low-resolution training block set of the corresponding position;

The deformation field matrix calculation module is used to calculate the low-resolution face image block to be processed, and the deformation field matrix from each block to the nearest neighbor;

The deformation module is used to calculate each adjacent block to the deformation block corresponding to the low-resolution face image block to be processed according to the deformation field matrix;

The weight coefficient acquisition module is used to calculate the weight coefficient between the low-resolution face image block to be processed and its adjacent deformation block;

A high-resolution image patch generation module for projecting weights onto a high-resolution space and recovering image patches based on reconstruction coefficients Obtain its corresponding high-resolution face image block

Stitching module for stitching high-resolution face image blocks according to position i high-resolution face images.

Compared with the prior art, the present invention has the following advantages and positive effects:

This method is based on learning face super-resolution technology. If you want to restore better results, you must use a larger sample library to cover more image block patterns, and solve the problem that the training samples in the manifold space are not dense enough (limited The training library is compared with the problem of representing the high-dimensional manifold space of facial feature information). However, establishing a face training sample library is a heavy and complicated project. In addition, the more images in the sample library, the higher the computational complexity of face super-resolution reconstruction. Therefore, how to enhance the expression ability of the existing training sample database face images has become an urgent problem to be solved in the current face super-resolution research.

This method mainly focuses on the face super-resolution method based on the variable local model. On the basis of the face position prior, by introducing the SIFT Flow feature to deform the image block of the library sample, expand the image block mode of the sample library, and enhance the existing The expressive ability of the image block makes the reconstruction result more accurate, further mining the relationship between the face image block in the sample library and the input face image block, and optimizing the result of the face super-resolution algorithm.

Description of drawings

Fig. 1 is a schematic flow chart of an embodiment of the present invention;

Fig. 2 is a schematic diagram of face image segmentation based on location according to an embodiment of the present invention.

detailed description

On the basis of the face position prior, the present invention introduces the SIFT Flow feature to deform the library sample image block, expands the sample library image block mode, and enhances the expressive ability of the existing image block, so that the reconstruction result has higher accuracy , to further mine the relationship between the face image blocks in the sample library and the input face image blocks, use the consistency of multiple expressions as a constraint, enhance the consistency and noise robustness of the image block representation, and improve the objective quality and similarity of the restoration results. Spend.

The present invention will be further described below in conjunction with specific embodiments and accompanying drawings.

The invention is oriented to extremely low-quality face images in a monitoring environment, and adopts a double-manifold assumption and a representation of image blocks constrained by consistency. During specific implementation, the technical solution of the present invention can use computer software technology to realize the automatic operation process.

Referring to Fig. 1, the concrete steps of the present invention are as follows:

S1: Construct a training library containing a high-resolution face image library and its corresponding low-resolution face image library;

The positions of the high-resolution face images in the high-resolution face image library Y are aligned, and the high-resolution face images are degraded to obtain corresponding low-resolution face images, thereby obtaining the low-resolution face image library X.

Before S2, the size of the low-resolution face image to be processed is the same as that of the image in the training library, and the positions are aligned.

In the specific implementation, first, the eyes and mouth positions of the high-resolution face image are aligned; then, the high-resolution face image is sequentially down-sampled, fuzzy window filtered, and up-sampled to obtain the corresponding high-resolution face image. low-resolution face images.

For the convenience of implementation reference, the following will provide the specific process of using the affine transformation method to achieve face image alignment:

The position alignment is performed by affine transformation method; the specific five positions include: two corners of the eyes, one tip of the nose, and two corners of the mouth.

Mark the feature points on the high-resolution face image, and the feature points are the edge points of the facial features, such as the corners of the eyes, the tip of the nose, the corners of the mouth, etc.; then, use the affine transformation method to align the feature points.

The affine transformation method is specifically:

Add all face images in the high-resolution face image library Y and divide by the number of samples to get the average face. Let (x′ _i , y′ _i ) be the coordinates of the i-th feature point on the average face, and ( _xi , y _i ) be the coordinates of the corresponding i-th feature point on the high-resolution face image to be aligned. Let the affine matrix Where a, b, c, d, e, f are affine transformation coefficients, Represents the relationship between the average face and the i-th feature point coordinates (x′ _i , y′ _i ) and ( _xi , y _i ) on the high-resolution face image to be aligned, and uses the direct linear transformation method to solve the affine transformation Matrix M. The coordinates obtained by multiplying all the coordinate points of the high-resolution face image to be aligned with the affine matrix M are the coordinates of the high-resolution face image after alignment.

Degrade the aligned high-resolution face image, for example, downsample the high-resolution face image by 4 times, blur window filter by 3*3, and upsample by 4 times to obtain a high-resolution face image Corresponding low-resolution face images, so as to obtain the low-resolution face image library X.

There is a one-to-one correspondence between the face images in the high-resolution face image database Y and the low-resolution face image database X, forming a pair of high-resolution and low-resolution face images. The high-resolution face image library Y and the low-resolution face image library X constitute the training library.

Make the low-resolution face image to be processed the same size as the image in the training library, and the positions are aligned.

The present invention is to process the low-resolution human face image x _in to be processed, estimate its corresponding high-resolution human face image, and record the estimated high-resolution human face image as the high-resolution human face image to be estimated y _out .

The low-resolution face image x _in to be processed is usually a low-resolution face image obtained in a noisy environment. For the low-resolution face image to be processed as input, it generally needs to be pre-processed, including cutting out the face part that meets the unified regulations, and upsampling the low-resolution face image x _in to be processed to make it consistent with the training The face images in the library are of the same size. Mark the feature points of the low-resolution face image x _in to be processed, and finally use the affine transformation method described in S1 to align the low-resolution face image x _in to be processed with the average face position. In this way, the face image in the training library and the low-resolution face image x _in to be processed are at the same level in size and eyebrow height. If there is insufficient light when the low-resolution face image x _in to be processed is collected, automatic brightness and contrast adjustment can be performed on the low-resolution face image x _in to be processed after position alignment, so that it is consistent with the low-resolution face in the training library Images are at similar brightness levels.

S2: Divide the low-resolution face image to be processed and the images in the training library into square image blocks with overlapping parts by using the same block division method; the image blocks are square image blocks with a side length of psize.

In this step, each image in the training library is divided into N square image blocks; at the same time, the low-resolution face image x _in to be processed is also divided into N image blocks. The image block set is used to represent the corresponding face image, and the high-resolution face image y _out to be estimated will be obtained by recovering the image blocks of the low-resolution face image x _in to be processed. The image block sets of the low-resolution face image x _in to be processed, the high-resolution face image y _out to be estimated, the low-resolution face image X in the training library, and the high-resolution face image Y in the training library are recorded as i represents the image block number, X ⁱ and Y ⁱ represent the low-resolution face image x _in to be processed, the high-resolution face image y _out to be estimated, the low-resolution face image X in the training library, and the high-resolution face image Y in the training library The i-th image block of .

As shown in Figure 2, the main basis for dividing face images into blocks is the idea of local manifolds, that is, face images are a special type of image, and these images have specific structural meanings, such as all small blocks at a certain position. It is the eyes, or the nose at a certain position, that is to say, the local small blocks at each position in the image are in a specific local geometric manifold. In order to ensure this local manifold, the image needs to be divided into several square image blocks. The size of the image block needs to have an appropriate size. If the block is too large, it will cause ghosting due to slight alignment problems; if the block is too small, the positional characteristics of each small block will be blurred and diluted. In addition, the size of overlapping blocks between image blocks also needs to be selected. Because if the image is simply divided into several small square blocks without overlapping blocks, then there will be grid effects between these square blocks due to incompatibility problems. Moreover, face images are not always square, so the size selection of overlapping blocks needs to pay attention to make the image as fully divided as possible.

The size of the image block is recorded as psize×psize, the width of the overlapping part between adjacent image blocks is recorded as D, and the position of the image block is expressed as i, i=1,2,...N, then:

Among them, height and width are the height and width of the face image respectively. In the embodiment, psize is 12, and D is 8.

S3: For each block of the low-resolution face image to be processed, search for its neighbor blocks in the low-resolution training block set at the corresponding position;

For the low-resolution image x _in to be processed, suppose the block at position i is The low-resolution image library is set to X, and the image block at position i on X is denoted as X ⁱ . The K ^nearest neighbor blocks on Xi, by Compared with the absolute value of each image block difference of X ⁱ one by one, the K low-resolution image blocks with the smallest absolute value of the difference are obtained as neighbors of

S4: Calculate the low-resolution face image block to be processed, and the deformation field matrix from each block to the neighbor;

In S4, each image block The deformation field matrix [u, v] of the image to be processed can be and Calculated by the following method:

We set p=(x,y) as the coordinates of pixel p on the image, let w(p)=(u(p),v(p)) be the flow vector of pixel p, we only allow u(p) and v(p) is the set of integers. W(p) is the flow vector at the pixel point of p, where u(p) and v(p) represent the displacement field of the pixel point in the horizontal and vertical directions respectively; let s ₁ and s ₂ denote and SIFT features. Then get the energy function E(w):

$\begin{array}{c}\mathrm{<span\; class="notranslate">\; E.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; p</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\\ \underset{\mathrm{<span\; class="notranslate">\; p</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; \&eta;</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\\ \underset{\mathrm{<span\; class="notranslate">\; p</span>}}{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; min</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; \&alpha;</span>}<span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; d</span>}<span\; class="notranslate">\; )</span>\end{array}$

Among them, the first line is a data item, which means that the SIFT feature is matched along the flow vector w(p) of the pixel point p; the second item is a displacement item, which means that when no other information is available, let the flow direction The amount w(p) is as small as possible; the third term is a smooth term, which makes adjacent pixels similar. In this objective function, the L1 criterion (absolute value addition) is used in the first row of data items and the third row of smoothing items to calculate matching outliers and flow vector discontinuities, and t and d are used as thresholds respectively; η, α is a balance parameter, assigned according to experience.

By minimizing the energy function E(w), we can obtain the deformation field matrix [u,v].

S5: According to the deformation field matrix, calculate each adjacent block to the deformation block corresponding to the low-resolution face image block to be processed;

In S5, according to the deformation field matrix, calculate each adjacent block to the deformation block corresponding to the low-resolution face image block to be processed, the specific process is:

We introduce [u,v], let Match low-resolution image patches get to the low-resolution image patch in the sample library The mapping relationship is defined as the feature operator sf( ), through the feature operator sf( ), we can deform the image block matched to the sample library to be similar to the input image block, then the deformation field [u ,v] expression:

$<span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span>$

The deformation matrix [u, v] we get, according to the high and low resolution image blocks have similar consistency in the manifold space, then we can get the deformation image block R ^k through the deformation matrix [u, v] (i,j), we define Deform( ) to represent the deformation process, then

${\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; D.</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; )</span>$

For low-resolution image patches in the training sample library Its deformed image block can be expressed as Similarly, for the high-resolution image block corresponding to the low-resolution image block, the deformed high-resolution image block can be expressed as

Therefore, this paper introduces the SIFT Flow feature operator sf( ) to obtain the deformation matrix [u,v]. Using the deformation matrix [u, v] to deform the image block, a deformed image block that is more similar to the input image block can be obtained, which makes the expression of the image block in the sample library more accurate and enhances the expressive ability of the samples in the sample library.

S6: Calculate the weight coefficient between the image block to be processed and the adjacent deformation block; specifically, the weight coefficient of each adjacent image block of each image block of the low-resolution face image

In step S6, the weight coefficient between the image block to be processed and the adjacent deformed block is calculated, and the specific process is as follows:

First introduce the Euclidean distance d _k (i,j):

${\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}$

That is, the input low-resolution image block with the low-resolution image patches in the training samples Euclidean distance of . Then select K high-resolution image blocks with the closest Euclidean distance for deformation, so the deformed image block can be expressed as:

${\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; D.</span>}\mathrm{<span\; class="notranslate">\; e</span>}\mathrm{<span\; class="notranslate">\; f</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; r</span>}\mathrm{<span\; class="notranslate">\; m</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span><span\; class="notranslate">\; \&lsqb;</span>\mathrm{<span\; class="notranslate">\; u</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; v</span>}<span\; class="notranslate">\; \&rsqb;</span><span\; class="notranslate">\; )</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; \&le;</span>\mathrm{<span\; class="notranslate">\; K</span>}<span\; class="notranslate">\; .</span>$

Input low-resolution face image patch From the sample training set K nearest low-resolution deformed image blocks The optimal representation weight w ^* (i,j) expressed is:

${\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{<span\; class="notranslate">\; *</span>}<span\; class="notranslate">\; =</span>\underset{{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}{\mathrm{<span\; class="notranslate">\; argmin</span>}}<span\; class="notranslate">\; \{</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; x</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; no</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}\mathrm{<span\; class="notranslate">\; L</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}_{\mathrm{<span\; class="notranslate">\; 2</span>}}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; \&tau;</span>}\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{<span\; class="notranslate">\; \&lsqb;</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&rsqb;</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; \}</span><span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; the\; s</span>}<span\; class="notranslate">\; .</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; .</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; K</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}$

in, is to use the low-resolution warped image patch from the sample library Linearly represents the weight coefficient of the input low-resolution image block; w _i is the element The vector; τ is a balance parameter, assigned according to experience.

S7: Project the weights onto the high-resolution space and restore the image blocks according to the reconstruction coefficients Obtain its corresponding high-resolution face image block

In step S7, the weights are projected onto the high-resolution space, and the image blocks are restored according to the reconstruction coefficients Obtain its corresponding high-resolution face image block The specific process is:

For the reconstructed target high-resolution image block It can be deformed by K nearest neighbor high-resolution image patches with its optimal weight synthesis:

${\mathrm{<span\; class="notranslate">\; the\; y</span>}}_{\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; u</span>}\mathrm{<span\; class="notranslate">\; t</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; K</span>}}{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}\mathrm{<span\; class="notranslate">\; h</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; i</span>}}$

in, for Corresponding high-resolution samples. .

S8: Stitching high-resolution face image blocks high-resolution face images.

In order to verify the technical effect of the present invention, the Chinese face database CAS-PEAL is used for verification. Select 1040 face samples from them, the resolution is 112*96, and use the affine transformation method to align the faces. Select 40 images from the face samples and downsample them by 4 times (resolution 24*28) and add Gaussian noise of 0.015 as test images. Using the remaining images of face samples as the training library, the traditional local face super-resolution method (method 1), the noise face illusion method based on data-driven local feature conversion (method 2), and the contour prior-based robust Sticky face super-resolution processing method (method 3) obtains subjective images.

It can be seen from the experimental results that although the resolution of methods 1 to 3 is improved compared with the interpolation method, there are serious errors and the similarity with the original image is very low. The result in method 2 is a global face architecture, and the global-based method often has shortcomings in detail restoration, so it is slightly inferior to the method of the present invention in this respect. Compared with the methods 1-3 and the bicubic interpolation method, the quality of the image restored by the method of the present invention is significantly improved.

Table 1 shows the corresponding objective quality of each image, including PSNR (peak signal-to-noise ratio) and SSIM value (structural similarity criterion). It can be seen from Table 1 that the method of the present invention also has a relatively obvious and stable improvement in the objective quality of the restored image.

Table 1 Comparison of objective quality of recovered images

PSNR value SSIM value method 1 20.6077 0.6006 Method 2 21.8865 0.6711 Method 3 21.6226 0.6569 The method of the invention 22.7622 0.7316

The method of the invention restores the low-quality human face image by combining the large-scale edge data automatically extracted from the original low-resolution human face image with the image features of the original scale. The experimental results have proved the effectiveness of the present invention from subjective quality to objective quality, that is, the introduction of edge data effectively weakens the influence of severe noise on super-resolution reconstruction, and the automatically extracted features avoid the negative effects caused by manual intervention (such as The processing results are unstable, inaccurate, etc.), thus improving the results of face super-resolution processing.

The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which the present invention belongs can make various modifications or supplements to the described specific embodiments or adopt similar methods to replace them, but they will not deviate from the spirit of the present invention or go beyond the definition of the appended claims range.

Claims (9)

1. A face super-resolution processing method based on a deformable local block is characterized by comprising the following steps:

s1: constructing a training library comprising a high-resolution face image library and a low-resolution face image library corresponding to the high-resolution face image library;

s2: dividing the low-resolution face image to be processed and the images in the training library into image blocks with overlapped parts in the same blocking mode, wherein the image blocks are square image blocks with side length of psme;

s3: searching a near block of each block of the low-resolution face image to be processed in a low-resolution training block set at a corresponding position;

s4: calculating a deformation field matrix from each block to the neighbor of the low-resolution human face image blocks to be processed;

s5: calculating a deformation block from each adjacent block to a corresponding to-be-processed low-resolution face image block according to the deformation field matrix;

s6: calculating a weight coefficient between a low-resolution face image block to be processed and a neighboring deformation block thereof;

s7: projecting the weight to high-resolution space, and restoring image block according to reconstruction coefficientObtaining the corresponding high-resolution human face image block

S8: splicing high-resolution human face image blocksAnd obtaining a high-resolution face image.

2. The face super-resolution processing method based on deformable local blocks as claimed in claim 1, characterized in that:

in said S1; aligning the positions of high-resolution face images in a high-resolution face image library, and performing degradation processing to obtain a corresponding low-resolution face image library, wherein the high-resolution face image library and the low-resolution face image library form a training library;

before S2, the low resolution face image to be processed is made to be the same size and position as the images in the training library.

3. The face super-resolution processing method based on deformable local blocks as claimed in claim 2, characterized in that:

the position alignment adopts an affine transformation method to perform position alignment; the specific five positions include: two canthi, one nose tip, two corners of the mouth;

the affine transformation method is that all face images in the high-resolution face image library are added and divided by the number of samples to obtain an average face; is (x'_i,y'_i) Is the ith feature point coordinate on the average face, (x)_i,y_i) The coordinates of the ith characteristic point corresponding to the high-resolution face image to be aligned are obtained; let affine matrixWherein a, b, c, d, e, f are affine transformation coefficients,representing the average face and the ith feature point coordinate (x ') on the high-resolution face image to be aligned'_i,y'_i) And (x)_i,y_i) Solving an affine transformation matrix M by adopting a direct linear transformation method; and multiplying all coordinate points of the high-resolution face image to be aligned with the affine matrix M to obtain coordinates, namely the coordinates of the aligned high-resolution face image.

4. The face super-resolution processing method based on deformable local blocks as claimed in claim 1, characterized in that:

in said S3, for the low resolution image x to be processed_inAssume that the block at position i isThe low resolution image library is X, and the image block at position i on X is marked as Xⁱ；At XⁱK adjacent blocks of (1) byAnd XⁱThe absolute value of the difference value of each image block is obtained by one-to-one comparison, and the K low-resolution image blocks with the minimum absolute value of the difference value are used asIs marked as

5. The face super-resolution processing method based on deformable local blocks as claimed in claim 1, characterized in that:

in said S4, each image blockDeformation field matrix u, v]Can pass through the image to be processedAndthe method is calculated by the following method;

let p be (x, y) the coordinates of pixel p on the image, let w (p) be (u (p), v (p)) the flow vector of pixel p point, and only allow u (p) and v (p) to be integer sets; let s₁And s₂To representAndSIFT feature of (a); w (p) is the flow vector of the pixel at p, where u (p) and v (p) represent the displacement fields of the pixel in both horizontal and vertical directions, respectively; obtaining an energy function E (w):

$\begin{array}{c}E\left(w\right)=\underset{p}{\mathrm{\&Sigma;}}\min \left(\left|\right|s_1\left(p\right)-s_2\left(p+w\left(p\right)\right)|{|}_1,t\right)+\\ \underset{p}{\mathrm{\&Sigma;}}\mathrm{\&eta;}\left(|u\left(p\right)-u\left(p\right)|\right)+\\ \underset{p}{\mathrm{\&Sigma;}}\min \left(\mathrm{\&alpha;}|u\left(p\right)-u\left(p\right)|,d\right)+\min \left(\mathrm{\&alpha;}|v\left(p\right)-v\left(p\right)|,d\right)\end{array}$

wherein the first row is a data item representing the matching of SIFT features along a flow vector w (p) of pixel p points; the second term is a displacement term, which means that the flow vector w (p) is made as small as possible when no other information is available; the third term is a smoothing term to make neighboring pixels similar; in this objective function, the L1 criterion is used among the first row of data items and the third row of smoothing items to compute matching outliers and flow vector discontinuities, t and d as thresholds, respectively; eta and alpha are balance parameters and are assigned according to experience;

by minimizing the energy function E (w), the deformation field matrix [ u, v ] is obtained.

6. The face super-resolution processing method based on deformable local blocks as claimed in claim 1, characterized in that:

in S5, according to the deformation field matrix, a deformation block from each neighboring block to the corresponding to-be-processed low-resolution face image block is calculated, which specifically includes the following steps:

by introducing [ u, v ]]Let aMatching low resolution image blocksTo obtainTo low resolution image blocks in a sample libraryDefining the relation as a characteristic operator sf (-) and transforming the image block matched with the sample library to be similar to the input image block through the characteristic operator sf (-) to obtain a deformation field [ u, v ]]Expression (c):

$\&lsqb;u,v\&rsqb;=sf(x_{in}^i,X_k^i)$

according to the similarity of the image blocks with high and low resolutions in the manifold space, the image blocks are processed by the deformation matrix u, v]Obtaining a deformed image block R^k(i, j), defining Deform (. cndot.) means to beIn the process of deformation of

$R_{kL}^i=Deform(X_k^i,\&lsqb;u,v\&rsqb;)$

For low resolution image blocks in a training sample libraryThe deformed image block is expressed asSimilarly, the high-resolution image block corresponding to the low-resolution image block is represented as the deformed high-resolution image block

Obtaining a deformation matrix [ u, v ] according to an introduced SIFT Flow characteristic operator sf (·); and (3) deforming the image block by using the deformation matrix [ u, v ] to obtain a deformation image block which is more similar to the input image block, so that the expression of the image block of the sample library is more accurate, and the expression capability of the sample in the sample library is enhanced.

7. The face super-resolution processing method based on deformable local blocks as claimed in claim 1, characterized in that:

in S6, the weight coefficient between the image block to be processed and the neighboring deformation block is calculated as follows:

firstly, introducing the Euclidean distance d_k(i,j)：

$d_k^i=\left|\right|x_{in}^i-X_k^i|{|}_2$

I.e. input low resolution image blockAnd low resolution image blocks in training samplesThe Euclidean distance of (c); then, K high-resolution image blocks with a close Euclidean distance are selected for deformation, so that the deformed image blocks can be expressed as follows:

$R_{kL}^i=Deform(X_k^i,\&lsqb;u,v\&rsqb;),1\&le;k\&le;K.$

input low resolution face image blockFrom K nearest low-resolution deformation image blocks in sample training setOptimal representation weight w of representation^*(i, j) is:

$w_i^{*}=\underset{w_i}{\mathrm{argmin}}\left\{\right||x_{in}^i-\underset{k=1}{\overset{K}{\&Sigma;}}{R}_{kL}^i{w}_i|{|}_2^2+\mathrm{\&tau;}\underset{k=1}{\overset{K}{\&Sigma;}}{\&lsqb;d_k^i{w}_k^i\&rsqb;}^2\},s.t.\underset{k=1}{\overset{K}{\&Sigma;}}{w}_k^i=1$

wherein,using low-resolution deformed image blocks in the sample libraryWeight coefficient, w, for linearly representing an input low resolution image block_iIs an element ofThe vector of (a); τ is a balance parameter, which is assigned empirically.

8. The face super-resolution processing method and system based on deformable local blocks as claimed in claim 1, characterized in that:

in the above-mentioned step S7, the first step,

for reconstructed target high-resolution image blockIt can be formed by K high-resolution deformation image blocks in nearest neighborWith its optimal weightSynthesizing:

$y_{out}^i={\mathrm{\&Sigma;}}_{k=1}^K{R}_{kH}^i{w}_k^i$

wherein, is composed ofCorresponding high resolution samples.

9. A face super-resolution processing system based on a deformable local block is characterized by comprising:

the training library construction model is used for constructing a training library containing a high-resolution face image library and a low-resolution face image library corresponding to the high-resolution face image library;

the system comprises a blocking module, a searching module and a processing module, wherein the blocking module is used for dividing a low-resolution face image to be processed and an image in a training library into image blocks with overlapped parts by adopting the same blocking mode, and the image blocks are square image blocks with side length of psme;

the neighbor acquisition module is used for searching a neighbor block of each block of the low-resolution face image to be processed in the low-resolution training block set at the corresponding position;

the deformation field matrix calculation module is used for calculating the low-resolution face image blocks to be processed and the deformation field matrix from each block to the neighbor;

the deformation module is used for calculating a deformation block from each adjacent block to a corresponding to-be-processed low-resolution face image block according to the deformation field matrix;

the weight coefficient acquisition module is used for calculating the weight coefficient between the low-resolution face image block to be processed and the adjacent deformation block;

a high-resolution image block generation module for projecting the weight to the high-resolution space and recovering the image block according to the reconstruction coefficientObtaining the corresponding high-resolution human face image block

A splicing module for splicing the high-resolution face image blocks according to the position iAnd obtaining a high-resolution face image.