CN110084752B - Image super-resolution reconstruction method based on edge direction and K-means clustering - Google Patents

Abstract

The invention discloses an image super-resolution reconstruction method based on edge direction and K-means clustering, belonging to the technical fields of machine vision and image processing. The processing steps of the present invention include: S1: building a training set of high and low resolution image blocks; S2: extracting edge direction feature vectors of low resolution image blocks; S3: using K-means clustering to cluster edge direction features; S4: Classify high- and low-resolution image patches; S5: use ridge regression to train a linear mapping matrix for each category; S6: scale up the input low-resolution image. The present invention makes full use of the edge amplitude and direction characteristics of image blocks, uses K-means clustering to classify image blocks more accurately, ensures the quality of reconstructed images, and the reconstruction process has low computational complexity, which is convenient and fast accomplish.

Description

An image super-resolution reconstruction method based on edge direction and K-means clustering

technical field

The invention belongs to the field of computer vision and image processing, in particular to an image super-resolution reconstruction method based on edge direction and K-means clustering.

Background technique

With the development of technology, more and more playback devices that support ultra-high-definition resolution images have appeared on the market. However, due to the high price of the acquisition equipment, there is a lack of sufficient ultra-high-definition resolution images in the application market. Image super-resolution is a technology that uses software methods to increase image resolution. It only requires low cost and can solve the above problems well. Therefore, it has important research value and Application prospect.

At present, many researchers at home and abroad have studied image super-resolution reconstruction technology. According to the different theoretical basis chosen by researchers, super-resolution methods can be divided into three categories: super-resolution methods based on interpolation, reconstruction and learning.

Among them, the image super-resolution algorithm based on interpolation is the most intuitive and basic type of method. This type of method generally infers the value of the pixel of the point to be interpolated by the value of the surrounding neighborhood pixels. The interpolation-based method usually has very little computational overhead, so it is often used in daily software to enlarge images, but the quality of the reconstructed image is not high, and the edges of the image are prone to jagged.

Since the interpolation technology can only improve the image resolution, it cannot meet the requirements of some applications, so the super-resolution processing technology based on reconstruction appears. Reconstruction-based methods usually utilize correlation information between multiple low-resolution images and add appropriate prior knowledge for super-resolution. Researchers have introduced theories of set theory and probability theory into the super-resolution algorithm in order to obtain relatively high-quality high-resolution images.

In recent years, learning-based super-resolution methods have become a research hotspot. Many learning-based super-resolution algorithms have been proposed, and all of them have better image reconstruction quality than traditional interpolation and reconstruction techniques. Because it is different from the former, the method based on machine learning learns to obtain the corresponding relationship between high and low resolution image blocks by training a large number of high and low resolution image sets. The prior knowledge obtained in this way is more accurate than artificial assumptions. The mapping relationship can also better reflect the connection between high and low resolution images. But learning-based methods usually have high computational overhead, making fast implementation on hardware difficult.

The interpolation method focuses on the speed of image reconstruction, and the learning method pursues the quality of the reconstructed image. However, it is difficult for existing super-resolution methods to achieve a good balance between reconstruction quality and computational overhead.

Contents of the invention

The object of the present invention is to provide an image super-resolution reconstruction method that achieves a good balance between reconstruction quality and calculation cost in view of the above-mentioned existing problems.

The image super-resolution reconstruction method based on edge direction and K-means clustering of the present invention comprises the following steps:

Step 1, collecting high-resolution image datasets;

Degrading the high and low resolution images in the high resolution image data set to obtain the corresponding low resolution image data set;

和低分辨图像块集合

其中，i表示图像块标识，t表示图像区分符，n表示分割得到的图像块的总个数，由采集到的高分辨率图像个数和大小决定，图像块之间组成高低分辨率图像块对

Convert the high and low resolution images to YUV images, segment the Y channel high and low resolution images, and obtain a set of high resolution image blocks with the same number of image blocks

and a set of low-resolution image patches

Among them, i represents the image block identifier, t represents the image distinguisher, n represents the total number of segmented image blocks, which is determined by the number and size of the collected high-resolution images, and the image blocks form high and low resolution image blocks right

Wherein, the segmentation method of the high-resolution image is: based on a preset image block size (such as 2×2, 3×3, etc.), the high-resolution image is divided into n image blocks of the same size and adjacent to each other;

The segmentation method of the low-resolution image is: based on the preset image block size (such as 3×3, 5×5, 7×7, 9×9, etc.), the low-resolution image is divided into n pieces of the same size and mutually overlapping image blocks;

Step 2, extracting the edge direction feature vectors of each low-resolution image block:

其中r表示分割得到的图像子块的总个数，由低分辨率图像块的大小决定，采用一阶梯度算子计算低分辨率子图像块的水平和垂直梯度值，使用水平和垂直梯度值计算得到低分辨率图像子块的边缘幅度m_j和边缘方向a_j，将同一个低分辨率图像块的全部子块的边缘幅度和方向结合起来组成边缘方向特征向量f＝[a₁ m₁…a_r m_r]^T；The low-resolution image block in step 1 is further segmented into low-resolution image sub-blocks

Where r represents the total number of segmented image sub-blocks, which is determined by the size of the low-resolution image block. The first-order gradient operator is used to calculate the horizontal and vertical gradient values of the low-resolution sub-image blocks, and the horizontal and vertical gradient values are used Calculate the edge magnitude m _j and edge direction a _j of the low-resolution image sub-block, and combine the edge magnitude and direction of all sub-blocks of the same low-resolution image block to form the edge direction feature vector f=[a ₁ m ₁ … a _r m _r ] ^T ;

Step 3: Clustering the edge direction feature vectors of the low-resolution image blocks using the K-means clustering method, and calculating a center point c _k for each category, k=1,...,K, where K is the number of center points , determined by the super-resolution reconstruction results, select the number of clusters that can obtain the best super-resolution reconstruction quality, and the center points of the clusters are the largest dispersion in the feature space, and save the center point set;

计算图像块对中的低分辨率图像块的边缘方向特征向量分别与K个中心点的距离，并将高低分辨率图像块对

分配到距离中心点最近的类别中；Step 4. For each pair of high and low resolution image blocks

Calculate the distances between the edge direction eigenvectors of the low-resolution image blocks in the image block pair and the K center points, and combine the high- and low-resolution image block pairs

Assigned to the category closest to the center point;

Step 5. For each category, calculate (for example, calculate by ridge regression) a linear mapping matrix m _k that can convert low-resolution image blocks into high-resolution image blocks, k=1,...,K and save it;

Step 6, input the low-resolution image to be reconstructed, and convert the low-resolution image into a YUV image after edge processing;

According to the segmentation method of the low-resolution image in step 1, the Y channel image is divided into low-resolution image blocks, and the edge direction feature vectors of each low-resolution image block are extracted, based on the edge direction feature vector and K center points Distance, assign the current low-resolution image block to the category closest to the center point;

Based on the linear mapping matrix m _k corresponding to each category, perform super-resolution reconstruction on the low-resolution image block of the Y channel to obtain a high-resolution image block of the Y channel, and combine the high-resolution image blocks to obtain a high-resolution image of the Y channel, For the low-resolution image of the UV channel, the bicubic interpolation method is used for super-resolution of the same multiple, and the high-resolution YUV image is converted into an RGB image to obtain the reconstruction result;

Further, in Step 6, based on the linear mapping matrix m _k corresponding to each category, the super-resolution reconstruction of the low-resolution image blocks of the Y channel is specifically: h _i =m _k l _i ; where l _i is the i-th The column vector formed after the vectorization of the low-resolution image blocks, m _k is the linear mapping matrix of the kth class, and h _i is the column vector formed after the i-th high-resolution image block is vectorized.

Further, in step 6, the edge-adding process is specifically: adding (e-1)/2 edges (e is the length of the low-resolution image block) around the low-resolution image, and the value of the added edge Either zero or the value of the outermost pixel of the low-resolution image.

In the present invention, in order to significantly reduce the running time of the reconstruction process and improve the reconstruction effect, only the reconstruction method described in step S2 of the present invention is considered for the Y channel to obtain the corresponding high-resolution image of the Y channel, while for the other two channels ( UV channel) uses the existing bicubic interpolation method to perform super-resolution reconstruction of the same multiple. Of course, for further consideration of the reconstruction effect, it is also possible to reconstruct the high-resolution image of the corresponding channel by using the reconstruction method of the high-resolution image of the Y channel for the UV channel, that is, for the high and low resolutions of the U and V channels respectively The image is segmented to obtain high and low resolution image block pairs with the same image block, the edge direction feature vector of the low resolution image block is extracted and K-means clustering is performed on it, and then the high and low resolution image block pairs are based on each clustering center ( The distance between the center point) is assigned to different categories; and the linear mapping matrix of the corresponding U and V channels in each category is constructed, and then based on the edge direction of the corresponding image block of the U and V channels of the low-resolution image to be reconstructed The feature vector is assigned to the corresponding category (the category closest to the center point), and then combined with the corresponding linear mapping matrix to obtain the reconstructed high-resolution image of the U and V channels. Only this processing method will lose part of the computational overhead, but its computational overhead is still lower than that of existing learning-based super-resolution algorithms.

In summary, owing to adopting above-mentioned technical scheme, the beneficial effect of the present invention is:

The present invention can make full use of the edge direction information of the sub-image blocks by calculating the edge amplitude and direction of the sub-image blocks and forming a joint feature vector, and the feature extraction method also has low complexity; Vector clustering can flexibly select the number of clusters to achieve better reconstruction results, and the calculation overhead of this method is small, which is convenient for rapid implementation on hardware.

Description of drawings

Fig. 1 is the training phase flowchart of the image super-resolution algorithm based on edge direction and K-means clustering of the present invention;

Fig. 2 is the flowchart of low-resolution image recovery of the present invention;

Fig. 3 is the low-resolution image used for embodiment, and its image width is 144, and height is 144;

Figure 4 is a high-resolution image used in the embodiment with an image width of 288 and a height of 288.

Detailed ways

In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the implementation methods and accompanying drawings.

The image super-resolution reconstruction method based on edge direction and K-means clustering of the present invention includes two parts: a training stage and a low-resolution image reconstruction. Referring to Fig. 1, the specific process of the training stage (step S1) is as follows;

Step S101: collecting a high-resolution image dataset, using a preselected image degradation model to generate a corresponding low-resolution dataset, converting the high- and low-resolution images from RGB channels to YUV channels, and segmenting the high- and low-resolution images of the Y channel, Based on the preset number of divided blocks n (determined by the number and size of high-resolution images) and the size of the image to be divided, set the specific division method.

低分辨率图像使用滑动窗口分割成为相互间重叠的7×7大小的图像块集合

即分割后的高、低分辨率图像的图像块数相同；In this specific embodiment, the high-resolution image in the high-resolution image data set is divided into a set of adjacent image blocks with a size of 2×2

The low-resolution image is divided into a set of overlapping 7×7 image blocks using a sliding window

That is, the number of image blocks of the segmented high and low resolution images is the same;

采用一阶梯度算子计算低分辨率子图像块的水平和垂直梯度值，使用水平和垂直梯度值计算得到低分辨率图像子块的边缘幅度m_j和边缘方向a_j，将同一个低分辨率图像块的全部子块的边缘幅度和方向结合起来组成边缘方向特征向量f＝[a₁ m₁…a₄ m₄]^T；即将边缘方向特征向量f作为每个低分辨率图像块的特征向量f。Step S102: For the middle area of the low-resolution image block in step S101 (in this specific embodiment, it is the middle area with a size of 3×3), use a sliding window to further divide it into four low-resolution images with a size of 2×2 Subblock

Use the first-order gradient operator to calculate the horizontal and vertical gradient values of the low-resolution sub-image block, use the horizontal and vertical gradient values to calculate the edge amplitude m _j and edge direction a _j of the low-resolution image sub-block, and use the same low-resolution The edge magnitude and direction of all sub-blocks of the high-resolution image block are combined to form the edge direction feature vector f=[a ₁ m ₁ …a ₄ m ₄ ] ^T ; the edge direction feature vector f is used as the feature of each low-resolution image block vector f.

Step S103: Set the number of clusters in the K-means clustering method. In this specific implementation, set K=512. Then adopt the K-means clustering method to cluster the features (low-resolution image block feature vector f) of all image blocks obtained in step S102, and calculate a center point c _k for each category, k=1,...,512 , and the center point of the cluster has the largest dispersion in the feature space, and the center point set is saved;

Step S104: For the high and low resolution image block pair obtained in step S101, adopt the method in step S102 to extract the edge direction feature vector of the low resolution image block in each image block pair, and calculate the edge direction feature vector and step S103 The distance between each center point of the corresponding low-resolution image block is based on the distance between the edge direction feature vector of the corresponding low-resolution image block and each center point, and the high- and low-resolution image block pairs are assigned to the nearest category;

Step S105: For each category, based on the high- and low-resolution image block pairs included in each category, use ridge regression to calculate the linear mapping matrix m _k that can convert low-resolution image blocks into high-resolution image blocks, k= 1,...,512 and save.

Step S2: Low-resolution image reconstruction.

Referring to Figure 2, first input the low-resolution image to be reconstructed, as shown in Figure 3, its size is 144×144, and after adding borders to the original image, it is converted into a YUV image.

In this specific implementation manner, the edge adding process specifically includes adding three edges around the original image, and the value of the added edges is zero or the value of the outermost pixel of the low-resolution image.

Then, divide the Y-channel image into low-resolution image blocks l _i , i=1,...,144 ² according to the method of step S101, classify the low-resolution image blocks according to the methods of step S102 and step S104, and use the corresponding The linear mapping function (that is, the linear mapping matrix corresponding to each category) performs super-resolution reconstruction on the Y-channel low-resolution image block to obtain the Y-channel high-resolution image block h _i , i=1,...,144 ² ; where the corresponding The formula for calculating the high-resolution image block of the channel is: h _i =m _k l _i ; where, l _i is the column vector formed after the i-th low-resolution image block is vectorized, and m _k is the first K-type linear mapping matrix, h _i is the column vector formed after the i-th high-resolution image block is vectorized.

Finally, the high-resolution image blocks are combined to obtain the high-resolution image of the Y channel. For the low-resolution image of the UV channel, the bicubic interpolation method is used for super-resolution of the same multiple, and the high-resolution YUV image is converted into an RGB image. The reconstructed high-resolution image, as shown in Figure 4, has a size of 288×288, and the reconstructed image has a good effect.

The above is only a specific embodiment of the present invention. Any feature disclosed in this specification, unless specifically stated, can be replaced by other equivalent or alternative features with similar purposes; all the disclosed features, or All method or process steps may be combined in any way, except for mutually exclusive features and/or steps.

Claims (6)

1. The image super-resolution reconstruction method based on the edge direction and the K-means clustering is characterized by comprising the following steps of:

step one, collecting a high-resolution image data set;

performing degradation treatment on high-resolution and low-resolution images in the high-resolution image dataset to obtain a corresponding low-resolution image dataset;

converting the high-low resolution image into a YUV image, and dividing the Y-channel high-low resolution image to obtain a high-resolution image block set with the same image block number

And low resolution image block set->

Wherein i represents image block identification, t represents image identifier, n represents total number of image blocks obtained by segmentation, and high-low resolution image block pair is formed between the image blocks>

The segmentation mode of the high-resolution image is as follows: dividing the high-resolution image into n image blocks which are identical in size and mutually adjacent based on a preset image block size;

the segmentation mode of the low resolution image is as follows: dividing the low-resolution image into n image blocks which are identical in size and overlap each other based on a preset image block size;

extracting edge direction feature vectors of each low-resolution image block:

aggregating low resolution image blocks

Each low resolution image block in the image is further divided into low resolution image sub-blocks

Wherein r represents the total number of the segmented image sub-blocks;

calculating horizontal and vertical gradient values of the low-resolution sub-image block by adopting a gradient operator, and calculating the edge amplitude m of the low-resolution sub-image block by using the horizontal and vertical gradient values _j And edge direction a _j The edge amplitude and the edge direction of all image subblocks of the same low-resolution image block are combined to form an edge direction feature vector f= [ a ] ₁ m ₁ … a _r m _r ] ^T ；

Clustering edge direction feature vectors of the low-resolution image blocks by adopting a K-means clustering method, and calculating each category to obtain a center point c _k K=1, …, K, and willThe center point set is stored, wherein K is the number of preset center points;

step four, for each pair of high-low resolution image block pairs

Calculating the distances between the edge direction feature vectors of the low-resolution image blocks in the image block pairs and K center points respectively, and adding the high-resolution image block pairs +.>

Assigning to the category nearest to the center point;

step five, for each category, calculating a linear mapping matrix m capable of converting the low resolution image block into the high resolution image block _k K=1, …, K and saved;

step six, inputting a low-resolution image to be reconstructed, and converting the low-resolution image into a YUV image after edge processing;

dividing the Y channel image into low-resolution image blocks according to the dividing mode of the low-resolution image in the first step, extracting edge direction feature vectors of the low-resolution image blocks, and distributing the current low-resolution image blocks into categories closest to the center points based on the distances between the edge direction feature vectors and K center points;

based on the linear mapping matrix m corresponding to each category _k And performing super-resolution reconstruction on the Y-channel low-resolution image block to obtain a Y-channel high-resolution image block, combining the high-resolution image block to obtain a Y-channel high-resolution image, performing super-resolution of the same multiple on the UV-channel low-resolution image by adopting a bicubic interpolation method, and converting the high-resolution YUV image into an RGB image to obtain a reconstruction result.

2. The method of claim 1, wherein in step six, the linear mapping matrix m corresponding to each category is based on _k The super-resolution reconstruction of the Y-channel low-resolution image block is specifically: h is a _i ＝m _k l _i The method comprises the steps of carrying out a first treatment on the surface of the Wherein l _i Column vector formed by vectorizing ith low-resolution image block, m _k A linear mapping matrix of the kth class, h _i And (5) vectorizing the ith high-resolution image block to form a column vector.

3. The method according to claim 1 or 2, wherein in step six, the edge processing is specifically: and (e-1)/2 edges are added to the periphery of the low-resolution image, wherein the added edges have zero value or are the values of the outermost pixels of the low-resolution image, and e is the length of the low-resolution image block.

4. The method according to claim 1, wherein the high resolution image reconstruction mode of the UV channel of the low resolution image to be reconstructed in step six is replaced by: and carrying out high-resolution image reconstruction of the corresponding channel by adopting a reconstruction mode of the high-resolution image of the Y channel.

5. The method of claim 1, wherein the edge direction feature vector of the low resolution image block is extracted by dividing a middle region of the low resolution image block into r low resolution image sub-blocks having the same size.

6. The method of claim 5, wherein the middle region of the low resolution image block is divided into r low resolution image sub-blocks having the same size and overlapping each other.

Images