KR101428531B1 - A Multi-Frame-Based Super Resolution Method by Using Motion Vector Normalization and Edge Pattern Analysis - Google Patents

Abstract

A method for generating a super resolution image by generating a super resolution image using a motion vector normalization and an edge pattern analysis using a low resolution image, the method comprising the steps of: (a) Selecting an image before and after the target time point; (b) expanding the subpixels of the target viewpoint and the forward / backward viewpoint image (hereinafter referred to as a target image) into pixels and interpolating by applying a 6-tap finite impulse response (FIR) filter (hereinafter referred to as a 6-tap filter); (c) estimating a block-by-block motion vector in the expanded image at the previous and following points on the basis of the expanded view of the target viewpoint; And (d) normalizing the estimated motion vector with a modulo of an expansion multiple of the target image. (e) verifying a matching point by a motion vector according to a contour pattern analysis; And (f) estimating a pixel value of the last sub-pixel of the extended image through the one-dimensional kernel estimation equation.
According to the above-mentioned method of generating a super-resolution image, objectively and objectively superior performance is achieved compared with a conventional bi-linear interpolation method, a single image-based super-resolution image generation method, and a multi- Lt; / RTI >

Description

Technical Field [0001] The present invention relates to a multi-image-based super resolution image generation method using motion vector normalization and contour pattern analysis,

The present invention relates to a multi-image-based super-resolution image generation method for generating a super-resolution image based on a plurality of images using motion vector normalization and pattern analysis of an edge.

Generally, a technique of inserting an appropriate value between existing pixels in order to enlarge the resolution of the image is called interpolation. Due to the increase of digital image acquisition media and the difference in performance between media such as digital cameras or cameras embedded in portable devices, the importance of image interpolation method that compensates low resolution images is increasing.

Conventional image interpolation is based on a weighted sum of pixels of a low-resolution image. Typical methods include nearest neighborhood interpolation, bi-linear interpolation, bi- cubic interpolation method [Non-Patent Document 1]. However, these methods are a kind of low-pass filter, and inevitably blurring occurs in the boundary of the image.

To solve this problem, various super resolution image generation techniques have been studied. The super resolution image generation technique uses the information of a plurality of consecutive low resolution images based on the target image, using the characteristic that the number of images reproduced in one second is usually 20 to 30, [Non-Patent Document 2].

In addition, recently, the meaning has expanded, and a super resolution image generation technique using a single image has also been proposed variously. As one of them, a method using discrete wavelet transform (DWT) has been proposed for an input image [Non-Patent Document 3]. This technique is characterized by omitting the down-sampling process that must be performed in the discrete wavelet transform. (LL), low frequency-high frequency (LH), high frequency-low frequency (HL), and high frequency-high frequency (HH) sub-bands of the same size as the original image. After performing various processes on the remaining high frequency subbands and performing inverse transform, enlarged images with high frequency components can be obtained. However, this method has limited performance in that it does not use additional external information like the conventional interpolation method.

In addition, super resolution image generation techniques based on a plurality of images have been proposed.

Generally, a super resolution image generation method assumes a process of generating a single low resolution image as shown in FIG. As shown in the figure, the super-resolution image generation technique assumes that a natural image is down-sampled in the course of digital signalization and a low-quality image is generated by noise added due to sensor malfunction, [Non-Patent Document 2].

Generally, a common flowchart of the multiple-image-based super-resolution image generation techniques is as shown in FIG.

As shown in FIG. 2, as a first step, a total of t images including a target image to which a super resolution image generation technique is applied are input. In this case, the low-resolution images y ₁ to y _t must be adjacent images, and scene transition should not occur. The input images thus determined should be matched in positional relationship with respect to the target image. Then, the matching points obtained based on the positional relation are used to estimate the pixel value of the high resolution image. Finally, a de-blocking filter or a de-blurring filter is applied as a post-process to obtain a single high-resolution image X [Non-Patent Document 7].

However, in the case of the proposed multi-image-based super-resolution image generation techniques, the input image to which the super-resolution image generation technique is applied is subjected to global translation such as simple rotation or global movement, (non-patent documents 4 and 5), several other techniques have been proposed to overcome these limitations [Non-Patent Documents 6 and 7] There is a problem that the internal texture of the object becomes ambiguous.

In another method, a super-resolution image generation technique has been proposed in which a single high-resolution image is applied as a key-frame to adjacent images based on this information [Non-Patent Document 8] This technique is disadvantageous in that it can not be applied unless there is at least one high-resolution image to be used as a core image.

[Non-Patent Document 1] G.B. Kang, Y.S. Yang and J.H. Kim, "A study on interpolation for enlarged still image," KIICE General Conference, pp. 643-648, Bukyung univ., Korea, May. 2001. [Non-Patent Document 2] S.C. Park, M.K. Park, and M.G. Kang, "Super resolution image reconstruction: a technical overview," IEEE Signal Processing Magazine, vol. 20, no. 3, pp. 21-36, May, 2003. [Non-Patent Document 3] J.M. Lim and J. Yoo, "Super Resolution Algorithm using Discrete Wavelet Transform for Single-image," KOSBE Journals, vol. 17, no. 2, pp. 344-353, Mar., 2012. [Non-Patent Document 4] S.C. Kwon and J. Yoo, "Super Resolution Algorithm by Motion Estimation with Sub-pixel Accuracy Using a 6-tap FIR Filter," KICS Journals, vol. 37, no. 6, pp. 464-472, Jun., 2012. [Non-Patent Document 5] V.K. Asari, M.N. Islam and M.A. Karim, "Super resolution enhancement technique for low resolution video," IEEE trans. on Consumer Electronics, vol. 56, no. 2, pp. 919-924, May, 2010. [Non-Patent Document 6] S. Farsiu and P. Milanfar, "Kernel regression for image processing and reconstruction," IEEE Trans. on Image Processing, vol. 16, no. 2, pp. 349-366, Feb., 2007. [Non-Patent Document 7] M. Elad, H. Takeda and P. Milanfar, "Generalizing the nonlocal-means to super resolution reconstruction," on Image Processing, vol. 18, no. 1, pp. 36-51, Jan., 2009. [Non-Patent Document 8] S.C. Jeong and Y.L. Choi, "Video super resolution algorithm using bi-directional overlapped block motion compensation and on-the-fly dictionary training," IEEE Trans. on Circuits and Systems for Video Technology, vol. 21, no. 3, pp. 274-285, Mar., 2011. [Non-Patent Document 9] T.H. Kim, Y.S. Moon and C.S. Han, "Estimation of real boundary with subpixel accuracy in digital imagery," KSPE Journals, vol. 16, no. 8, pp. 16-22. Aug., 1999. [Non-Patent Document 10] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, "Overview of the H.264 / AVC video coding standard," IEEE Trans. on circuits and systems for video technology, vol. 13, no. 7, pp. 560-576, Jul., 2003. [Non-Patent Document 11] N. Hirai, T. Kato, T. Song and T. Shimamoto, "An efficient architecture for spiral-type motion estimation for H.264 / AVC," IEEK General Conference, pp. 314-317, Je-ju, Korea, Jul., 2009. [Non-Patent Document 12] Y. Ismail and J.B. McNeely, M. Shaaban, H. Mahmoud and M.A. Bayoumi, "Fast motion estimation system using dynamic models for H.264 / AVC video coding," IEEE Trans. on Circuits and Systems for Video Technology, vol. 22, no. 1, pp. 28-42, Jan., 2012. [Non-Patent Document 13] H.M. Wong, O.C. Au, A. Chang, S.K. Yip and C.W. Ho, "Fast mode decision and motion estimation for H.264 (FMDME)," IEEE Int. Symposium on Circuits and Systems, pp. 21-24, Greece, May, 2006. [Non-Patent Document 14] S. Winkler, "Evolution of video quality measurement: From PSNR to hybrid metrics," IEEE trans. on Broadcasting, vol. 54, no. 3, pp. 660-668, Sep., 2008.

An object of the present invention is to solve the above-mentioned problems, and an object of the present invention is to provide a method and apparatus for generating a super resolution image based on a plurality of images using motion vector normalization and edge pattern analysis . Particularly, in the present invention, the contents of the second and third steps of matching and high-resolution interpolation are improved mainly in the process of generating the super-resolution image of FIG.

More specifically, it is an object of the present invention to solve the limitation of the motion per sub-pixel, which is a disadvantage of the conventional method, through a normalization technique of a motion vector, and to perform motion estimation on a 2x2 block basis based on contour pattern analysis Resolution image generation method that solves the limitation on translational movement.

According to an aspect of the present invention, there is provided a method for generating a super-resolution image by generating a super-resolution image by receiving a low-resolution image, the method comprising the steps of: (a) Selecting an image of the image; (b) expanding the subpixels of the target viewpoint and the forward / backward viewpoint image (hereinafter referred to as a target image) into pixels and interpolating by applying a 6-tap finite impulse response (FIR) filter (hereinafter referred to as a 6-tap filter); (c) estimating a block-by-block motion vector in the expanded image at the previous and following points on the basis of the expanded view of the target viewpoint; And (d) normalizing the estimated motion vector with a modulo of an expansion multiple of the target image. (e) verifying a matching point by a motion vector according to a contour pattern analysis; And (f) estimating a pixel value of the last sub-pixel of the extended image through the one-dimensional kernel estimation equation.

According to another aspect of the present invention, there is provided a method of generating a multi-image-based super-resolution image, wherein in step (b), a 6-tap filter is applied to the target images to obtain 1/2 pixel, And enlarges the target image by 16 times.

According to another aspect of the present invention, there is provided a method of generating a plurality of image-based super-resolution images, wherein in step (c), a motion estimation is performed on a 2x2 block basis, Value is used.

According to another aspect of the present invention, there is provided a method of generating a multi-image-based super-resolution image, wherein, in step (d), when the motion vector is not 0, the motion vector is modulo- And normalization.

In addition, the present invention is a method for generating a multi-image-based super-resolution image, wherein the contour pattern is analyzed in units of 2x2 blocks in step (e).

Further, the present invention is a method for generating a multi-image based super resolution image, wherein a nadaraya-watson kernel estimation equation is used in step (f).

As described above, according to the method of generating a plurality of image-based super-resolution images according to the present invention, the conventional bi-linear interpolation method, a single image-based super-resolution image generation technique, It is possible to obtain an effect that has superior performance objectively and subjectively than the technique.

FIG. 1 is a flowchart illustrating a process of generating a low-resolution image according to the related art.
BACKGROUND OF THE INVENTION 1. Field of the Invention [0001]
3 is a configuration diagram of the entire system for implementing the present invention;
FIG. 4 shows an example of aliasing occurring in the digital conversion (ADC) process according to the present invention.
FIG. 5 shows an example of the pixel shift according to the present invention, in which (a) pixel shift in an integer unit and (b) pixel shift in a sub- , And the second column is f (t) image.
6 is a flowchart illustrating a method of generating a multi-image-based super-resolution image according to an exemplary embodiment of the present invention.
7 is a pixel layout diagram for explaining a 6-tap FIR filter according to the present invention.
FIG. 8 is an example of using temporal redundancy between images in motion estimation according to the present invention, wherein (a) is a (t-1) th image and (b) (t) th image.
9 is a flow chart illustrating a method for normalizing a directive vector according to the present invention.
FIG. 10 shows an example of using matching points that do not consider contour lines according to the present invention, (a) the positions of various matching points, (b) before applying pattern analysis, and (c) after applying pattern analysis.
Figure 11 shows eight patterns of 2x2 blocks for contour line consideration in accordance with the present invention.
12 is an example of the positions of the matching points and the target interpolation point according to the present invention, (a) before interpolation and (b) after interpolation.
13 is a table for PSNR measurement result (db) of each experimental image according to the experiment of the present invention.
(A) original image, (b) double linearity, (c) SR [3], (d) SR [4], and (e) The resulting image according to the present invention.
(B) a double linear shape, (c) SR [3], (d) SR [4], and (e) the present invention The result image by.
(A) original image, (b) double linearity, (c) SR [3], (d) SR [4] The result image by.
(A) original image, (b) double linearity, (c) SR [3], (d) SR [4] The result image by.
FIG. 18 is a schematic diagram of a TEMpete (third image) according to an experiment of the present invention, in which (a) an original image, (b) a double linear shape, (c) SR [3] The result image by.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the drawings.

In the description of the present invention, the same parts are denoted by the same reference numerals, and repetitive description thereof will be omitted.

First, examples of the configuration of the entire system for implementing the present invention will be described with reference to FIG.

3, the multi-image-based super-resolution image generation method according to the present invention can be implemented as a program system on a computer terminal 30 that receives a low-resolution image 10 and generates a super-resolution image 20 . That is, the super-resolution image generation method may be implemented by a program and installed in the computer terminal 30 and executed. The program installed in the computer terminal 30 can operate as one program system 40. [

Meanwhile, as another embodiment, the method of generating a super-resolution image may be implemented by a single electronic circuit such as an ASIC (application-specific semiconductor) in addition to being operated by a general-purpose computer. Or a dedicated computer terminal 30 for processing only the generation of the super resolution image 20 by receiving the low resolution image 10. This is referred to as a super-resolution image generating apparatus 40. Other possible forms may also be practiced.

The low resolution image 10 is directly input to the computer terminal 30 and stored therein, and is processed by the super resolution image generation apparatus 40. [ Alternatively, the low-resolution image 10 may be stored in advance in the storage medium of the computer terminal 30, and may be read by inputting the low-resolution image 10 stored by the super-resolution image generation apparatus 40. [

The low-resolution image 10 and the super-resolution image 20 are composed of temporally consecutive frames (or consecutive frames of time). One frame has one image. Also, the images 10 and 20 may have one frame (or image). That is, the images 10 and 20 correspond to one image.

Generating a super resolution image in a low resolution image means generating a super resolution frame (or image) in a single low resolution frame (or image) immediately. Further, each super resolution frame And generates a super resolution image for the entire low resolution image.

Unless there is a need for special distinction below, the term video will be used in combination with the frame terminology.

Next, before describing the present invention, the concept of pixel shifting on a sub-pixel basis, which is basically used in generation of a super-resolution image according to the present invention, will be described in detail.

The natural image seen by the human eye is a continuous analog signal. In order to convert a video signal into a digital signal that can be processed by a computer, a process of digitizing the signal such as a voice signal is required. However, the arbitrary natural scene is inevitably aliasing due to the sampling interval between the sensors of the image acquisition device in the course of the low resolution image as shown in FIG. As a result, high frequency components of the original video signal are lost [Non-Patent Document 9].

Assuming that the current image (frame) is f (t) and the immediately preceding image (frame) is f (t-1) in a moving image composed of 20 to 30 images (or frames) per second, The two images will have only a slight difference from each other. FIG. 5 shows the motion of an object existing between images. In Fig. 5, the left column represents the image f (t-1) and the right column represents the f (t) image [Non Patent Document 4].

FIG. 5 (a) shows a case where the movement of an object between images moves in units of an integer, which is a sampling interval. In this case, the digitized image has the same outline information of the corresponding object of f (t-1) and f (t). FIG. 5B shows a case in which the inter-image object moves in units of sub-pixels smaller than the sampling interval. In this case, unlike the first case, the two images have different contour information for the same object [Non-Patent Document 2].

If a plurality of images have the same outline information as in the first case, it is impossible to apply a method of generating a super resolution image to the corresponding images. In this case, it is defined that an integer unit pixel shift occurs between images. However, in the second case, it is possible to apply a method of generating a super-resolution image by combining different outline information. In this case, it is defined that a pixel shift is performed in units of sub-pixels among the images [Non Patent Document 4]. Therefore, in order to apply a method of generating a super-resolution image based on a plurality of images, it is necessary to ensure movement in sub-pixels between input images.

Next, a multi-image-based super-resolution image generation method according to an embodiment of the present invention will be described in more detail with reference to FIG.

As shown in FIG. 6, the proposed method includes a step S10 of selecting a target viewpoint frame from a low resolution image, a subpixel extended interpolation step S20 through a 6-tap finite impulse response (FIR) filter; A block motion estimation step S30; A motion vector normalization step S40; A matching point verification step (S50) of a motion vector according to a pattern analysis of a block unit edge; And a final super resolution image generation step (S60) using a one-dimensional kernel estimation equation. In addition, an ultra-high resolution image can be generated for all view frames including repeating step S70 for all view points.

That is, when a low-resolution moving image is given as an input, a method of proposing each image unit (or frame unit) constituting the moving image is applied (S10). As an example, a total of seven images (frames) of three frames before and after each of the images (or frames) to which the method is to be applied is used.

Sub-pixel interpolation is applied using a 6-tap FIR filter for subpixel matching (S20). When this process is performed, each input low-resolution image is enlarged to a total of 16 times by 4 times in the horizontal and vertical directions. Next, a motion vector is extracted by applying a 2x2 block-based motion estimation (S30), and the extracted motion vector is normalized (S40). If the pattern of the corresponding block is defined by the proposed method, the matching points are selectively used for pixel value calculation according to the result (S50). Finally, the final pixel value is calculated by weighted sum according to the distance between the position of the target pixel and the position of the matching point through the one-dimensional kernel estimation equation (S60).

The images to which the method is applied at all the viewpoints (frames) are collected to output one high-resolution video (S70).

On the other hand, a post-process such as a de-blocking filter or a de-blurring filter may be applied to the finally generated image, but a description thereof will be omitted in the present invention.

First, a step S10 of selecting a target time point frame from a low resolution image will be described in more detail.

As described above, a low-resolution image is composed of an image (frame) continuous in time or an image (frame) in continuous view. When a low-resolution video is given as an input, one of the video units (or frames) constituting the video is selected. The image at this time is referred to as a target-point-of-view image (or target image).

A video at the target time point and an image up to a predetermined point M before and after the target time point are selected. (T-1), f (t), f (t + 1), and f (t) +2), ..., f (t + M) are used. For example, if the predetermined M view is up to 3, a total of 7 images are used, including the target view image and the forward and back view images.

Next, the subpixel extended interpolation step (S20) through the 6-tap FIR filter is described.

In order to be able to generate a super-resolution image between multiple images, input images must have sub-pixel motion vectors, and sub-pixel expansion is required for input images to detect sub-pixel motion. According to the present invention, since the matching method using quarter-pixel units of motion estimation is used, a total of 16 times of image expansion is required, four times as large as the input images and four times as large as the input images.

The methods used here are typically bi-linear, bi-cubic, Lanczos interpolation, etc. [Non-Patent Document 4]. Among these sub-pixel expansion methods, the interpolation method most suitable for the motion estimation and the super-resolution image generation method is a 6-tap FIR filter used in the H.264 / AVC standard and has been experimentally proved in the existing method [Non Patent Document 4].

Therefore, in the present invention, a sub-pixel expansion based on a 6-tap FIR filter is applied. Equation 1 is used in a half-pixel search process [Non-Patent Document 10].

[Equation 1]

Here, b and h are pixel values in units of 1/2 pixel, and A, C, G, M, R, T, E, F, H, .

In addition, the number multiplied by the pixel of each integer unit represents a weight value. 7 shows a layout of the pixel according to Equation (1). In FIG. 7, a gray square indicated by an upper case letter denotes an original integer pixel.

As shown in FIG. 7, a half-pixel is obtained, and a quarter-pixel can be obtained by applying a linear interpolation method as shown in Equation 2 [Non-Patent Document 10].

&Quot; (2) "

Next, a block-based motion estimation step S30 for the subpixel extended image will be described.

The block-based motion estimation is applied to the target image using the input images expanded to the sub-pixels by 4 times, 4 times, 4 times, and 16 times in total (S30) using Equations 1 and 2 described above.

The motion estimation has a compression principle of generating a current image by storing only a difference of a previous image by using a temporal redundancy of a current image and a next image as shown in FIG. 8 by a method proposed through original image compression [Non-Patent Document 11].

In the present invention, by using motion estimation of 2x2 block units, it is possible to perform strong motion matching between images or strong local motion. The search reference of the similar block between images uses the SAD (sum of absolute difference) value of Equation (3) [Non-Patent Document 11].

&Quot; (3) "

Here, SAD (i, j) represents the SAD value at the position (i, j) in the search area, and x and y represent the coordinates in the processing block and may have values of 0 and 1. B _t denotes a processing block in the target image, and B _p denotes a processing block in the p- _th input low resolution image.

Motion estimation requires a lot of computation. Various methods have been proposed to increase the speed of motion estimation. However, all of [Non-Patent Documents 12 and 13] reduce the search accuracy to some extent.

Accelerating motion estimation can use conventional techniques known in the art. In particular, since the result of the method of generating a super-resolution image greatly depends on the accuracy of matching, the present invention uses a full search method of motion estimation.

Next, the motion vector normalization step S40 will be described. It is necessary to find the motion vector between the images and normalize them.

As described above, in the method of generating a super-resolution image based on a plurality of images, movement of the target image between the input images must be guaranteed on a sub-pixel basis [Non-Patent Document 2]. That is, the motion of an object in an image must be displayed within a sub-pixel range within one pixel. However, these limitations make the image that can apply the super-resolution image generation method considerably restrictive. In other words, it is easy to acquire such an image if a moving object or a background is obtained as a single image medium. However, if the motion of the object is complicated or large, it is difficult to apply the existing super-resolution image generation method. In fact, typical videos do not fully satisfy these conditions.

Therefore, in the present invention, a normal resolution image generation method can be applied to a general case having a motion vector larger than one pixel through normalization of a motion vector.

9 is a flowchart of a motion vector normalization method according to the present invention.

First, the block motion estimation is performed for the remaining (p-1) chapters for one processing block in the target image in the input image of the p total chapters. As a result of the motion estimation, a total of 2 x 2 x (p-1) motion vectors can be obtained for one target interpolation point. That is, since the pixels constituting the block are 2 × 2, one block is obtained for each chapter as a result of the motion estimation in the remaining (p-1) chapters excluding the target viewpoint, so that the total number of vectors is 2 × 2 × p-1).

If this vector has a value of 0, the distance from the target interpolation point is 0. Therefore, the value is used without being normalized. If there are a plurality of pixels having a vector value of 0, the values of the pixels are averaged and substituted into the target interpolation point.

A pixel having a vector value of 0 means that a distance between the 'reference pixel' and 'the pixel that is the most similar among the pixels to which motion estimation is applied' is generated when the motion estimation is basically performed, and this distance (vector) is 0. That is, as in the case of a static moving picture with very low motion characteristics, a motion vector of 0 is generated. In addition, when the low-resolution image is transferred to the high-resolution image, the target interpolation point inevitably generates a vacancy (a portion to be interpolated) of the pixel, which is referred to as a target interpolation point.

Conversely, if the distance of the extracted motion vector is not 0, the motion vector is normalized by the remaining 4 modulo. Here, the reason for performing the remainder of 4 is because the subpixel extension of 4 times the width and 4 times the length is applied to the input images for the motion estimation with the 1/4 pixel accuracy.

If the result of the remainder operation is 0, the corresponding pixel is not used for target point interpolation because it has an integer unit movement amount instead of a sub-pixel unit. Except that the result of the remaining operation is 0, the possible values of the remaining operations of 4 are 1, 2, and 3, and subpixel shift amounts of 0.25, 0.5, and 0.75 are assigned to each, and the corresponding pixel value is stored .

That is, the image is normalized by moduloing the image with a modulo of an extended multiple (or an extended multiple, for example, four times).

The matching points and the pixel values of each input low-resolution image pointed to by the normalized motion vector thus obtained are used for target pixel interpolation through kernel estimation as described below.

Next, a description will be made of a step (S50) of verifying a matching point by a motion vector in accordance with the contour pattern analysis.

In the case of the conventional multiple image based super resolution image generation methods, the pixel values of the target interpolation point are obtained through various interpolation methods without considering the matching accuracy or the contour line pattern for the pixel values obtained as the matching result. However, if this part is not taken into consideration, even if the pixel value is not suitable, it can be used if only the motion condition within the sub-pixel is satisfied, which degrades the result quality of the proposed method.

This error is shown in FIG. As shown in FIG. 10 (a), the hatched portion represents an outline, and some of the matching points for using the block including the outline in the resolution restoration are shown in red and blue. Although the matching points belonging to the positions indicated in blue are suitable for use, the accuracy of the pixel values may be degraded and the lattice phenomenon of the pixel values may occur in some cases if the matching points of the positions indicated in red are used. The results before and after improvement of these errors are shown in Fig. 10 (b) and (c), respectively, and are indicated by a red circle.

In the present invention, a total of eight patterns of contour lines can be analyzed as shown in FIG. 11 by performing processing in units of 2x2 blocks.

The criteria for judgment are as follows.

For example, in the case of (1) in FIG. 11, the difference between the pixel values of a and b, c and d should be very small, and the difference between the pixel values of a and c, b and d should be sufficiently large. This difference value is also determined by determining the threshold value, and the sensitivity of the pattern analysis can be adjusted according to the threshold value. If these criteria are used to judge by the pattern of (1), the points within the range of c and d are not used for the collected matching points. The same applies to the remaining 7 contour patterns.

In addition to the method of pattern analysis of the outline, the matching points which differ from the target interpolation point obtained by the sub-pixel search using Equation (4) by a predetermined threshold value are not used.

&Quot; (4) "

Here, V (P _t ) is a source pixel value used as a reference for matching point search, and V (P _r ) is a pixel value of each matching point. The optimal T value can solve the lattice phenomenon that may be caused by the use of an incorrect matching point.

By using the two methods described in this step, it is possible to effectively remove the noise near the boundary of the result image.

In other words, the image when the matching point is obtained is an image obtained by expanding through Equations 1 and 2, and the reference pixel is a pixel inside f (t). The matching point pixel is a pixel obtained at the remaining time point (f (t-2), etc.). By discarding some matching points through contour pattern analysis and discarding the matching points when a difference exceeds a certain distance from the reference pixel. Through these two steps, we use the remaining highly reliable matching points for interpolation.

As a result, in the above embodiment, the target image is expanded by 16 times the subpixel, and whether the value of each extended subpixel is correctly obtained is verified through the images at the preceding and following points in time. At this time, it is verified by an outline pattern by a matching point through a motion vector. The subpixel expansion (Equation 1) is an extension image obtained by simply multiplying pixels constituting an original low-resolution target view image by a coefficient. Here, the values are corrected by motion estimation and contour pattern analysis with respect to other remaining time points based on temporarily filled pixels.

Next, a step S60 of estimating the pixel value of the last sub-pixel of the expanded image through the one-dimensional kernel estimation equation will be described.

The pixel values of the target interpolation point are estimated using the matching points selected in the previous step. The one-dimensional kernel estimation used in the present invention is an estimation method for obtaining a value of an arbitrary position based on the position of a plurality of samples and the corresponding values when data is given.

In the kernel estimation method, the accuracy is determined according to the degree of the estimation equation. However, since the complexity of the equation becomes very large as the order increases, the normalized estimation equation is used. As shown in Equation (5), when the degree of the estimation equation is normalized to zero, this is called a nadaraya-watson kernel estimation equation [Non Patent Document 14].

&Quot; (5) "

Here, K (u) means a Gaussian based kernel equation, and can be expressed by Equation (6).

&Quot; (6) "

In Equation (6), x _i -x represents the euclidean distance between the target interpolation point and the i-th matching point, Y _i represents the pixel value of the i-th matching point, h represents the bandwidth of the kernel estimation equation bandwidth.

The motion vectors obtained from the normalization process described in the previous step can have values of 0.25, 0.5, and 0.75 in the horizontal and vertical directions, respectively.

12 shows the positional relationship between the matching points and the target point to be interpolated. As shown in FIG. 12 (a), the Euclidean distance from the target interpolation point of each matching point and the corresponding pixel value are used in Equation (5) described above. Thus, the pixel value of the final target interpolation point can be obtained.

In summary, all (7) images are expanded 16 times by applying a 6-tap filter. Thereafter, if motion estimation is performed on the remaining six frames based on the target viewpoint image, the motion vectors (normalization applied) and the matching points indicated by the vectors are extracted. Instead of using the matching points as they are, The remaining matching points are used for interpolation. As a result, all pixels including the pixel (or the sub-pixel) obtained by the 6-tap filter (all the pixels without the front and rear viewpoints) can be candidates.

Next, the effects of the present invention will be described in detail through experiments.

Experimental conditions for confirming the performance of the method proposed by the present invention are as follows. First, 30 consecutive images of 176 × 144 (QCIF) images were recorded for 5 videos with a size of 352 × 288 (CIF) 'Mother and Daughter', 'News', 'Carphone', 'Coastguard', 'Tempete' Down-sampling to the size of the image. Thereafter, a bi-linear interpolation method and a discrete wavelet transform (DWT) -based super-resolution image generation method [Non-Patent Document 3] are applied to the downsampled images, The resolution image generation method [Non-Patent Document 4] and the proposed method of the present invention are applied to restore the original size, and then the experiment is conducted by subjectively and objectively comparing the results. For objective performance comparison, the present invention uses the PSNR of Equation (7) as a reference (Non-Patent Document 14).

&Quot; (7) "

Where f is the original image, g is the image to be compared, and M and N are the length and width of the image, respectively. The higher the value, the closer the image g to be compared to the original image f, and the better the performance of the method is defined.

The results of measurement of each PSNR (Peak Signal to Noise Ratio) according to the five experimental images are shown in the table of FIG. PSNR was measured on the first 30 experimental images and the average value was shown. As can be seen in the table, when the proposed method of the present invention is compared with the existing methods, it can be confirmed that the PSNR value is the highest, although it depends on each experimental image. Especially, the difference from the bi-linear interpolation method is larger than the average of 2.5dB. The 'Mother and daughter' and 'News' images show a high difference of more than 3.8db.

14 through 18, a subjective image quality comparison of each interpolation method according to an experimental image is shown. For a more detailed comparison, a portion of the image is magnified 8 times and shown together.

14 to 18, it can be seen that the difference in sharpness from the bilinear interpolation method is very large as in the PSNR difference in the table of FIG. In particular, in the enlarged images of FIGS. 14 to 16, it can be seen that the images have a clear contour line in comparison with the previously proposed methods in the face and the boundary of the image.

It is also possible to visually observe the noise that may appear in the case of using the false matching points mentioned in the chapter 2 at the lower part of the object enlarged in (d) of FIG. 17 and the result of removing it through the proposed method. Finally, in Fig. 18 (d), noise due to matching error is also observed, but it can be seen that the noise is removed from the figure (e). In the case of the method of generating a super-resolution image based on the discrete wavelet transform shown in FIGS. 14 to 18C (Non-Patent Document 3), the objective and subjective performance are better than the conventional bilinear interpolation method. However, The fact that there is a limit to the increase of the picture quality in that the external information can not be utilized can be confirmed through the objective numerical comparison.

The present invention proposes a new super-resolution image generation method that can mitigate various limitations of the conventional single-image super-resolution image generation method and the multi-image-based super-resolution image generation method. In the proposed method, the constraint that only the subpixel motion should be present between the images in the moving image is enlarged so that the method of generating the superresolution image can be applied to various moving pictures through the normalization of the motion vector, and the pattern of the edge Through the analysis, adaptive interpolation was applied to the contents of each image. Experimental results show that the proposed super-resolution image generation method has more objective and subjective superior performance than the existing bi-linear interpolation method, single-image-based super-resolution image generation method, and multi- .

Although the present invention has been described in detail with reference to the above embodiments, it is needless to say that the present invention is not limited to the above-described embodiments, and various modifications may be made without departing from the spirit of the present invention.

10: Low Resolution Image 20: Super Resolution Image
30: computer terminal 40: program system

Claims (6)

A multi-image-based super-resolution image generation method for generating a super-resolution image by receiving a plurality of temporally continuous low-resolution images,
(a) selecting an image at a target time point and an image at forward and backward points of time at the target point in the low-resolution image;
(b) expanding the subpixels of the target viewpoint and the forward / backward viewpoint image (hereinafter referred to as a target image) into pixels and interpolating by applying a 6-tap finite impulse response (FIR) filter (hereinafter referred to as a 6-tap filter);
(c) estimating a block-by-block motion vector in the expanded image at the previous and following points on the basis of the expanded view of the target viewpoint; And
(d) normalizing the estimated motion vector with a modulo of an expansion multiple of the target image;
(e) verifying a matching point by a motion vector according to a contour pattern analysis; And
(f) estimating a pixel value of a final sub-pixel of the extended image through a one-dimensional kernel estimation equation.
The method according to claim 1,
Wherein in step (b), a 6-tap filter is applied to the target images to obtain 1/2 pixel, linear interpolation is applied to obtain 1/4 pixels, and the target image is expanded 16 times Based super resolution image generation method.
The method according to claim 1,
Wherein the step (c) estimates a 2x2 block-by-block motion estimation, and uses a SAD (sum of absolute difference) value as a search criterion of an inter-image similar block.
The method according to claim 1,
Wherein, in the step (d), if the motion vector is not 0, the motion vector is normalized by modulo the motion vector with a horizontal or vertical extension of the image.
The method according to claim 1,
Wherein, in step (e), the outline pattern is analyzed in units of 2x2 blocks.
The method according to claim 1,
Wherein in the step (f), a nadaraya-watson kernel estimation equation is used.

Images