JP2011154418A - Super-high resolution image generating device, super-high resolution image generation method and program therefor - Google Patents

Abstract

An input image or video is enlarged naturally and with a small amount of computation so that the coarseness of the granularity is not noticeable.
An inclination calculating unit calculates a curve that approximates an intermediate color between adjacent pixels for each arrangement of pixels in an x direction and a y direction in an input image, and determines the curve at the pixel position of the curve. The inclination is calculated, and the x and y inclination information of each pixel point is stored in the point color / inclination information storage unit 14 together with the color information. The enlarged image calculation unit 15 calculates a coefficient in a predetermined approximate curved surface formula from the color information and the inclination information of the point of each pixel, and uses the obtained approximate curved surface formula to find a point between the points of the pixel before enlargement. Interpolation calculates the color information of the enlarged pixel point and generates an enlarged image.
[Selection] Figure 1

Description

  The present invention relates to an apparatus, a method, and a program for naturally enlarging an image when digital data of an image stored discretely is enlarged and displayed.

  As a simple method for enlarging a digital image, a neighborhood method, a linear interpolation method, or the like is used. The neighborhood method is a method of outputting the color of the nearest pixel, and the linear interpolation method is a method of outputting a color by performing linear interpolation from four nearby points.

  However, when a digital image is enlarged by these methods, there is a problem that the image becomes discontinuous and jagged.

  Non-Patent Document 1 proposes a method for solving this problem. Here, a super-resolution oversampling method that exceeds the conventional Shannon interpolation method is shown by formulating the image oversampling problem as an image restoration problem under various assumptions. However, the method disclosed in Non-Patent Document 1 requires a large amount of calculation necessary for image enlargement and takes a lot of time.

Takahiro Saito, “Super-resolution oversampling from a single image”, Journal of the Institute of Image Media and Technology Vol.62, No2, pp.181-189 (2008).

  The method of Non-Patent Document 1 has a problem that the calculation becomes enormous. It is an object of the present invention to realize a technique that can obtain a good result close to the processing result of Non-Patent Document 1 and can calculate at high speed by calculating and maintaining the inclination of each point.

  Simply enlarging the image will only make the grain coarser. In the present invention, the image is naturally enlarged to a high resolution so that the coarseness of the particle size is not noticeable. For this reason, at each point of the original image before enlargement, the color (luminance value or RGB value or other values in various color spaces) in the horizontal direction (also referred to as x direction) and vertical direction (also referred to as y direction) are determined. The degree of change is calculated as a slope, and this is held for each point. When the image is enlarged n times, a curved surface obtained from the color and inclination of each point is calculated, and the color of the point to be interpolated is calculated. If there is an edge in the original image, the edge is processed as a tilt change point.

  That is, the present invention inputs a digitized image (including video, the same applies hereinafter), stores storage means for the input image, and changes in color in the x and y directions for each point of the image. The slope information is calculated, the slope information is stored together with the color information, and the information is used to calculate the color of the enlarged point by interpolating between the points to generate an ultra-high resolution image. It comprises a calculation means and an output means for outputting the generated ultra-high resolution image.

  The present invention further comprises an edge detection means for detecting an edge of the input image, wherein the calculation means regards the edge as a discontinuous point in the calculation of the color gradient, and considers the edge between points. The color of the point to be enlarged is calculated by interpolating.

  According to the present invention, an image can be enlarged and displayed smoothly at high speed. In addition, it is sufficient to calculate in advance the process for calculating the color gradients in the x and y directions for each point (for example, the process up to step S204 in FIG. 2), and the image is actually enlarged. In this case, it is sufficient to start from the process of step S206 in FIG.

It is a block diagram which shows the apparatus structural example of this invention. It is a flowchart of the process which produces | generates an ultra high resolution image. It is a flowchart of the process which calculates | requires the inclination of each point. It is a figure which shows the example which calculates the value and inclination of an intermediate point. It is a flowchart of the process which calculates | requires the color of each enlarged point.

  Embodiments of the present invention will be described in detail with reference to the drawings. In the following, an example of enlarging a still image will be described, but a moving image can also be enlarged by applying the present invention to each frame image in the same manner. Therefore, the image described below includes an image constituting a video.

  FIG. 1 is a block diagram showing an apparatus configuration example of the present invention. As shown in FIG. 1, the ultra-high resolution image generating apparatus 1 includes an image input unit 10 that inputs an image or video, an image storage device 11 that stores the input image or video, a color space conversion unit 12, and an inclination calculation unit 13. , A point color / tilt information storage unit 14, an enlarged image calculation unit 15, and an enlarged image display unit 16. Reference numeral 2 denotes a display device, and 3 denotes an enlargement image information storage device.

  Although the output image of the enlarged image display unit 16 is displayed on the display device 2, it may be output to an external storage device (not shown) and recorded instead of being output to the display device 2. The enlargement image information storage device 3 is a device that stores the contents of the point color / inclination information storage unit 14 calculated by the inclination calculation unit 13 for, for example, enlargement at another magnification. It is provided according to.

The flow of processing executed by the apparatus shown in FIG. 1 will be described using the flowchart of FIG.
In step S <b> 201, the image input unit 10 inputs an image file and stores it in the image storage device 11.
In step S202, the color space conversion unit 12 converts the pixel value of the input image into a value defined in the CIE-Lab space or the like.
In step S203, the inclination calculation unit 13 detects an edge from the input image.
In step S204, the inclination calculation unit 13 calculates an inclination indicating a change in color in the x and y directions at each point of the image, and stores it in the point color / inclination information storage unit 14 together with the color information.
In step S205, the enlarged image calculation unit 15 inputs an enlargement ratio indicating how many times the image is enlarged.
In step S206, the enlarged image calculation unit 15 calculates and obtains a curved surface that interpolates the color between the points from the value and inclination of each point stored in the point color / inclination information storage unit 14. Based on the curved surface, the values between the points are calculated according to the input magnification, and further converted to RGB values.
In step S207, the enlarged image display unit 16 outputs the calculated numerical value as an image.

  Details of the edge detection and the process of obtaining the inclination of each point in steps S203 and S204 in FIG. 2 will be described with reference to the process flowchart shown in FIG. In the input image, the points where the colors are defined are called grid points.

  In the embodiment described below, the tilt calculating unit 13 uses the following method when calculating the tilt indicating the color change of each point (pixel) of the input image. For each row and column, set an appropriate initial value for the slope of the point, and use a cubic function that represents a curve that approximates the color between the points from the color (pixel value) and slope of the adjacent points. Calculate the coefficient. From this cubic function, calculate the color and inclination at a point between the points, and recalculate the color and inclination of the original point (or just the inclination) from the color and inclination of the intermediate point. , Update color and slope values. By repeating this process until a predetermined condition is satisfied, the finally obtained inclination is set as a point inclination.

  In addition, before calculating the slope of the point by the above processing, the edge of the image is detected, and the pixel point corresponding to the detected edge is regarded as a discontinuous point of the curve, and the x direction or the y direction is considered. A plurality of pieces of inclination information on the left side and right side or the upper side and lower side of the points are calculated and stored.

  In order to perform the above processing, in step S301, edge detection of an image is performed using a Sobel filter, a Cany filter, or the like. A point regarded as an edge is stored as a non-differentiable point. An end point may always be considered an edge.

  From step S302 to step S308, the calculation is repeated for each row of the image. First, the width of the image is N, and the following array is prepared.

u [N]: Stores color information w [N]: Stores gradient d [N]: Flag indicating whether or not differentiable w_rt [N]: Stores right gradient of non-differentiable point w_lt [N]: Non-differentiable point U_tmp [N−1]: Stores the color information of the intermediate point w_tmp [N−1]: Stores the gradient of the intermediate point In step S303, for the row of the image to be processed, u [i], Substitute the initial value of w [i] (i = 1, 2,..., N). The color on the grid point is stored in u [i]. Also, the initial value of the slope is stored in w [i].

The initial value of the slope is
w [i] = u [i + 1] -u [i-1]
And so on. In this case, if it is a non-differentiable point or an endpoint,
w_rt [i] = u [i + 1] -u [i]
w_lt [i] = u [i] -u [i-1]
And so on.

  The initial value of the slope may be set to w [i] = 0 with all zeros.

  Next, the processes from step S304 to S307 are repeated until the inclination converges. In step S305, a cubic function is used to interpolate between the i-th and i + 1-th points with a continuous function, and the values and slopes of the i-th and i + 1-th intermediate points are calculated.

  FIG. 4 shows an example in which values and slopes at the intermediate point i + 0.5 are calculated by interpolating between the values and slopes at the i-th and i + 1-th points with a continuous function f (x).

  Here, a continuous function for interpolation is a cubic function as follows.

f (x) = Ax ³ + Bx ² + Cx + D With this cubic function,
f (0) = u [i]
f (1) = u [i + 1]
f ′ (0) = w [i]
f ′ (1) = w [i + 1]
The coefficient is set to satisfy
u_tmp [i] = f (0.5)
w_tmp [i] = f ′ (0.5)
Is assigned.

Specifically, if both the i and i + 1th points are differentiable (not edges),
u_tmp [i] = (u [i] + u [i + 1]) / 2− (w [i + 1] −w [i]) / 8;
w_tmp [i] = (u [i + 1] −u [i]) * 3 / 2− (w [i] + w [i + 1]) / 4;
And

  When the i-th point is non-differentiable (is an edge), w_rt [i] is used instead of w [i], and when the i + 1-th point is non-differentiable (is an edge) Is calculated using w_lt [i] instead of w [i + 1].

In step S306, u [i] and w [i] are recalculated from the intermediate point information. actually,
u [i] = (u_tmp [i−1] + u_tmp [i]) / 2− (w_tmp [i−1] −w_tmp [i]) / 8;
w [i] = (u_tmp [i] −u_tmp [i−1]) * 3 / 2− (w_tmp [i−1] + w_tmp [i]) / 4;
And Note that u [i] need not be recalculated.

When i-th non-differentiable w_lt [i] = (u [i] −u_tmp [i]) / 2;
w_rt [i] = (u_tmp [i + 1] −u [i]) / 2;
And

  In step S307, the end of the iterative process is determined. Specifically, the process is terminated after confirming that the change in u [i] is equal to or less than a predetermined value, or is terminated by repeating a predetermined number of times (for example, three times). In step S308, if there is a next line, the processes in and after step S302 are repeated for the next line, and if the process for the last line of the image is completed, the process proceeds to step S309.

  In step S309, the processing performed in steps S302 to S308 is similarly performed in the column direction (y direction). Through the above processing, the inclination at each grid point is calculated. The calculated inclination information at each lattice point is stored in the point color / inclination information storage unit 14 together with the color information.

  Next, the flow of processing for obtaining the color of each enlarged point in step S206 of FIG. 2 will be described using the flowchart of FIG. When calculating the color of a point of the enlarged image, a coefficient in a predetermined approximate curved surface equation is calculated from the color information and inclination information of each point stored in the point color / inclination information storage unit 14; By interpolating between the points of the pixel before enlargement using the obtained approximate surface equation, the color information of the point of the pixel after enlargement is calculated, and an enlarged image is generated.

  For this reason, in steps S401 and S402, an iteration for sequentially processing the lattice points of the original image is started.

  Assume that the width of the image is N, the height is M, and the enlargement ratio is k times both vertically and horizontally. Prepare the following array.

u [N] [M]: Stores color information wx [N] [M]: Stores the gradient in the x direction wy [N] [M]: Stores the gradient in the y direction d [N] [M]: Differentiable W_rt [N] [M]: Stores the slope of the right side of the non-differentiable point w_lt [N] [M]: Stores the slope of the left side of the non-differentiable point w_up [N] [M]: Stores upper slope w_dn [N] [M]: Stores lower slope of non-differentiable point u_exp [kN] [kM]: Stores color information after enlargement x-direction subscript i (i = 1, 2,..., N), and the subscript in the y direction is j (j = 1, 2,..., M). The color of each point is held as u [i] [j], the inclination in the x direction is wx [i] [j], and the inclination in the y direction is wy [i] [j].

In step S403, in order to approximate a curved surface having a color value in the range from i to i + 1 in the x direction and from j to j + 1 in the x direction from the color and inclination of each grid point in the image, the curved surface function is expressed as f (x, y ) = Ax ³ y + Bxy ³ + Cx ³ + Dy ³
+ Ex ² y + Fxy ² + Gx ² + Hy ²
+ Ixy + Jx + Ky + L
As the coefficient. In particular,
f (0,0) = u [i] [j]
f (1, 0) = u [i + 1] [j]
f (0,1) = u [i] [j + 1]
f (1,1) = u [i + 1] [j + 1]
∂ _x f (0,0) = wx [i] [j]
∂ _x f (1, 0) = wx [i + 1] [j]
∂ _x f (0,1) = wx [i] [j + 1]
∂ _x f (1,1) = wx [i + 1] [j + 1]
∂ _y f (0,0) = wy [i] [j]
∂ _y f (1, 0) = wy [i + 1] [j]
∂ _y f (0,1) = wy [i] [j + 1]
∂ _y f (1,1) = wy [i + 1] [j + 1]
The coefficient is determined so as to satisfy

  When the pixel before enlargement is a point on the edge and there are a plurality of inclinations, the right inclination w_rt is used for wx [i] [j] and wx [i] [j + 1], and wx [i + 1 ] [J], wx [i + 1] [j + 1], the left slope w_lt is used. Further, the lower slope w_dn is used for wy [i] [j] and wy [i + 1] [j], and the upper slope w_up is used for wy [i] [j + 1] and wy [i + 1] [j + 1]. .

In the following, f (x, y) calculated for the point (i, j) will be represented as f _ij (x, y).

In step S404, using f _ij (x, y) calculated in step S403, color information of the image magnified k times according to the following equation is calculated.

u_exp [(i−1) k + p] [(j−1) k + q]
= F _ij ((p-1) / k, (q-1) / k)
Here, p and q are integers from 1 to k, respectively.

  In step S405, if the color information is written in a space other than RGB, the color information is converted into RGB values if necessary.

  In step S405, it is determined whether there is a next column. If there is a next column, the processing from step S402 onward is repeated for that column. In step S406, it is determined whether there is a next line. If there is a next line, the processes in and after step S401 are repeated for that line.

  If the contents of the point color / tilt information storage unit 14 are stored in the image information storage device 3 for enlargement such as an external storage device, when the same image is enlarged later, it is enlarged at a different magnification. In addition, the processing of the color space conversion unit 12 and the inclination calculation unit 13 can be omitted, and an enlarged image with an arbitrary magnification can be generated only by the processing of the enlarged image calculation unit 15, so that the processing can be further speeded up. Convenient.

  The above-described processing for generating an ultra-high resolution image can be realized by a computer and a software program, and the program can be recorded on a computer-readable recording medium or provided through a network.

DESCRIPTION OF SYMBOLS 1 Super high resolution image generation apparatus 10 Image input part 11 Image storage apparatus 12 Color space conversion part 13 Inclination calculation part 14 Point color and inclination information storage part 15 Enlarged image calculation part 16 Enlarged image display part 2 Display apparatus 3 Enlarged image Information storage device

Claims (4)

An ultra-high resolution image generator that enlarges an input image,
Image storage means for storing the input image;
For each arrangement of pixels in the x direction and y direction in the image, a curve that approximates an intermediate color between adjacent pixels is calculated, and an inclination that indicates a color change at the pixel position of the curve is calculated. An inclination calculating means;
Color / inclination information storage means for storing the inclination information in the x-direction and y-direction of each pixel point calculated by the inclination calculation means together with the color information;
A coefficient in a predetermined approximate curved surface expression is calculated from the color information and inclination information of the point of each pixel stored in the color / inclination information storage means of the point, and the pixel before enlargement is calculated using the obtained approximate curved surface expression. An enlarged image calculation means for calculating color information of the point of the enlarged pixel by interpolating between the points and generating an enlarged image;
An ultra-high resolution image generation apparatus comprising: an enlarged image output unit that outputs the generated enlarged image. The ultra-high resolution image generating device according to claim 1,
Before the inclination calculating means calculates the inclination, the apparatus includes edge detecting means for detecting an edge of the image,
The inclination calculation means regards a pixel corresponding to the detected edge as a discontinuous point of the curve, and calculates a plurality of pieces of inclination information on the left side and the right side or the upper side and the lower side of the pixel with respect to the x direction or the y direction. An ultra-high resolution image generation apparatus characterized by the above. An ultra-high resolution image generation method for enlarging an input image,
Storing the input image in the image storage means;
For each arrangement of pixels in the x direction and y direction in the image, a curve that approximates an intermediate color between adjacent pixels is calculated, and an inclination that indicates a color change at the pixel position of the curve is calculated. Storing the calculated x-direction and y-direction inclination information of each pixel point together with the color information in the point color / inclination information storage means;
A coefficient in a predetermined approximate curved surface expression is calculated from the color information and inclination information of the point of each pixel stored in the color / inclination information storage means of the point, and the pixel before enlargement is calculated using the obtained approximate curved surface expression. Calculating the color information of the point of the enlarged pixel by interpolating between the points and generating an enlarged image;
And a step of outputting the generated enlarged image.   An ultra-high resolution image generation program for causing a computer to function as each unit included in the ultra-high resolution image generation apparatus according to claim 1.

Images