JP2017010115A - Image resolution conversion apparatus, image resolution conversion method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To further simplify super-resolution processing using a light field camera image captured at a plurality of locations.SOLUTION: An image resolution conversion device comprises: an image acquisition unit for acquiring a plurality of light field images in which the same subject is captured at different view points; an image correction unit for correcting the luminance value of pixels of the correction object light field image on the basis of the luminance value of a pixel which, among pixels of another light field image, has acquired the same light beam with that of the correction object pixel and which is hard to be influenced by deterioration of vignetting components, lens distortion or the like; and a super-resolution unit for generating an image with higher resolution by executing super-resolution processing using a plurality of light field images corrected by the image correction unit.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image resolution conversion technique.

  It is widely known that an image having a resolution higher than the resolution at the time of imaging is generated by performing super-resolution processing on the captured image. A finer image can be obtained by using images taken at a plurality of different photographing points when performing the super-resolution processing. Such a technique is called multiframe super-resolution (see Non-Patent Document 1). In multi-frame super-resolution, a desired high-resolution image is treated as an ideal signal. An observation model is assumed in which a captured image as an observation signal is obtained by multiplying an ideal signal with an observation matrix composed of a camera imaging system. For this observation model, super-resolution processing is realized by estimating an ideal signal from information of the observation signal and the observation matrix.

  However, actual lenses have deterioration characteristics such as vignetting and lens distortion. Therefore, if these degradations are included, the observation matrix becomes complicated. For this reason, these estimations are known as defective setting inverse problems, and many problems remain, such as the solution falling into a local solution and requiring enormous calculation time. In addition, a method for performing super-resolution by using depth estimation has been proposed. However, obtaining the depth accurately is difficult in an actual scene. Therefore, the range in which such a method can be applied is limited (see Non-Patent Document 2).

Farsiu, Sina, et al. “Fast and robust multiframe super resolution.” Image processing, IEEE Transactions on 13.10 (2004): 1327-1344. Keishi Fukushima, Yutaka Ishibashi. “Image quality improvement of free viewpoint images by super-resolution processing.” IEICE Transactions D 93.9 (2010): 1700-1703.

In the multi-frame super-resolution technique, there is a problem that it is difficult to solve the inverse problem using the observation model because the observation matrix is complicated.
In view of the above circumstances, an object of the present invention is to more easily realize super-resolution processing using light field camera images taken at a plurality of points.

  In one embodiment of the present invention, an image acquisition unit that acquires a plurality of light field images obtained by imaging the same subject from different viewpoints, and luminance values of pixels of the light field image to be corrected are set to the pixels of the other light field images. Among these, the image correction unit that corrects based on the luminance value of a pixel that has acquired the same light beam as the correction target pixel and is less susceptible to deterioration such as vignetting component and lens distortion, and the image And a super-resolution unit that generates a higher-resolution image by executing a super-resolution process using a plurality of the light field images corrected by the correction unit.

  One aspect of the present invention is the above-described image resolution conversion device, in which the image correction unit is a pixel that is less susceptible to deterioration such as vignetting components and lens distortion, and the center of the microlens through which the light beam has passed. A pixel having a shorter distance is selected.

  According to one embodiment of the present invention, an image acquisition step of acquiring a plurality of light field images obtained by imaging the same subject from different viewpoints, and luminance values of pixels of the light field image to be corrected are set to the pixels of the other light field images. Among these, the image correction step for correcting based on the luminance value of a pixel that has acquired the same light beam as the pixel to be corrected and is less susceptible to deterioration such as vignetting component and lens distortion, and the image And a super-resolution step of generating a higher-resolution image by executing a super-resolution process using a plurality of the light field images corrected in the correction step.

  One embodiment of the present invention is a computer program for causing a computer to function as the image resolution conversion apparatus.

  According to the present invention, it is possible to more easily realize super-resolution processing using light field camera images taken at a plurality of points.

1 is a block diagram illustrating a configuration of an image resolution conversion apparatus 10. FIG. 5 is a flowchart illustrating a specific example of a processing flow of the image correction unit 102. It is a figure which shows the outline of deviation | shift amount. It is a figure which shows the concept of the process for selecting an optimal pixel value. 10 is a flowchart illustrating a specific example of a processing flow of the image super-resolution unit 103. It is a figure which shows the specific example of the observation matrix which the image super-resolution part 103 acquires.

  Hereinafter, an image resolution conversion apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the image resolution conversion apparatus 10. The image resolution conversion apparatus 10 includes a CPU (Central Processing Unit), a memory, an auxiliary storage device, and the like connected by a bus. The CPU of the image resolution conversion apparatus 10 executes an image resolution conversion program stored in advance in the memory of the image resolution conversion apparatus 10. When the CPU executes the image resolution conversion program, the image resolution conversion apparatus 10 functions as an apparatus including the image acquisition unit 101, the image correction unit 102, and the image super-resolution unit 103. Note that all or part of the functions of the image resolution conversion apparatus 10 may be realized using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). Good. The image resolution conversion program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. The image resolution conversion program may be transmitted via a telecommunication line.

  The image acquisition unit 101 inputs a plurality of light field images. The light field image is an image formed by arranging a plurality of images (hereinafter referred to as “element images”) obtained by capturing the same subject from slightly different viewpoints according to the position of the viewpoint. The light field image may be captured by a plenoptic camera, for example. A plenoptic camera is a camera having a microlens array between a main lens and an imaging device. When imaged with a plenoptic camera, an image imaged according to each microlens constituting the microlens array is an element image. In this case, the element image may be called a microlens image. In the following description, it is assumed that the image acquisition unit 101 acquires a light field image captured by a plenoptic camera.

  The plurality of light field images are images having different viewpoint positions at the time of imaging. A plurality of light field images are acquired by repeatedly performing an operation in which the plenoptic camera moves and images in a direction perpendicular to the optical axis of the camera.

  The image correction unit 102 corrects a plurality of light field images. FIG. 2 is a flowchart illustrating a specific example of the processing flow of the image correction unit 102.

  First, the image correction unit 102 estimates the position of the camera at the time of imaging for each light field image. Here, the camera position is, for example, parameters of a 1 × 3 matrix T that represents the position where the light field image is captured and a 3 × 3 matrix R that represents the rotation in the viewing direction. Since these matrices are expressed as translation components and rotation components of a known perspective projection matrix, detailed description thereof is omitted here.

  The position of the camera may be estimated, for example, by performing Structure from Motion on a microlens image (element image) obtained from a light field image. Alternatively, it may be estimated using camera calibration using an object having an existing shape. When shooting at a plurality of points while moving the camera at equal intervals in the horizontal direction, the position of the camera can be known.

  Next, the image correction unit 102 measures the shift amounts of the plurality of light field images (S101). FIG. 3 is a diagram showing an outline of the shift amount. The deviation amount is an amount represented by Δm, x and Δm, y when there is another image m (302) with respect to the reference image 1 (301). The image correction unit 102 calculates a positional shift amount having the highest similarity of the luminance values of pixels included in an area where the image 1 (301) and the image m (302) overlap (hereinafter referred to as “overlap area”: 303). get. As a measure of similarity, the sum of luminance differences (SAD: Sum of Absolute Difference), the sum of squares of luminance differences (SSD: Sum of Squared Difference), normalized cross-correlation (ZNCC) Etc. are used.

  Next, the image correction unit 102 superimposes a plurality of light field images on the reference image coordinate system (step S102). The plurality of light field images have independent image coordinate systems. Therefore, the image correction unit 102 aligns the image coordinate system with the reference image coordinate system using the shift amount of each image measured in step S102.

  A specific example of the processing in steps S101 and S102 is as follows. First, one reference light field image (hereinafter referred to as “reference image”) is set. Next, m light field images (m is a natural number and equal to the number of input images minus 1) are superimposed on the reference image. Then, a search is performed while changing the deviation amounts Δm, x, Δm, y where the pixel values of all the pixels included in the overlapping region 303 are most similar to the pixel values of the reference image.

  By this processing, the optimum positional deviation amounts Δm, x, Δm, y on the light field image can be calculated. Then, according to the positional deviation amounts Δm, x, Δm, y, light field images other than the light field image serving as the reference image are superimposed on the same image space as the reference image. By such processing, a shift amount is obtained for a plurality of light field images.

  Next, the image correction unit 102 selects an optimal pixel value from among a plurality of pixels having the same coordinates (hereinafter referred to as “overlapping pixels”) for all pixels. Then, the image correction unit 102 replaces the pixel value of each overlapping pixel with the selected optimal pixel value. By substituting an optimal pixel value selected from the overlapping pixels, it is possible to remove components related to deterioration such as vignetting components and lens distortion.

  FIG. 4 is a diagram illustrating a concept of processing for selecting an optimal pixel value. In the example of FIG. 4, it is assumed that there are three candidate images having overlapping pixels for the correction target pixel (405) in the image to be output (correction target image: 401). First, the image correction unit 102 acquires the same light beam from the pixels of the candidate image A (402), the candidate image B (403), and the candidate image C (404) to the correction target pixel (405). The image coordinates of the existing pixels (406, 407 and 408) are determined. The image correction unit 102 calculates a distance (409, 410, and 411) between each determined pixel and the center of the microlens through which the light that has reached each pixel has passed. The image correction unit 102 selects the pixel value of the pixel with the shortest distance among the calculated distances as the optimum pixel value. In the specific example of FIG. 4, the pixel value of the pixel 407 is selected as the optimum pixel value. The image correction unit 102 corrects the pixel value of the correction target pixel (405) to the selected pixel value.

  The pixel closest to the center of the microlens is the pixel that is least susceptible to degradation such as vignetting components and lens distortion. Therefore, by replacing the pixel value of the pixel to be corrected with the pixel value of such a pixel, it is possible to correct the pixel value so that the influence of deterioration such as vignetting component and lens distortion is small. Therefore, when performing the image super-resolution processing using the light field image corrected in this way, it is possible to ignore deterioration such as vignetting components and lens distortion, and to use a simpler process to achieve high accuracy. Image super-resolution processing can be realized. It should be noted that the optimal pixel value is selected by other methods as long as it is a method that can select the pixel value of a pixel that is not limited to the pixel closest to the center of the microlens and that is less affected by degradation such as vignetting components and lens distortion. May be.

  The image super resolving unit 103 generates a high resolution image using the light field image corrected by the image correcting unit 102 (hereinafter referred to as “corrected light field image”). Specifically, the image super resolving unit 103 first inputs a corrected light field image as an initial value for repeated calculation. The image super resolving unit 103 inputs an observation matrix when these images are captured and solves the inverse problem. By executing such processing (super-resolution processing), the image super-resolution unit 103 acquires a high-resolution image.

  FIG. 5 is a flowchart illustrating a specific example of the processing flow of the image super-resolution unit 103. Hereinafter, the processing flow of the image super-resolution unit 103 will be described again with reference to FIG.

  First, the image super resolving unit 103 acquires a plurality of corrected light field images (step S201). Next, the image super resolving unit 103 acquires an observation matrix (step S202). FIG. 6 is a diagram illustrating a specific example of the observation matrix acquired by the image super resolving unit 103. The observation matrix indicates a shift amount F between the down-sample element D, the pixel-level blur element H, and the input viewpoint image.

  Next, the image super resolving unit 103 updates the ideal signal X indicating the use image (step S203). The update process will be described in detail. The image super resolving unit 103 updates the ideal signal X based on the following equation. The ideal signal X is a signal indicating a high-resolution image that the image super-resolution unit 103 should acquire. The observation signal Y is a signal indicating an actually acquired image (corrected light field image in the present embodiment). n indicates the number of steps. A shows an observation matrix. I represents a unit matrix. β indicates a convergence step. λ indicates the strength of restraint on the smoothness of the image. α represents an attenuation parameter related to the distance of Bilateral Total Variation (BTV). S indicates a parallel movement amount parameter. In the present embodiment, a noise robust method is applied as the update method. However, other methods may be applied as the update method.

  Next, the image super resolving unit 103 uses the updated ideal signal X to calculate Y from the relational expression of Y = DHFX. Since Y is an observation signal, when the ideal signal X is correctly restored, the calculated observation signal Y is equal to the input image. Accordingly, the more accurately the ideal signal X is restored, the smaller the difference between the observation signal Y and the input image. The image super resolving unit 103 calculates an evaluation value representing the degree of similarity between the observation signal Y and the input image every time the ideal signal X is updated in step S203. The calculated evaluation value is larger as the difference between the observation signal Y and the input image is smaller. The image super resolving unit 103 ends the process when the update process in step S203 is repeated a predetermined number of times or when the evaluation value becomes a value equal to or greater than the threshold value (step S204—YES). On the other hand, when the number of repetitions of the update process in step S203 is less than the predetermined number and the evaluation value is less than the threshold value (step S204-NO), the image super-resolution unit 103 performs the process of step S203. Run.

  In addition, the process of step S203 and step S204 is a calculation method called the steepest descent method. The process performed by the image super resolving unit 103 is not limited to this process, and other optimization calculation methods may be applied.

DESCRIPTION OF SYMBOLS 10 ... Image resolution conversion apparatus, 101 ... Image acquisition part, 102 ... Image correction part, 103 ... Image super-resolution part

Claims (4)

An image acquisition unit that acquires a plurality of light field images obtained by imaging the same subject from different viewpoints;
The luminance value of the pixel of the light field image to be corrected is a pixel that has acquired the same light ray as the pixel to be corrected among the pixels of the other light field image, and the deterioration of vignetting components, lens distortion, etc. An image correction unit that performs correction based on a luminance value of a pixel that is less affected,
A super-resolution unit that generates a higher-resolution image by performing a super-resolution process using a plurality of the light field images corrected by the image correction unit;
An image resolution conversion apparatus comprising:   The image according to claim 1, wherein the image correction unit selects a pixel having a shorter distance from the center of the microlens through which the light beam has passed as a pixel that is less susceptible to deterioration such as vignetting components and lens distortion. Resolution converter. An image acquisition step of acquiring a plurality of light field images obtained by imaging the same subject from different viewpoints;
The luminance value of the pixel of the light field image to be corrected is a pixel that has acquired the same light ray as the pixel to be corrected among the pixels of the other light field image, and the deterioration of vignetting components, lens distortion, etc. An image correction step for correcting based on a luminance value of a pixel that is less affected,
A super-resolution step of generating a higher-resolution image by performing a super-resolution process using a plurality of the light field images corrected in the image correction step;
An image resolution conversion method comprising:   A computer program for causing a computer to function as the image resolution conversion apparatus according to claim 1.

Images