JP2009271725A - Image-restoring apparatus, image restoring method, and image-restoring program - Google Patents

Abstract

An image restoration apparatus is provided that generates a restored image with reduced ringing and noise by restoring a true image based on a known PSF and one deteriorated image.
A base image generation processing unit that generates a base image by performing base image generation processing based on a deteriorated image, and convolution integration of the generated base image from the base image generation processing unit and the PSF. A blurred image generation processing unit that generates a blurred image, a degraded residual image generation processing unit that generates a degraded residual image by calculating a difference between the generated blurred image and the degraded image, and An image restoration processing unit that generates a restored residual image by performing image restoration processing on the degraded residual image based on the generated base image so as to change the strength of the constraint. A restored image generation processing unit that generates a restored image by adding the generated base image to the restored residual image;
[Selection] Figure 3

Description

  The present invention relates to an image processing technique, and more particularly to an image restoration apparatus, an image restoration method, and an image restoration program for restoring a true image from a deteriorated image.

  Camera shake when shooting a subject with an imaging device or shooting when the subject is a moving object often causes blurring, that is, a deterioration phenomenon called blur. is there.

  In image processing technology, an image with blur (an image that has deteriorated due to blur), that is, a blurred image, is an original image without blur (hereinafter also referred to as a “true image”) and a blur. PSF estimation processing when restoring a true image from a blur image (hereinafter also referred to as “degraded image”) because it is formulated as a convolution integral with a point spread function (PSF) representing Then, it is necessary to perform image restoration processing using the estimated PSF (see Non-Patent Document 1).

  The PSF estimation process is a difficult problem by itself. Therefore, in many methods, linear blurring (blur) is assumed and PSF estimation processing is performed (see Non-Patent Document 2 and Non-Patent Document 3). However, the PSF of the blur that actually occurs often has a complicated shape that cannot be expressed by these assumptions (see Non-Patent Document 4), and there are cases where sufficient estimation results cannot be obtained. In recent years, a method for estimating a PSF having a complicated shape has also been proposed, and it has been reported that a good estimation result can be obtained with a small PSF (see Non-Patent Document 4 and Non-Patent Document 5).

Even if the PSF is known, the image restoration process is still a difficult problem. The degraded image lacks a high-frequency component or a specific frequency component, and an image restored from the degraded image (hereinafter also referred to as a “restored image”) has artifacts such as Ringing. It often happens. That is, there is a problem that ringing occurs corresponding to the edge strength of the restored image. Further, in the process of restoring the deteriorated frequency component, there arises a problem that the noise component is simultaneously increased.
JP 2006-221347 A JP 2007-183842 A Co-authored by D. Kundur and D. Hatzinakos, “Blind image deconvolution”, Signal Processing Magazine, IEEE, Vol. 13, No. 3, p.43-64, 1996 Co-authored by Y. Yitzhaky, I. Mor, A. Lantzman and NSKopeika, “Direct Method for Restoration Off Motion Blured Images (Direct method) for restoration of motion-blurred images), Journal of the Optical Society of America A, Vol. 15, p. 1512-1519, 1998 Yoneji, Tanaka, Okutomi, "Psf parameter estimation method for linear camera shake image restoration", IPSJ SIG, 2005, 38, p.47-52, 2005 Co-authored by R. Fergus, B. Singh, A. Hertzmann, STOWeis, WTFreeman, "Removing Camera""Removing camera shake from a single photograph", ACM Trans. Graph., Vol. 25, No. 3, p.787-794, 2006 J. Jia, “Single image motion deblurring using transparency”, Computer Vision and Pattern Recognition, 2007 CVPR'07. IEEE Conference and Pattern Recognition, 2007 CVPR'07. IEEE Conference on), p.1-8, 2007 WHR Richardson, “Bayesian-based iterative method of image restoration”, Journal of the Optical Society of America (1917- 1983), 62, p. 55-59, 1972 LBLucy, “An iterative technique for the rectification of observed distributions”, Astronomical Journal, Vol. 79, p. .745-754, 1974 By N. Wiener, “Extrapolation, Interpolation, and Smoothing of Stationary Time Series”, The MIT Press, 1964 Yoneji, Tanaka, Okutomi, "Proposal of human-friendly restoration filter", IEICE Transactions, Vol.10, p.2830-2839, 2007 Co-authored by B. Bascle, A. Blake, and A. Zisserman, “Motion deblurring and super-resolution from an image ECCV'96: Proceedings of the 4th European Conference on Computer Vision-Volume II, London, UK, ECCV'96: Proceedings of the 4th European Conference on Computer Vision-Volume II, London, UK, Springer-Verlag), p.573-582, 1996 Co-authored by A.Rav-Acha and S.Peleg, “Two motion-blurred images are better than one”, Pattern Recogna. Recogn. Lett.), 26, No. 3, p. 311-317, 2005 Co-authored by M. Ben-Ezra and SKNayar, “Motion deblurring using hybrid imaging”, cvpr, volume 01, p.657, 2003 Co-authored by L.Yuan, J.Sun, L.Quan, H.-Y.Shum, “Image Deblurring with Blared / Noise "Image deblurring with blurred / noisy image pairs", SIGGRAPH'07: ACM SIGGRAPH 2007 papers, New York, NY, USA, ACM, p.1, 2007 G. Petschnigg, R. Szeliski, M. Agrawala, M. Cohen, H. Hoppe, K. Toyama (K) Toyama), "Digital photography with flash and no-flash image pairs", ACM Trans. Graph., Vol. 23, No. 3 , p.664-672,2004 Co-authored by Y. Weiss and W. Freeman, “What makes a good model of natural images?”, Computer Vision and Pattern Recognition, 2007 CVPR '07. IEEE Conference on (Computer Vision and Pattern Recognition, 2007 CVPR'07. IEEE Conference on), p.1-8, 2007 Okutomi et al., “Digital Image Processing”, CG-ARTS Association

  Conventionally, PSF is known, and as an image restoration method for restoring a true image from a taken degraded image, Richardson-Lucy method (Richardson-Lucy method: RL method) (Non-patent Document 6 and Non-patent Document 7) And Wiener filters (see Non-Patent Document 8) are well known. Both the RL method and the Wiener filter perform image restoration processing by directly using the deteriorated image taken.

  Therefore, the edge strength of the restored image restored by the RL method or the Wiener filter is large, and as a result, there is a problem that a large ringing occurs in the restored image restored by the RL method or the Wiener filter.

  That is, if the restoration parameter is adjusted so that a sharp result can be obtained using the RL method or the Wiener filter, there arises a problem that the restoration result (restored image) includes large noise and ringing. In order to solve the problem, a method in which restoration parameters can be easily adjusted has been reported (see Non-Patent Document 9).

  In addition, since both the RL method and the Wiener filter perform uniform processing on the entire image, there is a problem that the influence of noise and ringing becomes more conspicuous particularly in the textureless region of the restored image.

  Furthermore, an image restoration method that restores a true image using a plurality of deteriorated images has been proposed (see Non-Patent Document 10, Non-Patent Document 11, and Non-Patent Document 12). Yuan et al. Also proposed an image restoration method that can obtain a restored image with reduced ringing by using a combination of an image that contains a large amount of noise but a small amount of blur and an image that has a small amount of noise but a large amount of blur. (See Non-Patent Document 13).

  Also, not only the problem of image restoration processing that restores a true image from a deteriorated image, but also an image quality improvement method using a plurality of images has been proposed, for example, using a pair of images with and without flash to reduce noise. An image quality improvement method to be suppressed (see Non-Patent Document 14) and the like. Although the image processing method using image pairs is expected to improve the image processing result as compared with the image restoration method using only one deteriorated image, it is necessary to perform alignment between images having different properties. In the first place, there is also a problem that a plurality of images need to be taken for one subject.

  The present invention has been made under the circumstances as described above, and an object of the present invention is to restore ringing and noise by restoring a true image based on a known PSF and one degraded image. An object of the present invention is to provide an image restoration apparatus, an image restoration method, and an image restoration program that generate an image.

The present invention relates to an image restoration device that generates a restored image with reduced ringing and noise by restoring a true image based on a known PSF and one deteriorated image. An image restoration device that generates a restored image for the degraded image based on one degraded image that has been degraded by blur and a known PSF representing the blur, and generates a base image based on the degraded image A base image generation processing unit that generates a base image by performing processing, and a blur image is generated by performing convolution integration of the generated base image from the base image generation processing unit and the PSF. An image generation processing unit, which generates a deteriorated residual image by calculating a difference between the generated blurred image and the deteriorated image. An image restoration process for generating a restored residual image by performing image restoration processing on the generated degraded residual image based on the generated base image so as to change the strength of the constraint. A processing unit and a restored image generation processing unit that generates the restored image by adding the generated base image to the generated restored residual image, or in the image restoration processing unit, Using the generated base image, a constraint parameter representing the strength of the constraint so that the constraint is weak when the gradient of the generated base image is large and the constraint is strengthened when the gradient of the generated base image is small. Is adaptively set, and the generated residual image is subjected to image restoration processing using the set adaptive constraint parameter to generate a restored residual image, The body specifically, by minimizing the following evaluation function I, thereby generating a restoration residual image Y (u, v),
Where ‖ · ‖ ₂ represents the L2 norm, * represents the convolution integral, X (u, v) represents the degraded residual image, B (u, v) represents the PSF, α (u, v, v) represents the constraint parameter, or the constraint parameter α (u, v) is set as follows:
However, R (u, v) represents the generated base image, ▽ R (u, v) represents the gradient of the generated base image, and γ and λ are effectively achieved by representing the adjustment parameters. The

  Further, the object of the present invention is an image restoration device that generates a restored image for the deteriorated image based on one deteriorated image that has deteriorated due to blur and a known PSF that represents the blur. A base image generation processing unit that generates a base image by performing a base image generation process based on the degraded image, and a convolution integral of the generated base image from the base image generation processing unit and the PSF. A blurred image generation processing unit that generates a blurred image, a degraded residual image generation processing unit that generates a degraded residual image by calculating a difference between the generated blurred image and the degraded image; An image restoration processing unit that generates a restored residual image by performing an image restoration process on the generated degraded residual image, and the generated restoration residual image A restored image generation processing unit that generates the restored image by adding only a base image, or in the image restoration processing unit, an image obtained by a Wiener filter with respect to the generated degraded residual image This is effectively achieved by performing restoration processing or image restoration processing by the Richardson-Lucy method to generate a restored residual image.

Further, the object of the present invention is an image restoration device that generates a restored image for the deteriorated image based on one deteriorated image that has deteriorated due to blur and a known PSF that represents the blur. A base image generation processing unit that generates a base image by performing a base image generation process based on the degraded image, and a constraint on the degraded image based on the generated base image from the base image generation processing unit An image restoration processing unit that generates the restored image by performing an image restoration process so as to change the intensity of the image, or the image restoration processing unit uses the generated base image. Thus, the constraint is weakened when the gradient of the generated base image is large, and the constraint is strengthened when the gradient of the generated base image is small. A constraint parameter representing the strength of the image is adaptively set, and image restoration processing is performed on the degraded image using the set adaptive constraint parameter to generate the restored image. Generating the restored image Z (u, v) by minimizing the evaluation function I of
Where ‖ · ‖ ₂ represents the L2 norm, * represents the convolution integral, G (u, v) represents the degraded image, B (u, v) represents the PSF, α (u, v) Represents the constraint parameter, or the constraint parameter α (u, v) is set as follows:
However, R (u, v) represents the generated base image, ▽ R (u, v) represents the gradient of the generated base image, and γ and λ are effectively achieved by representing the adjustment parameters. The

  According to another aspect of the present invention, the base image generation processing unit generates a first restored image by performing an image restoration first process on the degraded image; A filter processing unit that generates the base image by performing a filtering process for denoising on the generated first restored image, or in the image restoration first processing unit, This is effectively achieved by performing the first image restoration processing by the Wiener filter or the first image restoration processing by the Richardson-Lucy method on the deteriorated image to generate the first restored image.

  In the base image generation processing unit, the base image generation processing is an iterative process, and the base image is gradually generated by the repetition process, and the base image generation processing unit The first filter processing is performed on the deteriorated image to generate an initial value of the base image, and the first filter processing unit and the convolution integration of the base image and the PSF are performed to obtain the first value. A first degradation image that generates a first degradation residual image by calculating a difference between the first blur image generation processing unit that generates one blur image, the generated first blur image, and the degradation image. A residual image generation processing unit, an image restoration first processing unit that generates a first restored residual image by performing an image restoration first process on the generated first degraded residual image, and generation The A second filter processing unit that generates an update image by performing a second filter process using the first restored residual image and the base image, and adds the generated update image to the base image The base image update processing unit that updates the base image, and determines whether or not a predetermined iterative process end condition is satisfied, and when it is determined that the iterative process end condition is satisfied, the updated base image A repetitive process end determination unit, which is used as the generated base image, and when it is determined that the repetitive process end condition is not satisfied, the updated base image is the base image and the repetitive process is continued. The repetition processing includes processing performed by the first blurred image generation processing unit, processing performed by the first degradation residual image generation processing unit, The processing performed in the image restoration first processing unit, the processing performed in the second filter processing unit, the processing performed in the base image update processing unit, and the processing performed in the repetition processing end determination unit Alternatively, the image restoration first processing unit performs image restoration first processing by the Wiener filter or image restoration first processing by the Richardson-Lucy method on the generated first degraded residual image. And effectively achieving the first restored residual image.

In the base image generation processing unit, the base image generation processing is an iterative process, and the base image is gradually generated by the repetition process, and the base image generation processing unit The first filter processing is performed on the deteriorated image to generate an initial value of the base image, and the first filter processing unit and the convolution integration of the base image and the PSF are performed to obtain the first value. A first degradation image that generates a first degradation residual image by calculating a difference between the first blur image generation processing unit that generates one blur image, the generated first blur image, and the degradation image. A first image restoration process is performed on the residual image generation processing unit and the generated first deteriorated residual image based on the base image so as to change the strength of the constraint. An image restoration first processing unit that generates an original residual image, an updated image is generated by performing a second filter process using the generated first restoration residual image and the base image, Two filter processing units, a base image update processing unit that updates the base image by adding the generated update image to the base image, a determination as to whether or not a predetermined iterative processing end condition is satisfied, When it is determined that the iterative process end condition is satisfied, the updated base image is set as the generated base image, and when it is determined that the iterative process end condition is not satisfied, the updated base image is A repetition process end determination unit that uses the base image and continues the repetition process, the repetition process being a process performed by the first blurred image generation processing unit The processing performed in the first degradation residual image generation processing unit, the processing performed in the image restoration first processing unit, the processing performed in the second filter processing unit, and performed in the base image update processing unit Or the processing performed by the repetitive processing end determination unit, or the image restoration first processing unit uses the base image to restrain the base image where the gradient of the base image is large. The constraint parameter indicating the strength of the constraint is adaptively set so that the constraint is weak and the constraint is strengthened at a small gradient of the base image, and is set for the generated first degradation residual image. adaptive restraint parameters performs image restoration first treatment with a, to generate a first recovered residue image, specifically, by minimizing the following evaluation function I, the first restored residue image y ₀ ( u, v) And,
Where ‖ · ‖ ₂ represents the L2 norm, * represents the convolution integral, x ₀ (u, v) represents the first degraded residual image, B (u, v) represents the PSF, α (u, v) represents the constraint parameter, or the constraint parameter α (u, v) is set as follows:
However, r ₀ (u, v) represents the base image, ▽ r ₀ (u, v) represents the gradient of the base image, and γ and λ are effectively achieved by representing the adjustment parameters.

  The present invention performs image restoration processing on a deteriorated image based on one deteriorated image that has deteriorated due to blur and a known PSF that represents blur, and can generate a restored image that suppresses ringing, noise, and the like. The present invention relates to an image restoration apparatus, an image restoration method, and an image restoration program.

  According to the feature of “image restoration using degraded residual image” of the present invention, the degraded image is divided into a base image and a degraded residual image, and image restoration processing is performed on the degraded residual image. As a result, the amplitude (variance) of the restored residual image (restored residual image) is reduced, and as a result, the ringing can be reduced.

  Further, according to the feature of the present invention, which is “base image generation by repetitive processing”, in the process of progressively generating a base image from one deteriorated image, a restored residual image (restored residual image) The base image is updated based on the updated base image, and further, the degradation residual image is also updated based on the updated base image, and the base image is gradually updated by repeating the image restoration processing on the updated degradation residual image. It is possible to generate a sharper restored image by using the base image approaching the true image and the restored residual image restored based on the base image. There is an effect.

  Further, according to the feature of the present invention, which is “image restoration by adaptive constraint parameter”, when performing image restoration on the degraded residual image, the constraint parameter is based on the relationship between the base image and the degraded residual image. Adaptively set the constraint parameters so that the constraint is weaker when the base image gradient is large and the constraint is stronger when the base image gradient is small. By doing so, it is possible to generate a restored image with reduced ringing and noise while preserving edges.

  In other words, according to the feature of the present invention “image restoration by adaptive constraint parameters”, the edge portion is sharpened and there is an effect of noise reduction in the textureless region. Further, the ringing can be effectively reduced in the textureless region where the ringing is conspicuous.

  Thus, according to the present invention, the effect achieved by the image restoration method using a plurality of deteriorated images proposed by Yuan et al. Can be realized by using only one deteriorated image.

  In other words, according to the present invention, it is possible to generate a restored image with reduced ringing and noise by using only one deteriorated image without using a plurality of deteriorated images like Yuan et al. There is a remarkable effect of being able to.

  First, the focus of the present invention will be described.

  As described in the background art, in the image restoration technique for restoring a true image from a degraded image, there is a problem that ringing occurs corresponding to the edge strength of the restored image. The inventors of the present invention pay attention to this point and reduce the influence of ringing in the restored image by reducing the edge strength of the image restored from the degraded image (restored image). Invented an image restoration technique that can obtain a restored image with reduced image quality.

  The present invention relates to an image restoration device that generates a restored image in which ringing and noise are suppressed for a deteriorated image based on one deteriorated image that has been deteriorated by blur and a known PSF that represents blur. The present invention relates to an image restoration method and an image restoration program.

  In the present invention, it is assumed that the blur in the deteriorated image is uniform. In addition, when applying the present invention, it is also a premise that PSF is known. Even if the PSF is unknown, in order to estimate the PSF, for example, the PSF may be measured using a measurement sensor such as a gyro sensor, or a method for estimating the PSF from the deteriorated image is used. It is also possible. As a method for estimating the PSF from the deteriorated image, for example, the “blurring information detection method” disclosed in Patent Document 1 or the method described in Non-Patent Document 4 can be used.

In the present invention, a novel technical idea of decomposing a degraded image into a “base image” and a “degraded residual image” is proposed. In other words, a true image is decomposed into a “base image” and a “residual image”. Proposing a novel technical idea to do, and a technique called “image restoration using degraded residual images”, a technique called “base image generation by iterative processing”, and “degradation” built based on that technical idea A major feature is a technique called “image restoration by adaptive constraint parameters set according to the conspicuousness of ringing in an image”.

<1> Image Restoration Using Degraded Residual Images First, “image restoration using degraded residual images” will be described.

  In the image processing technology field, in image restoration in which a true image is restored from a deteriorated image, an artifact called ringing that occurs in the restored image becomes a problem.

  FIG. 1 shows an example of ringing that occurs in a deteriorated image (blurred image) and a restored image in a one-dimensional image. As can be seen from FIG. 1, a wavy artifact called ringing occurs around the edge of the restored image. Ringing is an artifact that appears prominently at discontinuities such as edges, and is considered to be a kind of Gibbs phenomenon. The size of this ringing is characterized by being proportional to the size of the gap of discontinuities such as edges.

  For this reason, the inventors of the present invention have come up with the idea that if the gap at the discontinuity of the image to be restored can be reduced by some method as shown in FIG. 2, the influence of ringing can also be reduced. It was.

  In other words, the present invention has been made in view of the fact that the ringing can be reduced if the amplitude of the image to be restored can be reduced. As can be seen from the definition of the “degraded residual image” in the present invention, the amplitude is smaller than that of the original image (true image). As a result, image restoration using the degraded residual image is performed. This makes it possible to obtain a restored image in which the influence of ringing is suppressed.

Here, the image restoration problem using the degradation residual image is formulated.
Is the vector representing the coordinates on the image, and the true image (original image)
And a degraded image (blurred image)
Then, the image restoration problem of restoring the original image from the deteriorated image can be formulated by the following equation (1).

here,
Is the PSF representing blur,
Represents noise, and * represents convolution integral.

Some base image
Is obtained, the original image
The base image
And residual image
Is expressed as the following formula 2.

Using the above equation 2, the “image restoration problem for restoring the original image” formulated in the above equation 1 is formulated as the following equation 3 as an “image restoration problem for restoring the residual image”. be able to.

here,
Represents a degradation residual image and is defined as shown in Equation 4 below.

Base image
Is known, that is, if a base image is generated from a degraded image, the residual image can be restored from the degraded residual image based on Equation 3 above, and thus the original image can be restored. .

  In other words, the “image restoration problem of restoring the original image from the degraded image” formulated in Equation 1 is replaced with the “image for restoring the residual image from the degraded residual image” formulated in Equation 3 through the base image. Redefining it as a “restoration problem” is one of the major features of the present invention.

  In the present invention, image restoration is performed on the degraded residual image by performing image restoration on the degraded residual image, and the amplitude of the degraded residual image is smaller than the amplitude of the original image. A restored image with reduced ringing can be generated.

  Further, as can be seen by comparing Equations 1 and 3, Equations 1 and 3 are formally completely different, although there are differences between the original image and the residual image, and there are differences between the degraded image and the degraded residual image. The same.

In other words, in the present invention, the image restoration method that restores a true image from a normal degraded image that has been proposed in the past is applied as it is to image restoration processing that restores a residual image from a degraded residual image. Of course it is possible.

<2> Base Image Generation by Iterative Processing Next, the generation of a base image by repetition processing using only one deteriorated image will be described. That is, the present invention is characterized in that an image close to a true image called a “base image” is gradually generated from one deteriorated image while being repeatedly updated.

  In other words, the “base image” referred to in the present invention is preferably an image that does not contain ringing or noise. The base image preferably represents the approximate shape of a true image (original image). Although it is desirable that the base image and the original image are close to each other, they do not have to coincide with each other, and the rough structure and the edges may be similar.

  In the present invention, the technical idea of “image restoration using a degraded residual image” described in <1> is used even when a base image is gradually generated from a single degraded image.

  In other words, a first base image is generated from the deteriorated image by some method, a difference between the deteriorated image and the generated base image is calculated to generate a deteriorated residual image, and the generated deteriorated residual image Restore the residual image from. The restored residual image (hereinafter referred to as “restored residual image”) contains some noise and ringing, but retains sharp edges and is closer to the original image than the degraded image. is there.

  In the present invention, based on the restored residual image, the base image is updated, a degraded residual image is generated using the updated base image and degraded image, and the generated degraded residual image is again generated. A base image is gradually generated by using an iterative method of performing image restoration.

  First, the first base image generation method will be described. In the first base image, it is only necessary to represent the approximate shape of the original image, that is, the low frequency component, without ringing or noise. In the present invention, it is desirable to generate the first base image by applying a bilateral filter to the degraded image.

Next, the n-th repeated base image, that is, the n-th base image
(N + 1) th base image
The updating will be specifically described.

  Although the base image is updated based on the restored residual image, it is not desirable to use the restored residual image as it is as the (n + 1) th base image because the restored residual image includes noise and ringing.

In the image restoration using the degradation residual image, the restored image is the nth base image.
And the nth restored residual image, which is the result of restoring the residual image from the degraded residual image
The sum of This nth restored residual image
Includes the effects of ringing and noise, although the amplitude is relatively small.

  In the present invention, after removing or reducing the influence of ringing or noise included in the restored residual image, the restored residual image from which the influence of ringing or noise is removed or reduced is added to the base image. Update.

  In the present invention, various methods can be used as a method for reducing the influence of ringing and noise included in the restored residual image. As a preferred example, a joint bilateral filter using a base image is used. (See Non-Patent Document 14).

  The joint bilateral filter is known to be effective in improving the image quality of an image pair, and is defined by the following equation (5).

here,
Is a function representing a joint bilateral filter. G _d and G _r are Gaussian functions having variances σ _d and σ _r , respectively. Also,
Represents the neighborhood window of
Represents a normalization term.

In a normal bilateral filter, a weight based on a difference between pixel values of a restored residual image is used. On the other hand, with the joint bilateral filter, the difference in pixel values of the base image
Use weights based on.

  In the present invention, the base image represents the approximate shape of the original image and has the property of being less affected by ringing and noise. Therefore, by using a joint bilateral filter, the residual image can be effectively restored. The influence of ringing and noise included in the difference image can be reduced.

(N + 1) th base image using joint bilateral filter
Is updated as shown in Equation 6 below.

here,
Is an n-th residual image (n-th restored residual image).

Updated base image
Using (), the (n + 1) th restored residual image
Is obtained. Thus, in the present invention, the base image is generated while being gradually updated.

<3> Image Restoration Using Adaptive Constraint Parameters In <2>, base image generation by iterative processing has been described in detail. In the present invention, during the process of generating the base image by the progressive repetitive processing, the base image is gradually generated by repeatedly performing the image restoration on the degraded residual image.

  By the way, the image restoration process for the degraded residual image can be performed in the same manner as the image restoration for the normal degraded image. Therefore, in the present invention, based on the MAP (Maximum A Posteriori) method, image restoration is performed on the degraded residual image.

  The MAP method in image restoration varies depending on the definition of likelihood distribution and prior probability distribution, etc., but the most typical MAP method is formulated as an optimization problem of the evaluation function E expressed by the following equation 7, It is formulated as an optimization function of an evaluation function consisting of an error term and a constraint term.

Here, α is a constraint term parameter (constant constraint parameter) corresponding to the prior probability, representing the strength of the constraint. As a result, the image restoration process based on the evaluation function of Equation 7 can be considered to be an image restoration process using a Wiener filter when the ratio of the power spectrum of the noise to the power spectrum of the original image is assumed to be constant (non-conversion). (See Patent Document 8). Also residual image
It is possible to interpret that the prior probability distribution of is modeled by a Gaussian function. On the other hand, since the prior probability distribution of a natural image is different from a Gaussian function, it is known that when the prior probability distribution is modeled with a Gaussian function, the restored image becomes excessively smooth ( (Refer nonpatent literature 15).

  The inventors of the present invention adaptively set the constraint parameter based on the gradient of the base image instead of fixing the constraint parameter with a constant when performing image restoration on the degraded residual image based on the MAP method. Invented changing (spatially).

  That is, in the present invention, the constraint parameter is set to be large when the variance of the degraded residual image is expected to be small, and the constraint parameter is set to be small when the variance of the degraded residual image is expected to be large.

  Furthermore, in the present invention, since the base image corresponds to the approximate shape of the original image, that is, the low frequency component, and the degradation residual image corresponds to the high frequency component, the base image has a large gradient. The variance of the degradation residual image is large, and the variance of the degradation residual image is expected to be small where the gradient of the base image is small.

  Therefore, in the present invention, the constraint parameter is adaptively set based on the gradient of the base image, that is, the constraint is weak when the gradient of the base image is large and the constraint is strengthened when the gradient of the base image is small. .

  The evaluation function having the “constraint parameter that can be set adaptively” according to the present invention is expressed by the following equation (8).

here,
Represents the gradient of the base image.
Is a constraint parameter that can be set adaptively. Γ and λ are the constraint parameters that can be set adaptively.
Is an adjustment parameter for designing (setting).

  Although it is theoretically possible to express the evaluation function of Equation 8 in vector and analytically obtain a restored residual image, the unknown is a large-scale problem that is the same as the number of pixels. A restored residual image is obtained by the gradient method.

Further, as human vision, there is a phenomenon that ringing is hardly perceived in an area where texture is abundant, but ringing is perceived remarkably in an area where there is no texture. According to the evaluation function of the above formula 8, as a result, there is also an effect of suppressing ringing in a region having no texture. Therefore, it can be said that the image restoration method according to the present invention is a visually reasonable method.

Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

  FIG. 3 is a block diagram showing the first embodiment of the image restoration apparatus according to the present invention.

  As shown in FIG. 3, the image restoration apparatus according to the first embodiment of the present invention (hereinafter also simply referred to as “image restoration apparatus 1”) includes a base image generation processing unit 10, a blurred image generation processing unit 20, and the like. The degradation residual image generation processing unit 30, the image restoration processing unit 40, and the restoration image generation processing unit 50 are composed of a single degraded image that has deteriorated due to blur and a known PSF that represents blur. Based on this, a restored image for the deteriorated image is generated.

  In the image restoration apparatus 1 of the present invention, first, the base image generation processing unit 10 generates a base image by performing base image generation processing based on the deteriorated image. The generated base image generated by the base image generation processing unit 10 has an approximate shape similar to a true image and does not include ringing or noise. The generated base image is ideal as it is closer to a true image, but can be used in the present invention even when it is not so close.

  Next, the blurred image generation processing unit 20 performs convolution integration between the generated base image output from the base image generation processing unit 10 and the PSF, that is, the generated base image is converted into a PSF representing blur. By blurring, a blurred image is generated.

  Then, the degradation residual image generation processing unit 30 generates a degradation residual image by calculating a difference image between the blurred image generated by the blurred image generation processing unit 20 and the degradation image.

  Next, the image restoration processing unit 40 performs image restoration processing on the deterioration residual image generated by the deterioration residual image generation processing unit 30 so as to change the strength of the constraint based on the generated base image. By doing so, a restored residual image is generated.

  Finally, the restored image generation processing unit 50 generates a restored image by adding the generated base image to the restored residual image generated by the image restoration processing unit 40.

  FIG. 4 is a block diagram showing a second embodiment of the image restoration apparatus according to the present invention.

  As shown in FIG. 4, an image restoration apparatus according to the second embodiment of the present invention (hereinafter also simply referred to as “image restoration apparatus 2”) includes a base image generation processing unit 10, a blurred image generation processing unit 20, and The degradation residual image generation processing unit 30, the image restoration processing unit 41, and the restoration image generation processing unit 50 are composed of one degradation image that has been degraded by blur and a known PSF that represents blur. Based on this, a restored image for the deteriorated image is generated.

  In the image restoration apparatus 2 of the present invention, first, the base image generation processing unit 10 generates a base image by performing base image generation processing based on the deteriorated image. The generated base image generated by the base image generation processing unit 10 has an approximate shape similar to a true image and does not include ringing or noise. The generated base image is ideal as it is closer to a true image, but can be used in the present invention even when it is not so close.

  Next, the blurred image generation processing unit 20 performs convolution integration between the generated base image output from the base image generation processing unit 10 and the PSF, that is, the generated base image is converted into a PSF representing blur. By blurring, a blurred image is generated.

  Then, the degradation residual image generation processing unit 30 generates a degradation residual image by calculating a difference image between the blurred image generated by the blurred image generation processing unit 20 and the degradation image.

  Next, the image restoration processing unit 41 performs image restoration processing on the degraded residual image generated by the degraded residual image generation processing unit 30, thereby generating a restored residual image. Further, the image restoration processing unit 41 performs image restoration processing using an existing image restoration method, for example, a Wiener filter, an RL method, or an image restoration method described in Patent Document 2.

  Finally, the restored image generation processing unit 50 generates a restored image by adding the generated base image to the restored residual image generated by the image restoration processing unit 40.

  FIG. 5 is a block diagram showing the first embodiment of the base image generation processing unit in the image restoration apparatus according to the present invention.

  As shown in FIG. 5, a base image generation processing unit 10 (hereinafter also simply referred to as “base image generation processing device 1”) according to the first embodiment of the present invention includes an image restoration first processing unit 101, a filter, And a processing unit 102.

  In the base image generation processing device 1 of the present invention (that is, the base image generation processing unit 10 shown in FIG. 5), first, the image restoration first processing unit 101 performs the image restoration first processing on the degraded image. A first restored image is generated. The image restoration first process performed by the image restoration first processing unit 101 is an image restoration process using an existing image restoration method. Examples of the existing image restoration method include a Wiener filter, an RL method, and an image restoration method described in Patent Document 2.

  Next, the filter processing unit 102 performs a filtering process for denoising on the first restored image generated by the image restoration first processing unit 101, thereby generating a base image. The filter processing performed by the filter processing unit 102 is filter processing using, for example, a low-pass filter, a median filter, or a bilateral filter.

  In the present invention, the base image generation processing device 1 (that is, the base image generation processing unit 10 shown in FIG. 5) is used as the base image generation processing unit 10 in the image restoration device 1 and the base image generation in the image restoration device 2. It can also be used as the processing unit 10.

  FIG. 6 is a block diagram showing a second embodiment of the base image generation processing unit in the image restoration apparatus according to the present invention.

  As shown in FIG. 6, the base image generation processing unit 10 (hereinafter also simply referred to as “base image generation processing device 2”) according to the second embodiment of the present invention includes a first filter processing unit 111 and a first filter processing unit 111. The blurred image generation processing unit 112, the first degradation residual image generation processing unit 113, the image restoration first processing unit 114, the second filter processing unit 115, the base image update processing unit 116, and the iterative processing end determination unit 118.

  The base image generation processing performed in the base image generation processing device 2 (that is, the base image generation processing unit 10 shown in FIG. 6) of the present invention is a repetitive processing, and the base image is gradually generated by the repetitive processing. The

  In the base image generation processing device 2 of the present invention (that is, the base image generation processing unit 10 shown in FIG. 6), first, the first filter processing unit 111 performs the first filter processing on the deteriorated image, so that the base Generate the initial value of the image. The first filter processing performed by the first filter processing unit 111 is filter processing using, for example, a low-pass filter, a median filter, or a bilateral filter. Note that the first filter processing unit 111 may perform the first filter processing on the deteriorated image without changing the deteriorated image itself as the initial value of the base image.

  Next, the first blurred image generation processing unit 112 generates a first blurred image by performing convolution integration between the base image and the PSF.

  Then, the first deterioration residual image generation processing unit 113 calculates a difference image between the first blurred image generated by the first blurred image generation processing unit 112 and the deteriorated image, and thereby the first deterioration residual image. Is generated.

  Next, the image restoration first processing unit 114 performs the image restoration first process on the first degradation residual image generated by the first degradation residual image generation processing unit 113, whereby the first restoration residual is performed. A difference image is generated.

  And the 2nd filter process part 115 produces | generates an update image by performing a 2nd filter process using the 1st decompression | restoration residual image and base image which were produced | generated by the image restoration 1st process part 114. FIG. The second filter processing performed by the second filter processing unit 115 is filter processing using, for example, a joint bilateral filter, a bilateral filter, or a low-pass filter. The details of the joint bilateral filter are disclosed in Non-Patent Document 14.

  Next, the base image update processing unit 116 updates the base image by adding the update image generated by the second filter processing unit 115 to the base image.

  Finally, the iterative process end determination unit 118 determines whether or not a predetermined iterative process end condition is satisfied, and if it is determined that the iterative process end condition is satisfied, the updated base image is set as a generated base image. If it is determined that the repetitive processing end condition is not satisfied, the renewed base image is set as the base image and the repetitive processing is continued. Further, the predetermined iterative processing end condition is a condition 1 in which there is no change before and after the update in the base image, a condition 2 in which a predetermined number of repetitive processes is reached, or a square sum of pixel values of the updated image is equal to or less than a predetermined value Condition 3

  In addition, the repetitive processing performed in the base image generation processing device 2 (that is, the base image generation processing unit 10 shown in FIG. 6) of the present invention is processing performed in the first blurred image generation processing unit 112, first degradation. Processing performed by the residual image generation processing unit 113, processing performed by the image restoration first processing unit 114, processing performed by the second filter processing unit 115, processing performed by the base image update processing unit 116, And a process performed by the repeated process end determination unit 118.

  In the present invention, the base image generation processing device 2 (that is, the base image generation processing unit 10 shown in FIG. 6) is used as the base image generation processing unit 10 in the image restoration device 1 and the base image generation in the image restoration device 2. It can also be used as the processing unit 10.

  FIG. 7 is a block diagram showing a third embodiment of the base image generation processing unit in the image restoration apparatus according to the present invention.

  As shown in FIG. 7, the base image generation processing unit 10 (hereinafter also simply referred to as “base image generation processing device 3”) according to the third embodiment of the present invention includes a first filter processing unit 111 and a first filter processing unit 111. The blurred image generation processing unit 112, the first deterioration residual image generation processing unit 113, the image restoration first processing unit 117, the second filter processing unit 115, the base image update processing unit 116, and the iterative processing end determination unit 118.

  The base image generation processing performed in the base image generation processing device 3 (that is, the base image generation processing unit 10 shown in FIG. 7) of the present invention is a repetitive processing, and the base image is gradually generated by the repetitive processing. The

  In the base image generation processing device 3 of the present invention (that is, the base image generation processing unit 10 shown in FIG. 7), first, the first filter processing unit 111 performs the first filter processing on the degraded image, so that the base Generate the initial value of the image. The first filter processing performed by the first filter processing unit 111 is filter processing using, for example, a low-pass filter, a median filter, or a bilateral filter. Note that the first filter processing unit 111 may perform the first filter processing on the deteriorated image without changing the deteriorated image itself as the initial value of the base image.

  Next, the first blurred image generation processing unit 112 generates a first blurred image by performing convolution integration between the base image and the PSF.

  Then, the first deterioration residual image generation processing unit 113 calculates a difference image between the first blurred image generated by the first blurred image generation processing unit 112 and the deteriorated image, and thereby the first deterioration residual image. Is generated.

  Next, the image restoration first processing unit 117 changes the restraint strength based on the base image with respect to the first deterioration residual image generated by the first deterioration residual image generation processing unit 113. A first restored residual image is generated by performing the first image restoration process.

  And the 2nd filter process part 115 produces | generates an update image by performing a 2nd filter process using the 1st decompression | restoration residual image and base image which were produced | generated by the image restoration 1st process part 114. FIG. The second filter processing performed by the second filter processing unit 115 is filter processing using, for example, a joint bilateral filter, a bilateral filter, or a low-pass filter. The details of the joint bilateral filter are disclosed in Non-Patent Document 14.

  Next, the base image update processing unit 116 updates the base image by adding the update image generated by the second filter processing unit 115 to the base image.

  Finally, the iterative process end determination unit 118 determines whether or not a predetermined iterative process end condition is satisfied, and if it is determined that the iterative process end condition is satisfied, the updated base image is set as a generated base image. If it is determined that the repetitive processing end condition is not satisfied, the renewed base image is set as the base image and the repetitive processing is continued. Further, the predetermined iterative processing end condition is a condition 1 in which there is no change before and after the update in the base image, a condition 2 in which a predetermined number of repetitive processes is reached, or a square sum of pixel values of the updated image is equal to or less than a predetermined value Condition 3

  In addition, the repetitive processing performed in the base image generation processing device 3 (that is, the base image generation processing unit 10 shown in FIG. 7) of the present invention is processing performed in the first blurred image generation processing unit 112, first degradation. Processing performed by the residual image generation processing unit 113, processing performed by the image restoration first processing unit 117, processing performed by the second filter processing unit 115, processing performed by the base image update processing unit 116, And a process performed by the repeated process end determination unit 118.

  In the present invention, the base image generation processing device 3 (that is, the base image generation processing unit 10 shown in FIG. 7) is used as the base image generation processing unit 10 in the image restoration device 1 and the base image generation in the image restoration device 2. It can also be used as the processing unit 10.

  FIG. 8 is a block diagram showing a third embodiment of the image restoration apparatus according to the present invention.

  As shown in FIG. 8, an image restoration apparatus according to the third embodiment of the present invention (hereinafter also simply referred to as “image restoration apparatus 3”) includes a base image generation processing unit 10 and an image restoration processing unit 42. A restored image for the deteriorated image is generated based on one deteriorated image that has been configured and deteriorated due to blur and a known PSF that represents blur.

  In the image restoration apparatus 3 of the present invention, first, the base image generation processing unit 10 generates a base image by performing a base image generation process based on the deteriorated image. The generated base image generated by the base image generation processing unit 10 has an approximate shape similar to a true image and does not include ringing or noise. The generated base image is ideal as it is closer to a true image, but can be used in the present invention even when it is not so close.

  Next, the image restoration processing unit 42 performs image restoration processing on the degraded image so as to change the strength of the constraint based on the generated base image output from the base image generation processing unit 10. A restored image is generated.

  In the present invention, as the base image generation processing unit 10 in the image restoration device 3, the base image generation processing device 1 (that is, the base image generation processing unit 10 shown in FIG. 5) and the base image generation processing device 2 (that is, in FIG. 6). The base image generation processing unit 10) or the base image generation processing device 3 (that is, the base image generation processing unit 10 shown in FIG. 7) can be used.

  In the image restoration processing unit 42 in the image restoration device 3 of the present invention, the generated base image is used, and the restriction is weak when the gradient of the generated base image is large, and is restricted when the gradient of the generated base image is small. A constraint parameter indicating the strength of the constraint is adaptively set so as to be strengthened, and an image restoration process is performed on the deteriorated image using the set adaptive constraint parameter to generate a restored image.

  Specifically, the restored image Z (u, v) is generated by minimizing the evaluation function I expressed by the following equation (10).

Here, ‖ · ‖ ₂ represents the L2 norm, and * represents the convolution integral. G (u, v) represents a degraded image. B (u, v) represents PSF. α (u, v) represents an adaptive constraint parameter, and is set as shown in Equation 11 below.

Here, R (u, v) represents a generated base image. R (u, v) represents the gradient of the generated base image. Further, γ and λ represent adjustment parameters for designing (setting) a constraint parameter α (u, v) that can be set adaptively.

The image restoration apparatus according to the present invention can be implemented by software (computer program) using a computer system, and can be implemented by an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or a field programmable gate (FPGA). Of course, it can also be implemented by hardware such as Array).

Next, a preferred embodiment of the image restoration apparatus according to the present invention will be described in detail.
<Example 1>
The image restoration apparatus according to the first embodiment of the present invention has the configuration of the image restoration apparatus 1 shown in FIG. 3 and the base image generation processing unit 10 shown in FIG.

  The flow of the image restoration process performed in the image restoration apparatus according to the first embodiment of the present invention is as follows.

First, a generated base image is generated from a deteriorated image by processing performed by the base image generation processing unit 10 shown in FIG. Note that the pixel value of the image takes a value from 0 to 255. In the case of an RGB full color image, after the conversion to the YCrCb color space, the processing described below is performed for each color channel, and finally the RGB color space. Converted again and restored the image. In the case of a simulation image, the PSF uses the PSF used for generating a deteriorated image, and in the case of an actual image, the PSF estimated by the “blurring information detection method” disclosed in Patent Document 1. used.

(A1)
First, the first filter process is performed. In the first embodiment, a bilateral filter defined by the following formula 12 is used in the first filter processing.

However, (u, v) represents coordinates on the image. r ₀ (u, v) is an output result of the bilateral filter, that is, an initial value of the generated base image. w ₂ (i, j) represents a two-dimensional Gaussian function, and w ₁ () represents a one-dimensional Gaussian function. G (u, v) represents a degraded image. At this time, the standard deviation of the two-dimensional Gaussian function is set to 3.0, and the standard deviation of the one-dimensional Gaussian function is set to 25.0.

The image obtained by the first filter processing is used as the initial value of the base image, that is, the initial base image.

(A2)
Next, the first deteriorated residual image x ₀ (u, v) corresponding to the initial base image has the following number as the difference between the result of the convolution integration of the initial base image and the PSF (first blurred image) and the deteriorated image: 13 is calculated.

Here, B (u, v) represents PSF. * Represents a convolution integral.

(A3)
Next, image restoration processing is performed on the first degraded residual image using the base image. That is, the restriction is weak when the gradient of the base image is large, and the constraint is strong when the gradient of the base image is small, and the first deteriorated residual image is restored.

Specifically, the first restored residual image y ₀ (u, v) is generated by minimizing the evaluation function I expressed by the following _equation (14).

Here, ‖ · ‖ ₂ is an L2 norm, and α (u, v) is an adaptive constraint parameter indicating the strength of the constraint. The constraint parameter α (u, v) is set to be small when the gradient of the base image is large and large when the gradient of the base image is small. In Example 1, the following formula 15 was set.

Here, r ₀ (u, v) represents the base image, and ▽ r ₀ (u, v) represents the gradient of the base image. Γ and λ represent adjustment parameters for designing (setting) a constraint parameter α (u, v) that can be set adaptively. In Example 1, γ = 0.006 and λ = 2.5 were set.

(A4)
The second filter process of FIG. 7 is performed using the base image r ₀ (u, v) corresponding to the first restored residual image y ₀ (u, v) thus obtained. In Example 1, the joint bilateral filter defined by the following equation 16 was used.

Here, z ₀ (u, v) is the output result of the joint bilateral filter, that is, the generated updated image, w ₂ (i, j) represents a two-dimensional Gaussian function, and w ₁ () is Represents a one-dimensional Gaussian function. At this time, the standard deviation of the two-dimensional Gaussian function is set to 5.0, and the standard deviation of the one-dimensional Gaussian function is set to 25.0.

(A5)
By adding the output result z ₀ (u, v) of the second filter processing to the base image r ₀ (u, v), the base image is updated as shown in Equation 17 below.

Here, r ₁ (u, v) is an updated base image (updated base image).

A base image obtained by repeating the processes (a2) to (a5) a predetermined number of times is output as an output (generated base image) of the base image generation processing unit 10. In the first embodiment, the predetermined number of times is set to seven. R (u, v) represents a generated base image output from the base image generation processing unit 10.

Next, using the generated base image R (u, v) output from the base image generation processing unit 10, image restoration processing is performed based on processing performed by the image restoration device 1 illustrated in FIG. The corresponding degradation residual image X (u, v) is a difference image between the generated base image R (u, v) and PSF convolution integration result (blurred image) and the degradation image G (u, v). The following formula 18 is calculated.

Next, the degraded residual image is restored using the generated base image R (u, v). That is, the restriction is weak when the gradient of the generated base image is large, and the constraint is strong when the gradient of the generated base image is small, and the degraded residual image is restored.

  Specifically, the reconstructed residual image Y (u, v) is generated by minimizing the evaluation function I expressed by the following equation (19).

Here, ‖ · ‖ ₂ is an L2 norm, and α (u, v) is an adaptive constraint parameter indicating the strength of the constraint. The constraint parameter α (u, v) is set to be small when the gradient of the generated base image R (u, v) is large and large when the gradient of the generated base image R (u, v) is small. In Example 1, the following equation 20 was set.

Here, R (u, v) represents the generated base image, and ▽ R (u, v) represents the gradient of the generated base image. γ and λ represent adjustment parameters for designing (setting) a constraint parameter α (u, v) that can be set adaptively. In Example 1, γ = 0.06 and λ = 2.5 were set.

Finally, the final restored image Z (u, v) generated by the image restoration device 1 as the output result of the image restoration device 1 is the restored residual image (restored residual image) Y (u, v ) And the generated base image R (u, v) output from the base image generation processing unit 10 is calculated as shown in Equation 21 below.

<Example 2>
The image restoration apparatus according to the second embodiment of the present invention has the configuration of the image restoration apparatus 2 shown in FIG. 4 and the base image generation processing unit 10 shown in FIG.

  The flow of the image restoration process performed in the image restoration apparatus according to the second embodiment of the present invention is as follows.

First, a generated base image is generated from a deteriorated image by processing performed by the base image generation processing unit 10 shown in FIG. Note that the pixel value of the image takes a value from 0 to 255. In the case of an RGB full color image, after the conversion to the YCrCb color space, the processing described below is performed for each color channel, and finally the RGB color space. Converted again and restored the image. In the case of a simulation image, the PSF uses the PSF used for generating a deteriorated image, and in the case of an actual image, the PSF estimated by the “blurring information detection method” disclosed in Patent Document 1. used.

(B1)
First, the first filter process is performed. In the second embodiment, a bilateral filter defined by the following Expression 22 is used in the first filter processing 1.

However, (u, v) represents coordinates on the image. r ₀ (u, v) is an output result of the bilateral filter, that is, an initial value of the generated base image. w ₂ (i, j) represents a two-dimensional Gaussian function, and w ₁ () represents a one-dimensional Gaussian function. G (u, v) represents a degraded image. At this time, the standard deviation of the two-dimensional Gaussian function is set to 3.0, and the standard deviation of the one-dimensional Gaussian function is set to 25.0.

The image obtained by the first filter processing is used as the initial value of the base image, that is, the initial base image.

(B2)
Next, the first deteriorated residual image x ₀ (u, v) corresponding to the initial base image has the following number as the difference between the result of the convolution integration of the initial base image and the PSF (first blurred image) and the deteriorated image: 23 is calculated.

Here, B (u, v) represents PSF. * Represents a convolution integral.

(B3)
Next, an image restoration process using a Wiener filter is performed on the first deteriorated residual image. A first restored residual image y ₀ (u, v), which is a restoration result of the first degraded residual image, is generated as shown in the following Expression 24.

However, F- ¹ represents an inverse Fourier transform. B (u, v) represents PSF,
Represents the Fourier transform of B (u, v).
Represents the complex conjugate of
Represents the size of. Also,
Represents a Fourier transform of the first degradation residual image x ₀ (u, v), and Γ represents an adjustment parameter.

Details of the Wiener filter are described in Chapter 8-1-3 of Non-Patent Document 16. The value of the adjustment parameter Γ is 0.01, 0.02, 0.006 depending on the image.

(B4)
The second filter process of FIG. 6 is performed using the base image r ₀ (u, v) corresponding to the first restored residual image y ₀ (u, v) thus obtained. In Example 2, a joint bilateral filter defined by the following Equation 25 was used.

Here, z ₀ (u, v) is the output result of the joint bilateral filter, that is, the generated updated image, w ₂ (i, j) represents a two-dimensional Gaussian function, and w ₁ () is Represents a one-dimensional Gaussian function. At this time, the standard deviation of the two-dimensional Gaussian function is set to 5.0, and the standard deviation of the one-dimensional Gaussian function is set to 25.0.

(B5)
By adding the output result z ₀ (u, v) of the second filter processing to the base image r ₀ (u, v), the base image is updated as shown in the following Expression 26.

Here, r ₁ (u, v) is an updated base image (updated base image).

A base image obtained by repeating the processes (b2) to (b5) a predetermined number of times is output as an output (generated base image) of the base image generation processing unit 10. In the second embodiment, the predetermined number of times is set to seven. R (u, v) represents a generated base image output from the base image generation processing unit 10.

Next, using the generated base image R (u, v) output from the base image generation processing unit 10, image restoration processing is performed based on processing performed by the image restoration device 2 illustrated in FIG. The corresponding degradation residual image X (u, v) is a difference image between the generated base image R (u, v) and PSF convolution integration result (blurred image) and the degradation image G (u, v). The following equation 27 is calculated.

Next, image restoration processing using a Wiener filter is performed on the degraded residual image X (u, v). A restored residual image Y (u, v), which is a restoration result of the degraded residual image, is generated as shown in the following Expression 28.

However, F- ¹ represents an inverse Fourier transform. B (u, v) represents PSF,
Represents the Fourier transform of B (u, v).
Represents the complex conjugate of
Represents the size of. Also,
Represents the Fourier transform of the degraded residual image X (u, v). Γ represents an adjustment parameter.

Details of the Wiener filter are described in Chapter 8-1-3 of Non-Patent Document 16. The value of the adjustment parameter Γ is 0.01, 0.02, 0.006 depending on the image.

Finally, the final restored image Z (u, v) generated by the image restoration device 2 as the output result of the image restoration device 2 is the restored residual image (restored residual image) Y (u, v ) And the generated base image R (u, v) output from the base image generation processing unit 10, the following equation 29 is calculated.

FIG. 9 shows an example of an image in the base image generation process in the base image generation processing unit in the image restoration apparatus according to the present invention. FIG. 9A shows a deteriorated image. The first base image can be generated by performing the first filter processing with the bilateral filter on the degraded image in FIG.

  After the first base image is generated, the base image and the first restored residual image are updated by iterative processing, and when a predetermined iterative processing end condition is satisfied, the iterative processing ends and the base image ( A generated base image) is generated.

  In the n-th repeated processing, the n-th first degradation residual image can be obtained from the degraded image and the n-th base image.

  When the n-th base image shown in FIG. 9B is given, the n-th first deterioration residual image shown in FIG. 9C is generated. Image restoration processing is performed on the n-th first degraded residual image, and an n-th first restored residual image shown in FIG. 9D is obtained.

  By performing a second filter process using a joint bilateral filter on the n-th base image and the n-th first restored residual image, an update image that is an update component of the base image shown in FIG. 9F is generated. The

By adding the updated image in FIG. 9F to the nth base image, the (n + 1) th base image shown in FIG. 9E, that is, an updated base image is generated.

In the following, the effectiveness of the present invention is demonstrated by simulation experiments and real image experiments.
<1> Simulation Experiment First, in order to quantitatively confirm the image restoration effect, a simulation experiment is performed in which a blurred image (degraded image) is created using a known original image and the original image is restored from the created degraded image. went.

  A known original image (true image) shown in FIG. 10 was used as the known original image. A straight blur with an angle of 30 [deg] and a length of 20 [pixels] was added as the PSF of blur, and Gaussian noise with a standard deviation of 3 was added independently as R, G, and B as noise.

  As the image restoration method of the present invention, the image restoration method according to Example 1 and Example 2 was used. For comparison, an RL method and a Wiener filter are used as existing image restoration methods. Note that the RMSE (Root Mean Square Error) with the original image is used as the evaluation method for the image restoration result, and the value is adjusted so that the restoration result is the best in the method that requires setting the parameters and the number of repetitions. Set.

  Table 1 shows the image restoration result and the RMSE of the correct image.

The rows in Table 1 show the types of known original images used, and the columns in the table show the types of image restoration methods used for image restoration. Comparing the RMSE of the image restoration method of the present invention and the existing image restoration method, the image restoration method of the present invention shows better results than the two existing image restoration methods for all images.

  Further, when the image restoration methods of the present invention are compared, the image restoration method of the first embodiment that adaptively sets the constraint parameters shows better results than the image restoration method of the second embodiment that uses constant constraint parameters. ing. In particular, in an image having a wide area with little texture, such as a known original image “Lena” and a known original image “Peppers”, the RMSE is greatly improved by an adaptive constraint parameter.

  From Table 1, it is apparent that the RMSE that evaluates the difference between the restored image and the true image is smaller in the image restoration method of the present invention than the conventional image restoration method.

  FIG. 11 shows an enlarged view of the image restoration result of the known original image “Lena”. Comparing the ringing generated around the edge, it can be seen that the result image (restored image) using the present invention is significantly suppressed in ringing compared to the existing method. In particular, in the restored image (FIG. 11A) by the image restoration method according to the first embodiment that adaptively uses constraint parameters, the occurrence of ringing is almost completely suppressed.

From the above results, by using the image restoration method of the present invention, a restored image with good RMSE can be obtained while suppressing ringing. Further, according to the present invention, it is possible to confirm the improvement of the restoration result centering on the region with less texture by adaptively designing the constraint parameter in the image restoration.

<2> Real image experiment The present invention was applied to a real image taken by a handheld camera, and the effectiveness of the present invention was verified. As the present invention, the image restoration method according to the first embodiment that adaptively designs constraint parameters is used. The PSF used was estimated by the method of Yoneji et al. (See Non-Patent Document 3).

  12 to 14 show three types of real images (degraded images) and their restoration results (restored images), respectively. For comparison, an enlarged view of a region surrounded by a square is also shown. It can be seen that the sharpness of the restored edge is the same, but the present invention suppresses ringing occurring in the restoration result of the existing method.

  In addition, in the restored image according to the present invention, the noise component of the entire image that has been increased by the existing method is also suppressed to a small value. In any image, these effects can be remarkably confirmed particularly in a region having a small texture.

  As described above, the effectiveness of the present invention was verified through image restoration experiments of simulation images and real images. In particular, it has been confirmed that the present invention is effective in suppressing ringing.

It is a figure which shows the example of the ringing which generate | occur | produces in the degradation image in a one-dimensional image, and a decompression | restoration image. It is a figure which shows the degradation image and restored image in a one-dimensional image for demonstrating the ringing in an image with a small amplitude. 1 is a block configuration diagram showing a first embodiment of an image restoration apparatus according to the present invention. It is a block block diagram which shows 2nd Embodiment of the image restoration apparatus which concerns on this invention. It is a block block diagram which shows 1st Embodiment of the base image generation process part in the image restoration apparatus which concerns on this invention. It is a block block diagram which shows 2nd Embodiment of the base image generation process part in the image restoration apparatus which concerns on this invention. It is a block block diagram which shows 3rd Embodiment of the base image generation process part in the image restoration apparatus which concerns on this invention. It is a block block diagram which shows 3rd Embodiment of the image restoration apparatus which concerns on this invention. It is a figure which shows the example of the image in a base image generation process in the base image generation process part in the image restoration apparatus which concerns on this invention. It is a figure which shows the known original image (true image) used for the image restoration experiment. It is a figure which shows the decompression | restoration image by Example 1 of this invention, Example 2, the conventional RL method, and the conventional Wiener filter using known original image "Lena". It is the figure which shows the restored image by Example 1 of this invention, the conventional RL method, and the conventional Wiener filter using the real image 1 which is a degradation image, and the real image 1. FIG. It is the figure which shows the restored image by Example 1 of this invention, the conventional RL method, and the conventional Wiener filter using the real image 2 which is a degradation image, and the real image 2. FIG. It is the figure which shows the restoration image by Example 1 of this invention, the conventional RL method, and the conventional Wiener filter using the real image 3 which is a degradation image, and the real image 3. FIG.

Explanation of symbols

1, 2, 3 Image Restoration Device 10 Base Image Generation Processing Unit 20 Blurred Image Generation Processing Unit 30 Degradation Residual Image Generation Processing Units 40, 41, 42 Image Restoration Processing Unit 50 Restored Image Generation Processing Units 101, 114, 117 Image Restoration First processing unit 102 Filter processing unit 111 First filter processing unit 112 Blurred image first generation processing unit 113 Degraded residual image first generation processing unit 115 Second filter processing unit 116 Base image update processing unit 118 Repeat processing end determination unit

Claims (43)

An image restoration device that generates a restored image for the deteriorated image based on one deteriorated image that has deteriorated due to blur and a known PSF representing the blur,
A base image generation processing unit that generates a base image by performing base image generation processing based on the degraded image;
A blurred image generation processing unit that generates a blurred image by performing convolution integration between the generated base image from the base image generation processing unit and the PSF;
A degradation residual image generation processing unit that generates a degradation residual image by calculating a difference between the generated blurred image and the degradation image;
An image restoration processing unit that generates a restored residual image by performing image restoration processing on the generated degraded residual image based on the generated base image so as to change the strength of constraint.
An image restoration apparatus comprising: a restored image generation processing unit configured to generate the restored image by adding the generated base image to the generated restored residual image. In the image restoration processing unit, using the generated base image, the constraint is weak when the gradient of the generated base image is large, and the constraint is strengthened when the gradient of the generated base image is small. A constraint parameter representing the strength of the constraint is adaptively set, and image restoration processing is performed on the generated degraded residual image using the set adaptive constraint parameter to generate a restored residual image. ,
Specifically, by generating the residual image Y (u, v) by minimizing the next evaluation function I,
Where ‖ · ‖ ₂ represents the L2 norm, * represents the convolution integral, X (u, v) represents the degraded residual image, B (u, v) represents the PSF, α (u, v, The image restoration apparatus according to claim 1, wherein v) represents the constraint parameter. The constraint parameter α (u, v) is set as follows:
3. The image according to claim 2, wherein R (u, v) represents the generated base image, ▽ R (u, v) represents a gradient of the generated base image, and γ and λ represent adjustment parameters. Restore device. An image restoration device that generates a restored image for the deteriorated image based on one deteriorated image that has deteriorated due to blur and a known PSF representing the blur,
A base image generation processing unit that generates a base image by performing base image generation processing based on the degraded image;
A blurred image generation processing unit that generates a blurred image by performing convolution integration between the generated base image from the base image generation processing unit and the PSF;
A degradation residual image generation processing unit that generates a degradation residual image by calculating a difference between the generated blurred image and the degradation image;
An image restoration processing unit that generates a restored residual image by performing image restoration processing on the generated degraded residual image;
An image restoration apparatus comprising: a restored image generation processing unit configured to generate the restored image by adding the generated base image to the generated restored residual image.   5. The image restoration processing unit performs image restoration processing using a Wiener filter or image restoration processing using a Richardson-Lucy method on the generated degraded residual image to generate a restored residual image. The image restoration apparatus described. An image restoration device that generates a restored image for the deteriorated image based on one deteriorated image that has deteriorated due to blur and a known PSF representing the blur,
A base image generation processing unit that generates a base image by performing base image generation processing based on the degraded image;
An image restoration processing unit that generates the restored image by performing image restoration processing on the deteriorated image based on the generated base image from the base image generation processing unit so as to change the strength of the constraint. An image restoration apparatus comprising: In the image restoration processing unit, using the generated base image, the constraint is weak when the gradient of the generated base image is large, and the constraint is strengthened when the gradient of the generated base image is small. A constraint parameter representing the strength of the constraint is adaptively set, and the degraded image is subjected to an image restoration process using the set adaptive constraint parameter to generate the restored image,
Specifically, the restored image Z (u, v) is generated by minimizing the next evaluation function I,
Where ‖ · ‖ ₂ represents the L2 norm, * represents the convolution integral, G (u, v) represents the degraded image, B (u, v) represents the PSF, and α (u, v) The image restoration device according to claim 6, which represents the constraint parameter. The constraint parameter α (u, v) is set as follows:
8. The image according to claim 7, wherein R (u, v) represents the generated base image, ▽ R (u, v) represents a gradient of the generated base image, and γ and λ represent adjustment parameters. Restore device. The base image generation processing unit
An image restoration first processing unit that generates a first restored image by performing an image restoration first process on the degraded image;
The image restoration according to any one of claims 1 to 8, further comprising: a filter processing unit configured to generate the base image by performing a filtering process for denoising on the generated first restored image. apparatus.   The image restoration first processing unit generates a first restored image by performing image restoration first processing by a Wiener filter or image restoration first processing by a Richardson-Lucy method on the degraded image. 10. The image restoration device according to 9. In the base image generation processing unit, the base image generation process is a repetitive process, and the base image is gradually generated by the repetitive process,
The base image generation processing unit
A first filter processing unit configured to generate an initial value of the base image by performing a first filter process on the degraded image;
A first blurred image generation processing unit that generates a first blurred image by performing convolution integration of the base image and the PSF;
A first deteriorated residual image generation processing unit that generates a first deteriorated residual image by calculating a difference between the generated first blurred image and the deteriorated image;
An image restoration first processing unit that generates a first restored residual image by performing an image restoration first process on the generated first degraded residual image;
A second filter processing unit that generates an updated image by performing a second filter process using the generated first restored residual image and the base image;
A base image update processing unit that updates the base image by adding the generated update image to the base image;
It is determined whether or not a predetermined iterative process end condition is satisfied, and if it is determined that the iterative process end condition is satisfied, the updated base image is set as the generated base image, and the iterative process end condition is When it is determined that it does not satisfy, the updated base image is set as the base image, and the iterative processing end determining unit continues the repetitive processing;
With
The repetitive processing includes processing performed by the first blurred image generation processing unit, processing performed by the first degradation residual image generation processing unit, processing performed by the image restoration first processing unit, 9. The process according to claim 1, which is a process performed by a second filter processing unit, a process performed by the base image update processing unit, and a process performed by the repetitive process end determination unit. Image restoration device.   The image restoration first processing unit performs image restoration first processing by the Wiener filter or image restoration first processing by the Richardson-Lucy method on the generated first deteriorated residual image, thereby performing the first restoration. The image restoration device according to claim 11, which generates a residual image. In the base image generation processing unit, the base image generation process is a repetitive process, and the base image is gradually generated by the repetitive process,
The base image generation processing unit
A first filter processing unit configured to generate an initial value of the base image by performing a first filter process on the degraded image;
A first blurred image generation processing unit that generates a first blurred image by performing convolution integration of the base image and the PSF;
A first deteriorated residual image generation processing unit that generates a first deteriorated residual image by calculating a difference between the generated first blurred image and the deteriorated image;
The first restoration residual image is generated by performing image restoration first processing on the generated first deteriorated residual image based on the base image so as to change the strength of the constraint. A first processing unit;
A second filter processing unit that generates an updated image by performing a second filter process using the generated first restored residual image and the base image;
A base image update processing unit that updates the base image by adding the generated update image to the base image;
It is determined whether or not a predetermined iterative process end condition is satisfied, and if it is determined that the iterative process end condition is satisfied, the updated base image is set as the generated base image, and the iterative process end condition is When it is determined that it does not satisfy, the updated base image is set as the base image, and the iterative processing end determining unit continues the repetitive processing;
With
The repetitive processing includes processing performed by the first blurred image generation processing unit, processing performed by the first degradation residual image generation processing unit, processing performed by the image restoration first processing unit, 9. The process according to claim 1, which is a process performed by a second filter processing unit, a process performed by the base image update processing unit, and a process performed by the repetitive process end determination unit. Image restoration device. The first image restoration processing unit uses the base image to increase the constraint strength so that the constraint is weak when the gradient of the base image is large and the constraint is strengthened when the gradient of the base image is small. And adaptively set a constraint parameter representing the image, and perform a first image restoration process using the set adaptive constraint parameter on the generated first deteriorated residual image to obtain a first restored residual image. Generate
Specifically, by generating the first restored residual image y ₀ (u, v) by minimizing the next evaluation function I,
Where ‖ · ‖ ₂ represents the L2 norm, * represents the convolution integral, x ₀ (u, v) represents the first degraded residual image, B (u, v) represents the PSF, α The image restoration apparatus according to claim 13, wherein (u, v) represents the constraint parameter. The constraint parameter α (u, v) is set as follows:
However, r ₀ (u, v) represents the base _{image, ▽ r 0 (u, v} ) is an image restoration apparatus according to claim 14 wherein represents the gradient of the base image, the γ and λ representing an adjustment parameter .   The predetermined repetitive processing end condition is a condition 1 in which there is no change before and after the update in the base image, a condition 2 in which the predetermined repetitive processing count is reached, or a square sum of pixel values of the updated image is equal to or less than a predetermined value. The image restoration device according to claim 11, wherein Condition 3 is satisfied. In the first filter processing, a bilateral filter or a low-pass filter is used,
When a bilateral filter is used for the first filter processing, the bilateral filter is defined by the following expression for the degraded image G (u, v):
Here, (u, v) represents coordinates on the image, r ₀ (u, v) is an output result of the bilateral filter, that is, an initial value of the generated base image, and w ₂ (i, v j) represents a two-dimensional Gaussian function, w ₁ () represents a one-dimensional Gaussian function,
In the second filter processing, a joint bilateral filter, a bilateral filter or a low-pass filter is used.
When a joint bilateral filter is used for the second filter processing, the first restored residual image y ₀ (u, v) and the corresponding base image r ₀ (u, v) are jointed. The bilateral filter is defined by
Here, z ₀ (u, v) is the output result of the joint bilateral filter, that is, the generated updated image, w ₂ (i, j) represents a two-dimensional Gaussian function, and w ₁ () is The image restoration apparatus according to claim 11, which represents a one-dimensional Gaussian function. An image restoration method for generating a restored image for the deteriorated image based on one deteriorated image that has deteriorated due to blur and a known PSF representing the blur,
A base image generation processing step of generating a base image by performing base image generation processing based on the degraded image;
A blurred image generation processing step for generating a blurred image by performing convolution integration between the generated base image and the PSF generated in the base image generation processing step;
A degradation residual image generation processing step of generating a degradation residual image by calculating a difference between the generated blurred image and the degradation image;
An image restoration processing step of generating a restored residual image by performing image restoration processing on the generated degraded residual image based on the generated base image so as to change the strength of the constraint;
An image restoration method comprising: a restored image generation processing step of generating the restored image by adding the generated base image to the generated restored residual image. In the image restoration processing step, using the generated base image, the constraint is weak when the gradient of the generated base image is large, and the constraint is strengthened when the gradient of the generated base image is small. A constraint parameter representing the strength of the constraint is adaptively set, and image restoration processing is performed on the generated degraded residual image using the set adaptive constraint parameter to generate a restored residual image. ,
Specifically, by generating the residual image Y (u, v) by minimizing the next evaluation function I,
Where ‖ · ‖ ₂ represents the L2 norm, * represents the convolution integral, X (u, v) represents the degraded residual image, B (u, v) represents the PSF, α (u, v, The image restoration method according to claim 18, wherein v) represents the constraint parameter. The constraint parameter α (u, v) is set as follows:
20. The image according to claim 19, wherein R (u, v) represents the generated base image, ▽ R (u, v) represents a gradient of the generated base image, and γ and λ represent adjustment parameters. Restoration method. An image restoration method for generating a restored image for the deteriorated image based on one deteriorated image that has deteriorated due to blur and a known PSF representing the blur,
A base image generation processing step of generating a base image by performing base image generation processing based on the degraded image;
A blurred image generation processing step for generating a blurred image by performing convolution integration between the generated base image and the PSF generated in the base image generation processing step;
A degradation residual image generation processing step of generating a degradation residual image by calculating a difference between the generated blurred image and the degradation image;
An image restoration processing step for generating a restored residual image by performing an image restoration process on the generated degraded residual image;
An image restoration method comprising: a restored image generation processing step of generating the restored image by adding the generated base image to the generated restored residual image.   The image restoration processing step performs image restoration processing using a Wiener filter or image restoration processing using a Richardson-Lucy method on the generated degraded residual image to generate a restored residual image. The image restoration method described. An image restoration method for generating a restored image for the deteriorated image based on one deteriorated image that has deteriorated due to blur and a known PSF representing the blur,
A base image generation processing step of generating a base image by performing base image generation processing based on the degraded image;
An image that generates the restored image by performing image restoration processing on the deteriorated image so as to change the strength of restraint based on the generated base image generated in the base image generation processing step. An image restoration method comprising: a restoration processing unit. In the image restoration processing step, using the generated base image, the constraint is weak when the gradient of the generated base image is large, and the constraint is strengthened when the gradient of the generated base image is small. A constraint parameter representing the strength of the constraint is adaptively set, and the degraded image is subjected to an image restoration process using the set adaptive constraint parameter to generate the restored image,
Specifically, the restored image Z (u, v) is generated by minimizing the next evaluation function I,
Where ‖ · ‖ ₂ represents the L2 norm, * represents the convolution integral, G (u, v) represents the degraded image, B (u, v) represents the PSF, α (u, v) The image restoration device according to claim 23, wherein represents the constraint parameter. The constraint parameter α (u, v) is set as follows:
25. The image according to claim 24, wherein R (u, v) represents the generated base image, ▽ R (u, v) represents a gradient of the generated base image, and γ and λ represent adjustment parameters. Restoration method. The base image generation processing step includes:
An image restoration first processing step of generating a first restored image by performing an image restoration first process on the degraded image;
The image restoration according to any one of claims 18 to 25, further comprising: a filtering process step of generating the base image by performing a filtering process for denoising on the generated first restored image. Method.   The image restoration first processing step performs image restoration first processing by a Wiener filter or image restoration first processing by a Richardson-Lucy method on the deteriorated image to generate a first restoration image. 26. The image restoration method according to 26. In the base image generation process step, the base image generation process is a repetitive process, and the base image is gradually generated by the repetitive process,
The base image generation processing step includes:
A first filter processing step of generating an initial value of the base image by performing a first filter process on the degraded image;
A first blurred image generation processing step of generating a first blurred image by performing convolution integration of the base image and the PSF;
A first deteriorated residual image generation processing step of generating a first deteriorated residual image by calculating a difference between the generated first blurred image and the deteriorated image;
An image restoration first processing step of generating a first restored residual image by performing an image restoration first process on the generated first degraded residual image;
A second filter processing step of generating an updated image by performing a second filter process using the generated first restored residual image and the base image;
A base image update processing step of updating the base image by adding the generated update image to the base image;
It is determined whether or not a predetermined iterative process end condition is satisfied, and if it is determined that the iterative process end condition is satisfied, the updated base image is set as the generated base image, and the iterative process end condition is When it is determined that the base image is not satisfied, the updated base image is set as the base image, and the repetitive processing is continued.
Have
The repetitive processing includes processing performed in the first blurred image generation processing step, processing performed in the first degradation residual image generation processing step, processing performed in the image restoration first processing step, The process performed in the second filter processing step, the process performed in the base image update processing step, and the process performed in the iterative process end determination step. Image restoration method.   In the first image restoration processing step, the first restoration residual image is subjected to image restoration first processing by a Wiener filter or image restoration first processing by Richardson-Lucy method, and first restoration is performed. The image restoration method according to claim 28, wherein a residual image is generated. In the base image generation process step, the base image generation process is a repetitive process, and the base image is gradually generated by the repetitive process,
The base image generation processing step includes:
A first filter processing step of generating an initial value of the base image by performing a first filter process on the degraded image;
A first blurred image generation processing step of generating a first blurred image by performing convolution integration of the base image and the PSF;
A first deteriorated residual image generation processing step of generating a first deteriorated residual image by calculating a difference between the generated first blurred image and the deteriorated image;
The first restoration residual image is generated by performing image restoration first processing on the generated first deteriorated residual image based on the base image so as to change the strength of the constraint. A first processing step;
A second filter processing step of generating an updated image by performing a second filter process using the generated first restored residual image and the base image;
A base image update processing step of updating the base image by adding the generated update image to the base image;
It is determined whether or not a predetermined iterative process end condition is satisfied, and if it is determined that the iterative process end condition is satisfied, the updated base image is set as the generated base image, and the iterative process end condition is When it is determined that the base image is not satisfied, the updated base image is set as the base image, and the repetitive processing is continued.
Have
The repetitive processing includes processing performed in the first blurred image generation processing step, processing performed in the first degradation residual image generation processing step, processing performed in the image restoration first processing step, The process performed in the second filter processing step, the process performed in the base image update processing step, and the process performed in the iterative process end determination step. Image restoration method. In the first image restoration processing step, using the base image, the constraint strength is weakened so that the constraint is weak when the gradient of the base image is large and the constraint is strengthened when the gradient of the base image is small. And adaptively set a constraint parameter representing the image, and perform a first image restoration process using the set adaptive constraint parameter on the generated first deteriorated residual image to obtain a first restored residual image. Generate and
Specifically, by generating the first restored residual image y ₀ (u, v) by minimizing the next evaluation function I,
Where ‖ · ‖ ₂ represents the L2 norm, * represents the convolution integral, x ₀ (u, v) represents the first degraded residual image, B (u, v) represents the PSF, α The image restoration method according to claim 30, wherein (u, v) represents the constraint parameter. The constraint parameter α (u, v) is set as follows:
However, r ₀ (u, v) represents the base _{image, ▽ r 0 (u, v} ) is the represents the gradient of the base image, the image restoration method according to claim 31 γ and λ representing an adjustment parameter .   The predetermined repetitive processing end condition is a condition 1 in which there is no change before and after the update in the base image, a condition 2 in which the predetermined repetitive processing count is reached, or a square sum of pixel values of the updated image is equal to or less than a predetermined value. The image restoration method according to claim 28, wherein Condition 3 is satisfied. In the first filter processing, a bilateral filter or a low-pass filter is used,
When a bilateral filter is used for the first filter processing, the bilateral filter is defined by the following expression for the degraded image G (u, v):
Here, (u, v) represents coordinates on the image, r ₀ (u, v) is an output result of the bilateral filter, that is, an initial value of the generated base image, and w ₂ (i, v j) represents a two-dimensional Gaussian function, w ₁ () represents a one-dimensional Gaussian function,
In the second filter processing, a joint bilateral filter, a bilateral filter or a low-pass filter is used.
When a joint bilateral filter is used for the second filter processing, the first restored residual image y ₀ (u, v) and the corresponding base image r ₀ (u, v) are jointed. The bilateral filter is defined by
Here, z ₀ (u, v) is the output result of the joint bilateral filter, that is, the generated updated image, w ₂ (i, j) represents a two-dimensional Gaussian function, and w ₁ () is The image restoration method according to any one of claims 28 to 33, which represents a one-dimensional Gaussian function. An image restoration program for generating a restored image for the deteriorated image based on one deteriorated image that has deteriorated due to blur and a known PSF representing the blur,
A1. A procedure for generating a base image by performing base image generation processing based on the degraded image;
A2. A procedure of generating a blurred image by performing convolution integration between the generated base image and the PSF generated in step A1;
A3. A procedure for generating a degraded residual image by calculating a difference between the generated blurred image and the degraded image;
A4. A procedure for generating a restored residual image by performing image restoration processing on the generated degraded residual image based on the generated base image so as to change the strength of the constraint;
A5. Generating the restored image by adding the generated base image to the generated restored residual image;
A program that causes a computer to execute. In step A4, using the generated base image, the constraint is strengthened so that the constraint is weak when the gradient of the generated base image is large and the constraint is strengthened when the gradient of the generated base image is small. A constraint parameter indicating the length is adaptively set, and the generated residual image is subjected to image restoration processing using the set adaptive constraint parameter to generate a restored residual image,
Specifically, by generating the residual image Y (u, v) by minimizing the next evaluation function I,
Where ‖ · ‖ ₂ represents the L2 norm, * represents the convolution integral, X (u, v) represents the degraded residual image, B (u, v) represents the PSF, α (u, v, 36. The image restoration program according to claim 35, wherein v) represents the constraint parameter. The constraint parameter α (u, v) is set as follows:
37. The image according to claim 36, wherein R (u, v) represents the generated base image, ▽ R (u, v) represents a gradient of the generated base image, and γ and λ represent adjustment parameters. Restore program. An image restoration program for generating a restored image for the deteriorated image based on one deteriorated image that has deteriorated due to blur and a known PSF representing the blur,
B1. A procedure for generating a base image by performing base image generation processing based on the degraded image;
B2. A procedure for generating the restored image by performing image restoration processing on the degraded image based on the generated base image generated in step B1 so as to change the strength of the constraint;
A program that causes a computer to execute. In step B2, using the generated base image, the constraint is weakened so that the constraint is weak when the gradient of the generated base image is large and the constraint is strengthened when the gradient of the generated base image is small. Adaptively set a constraint parameter representing the length, and perform an image restoration process with the set adaptive constraint parameter for the degraded image, to generate the restored image,
Specifically, the restored image Z (u, v) is generated by minimizing the next evaluation function I,
Where ‖ · ‖ ₂ represents the L2 norm, * represents the convolution integral, G (u, v) represents the degraded image, B (u, v) represents the PSF, α (u, v) The image restoration program according to claim 38, wherein represents the constraint parameter. The constraint parameter α (u, v) is set as follows:
40. The image of claim 39, wherein R (u, v) represents the generated base image, ▽ R (u, v) represents the slope of the generated base image, and γ and λ represent adjustment parameters. Restore program. In the procedure A1 or the procedure B1, the base image generation process is a repetitive process, and the base image is gradually generated by the repetitive process,
Procedure A1 or Procedure B1
C1. A procedure for generating an initial value of the base image by performing a first filter process on the degraded image;
C2. Generating a first blurred image by performing convolution integration of the base image and the PSF;
C3. A procedure for generating a first degraded residual image by calculating a difference between the generated first blurred image and the degraded image;
C4. A procedure for generating a first restored residual image by performing image restoration first processing on the generated first degraded residual image based on the base image so as to change the strength of the constraint;
C5. A procedure for generating an updated image by performing a second filter process using the generated first restored residual image and the base image;
C6. Updating the base image by adding the generated update image to the base image;
C7. It is determined whether or not a predetermined iterative process end condition is satisfied, and if it is determined that the iterative process end condition is satisfied, the updated base image is set as the generated base image, and the iterative process end condition is If it is determined that the base image is not satisfied, the updated base image is set as the base image, and the repetitive processing is continued.
Including
The repetitive processing includes processing performed in procedure C2, processing performed in procedure C3, processing performed in procedure C4, processing performed in procedure C5, processing performed in procedure C6, and procedure C7. The image restoration program according to any one of claims 35 to 40, wherein In step C4, using the base image, a constraint parameter representing the strength of the constraint is set so that the constraint is weak when the base image has a large gradient and the constraint is strengthened when the base image has a small gradient. Adaptively setting and performing a first image restoration process based on the set adaptive constraint parameter on the generated first degraded residual image to generate a first restored residual image;
Specifically, by generating the first restored residual image y ₀ (u, v) by minimizing the next evaluation function I,
Where ‖ · ‖ ₂ represents the L2 norm, * represents the convolution integral, x ₀ (u, v) represents the first degraded residual image, B (u, v) represents the PSF, α 42. The image restoration program according to claim 41, wherein (u, v) represents the constraint parameter. The constraint parameter α (u, v) is set as follows:
However, r ₀ (u, v) represents the base _{image, ▽ r 0 (u, v} ) is the represents the gradient of the base image, the image restoration program according to claim 42 γ and λ representing an adjustment parameter .