JP2017010093A - Image processing apparatus, imaging apparatus, image processing method, image processing program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus that can execute accurate denoising of estimated blur while suppressing an increase in the amount of calculation.SOLUTION: An image processing apparatus comprises: creation means that acquires at least part of a blur estimation area in a blur image and creates a plurality of pieces of two-dimensional estimated blur from the blur estimation area; and denoising means that performs denoising processing while maintaining the dimensionality of the estimated blur by using the plurality of pieces of two-dimensional estimated blur and creates two-dimensional noise reduction estimated blur. The creation means creates the plurality of pieces of estimated blur by performing repeated arithmetic processing of repeating correction processing of correcting blur included in information on a signal in the blur estimation area to create information on a correction signal, and estimation processing of estimating two-dimensional blur on the basis of information on the signal and information on the correction signal, and creates each of the plurality of pieces of estimated blur by using a plurality of areas determined as blur estimation areas and using a plurality of arithmetic expressions different from each other in the repeated arithmetic processing.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing method for estimating blur, which is a degradation component acting on a blurred image, from a single blurred image.

  In recent years, with higher definition of display devices, higher quality of captured images is desired. In the photographed image, information on the subject space is lost due to deterioration factors such as aberration and diffraction of an optical system used for photographing or camera shake during photographing. For this reason, conventionally, a method has been proposed in which deterioration of a captured image based on these factors is corrected to obtain a higher quality image. Examples of such a method include a Wiener filter and a Richardson-Lucy method. However, with these methods, when the deterioration component (blur) acting on the image is not known, a high correction effect cannot be obtained.

  On the other hand, a method for estimating a camera shake component from a single image deteriorated due to camera shake (a type of blur) has been proposed. Patent Document 1 discloses a method for estimating a camera shake component by using statistical information related to a known intensity gradient distribution of a natural image from a single image deteriorated by camera shake. Here, the natural image means an image that is seen naturally by a modern human being. For this reason, the natural image is not limited to an image showing a tree or an animal, but includes an image showing a person, a building, an electronic device, or the like. As a nature of a natural image, it is known that a histogram related to the intensity gradient of a signal (intensity gradient histogram) follows a heavy tailed distribution according to the intensity of the gradient. Patent Document 1 discloses a method for estimating a camera shake correction image from only a camera shake image by constraining the intensity gradient histogram of the image with the camera shake corrected to follow a heavy distribution. Then, a shake component is estimated based on a comparison result between the camera shake correction image and the camera shake image. At this time, the blur component estimation accuracy increases as more edges are included in the hand shake image.

  Patent Document 2 discloses a technique for estimating camera shake with high accuracy from a camera shake image including noise. In this method, a denoising filter in a one-dimensional direction is applied to a camera shake image, noise in that direction is reduced, and camera shake is estimated. Then, the estimated camera shake is integrated (Radon transform) in a direction parallel to the denoising filter to generate a one-dimensional camera shake. By performing this operation in various directions to obtain a plurality of one-dimensional camera shakes and performing inverse Radon transform on them, it is possible to estimate two-dimensional camera shakes with less noise.

U.S. Pat. No. 7,616,826 US Published Application No. 2014/133775

  However, with the conventional method, it is impossible to perform highly accurate denoising of the estimated blur while suppressing an increase in the amount of calculation. Since the blurred image includes noise, the noise estimated from the blurred image is also mixed with noise. In order to estimate blur with high accuracy, it is important how to reduce the noise of the estimated blur.

  In Patent Document 1, a hard threshold method is used for denoising of estimated camera shake. In this method, since the intensity of a component that is equal to or lower than a predetermined intensity (threshold) is set to 0, there is a problem that a weak component that is inherent to camera shake is erased simultaneously with noise. In the method of Patent Document 2, it is necessary to reconstruct a two-dimensional camera shake from the estimated one-dimensional camera shake. For this reason, a very large amount of one-dimensional camera shake (similar to or more than the components included in the dropped dimension) must be generated. With a small number of one-dimensional camera shake, the accuracy of reconstruction is greatly reduced, or artifacts are generated. For this reason, there is a problem that the amount of calculation for estimating the denominated camera shake becomes large.

  Therefore, the present invention provides an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium that can execute denoising with high accuracy that is estimated blurring while suppressing an increase in calculation amount.

  An image processing apparatus according to one aspect of the present invention includes: a generation unit that acquires at least a part of a blur estimation region in a blur image, and generates a plurality of two-dimensional estimation blurs from the blur estimation region; Denoising means for generating a two-dimensional noise reduction estimated blur by performing a denoising process using the estimated blur of the image while maintaining the number of dimensions of the estimated blur, and the generating means includes a signal in the blur estimation area. Iterating to repeat correction processing for correcting blur included in information related to generating correction signal information, and estimation processing for estimating two-dimensional blur based on the signal information and the correction signal information Performing arithmetic processing to generate the plurality of two-dimensional estimated blurs, using a plurality of regions determined as the blur estimation regions, or in the iterative arithmetic processing Using different multiple arithmetic expression to have to generate the plurality of two-dimensional estimation blurring, respectively.

  An imaging device according to another aspect of the present invention includes an imaging device that photoelectrically converts an optical image formed through an optical system and outputs an image signal, and at least one of a blurred image generated based on the image signal. Generating a plurality of two-dimensional estimation blurs from the blur estimation region, and maintaining the number of dimensions of the estimation blur using the plurality of two-dimensional estimation blurs. Denoising means for performing a denoising process and generating a two-dimensional noise reduction estimation blur, and the generation means corrects the blur included in the information about the signal in the blur estimation area and generates information about the correction signal. The plurality of two-dimensional processes are performed by performing an iterative calculation process that repeats a correction process, an estimation process for estimating a two-dimensional blur based on information on the signal and information on the correction signal. Generate an estimated blur, and generate each of the plurality of two-dimensional estimated blurs using a plurality of regions determined as the blur estimation region, or using a plurality of arithmetic expressions different from each other in the iterative calculation process. .

  According to another aspect of the present invention, there is provided an image processing method, comprising: obtaining at least a part of a blur estimation region in a blur image; generating a plurality of two-dimensional estimation blurs from the blur estimation region; Generating a two-dimensional noise reduction estimated blur by performing a denoising process using the estimated blur of the image while maintaining the number of dimensions of the estimated blur, and generating the plurality of two-dimensional estimated blurs. The correction processing for correcting the blur included in the information on the signal in the blur estimation region to generate information on the correction signal, and estimating the two-dimensional blur based on the information on the signal and the information on the correction signal Generating a plurality of two-dimensional estimated blurs by performing an iterative calculation process that repeats the estimation process, and using the plurality of regions determined as the blur estimation regions, Other uses the iterative processing a plurality of different calculation formula to generate the plurality of two-dimensional estimation blurring, respectively.

  According to another aspect of the present invention, there is provided a program for acquiring at least a part of a blur estimation region in a blur image, generating a plurality of two-dimensional estimation blurs from the blur estimation region, and the plurality of two-dimensional estimations. An image processing program configured to cause a computer to perform a denoising process using blur to maintain a number of dimensions of the estimated blur and generate a two-dimensional noise reduction estimated blur, In the step of generating the plurality of two-dimensional estimated blurs, a correction process for correcting blur included in the information related to the signal in the blur estimation region to generate information related to the correction signal, the information related to the signal, and the correction signal An estimation process for estimating a two-dimensional blur based on the information, and an iterative calculation process for repeating the plurality of two-dimensional estimation blurs Generated by using a plurality of regions determined as the blur estimation region, or, using the iterative processing a plurality of different calculation formula to generate the plurality of two-dimensional estimation blurring, respectively.

  A storage medium according to another aspect of the present invention stores the program.

  Other objects and features of the present invention are illustrated in the following examples.

  According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium capable of executing highly accurate denoising of estimated blur while suppressing an increase in calculation amount. Can do.

3 is a flowchart illustrating a noise reduction estimation blur generation method according to the first embodiment. 1 is a block diagram of an image processing system in Embodiment 1. FIG. 1 is an external view of an image processing system in Embodiments 1 and 2. FIG. 3 is a flowchart illustrating an image processing method according to the first exemplary embodiment. 6 is a diagram illustrating an example of acquiring a blur estimation area in Embodiment 1. FIG. 6 is a diagram illustrating an example of generation of noise reduction estimation blur in Embodiment 1. FIG. It is a figure which shows the production | generation example of the noise reduction estimation blur in Examples 1-3. 6 is a block diagram of an image processing system in Embodiment 2. FIG. 10 is a flowchart illustrating an image processing method according to the second exemplary embodiment. 10 is a diagram illustrating an example of generation of noise reduction estimation blur in Embodiment 2. FIG. FIG. 6 is a block diagram of an imaging system in Embodiment 3. FIG. 6 is an external view of an imaging system in Embodiment 3. 10 is a flowchart illustrating an image processing method according to the third exemplary embodiment. 10 is a diagram illustrating an example of generation of noise reduction estimation blur in Embodiment 3. FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted.

  In the present embodiment, the factors that degrade the information of the subject space are collectively referred to as “blur”, but the factors of blur include diffraction, aberration, defocus, blurring, and disturbance. Before describing the present embodiment, these blurring factors will be described in detail.

  Diffraction is degradation caused by diffraction, which occurs in the optical system of the imaging device that captured the image. This occurs because the aperture diameter of the optical system is finite.

  Aberration is deterioration caused by deviation from an ideal wavefront generated in an optical system. Aberrations that occur due to design values of the optical system are called design aberrations, and aberrations that occur due to optical system manufacturing errors and environmental changes are called error aberrations. The environmental change indicates changes in temperature, humidity, atmospheric pressure, etc., and the performance of the optical system also changes in accordance with these changes. When referring simply to aberrations, both design aberrations and error aberrations are included.

  Defocus is deterioration caused by the fact that the focus of the optical system and the subject do not match. A case where defocusing extends over the entire image is called out-of-focus, and a case where defocusing is only part of the image is called defocusing blur. Specifically, defocus blur refers to a component that degrades background information when the main subject and background at different distances exist in the image and the main subject is in focus. The magnitude of this degradation varies depending on how far the background is from the main subject in the depth direction. Defocus blur is used, for example, as an expression method that makes a main subject in an image stand out. In the case of simply defocusing, both defocusing and defocus blur are included.

  The blur is deterioration caused by a change in the relative relationship (position and angle) between the subject and the imaging apparatus during exposure during shooting. Among the blurs, those that degrade the entire image are called camera shakes, and those that partially degrade the image are called subject blurs. In the case of simply blurring, both camera shake and subject blur are included.

  Disturbance is deterioration caused by fluctuation of a substance existing between a subject and an imaging device during photographing. For example, atmospheric fluctuations and water fluctuations in underwater photography can be mentioned. When disturbance occurs during short-second exposure, the straight line in the subject space becomes a curved curve in the blurred image. In order to correct this bending, a plurality of frame (or continuous shooting) images having different disturbance components may be synthesized. However, in such a synthesis process, even if the edge curvature can be corrected, the deterioration of the frequency component remains. The blurred image handled here is not only an image (long-second exposure image) in which the degradation of the frequency component due to the disturbance occurred during one exposure, but also a composite of multiple frames (or continuous shots) as described above. Includes images.

  In the present embodiment, the point image intensity distribution is referred to as PSF (Point Spread Function). In this embodiment, processing for a monochrome image will be described. However, the present invention can be similarly applied to a multidimensional (for example, RGB) image, and processing may be performed for each of RGB channels. If there is no difference between channels or the blur is negligible, the number of channels may be reduced (eg, RGB is converted into monochrome) and processed. When each channel acquires a different wavelength, blurring is an example of a blur that has no difference between channels.

  On the other hand, the aberration, diffraction, defocus, and disturbance change in blur depending on the wavelength. Here, with regard to defocusing, even if it is far away from the focus position, the spread of blur varies depending on the wavelength due to the influence of axial chromatic aberration. However, even for these blurs having wavelength dependency, if the performance difference between channels is sufficiently small with respect to the sampling frequency of the image sensor used for imaging, it may be considered that the difference between channels can be ignored. However, when estimating blur where the difference between these channels can be ignored, it is preferable to estimate one blur with a plurality of channels, rather than performing the above-described process of reducing the number of channels. Since blur estimation is performed using information related to the signal gradient of the image, the estimation accuracy improves as the information increases. In other words, signal gradient information increases when estimating with multiple images without reducing the number of channels (however, if the images of each channel match their proportional multiples, the signal gradient information does not increase) The blur can be estimated with higher accuracy. In the present embodiment, an example in which wavelengths with different channels are photographed is given, but other parameters (polarized light, etc.) may be considered in the same manner.

  Here, before describing specific examples, a brief overview of the present embodiment will be described. A blur estimation area for estimating blur is obtained from the blur image, and an estimated blur is generated by comparing a signal gradient in the blur estimation area with a corrected signal gradient in which the blur of the signal gradient is corrected. At this time, if noise is included in the blur estimation area, noise is also generated in the estimated blur due to the influence. Therefore, in the present embodiment, a plurality of blur estimation regions existing at substantially the same position are acquired, and a plurality of estimated blurs are generated. Alternatively, a plurality of estimated blurs are generated using different arithmetic expressions (different processes or different parameters) for one blur estimation region. However, a plurality of estimated blurs may be generated by acquiring a plurality of blur estimation regions and using different processes or parameters.

  Regarding the estimated blur generated in this way, the blur components have substantially the same value, but the noises are different from each other. Therefore, by using a plurality of estimated blurs (for example, averaging a plurality of estimated blurs), it is possible to obtain a denoising noise reduction estimated blur. Since this method does not include processing for reducing the number of dimensions (processing is performed with the same number of dimensions), noise reduction estimated blur can be obtained even from a small number (for example, two or three) of estimated blurs. it can. For this reason, compared with the conventional method, the increase in calculation amount can be suppressed (or the calculation amount can be reduced).

  First, the image processing system according to the first embodiment of the present invention will be described with reference to FIGS. FIG. 2 is a block diagram of the image processing system 100 in the present embodiment. FIG. 3 is an external view of the image processing system 100. The image processing system 100 according to the present embodiment acquires a plurality of blur estimation regions and generates a plurality of estimated blurs.

  The image processing system 100 includes an imaging device 101, a recording medium 102, a display device 103, an output device 104, and an image processing device 105. The image processing apparatus 105 includes a communication unit 106, a storage unit 107, and a blur correction unit 108 (image processing unit). The blur correction unit 108 includes an acquisition unit 1081 (acquisition unit), a generation unit 1082 (generation unit), a denoising unit 1083 (denoising unit), and a correction unit 1084 (correction unit). Each unit of the blur correction unit 108 executes the image processing method of the present embodiment as will be described later. Note that the image processing apparatus 105 (blur correction unit 108) may be provided inside the imaging apparatus 101.

  The imaging apparatus 101 includes an optical system 1011 (imaging optical system) and an imaging element 1012. The optical system 1011 forms an image of light rays from the subject space on the image sensor 1012. The image sensor 1012 has a plurality of pixels, photoelectrically converts an optical image (subject image) formed via the optical system 1011, and outputs an image signal. The imaging device 101 generates a captured image (blurred image) based on the image signal output from the image sensor 1012. The blurred image obtained by the imaging device 101 is output to the image processing device 105 via the communication unit 106. Regarding the blurred image, information on the subject space is deteriorated by the action of at least one of a plurality of types of blur as described above.

  The storage unit 107 stores a blurred image input to the image processing apparatus 105 and information regarding shooting conditions when the blurred image is shot. Here, the shooting conditions are the focal length, aperture, shutter speed, ISO sensitivity, and the like of the imaging apparatus 101 at the time of shooting. The blur correction unit 108 estimates and corrects a specific blur component based on the blur image, and generates a blur correction image. The blur correction image is output to any one or more of the display device 103, the recording medium 102, and the output device 104 via the communication unit 106. The display device 103 is, for example, a liquid crystal display or a projector. The user can perform work while confirming an image being processed via the display device 103. The recording medium 102 is a semiconductor memory, a hard disk, a server on a network, or the like. The output device 104 is a printer or the like. The image processing apparatus 105 may have a function of performing development processing and other image processing as necessary.

  In order to implement the image processing method of the present embodiment, software (image processing program) can be supplied to the image processing apparatus 105 via a network or a storage medium such as a CD-ROM. At this time, the image processing program is read by the computer (or CPU, MPU, etc.) of the image processing apparatus 105 and executes the function of the blur correction unit 108.

  The blur correction unit 108 performs image processing shown in the flowcharts of FIGS. 1 and 4. Before explaining the image processing, a method for correcting the blur of the signal gradient in the blur estimation region (at least a part of the blur image) (correction processing) and a method for estimating the blur (estimation processing) will be described. In the present embodiment, the signal gradient is information related to the signal in the blur estimation region, and information related to the image itself (the luminance distribution of the image) or information related to the nth derivative (n is a natural number including 0) (related to the differential value). Information).

  In this embodiment, the blur (deterioration component) is expressed in the form of PSF, but is not limited to this, and may be expressed in the form of, for example, OTF (Optical Transfer Function). First, an appropriate initial value of PSF (Gaussian distribution, one-dimensional line, etc.) is given, and the signal gradient in the blur estimation region is corrected with the PSF to generate a corrected signal gradient. Subsequently, the PSF is estimated based on the signal gradient and the correction signal gradient. Using the estimated PSF, the signal gradient in the blur estimation area is corrected again to generate a new corrected signal gradient, and the PSF is estimated. By repeating this (loop processing), blur can be estimated only from the blur image (blur estimation region).

  Next, the signal gradient blur correction method (correction process) will be specifically described. The relationship between the blur estimation area and the blur is expressed as the following formula (1).

In equation (1), b- _i is the signal distribution in the blur estimation region in the i-th loop processing, k _i is blur, a _i is a signal distribution that is not deteriorated by blur k _i , and n _i is noise. “*” Represents a convolution operation.

By solving the optimization problem is represented by the following equation (2), _{(corresponding} to a correction signal distribution) estimate _d-i of a _i in the i-th loop process to estimate.

In Expression (2), L is a loss function, and Φ (d _i ) is a regularization term for d _i , and specific examples of each will be described later. The loss function L has an effect of fitting the solution to a model (here, the equation (1) is indicated). The regularization term Φ has the effect of converging the solution to a plausible value. For the regularization term, a property that the solution (a _i ) called prior knowledge should have is used. Moreover, regularization term has a role of preventing excessive fitting experience when considering only loss function (to the influence of noise n _i will be reflected to d _i). When the blur estimation area is acquired from a plurality of channels (for example, RGB), Equation (2) is solved for each channel.

  In this embodiment, the correction signal gradient is obtained by solving Equation (2), but other methods may be used. For example, a method using an inverse filter such as a Wiener filter or a super-resolution method such as a Richardson-Lucy method can be used. Further, a correction signal gradient may be generated by applying a sharpening filter such as a shock filter in combination with an inverse filter or a super-resolution technique. Furthermore, when a sharpening filter is used in combination, it is preferable to suppress noise and ringing by using a smoothing filter that maintains edge resolution such as a bilateral filter or a guided filter. When a plurality of correction signal gradients are generated, they may be generated using different methods (for example, optimization of expression (2) and inverse filter).

  Next, specific examples of the loss function L and the regularization term Φ in Expression (2) will be described. As the loss function L, a function represented by the following equation (3) is conceivable.

In the expression (3), a symbol represented as the following expression (4) represents a p-order average norm, and in the case of p = 2, represents a Euclidean norm.

  As an example of the regularization term Φ, there is a first-order average norm represented by the following equation (5).

In Equation (5), λ is a parameter that represents the weight of the regularization term Φ, and ψ is a function that represents a basis transformation for the image, examples of which include wavelet transformation and discrete cosine transformation. The regularization term Φ in equation (5) is expressed by a smaller number of signals because the image component is subjected to base transformation such as wavelet transformation or discrete cosine transformation, so that the signal component becomes sparse. Based on the nature of being able to. This is described, for example, in “Richard G. Baraniuk,“ Compressive Sensing ”, IEEE SIGNAL PROCESSING MAGAZINE [118] JULY 2007”. In addition, as examples of other regularization terms, a Tikhonov regularization term, a TV (Total Variation) norm regularization term, or the like may be used.

  In order to solve the estimation equation represented by Expression (2), which is an optimization problem, a technique using an iterative operation is used. For example, when a Tikhonov regularization term is employed, a conjugate gradient method or the like may be used. . In addition, when adopting the formula (5) or the TV norm regularization term, TwIST (Two-step Iterative Shrinkage / Thresholding) or the like may be used. Regarding TwIST, “J. M. Bioucas-Dias, et al.,“ A new TwIST: two-step iterative shrinkage / thresholding algorithms for image restoration ”, IEEE Trave. Explained.

  In addition, when performing these repetitive calculations, parameters such as regularization weights may be updated each time it is repeated. The expressions (1) and (2) are described for the image (signal distribution), but the same holds true for the differentiation of the image. Therefore, in place of the image, blur correction may be performed on the differentiation of the image (which is expressed as a signal gradient including both).

  Next, the blur estimation method (estimation process) will be specifically described. PSF can also be estimated using the following equation (6), as in equation (2).

When the blur estimation area is acquired from a plurality of channels (the blur due to the channel is not changed or can be ignored), the equation (6) is transformed as shown in the following equation (6a).

In the formula (6a), H is the total number of channels included in blur estimation _{region, d i, h} is _d i, _{v h} in the h-th channel indicate the weight. In the case of signal gradient correction, equation (2) may be solved for each channel, but in blur estimation, the form is summarized for all channels as in equation (6a). This is because the target to be estimated is different for each channel in Equation (2), but is common in Equation (6a) (the same PSF).

  As the loss function L of the equations (6) and (6a), the following equation (7) can be considered.

In Expression (7), _{ｊ j} represents a differential operator. _{０ 0} (j = 0) is an identity operator, and ∂ _x and ∂ _y represent the horizontal and vertical differentiations of the image, respectively. Further, the higher-order differentiation is expressed as ∂ _xx or ∂ _xyy , for example. Note that the signal gradient in the blur estimation area includes all these j (j = 0, x, y, xx, xy, yy, yx, xxx,...). In this embodiment, j = x, y Think only. u _j is a weight.

  As the regularization term Φ in the equation (6), one that conforms to the property of the PSF to be estimated can be used. For example, when the PSF is in a collapsed form such as defocus, it is preferable to use TV norm regularization represented by the following equation (8).

In equation (8), ζ is the weight of the regularization term.

  In addition, when the PSF has a linear shape such as blurring, it is preferable to use a first-order average norm regularization represented by the following formula (9).

  Further, Tikhonov regularization may be used as a regularization term that is not dependent on the shape of the PSF, ensures a certain degree of accuracy, and can be solved at high speed. When generating a plurality of correction blurs, they may be generated using different methods (for example, TV norm regularization and Tikhonov regularization of Expression (8)).

  Next, image processing performed by the blur correction unit 108 will be described with reference to FIG. FIG. 4 is a flowchart showing the image processing method of this embodiment. Each step in FIG. 4 is executed by the acquisition unit 1081, the generation unit 1082, the denoising unit 1083, and the correction unit 1084 of the blur correction unit 108.

  First, in step S101, the acquisition unit 1081 of the blur correction unit 108 acquires a blurred image (captured image). Subsequently, in step S102, the generation unit 1082 of the blur correction unit 108 determines the number of estimated blurs to be generated (the number of estimated blurs generated) based on the amount of noise in the blur image. The greater the amount of noise in the blurred image, the greater the estimated blur noise. Therefore, preferably, the generation unit 1082 increases the number of estimated blur generations as the amount of noise in the blur image increases. The generation unit 1082 determines the noise amount of the blurred image based on, for example, the ISO sensitivity that is the shooting condition or by estimating from the blurred image. A method for estimating the amount of noise included in a blurred image from a blurred image is described in, for example, “C. Liu, et al.,“ Noise Estimation from a Single Image ”, proc. IEEE Computer Vision and Pattern Recognition, pp. 901-908. 2006) ”.

  Further, the generation unit 1082 may acquire the noise amount from the blur estimation area acquired in step S105 (more precisely, step S201 in FIG. 1) described later, instead of acquiring the noise amount from the blurred image. Here, the generation unit 1082 determines the number of generations of estimated blur from the amount of noise. However, the generation unit 1082 may always generate a predetermined number (a constant number) of estimated blurs. However, in order to balance the noise reduction effect and the calculation load, it is preferable to determine the number of generations of estimated blur from the amount of noise.

  Subsequently, in step S103, the generation unit 1082 performs denoising processing on the blurred image. If noise is included in a blurred image (or a blur estimation area), noise is also generated in the estimated blur, and the blur estimation accuracy decreases. However, the estimated blur noise can be reduced through step S103, but it cannot be completely eliminated. This is because only the noise component of the blurred image cannot be completely eliminated by the denoising process. Similarly to step S102, instead of step S103, denoising processing may be performed on the blur estimation area after step S105 (more precisely, after step S201 in FIG. 1). Examples of denoising techniques include bilateral filters and NLM (Non Local Means) filters.

  Preferably, the generation unit 1082 denoises the blur estimation area by the following method. First, the generation unit 1082 generates a frequency-resolved blur estimation region by performing frequency decomposition on the blur estimation region. Then, the generation unit 1082 denoises the frequency-resolved blur estimation area based on the amount of noise in the blur estimation area. Next, the generation unit 1082 obtains a noise-reduced blur estimation region by recombining the frequency-resolved blur estimation region. In general, the denoising process has a problem of reducing noise from an image and blurring the image. If the blur estimation area is blurred by denoising, PSF in which the blur that originally deteriorated the image and the blur due to denoising is mixed is estimated when performing the subsequent estimation process. For this reason, it is preferable to use a denoising technique with a small blur on the image. As such a technique, a denoising process using frequency decomposition of an image is applied. Here, an example using wavelet transform as frequency decomposition will be described. The details are described in “Donoho DL,“ De-noising by soft-thresholding ”, IEEE Trans. On Inf. Theory, 41, 3, pp. 613-627.

  The wavelet transform is a transform in which a frequency analysis is performed for each position of an image using a localized small wave (wavelet) and a signal is decomposed into a high frequency component and a low frequency component. In the wavelet transform of an image, the wavelet transform is performed on the horizontal direction of the image to decompose it into a low frequency component and a high frequency component. Do. By the wavelet transform, the image is divided into four and is frequency-resolved into four subband images having different frequency bands. At this time, the subband image of the upper left low frequency band component (scaling coefficient) is LL1, and the subband image of the lower right high frequency band component (wavelet coefficient) is HH1. The upper right (HL1) and lower left (LH1) subband images are obtained by taking a high frequency band component in the horizontal direction and taking out a low frequency band component in the vertical direction, and taking a low frequency band component in the horizontal direction, respectively. A high frequency band component is extracted in the vertical direction.

  Further, when the subband image LL1 is wavelet transformed, the image size can be halved and decomposed into subband images LL2, HL2, LH2, and HH2, and the conversion level for the subband image LL obtained by the decomposition can be reduced. It can be disassembled as many times as possible.

Thresholding is known as a method for performing noise reduction processing using wavelet transform. In this method, a component having an amount smaller than a set threshold value is regarded as noise, and the noise is reduced. The threshold processing on the wavelet space is performed on the subband images other than the subband image LL, and the wavelet coefficient w having an absolute value equal to or smaller than the threshold is represented by the following equation (10). _Denoising is performed by replacing _subband (x, y) with 0.

In Expression (10), x and y are the vertical and horizontal coordinates of the image, ρ _subband is a weight parameter, and σ is a standard deviation of noise. The amount of noise σ included in the blur estimation area is obtained by measuring or estimating from the blur estimation area. When the noise is white Gaussian noise that is uniform in real space and frequency space, a method for estimating noise in the blur estimation region from MAD (Media Absolute Deviation) as shown in the following equation (11) is known. Yes.

The MAD is obtained by using the median (median value) of the wavelet coefficient w _HH1 in the subband image HH1 obtained by wavelet transforming the blur estimation area. Since the standard deviation and the MAD have the relationship shown in the following formula (12), the standard deviation of the noise component can be estimated.

In place of the equations (11) and (12), the noise amount σ may be acquired based on the shooting condition (ISO sensitivity at the time of shooting).

  Subsequently, in step S104, the generation unit 1082 acquires a position where blur estimation is performed from the blurred image. The generation unit 1082 acquires a plurality of blur estimation regions in the subsequent processing from the vicinity of the position acquired in step S104. Subsequently, in step S105, the generation unit 1082 and the denoising unit 1083 generate an estimated blur after the denoising process corresponding to the position acquired in step S104. Details of step S105 are shown in the flowchart of FIG. 1, and the description thereof will be described later.

  Subsequently, in step S106, the generation unit 1082 determines whether or not generation of noise reduction estimation blur has been completed for a predetermined region (for example, the entire blurred image) in the blurred image. If the generation of the noise reduction estimation blur has been completed, the process proceeds to step S107. On the other hand, if the generation of the noise reduction estimated blur has not been completed, the process returns to step S104, and the generation unit 1082 acquires, as a new estimated position, a portion where the estimated blur has not yet been generated from the predetermined area in the blurred image. To do.

  In step S107, the denoising unit 1083 (or the generation unit 1082) outputs the generated noise reduction estimation blur. When a plurality of blur estimated positions are included in a predetermined area in the blurred image, there are a plurality of noise reduction estimated blurs. Therefore, the denoising unit 1083 outputs the plurality of noise reduction estimation blurs. In this embodiment, the noise reduction estimation blur is PSF data. However, the present embodiment is not limited to this, and may be output in the form of coefficient data obtained by fitting OTF, PSF or OTF with a predetermined basis, or an image obtained by converting PSF or OTF into image data. Good.

  Subsequently, in step S108, the correction unit 1084 of the blur correction unit 108 corrects the blur of the blurred image using the output noise reduction estimated blur. For the correction, for example, a method using an inverse filter such as a Wiener filter or a super-resolution method such as a Richardson-Lucy method can be used. When there are a plurality of positions where blur is estimated in the blurred image, the correction unit 1084 corrects the vicinity of the estimated position where each corresponding noise reduction estimated blur is obtained using the noise reduction estimated blur. . Further, before all noise reduction estimation blurs are output, the noise reduction estimation blur generated in step S105 is used to sequentially correct blurs in the vicinity of the estimated position, and finally match them. A blur correction image may be generated.

  Next, the noise reduction estimation blur generation process (step S105) will be described in detail with reference to FIG. FIG. 1 is a flowchart illustrating a method for generating noise reduction estimation blur. Each step in FIG. 1 is executed by the generation unit 1082 and the denoising unit 1083 of the blur correction unit 108.

  First, in step S201, the generation unit 1082 acquires a plurality of blur estimation regions (a plurality of regions as a plurality of blur estimation regions) from the vicinity of the blur estimation position acquired in step S104. The generation unit 1082 estimates blur assuming that the same blur is acting in each blur estimation region. The blur estimation area may be an area specified by the user or automatically acquired. When blur is constant in the entire blur image (referred to as Shift-invariant), the entire blur image may be set as one blur estimation area. Therefore, in this embodiment, shift-invariant blur is not handled. In this case, the method described in the second embodiment or the third embodiment may be used. In this embodiment, the case where the blur changes according to the position in the blurred image (Shift-variant) is handled.

  In the vicinity of the estimated blur position acquired in step S104, the blur has substantially the same shape. Therefore, the generation unit 1082 acquires a plurality of blur estimation areas (a plurality of areas) from the vicinity of the blur estimation position. The generation unit 1082 preferably acquires each blur estimation area so as to include the estimated blur position. The number of acquired blur estimation regions is the same as the number of estimated blurs determined in step S102. However, by combining the second embodiment or the third embodiment with the method described later, the number of blur estimation area acquisitions can be made smaller than the number of estimated blurs.

  FIG. 5 is a diagram illustrating an example of obtaining the blur estimation area in the present embodiment. As shown in FIG. 5, an estimated blur position 202 is specified in the blurred image 201. The generation unit 1082 acquires the three blur estimation regions 203a to 203c from the vicinity of the blur estimation position 202. However, the number, position, and shape (including size) of the blur estimation area are not limited to this. The blur estimation regions 203a to 203c have different positions or shapes.

  Preferably, generation unit 1082 determines a plurality of blur estimation regions so as to satisfy the following conditions. That is, the blur estimation area having the smallest area among the plurality of blur estimation areas is referred to as a minimum blur estimation area. At this time, the area of the area where the minimum blur estimation area and each of the remaining blur estimation areas (all remaining blur estimation areas) among the plurality of blur estimation areas overlap each other is more than half the area of the minimum blur estimation area It is preferable that As described above, in this embodiment, shift-variant blur is targeted, and blur differs depending on the position or shape of the blur estimation area. For this reason, if a plurality of blur estimation regions are greatly shifted from each other, a completely different blur may be estimated. By satisfying the above-described conditions, the divergence between the plurality of blur estimation regions is suppressed, and substantially the same blur can be estimated. Also in FIG. 5, the area of the area where the blur estimation area 203a (minimum blur estimation area) and the blur estimation area 203b (or the blur estimation area 203c) overlap is more than half the area of the blur estimation area 203a. Yes.

  Subsequently, in step S202, the generation unit 1082 reduces the resolution of each of the plurality of blur estimation regions, and generates a plurality of low-resolution blur estimation regions. By reducing the resolution of the blur estimation area, the resolution of the blur to be estimated is similarly reduced. As a result, the convergence of the blur estimation described later can be improved, and the possibility that the estimation result falls into a local solution different from the optimal solution can be reduced. The rate of reducing the resolution of the blur estimation area is determined according to the downsampling parameter. Step S202 is executed a plurality of times by loop processing (repetitive calculation), but a predetermined downsampling parameter is used at the first execution. In the second and subsequent executions, a low-resolution blur estimation region is generated using the downsampling parameters set in step S206 described later. Each time the loop is repeated, the amount of decrease in resolution decreases, and the resolution of the low-resolution blur estimation area gradually approaches the blur estimation area. That is, at first, low-resolution blur is estimated, and the estimation result is used as a new initial value, and the estimation is repeated while gradually increasing the resolution. Thereby, it is possible to estimate the optimum blur while avoiding the local solution. Note that the resolution of the low-resolution blur estimation area is less than or equal to the resolution of the blur estimation area, and the resolutions of both may be the same.

  Subsequently, in step S203, the generation unit 1082 corrects the blur of the signal gradient (information about the signal) in each of the plurality of low-resolution blur estimation regions, and generates a plurality of correction signal gradients (information about the correction signal). (Correction process). For example, as illustrated in FIG. 6, the generation unit 1082 generates a plurality of correction signal gradients 205 a to 205 c for the signal gradients 204 a to 204 c of the plurality of low resolution blur estimation regions. However, the number of low-resolution blur estimation regions is not limited to this. As described above, the correction using the equation (2), other super-resolution, an inverse filter, and the like are used to generate the correction signal gradient. For each low-resolution blur estimation region, a correction signal gradient may be generated by a different method. The PSF used for blur correction uses the result (blur) estimated from each low-resolution blur estimation region in step S204 in the previous loop. At the first time of the loop, an appropriately shaped PSF (Gauss distribution, one-dimensional line, etc.) is used. This may be common for each low-resolution blur estimation region, or may be different. Subsequently, in step S204, the generation unit 1082 estimates a plurality of blurs for each of the plurality of low-resolution blur estimation regions based on the signal gradient and the correction signal gradient (blur estimation process). Formula (6) or formula (6a) is used for blur estimation. Note that, for each low-resolution blur estimation region, blur may be estimated using different arithmetic expressions (regularization type, weight parameter value, etc.).

  Subsequently, in step S205, the generation unit 1082 determines whether or not the iterative (loop) calculation has been completed, that is, whether or not the resolution difference between the blur estimation area and the low-resolution blur estimation area is smaller than a predetermined value. To do. This determination is performed by comparing the resolution of the blur estimation area with the resolution of the low resolution blur estimation area. When the resolution difference between the two is smaller than a predetermined value, the iterative calculation is terminated. Then, the plurality of blurs estimated in step S204 are regarded as the final plurality of estimated blurs, and the process proceeds to step S207. Whether or not the resolution difference between the two is smaller than a predetermined value is not limited to whether or not the absolute value of the resolution difference between the two is smaller than the predetermined value, and the ratio of the number of pixels of both is smaller than the predetermined value. You may determine by whether it is close to 1. If the predetermined condition is not satisfied (for example, if the absolute value of the resolution difference between the two is larger than the predetermined value), the estimated blur resolution is still insufficient, and thus the process proceeds to step S206, and the iterative calculation is performed.

  In step S206, the generation unit 1082 sets downsampling parameters used in step S202. In order to increase the resolution in the process of repeating steps S202 to S205, parameters are set so that the degree of downsampling is weaker than that of the previous loop (so that the amount of resolution decrease is reduced). In the iterative calculation, the blur estimated in step S204 of the previous loop is used as a new initial solution. At this time, it is necessary to increase the resolution of the blur. In order to improve the resolution, it is preferable to use bilinear interpolation or bicubic interpolation.

  In step S207, the denoising unit 1083 performs denoising of a plurality of blurs (a plurality of estimated blurs) estimated in step S204. FIG. 6 is a diagram illustrating a generation example of noise reduction estimation blur in the present embodiment. As in the example illustrated in FIG. 6, the denoising unit 1083 uses the noise reduction estimation blur 207 from the estimation blurs 206a to 206c generated for each of the plurality of blur estimation regions (a plurality of correction signal gradients 205a to 205c). Is generated. In this embodiment, the denoising unit 1083 generates a noise reduction estimated blur by taking an average value (or a weighted average value with a weight) of a plurality of estimated blurs. However, in addition to the average value (simple average value) or the weighted average value, the noise reduction estimation blur can be generated by obtaining the minimum value, the median value, the mode value, or the maximum likelihood value. Further, the denoising unit 1083 may generate one noise reduction estimation blur by combining the plurality of values. The plurality of estimated blurs include different outgoing noises in a region other than the original blur component (that is, a region where the component is correctly zero). For this reason, by obtaining the minimum values of a plurality of estimated blurs, it is possible to employ an estimated blur having the smallest noise (or no noise) among them.

  Preferably, the denoising unit 1083 generates the noise reduction estimated blur so that a component at a position where at least one of the plurality of estimated blurs is zero is zero. This will be described with reference to FIG. FIG. 7 is a diagram illustrating a generation example of noise reduction estimation blur in the present embodiment. 7A to 7C, the horizontal axis represents spatial coordinates, and the vertical axis represents intensity. FIGS. 7A and 7B are one-dimensional cross sections of a plurality of estimated blurs. Here, for the sake of simplicity, the number of a plurality of estimations is two, but this is not a limitation. The position in the vicinity of the center of the spatial coordinates is a region where the original blur component and the noise component are mixed, and the positions at both ends where the intensity is near zero are regions containing only the noise component. As can be seen from FIGS. 7 (A) and 7 (B), the noises are different from each other, and there is a portion where noise is present on one side but zero on the other side. When FIGS. 7A and 7B are simply averaged, a region where only one of the noises is zero has finite noise. However, in the component in which the intensity of either one of FIG. 7A or FIG. 7B is zero, noise reduction estimation blur component is set to zero to perform more efficient noise reduction. Can do. In a portion where both of FIGS. 7A and 7B have strength, it is preferable to obtain an average value thereof. FIG. 7C shows noise reduction estimation blur obtained by performing such processing. As shown in FIG. 7C, a noise reduction effect higher than that of simple averaging can be exhibited.

  Preferably, when performing the denoising process using a plurality of estimated blurs, the denoising unit 1083 aligns the plurality of estimated blurs to generate a noise reduction estimated blur. The plurality of estimated blurs may cause a phase shift because the positions of the blur estimation regions are different from each other. In addition, when different methods are used in the generation of the correction signal gradient and the estimation of the blur, the estimated PSF may be shifted by different amounts. At this time, if the process such as averaging is performed as it is, the PSF is blurred. Therefore, it is preferable to align a plurality of estimated blurs.

  More preferably, the denoising unit 1083 aligns the position of the peak value in each of a plurality of estimated blurs as alignment (alignment reference). Note that the position of the center of gravity or the like can be used as a reference for alignment instead of the peak position. However, when the center of gravity is used as a reference, the amount of PSF alignment (shift) can be non-integer, so interpolation is required. If the interpolation process is performed when the PSF has a very sharp peak, the peak is crushed and the PSF tends to be slightly widened (blurred). If a blurred image is corrected with a PSF that is originally blurred, overcorrection may occur, which may cause adverse effects such as ringing. For this reason, it is preferable to use a peak position that does not require interpolation as a reference.

  Further, the denoising unit 1083 may use a thresholding, an opening process for removing isolated points, or a denoising using a smoothing filter or the above-described frequency decomposition when generating noise reduction estimation blur.

  According to the present embodiment, it is possible to provide an image processing system capable of executing highly accurate denoising of estimated blur while suppressing an increase in the amount of calculation.

  Next, an image processing system according to the second embodiment of the present invention will be described with reference to FIG. FIG. 8 is a block diagram of the image processing system 300 in the present embodiment.

  The image processing system 300 according to the present exemplary embodiment includes the image processing apparatus 305 having the noise reduction estimated blur generation unit 308 instead of the image processing apparatus 105 having the blur correction unit 108. Different from the system 100. The image processing system 300 according to the present embodiment generates a plurality of correction signal gradients from one blur estimation region, and acquires a plurality of estimated blurs. The noise reduction estimation blur generation unit 308 estimates a blur component from the blur image captured by the imaging device 101, further performs a denoising process, and outputs an estimated blur with reduced noise. The noise reduction estimation blur generation unit 308 includes an acquisition unit 3081 (acquisition unit), a generation unit 3082 (generation unit), and a denoising unit 3083 (denoising unit). Since other parts of the image processing system 300 are the same as those of the image processing system 100 of the first embodiment, description thereof is omitted.

  Next, image processing performed by the noise reduction estimated blur generation unit 308 will be described with reference to FIG. FIG. 9 is a flowchart showing the image processing method of the present embodiment. Each step in FIG. 9 is executed by the acquisition unit 3081, the generation unit 3082, and the denoising unit 3083 of the noise reduction estimation blur generation unit 308.

  First, in step S301, the acquisition unit 3081 acquires a blurred image. Step S301 is the same as step S101 in the first embodiment described with reference to FIG.

  Subsequently, in step S302, the generation unit 3082 acquires a blur estimation area from the blur image. In the present embodiment, the generation unit 3082 and the denoising unit 3083 generate a plurality of estimated blurs for one blur estimation region and perform a denoising process. Therefore, it is effective even when the blur is Shift-invariant. Examples of the shift-invariant blur include a shake in which the entire image pickup apparatus 101 shifts during exposure and diffraction by an optical system in the image pickup apparatus 101 (assuming that vignetting is sufficiently small). However, in the present embodiment, the generation unit 3082 may acquire a plurality of blur estimation regions.

  Subsequently, in step S303, the generation unit 3082 corrects the signal gradient (information related to the signal) in the blur estimation region, and generates a plurality of correction signal gradients (information related to the plurality of correction signals) (correction processing). In the present embodiment, the generation unit 3082 always generates a predetermined number of correction signal gradients as a plurality of correction signal gradients. The number of correction signal gradients is equal to the number of estimated blurs. However, the present embodiment is not limited to this, and the number of correction signal gradients may be determined from the amount of noise as in the first embodiment. When generating the correction signal gradient, different arithmetic expressions, that is, different correction processes or parameters (correction parameters) are used.

  The correction processing refers to a method used for blur correction (optimization based on equation (2), an inverse filter such as a Wine filter, or a super-resolution method such as a Richardson-Lucy method). Alternatively, the correction processing refers to the type of regularization term used in the optimization of Expression (2) (primary average norm regularization or TV norm regularization of Expression (5)). The presence or absence of filtering after blur correction (sharpening filter and smoothing filter) is also included in the correction processing. The correction parameter is a weight of the regularization term in Expression (2) (for example, λ in Expression (5)), a high-resolution parameter in a super-resolution technique such as the Richardson-Lucy method, or sharpening or smoothing in filtering. This parameter represents the strength of the conversion. If the generation unit 3082 generates a plurality of correction signal gradients by changing the correction process or the correction parameter (that is, using different correction processes or correction parameters), the manner in which the correction is performed differs, and thus the behavior of the signal change, Also, there is a difference in how noise is generated. For this reason, the blur estimated from the plurality of correction signal gradients has different noises.

  When there are a plurality of correction signal gradients, the equations (6) and (6a) are respectively expressed by the following equations (13) and (13a).

Equation (13), in (13a), G is the total number of the correction signal gradients used to estimate the common blur, _{t g} denotes a weight.

  Subsequently, in step S304, the generation unit 3082 estimates a plurality of blurs based on the signal gradient in the blur estimation region and the plurality of correction signal gradients (blur estimation process). The blur estimation process in the present embodiment is the same as step S204 in the first embodiment described with reference to FIG.

  Subsequently, in step S305, the generation unit 3082 determines whether the blur estimated in step S304 has converged. If the blur has converged, the process proceeds to step S306. On the other hand, if the blur has not converged, the process returns to step S303. When returning to step S303, the generation unit 3082 newly generates a plurality of correction signal gradients in step S303 using the blur estimated in step S304. Whether or not the estimated blur has converged can be determined by, for example, obtaining a difference or ratio between a value obtained by degrading the correction signal gradient from the estimated blur and a signal gradient in the blur estimation region and comparing it with a predetermined value. is there. Alternatively, when blur is estimated by iterative calculation such as the conjugate gradient method in step S304, the determination may be made based on whether or not the update amount of blur by the iterative calculation is smaller than a predetermined value. When the blur converges, the generation unit 3082 sets each of the estimated blurs as an estimated blur in the blur estimation region.

  In step S306, the denoising unit 3083 generates a noise reduction estimated blur using a plurality of estimated blurs. The noise reduction estimation blur generation method is the same as that in step S207 of the first embodiment described with reference to FIG. FIG. 10 is a diagram illustrating a generation example of noise reduction estimation blur in the present embodiment, and is a schematic diagram of steps S303 to S306 in FIG. A plurality of correction signal gradients 402a to 402c are generated from the signal gradient 401 of one blur estimation region. Then, a plurality of estimated blurs 403 a to 403 c are generated from the comparison between the plurality of correction signal gradients 402 a to 402 c and the signal gradient 401. The denoising unit 3083 generates the noise reduction estimated blur 404 using the plurality of estimated blurs 403a to 403c. However, in the present embodiment, the correction signal gradient and the number of estimated blurs are not limited to this, respectively. Moreover, you may combine with the method of acquiring a some blur estimation area like Example 1. FIG.

  Subsequently, in step S307, the generation unit 3082 determines whether or not generation of noise reduction estimated blur has been completed for a predetermined region (for example, the entire blur image) in the blur image. If the generation of the noise reduction estimation blur has been completed, the process proceeds to step S308. On the other hand, when the generation of the noise reduction estimation blur is not completed, the process returns to step S302. Step S307 is the same as step S106 of the first embodiment described with reference to FIG.

  In step S308, the denoising unit 3083 (or the generation unit 3082) outputs the generated noise reduction estimation blur. When a plurality of blur estimation regions are included in a predetermined region in a blurred image (that is, Shift-variant), since there are a plurality of noise reduction estimation blurs, the denoising unit 3083 outputs a plurality of noise reduction estimation blurs. To do. At this time, the noise may be output sequentially while obtaining a plurality of noise reduction estimation blurs. The output noise reduction estimated blur can be used for correcting a blurred image, measuring the optical performance of a photographed optical system, or analyzing camera shake during photographing.

  According to the present embodiment, it is possible to provide an image processing system capable of executing highly accurate denoising of estimated blur while suppressing an increase in the amount of calculation.

  Next, with reference to FIG. 11 and FIG. 12, the imaging system in Example 3 of this invention is demonstrated. FIG. 11 is a block diagram of an imaging system 500 in the present embodiment. FIG. 12 is an external view of the imaging system 500.

  The imaging system 500 includes an imaging device 501, a network 502, and a server 503 (image processing device). The imaging device 501 and the server 503 are connected wirelessly, and an image (blurred image) from the imaging device 501 is transferred to the server 503, and the server 503 performs blur estimation and correction. Further, the server 503 generates a plurality of estimated blurs for the signal gradient and the correction signal gradient of the pair of blur estimation regions, and acquires the noise reduction estimated blur.

  The server 503 includes a communication unit 504, a storage unit 505, and a noise reduction estimation blur generation unit 506 (image processing unit). A communication unit 504 of the server 503 is connected to the imaging device 501 via the network 502. In this embodiment, the imaging device 501 and the server 503 are connected wirelessly, but the present invention is not limited to this, and may be connected by wire. The communication unit 504 of the server 503 is configured to receive a blurred image from the imaging device 501. When shooting is performed by the imaging device 501, a blurred image (input image or captured image) is automatically or manually input to the server 503 and sent to the storage unit 505 and the noise reduction estimated blur generation unit 506. The storage unit 505 stores information regarding a blurred image and shooting conditions for shooting the blurred image. The noise reduction estimated blur generation unit 506 generates an estimated blur based on the blurred image. Then, the noise reduction estimated blur generation unit 506 generates a blur correction image based on the estimated blur. The blur correction image is stored in the storage unit 505 or is sent to the imaging device 501 via the communication unit 504. The noise reduction estimation blur generation unit 506 includes an acquisition unit 5061 (acquisition unit), a generation unit 5062 (generation unit), and a denoising unit 5063 (denoising unit).

  In order to realize the image processing method of the present embodiment, software (image processing program) can be supplied to the server 503 via a network or a storage medium such as a CD-ROM. At this time, the image processing program is read by a computer (or CPU, MPU, etc.) of the server 503 and executes the function of the server 503.

  Next, image processing performed by the noise reduction estimation blur generation unit 506 will be described with reference to FIG. FIG. 13 is a flowchart showing an image processing method in the present embodiment. Each step in FIG. 13 is executed by the acquisition unit 5061, the generation unit 5062, and the denoising unit 5063 of the noise reduction estimation blur generation unit 506. 13 are the same as steps S301, S302, and S305 to S308 in FIG. 9 in the second embodiment, and therefore their explanations are as follows. Omitted.

  In step S403, the generation unit 5062 generates one correction signal gradient from the signal gradient in one blur estimation region (correction processing). However, the generation unit 5062 may generate one correction signal gradient from a plurality of signal gradients with respect to a plurality of blur estimation regions, or may generate a plurality of correction signal gradients from one signal gradient.

  Subsequently, in step S404, the generation unit 5062 estimates a plurality of blurs using different arithmetic expressions, that is, processing (estimation processing) or parameters (estimation parameters), based on the pair of signal gradients and the correction signal gradient. Yes (blur estimation process). The blur estimation process uses the optimization of formula (6) or formula (6a), etc., uses different regularization terms (processes) such as formula (8) and formula (9), or different regularizations. Are used to estimate a plurality of blurs. Since these plural blurs have different noises, the noises included in the estimated blur that is a result of the convergence thereof are also different from each other. For this reason, noise reduction estimation blur can be obtained from a plurality of estimation blurs.

  FIG. 14 is a diagram illustrating an example of generation of noise reduction estimation blur in the present embodiment, and illustrates a schematic diagram of generation of noise reduction estimation blur. The generation unit 5062 generates a plurality of estimated blurs 603 a to 603 c by using different processes or parameters for the signal gradient 601 and the correction signal gradient 602 in the blur estimation region. The denoising unit 5063 generates a noise reduction estimated blur 604 from the plurality of estimated blurs 603a to 603c. However, the number of estimated blurs is not limited to this, and a method of acquiring a plurality of blur estimation regions as in the first embodiment or a plurality of correction signal gradients from one signal gradient as in the second embodiment. You may use together the method of producing | generating.

  As described above, in each embodiment, the image processing apparatus (the image processing apparatuses 105 and 305 and the server 503) includes a generation unit (generation units 1082, 3082, and 5062) and a denoising unit (denoising units 1083, 3083, and 5603). The generation unit acquires at least a part of the blur estimation area in the blur image (captured image or input image), and generates a plurality of two-dimensional estimation blurs from the blur estimation area. The denoising means performs a denoising process using a plurality of two-dimensional estimation blurs while maintaining the number of estimated blur dimensions, and generates a two-dimensional noise reduction estimation blur. The generation unit generates a plurality of two-dimensional estimation blurs by performing an iterative calculation process that repeats the correction process and the estimation process. Here, the correction process is a process of generating information (correction signal gradient) related to the correction signal by correcting blur (signal gradient) included in the information related to the signal in the blur estimation region. The estimation process is a process for estimating a two-dimensional blur based on the information on the signal and the information on the correction signal. Then, the generation unit uses a plurality of regions determined as blur estimation regions (Example 1) or uses a plurality of arithmetic expressions different from each other in the iterative calculation process (Examples 2 and 3). Generate an estimated blur for each dimension.

  Preferably, the denoising means uses at least one of a weighted average value, a minimum value, a median value, a mode value, or a maximum likelihood value of the plurality of two-dimensional estimation blurs using the plurality of two-dimensional estimation blurs. To generate noise reduction estimation blur. Further preferably, the denoising means sets a component at a position where at least one of a plurality of two-dimensional estimation blurs is zero to zero. Preferably, the denoising means aligns a plurality of two-dimensional estimated blurs to generate a noise reduction estimated blur. Preferably, the denoising means aligns the position of the peak value in each of the plurality of two-dimensional estimated blurs as the alignment.

  Preferably, the generation unit acquires a plurality of areas as the blur estimation area, and generates information regarding a plurality of correction signals based on information (including blur) regarding the plurality of signals in the plurality of areas in the correction process. To do. In the estimation process, the generation unit estimates a two-dimensional blur based on each of the information on the plurality of signals and the information on the plurality of correction signals (Example 1, FIG. 6). More preferably, the generation unit acquires a plurality of regions having different positions or shapes as the blur estimation region. More preferably, the generation unit is configured such that an area of the minimum blur estimation area having the smallest area among the plurality of areas and an area where each of the remaining blur estimation areas of the plurality of areas overlap each other is an area of the minimum blur estimation area. A plurality of areas are acquired so as to be more than half of the above.

  Preferably, the generation unit generates information on the plurality of correction signals using a plurality of arithmetic expressions based on information on the signal in one area acquired as the blur estimation area in the correction process. In the estimation process, the generation unit estimates a two-dimensional blur based on each of the information on the signal and the information on the plurality of correction signals (Example 2, FIG. 10).

  Preferably, the generation unit generates information regarding the correction signal based on information regarding the signal in one area acquired as the blur estimation area in the correction process. Further, in the estimation process, the generation means estimates a plurality of two-dimensional blurs using a plurality of arithmetic expressions based on the information on the signal and the information on the correction signal (Example 3, FIG. 14).

  Preferably, the generation unit determines the number of the plurality of two-dimensional estimated blurs based on the noise amount of the blur image. Preferably, the generation unit generates a low-resolution blur estimation area by reducing the resolution of the blur estimation area, and corrects information about the signal in the low-resolution blur estimation area to generate information about the correction signal. Then, the generation unit brings the resolution of the low-resolution blur estimation area close to the resolution of the blur estimation area in the process of the iterative calculation process (S206). Preferably, the generation unit acquires the amount of noise included in the blur estimation area, and frequency-decomposes the blur estimation area to generate a frequency-resolved blur estimation area. Then, the generation unit performs denoising processing on the frequency-resolved blur estimation region based on the amount of noise, and re-synthesizes the frequency-resolved blur estimation region after the denoising process (S103). Preferably, the image processing apparatus includes a correcting unit (correcting unit 1084) that corrects the blurred image using the noise reduction estimated blur.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  According to each embodiment, there are provided an image processing device, an imaging device, an image processing method, an image processing program, and a storage medium capable of executing highly accurate denoising of estimated blur while suppressing an increase in calculation amount. be able to.

  As mentioned above, although the preferable Example of this invention was described, this invention is not limited to these Examples, A various deformation | transformation and change are possible within the range of the summary.

105, 305 Image processing devices 1082, 3082, 5062 Generation unit (generation means)
1083, 3083, 5603 Denoising unit (denoising means)

Claims (18)

Generating means for obtaining at least a part of a blur estimation area in a blur image and generating a plurality of two-dimensional estimation blurs from the blur estimation area;
Denoising means for generating a two-dimensional noise reduction estimated blur by performing a denoising process using the plurality of two-dimensional estimated blurs while maintaining the number of dimensionality of the estimated blurs,
The generating means includes
Correction processing for correcting blur included in information on a signal in the blur estimation region to generate information on a correction signal, and estimation processing for estimating two-dimensional blur based on the information on the signal and the information on the correction signal To generate the plurality of two-dimensional estimated blurs by performing an iterative calculation process that repeats
The plurality of two-dimensional estimated blurs are respectively generated using a plurality of areas acquired as the blur estimation area or using a plurality of arithmetic expressions different from each other in the iterative calculation process. Image processing device.   The denoising means uses the plurality of two-dimensional estimation blurs to at least one of a weighted average value, a minimum value, a median value, a mode value, or a maximum likelihood value of the plurality of two-dimensional estimation blurs. The image processing apparatus according to claim 1, wherein the noise reduction estimation blur is calculated.   The image processing apparatus according to claim 1, wherein the denoising unit sets a component at a position where at least one of the plurality of two-dimensional estimation blurs is zero to zero.   The image processing apparatus according to claim 1, wherein the denoising unit performs alignment of the plurality of two-dimensional estimated blurs to generate the noise reduction estimated blur.   The image processing apparatus according to claim 4, wherein the denoising unit aligns a position of a peak value in each of the plurality of two-dimensional estimated blurs as the alignment. The generating means includes
Obtaining the plurality of regions as the blur estimation region;
In the correction process, based on information on the plurality of signals in the plurality of regions, information on the plurality of correction signals is generated,
The said estimation process WHEREIN: Based on each of the information regarding these signals, and the information regarding these correction signals, the said two-dimensional blur is estimated, The any one of Claim 1 thru | or 5 characterized by the above-mentioned. The image processing apparatus described.   The image processing apparatus according to claim 6, wherein the generation unit acquires the plurality of regions having different positions or shapes as the blur estimation region.   The generation unit is configured such that an area of a region where a minimum blur estimation region having the smallest area among the plurality of regions and a remaining blur estimation region among the plurality of regions overlap each other is an area of the minimum blur estimation region. The image processing apparatus according to claim 7, wherein the plurality of regions are acquired so as to be more than half of the image processing apparatus. The generating means includes
In the correction process, based on the information about the signal in one area acquired as the blur estimation area, generate information about the plurality of correction signals using the plurality of arithmetic expressions,
The said estimation process WHEREIN: Based on each of the information regarding the said signal, and the information regarding these correction | amendment signals, the said two-dimensional blur is estimated, The any one of Claim 1 thru | or 5 characterized by the above-mentioned. Image processing device. The generating means includes
In the correction process, based on the information about the signal in one area acquired as the blur estimation area, information about the correction signal is generated,
6. The estimation process according to claim 1, wherein a plurality of the two-dimensional blurs are estimated using the plurality of arithmetic expressions based on the information on the signal and the information on the correction signal. The image processing apparatus according to claim 1.   The image processing apparatus according to claim 1, wherein the generation unit determines the number of the two-dimensional estimated blurs based on a noise amount of the blur image. The generating means includes
Reducing the resolution of the blur estimation area to generate a low-resolution blur estimation area;
In the low-resolution blur estimation region, information related to the signal is corrected to generate information related to the corrected signal,
The image processing apparatus according to claim 1, wherein the resolution of the low-resolution blur estimation area is made closer to the resolution of the blur estimation area in the process of the iterative calculation process. The generating means includes
Obtain the amount of noise included in the blur estimation area,
Frequency-resolving the blur estimation region to generate a frequency-resolved blur estimation region;
Based on the amount of noise, performing a denoising process on the frequency-resolved blur estimation region,
The image processing apparatus according to any one of claims 1 to 12, wherein the frequency-resolved blur estimation region after the denoising process is re-synthesized.   The image processing apparatus according to claim 1, further comprising a correction unit that corrects at least a part of the blurred image using the noise reduction estimation blur. An image sensor that photoelectrically converts an optical image formed through the optical system and outputs an image signal;
Generating means for acquiring at least a part of a blur estimation area in a blur image generated based on the image signal and generating a plurality of two-dimensional estimation blurs from the blur estimation area;
Denoising means for generating a two-dimensional noise reduction estimated blur by performing a denoising process using the plurality of two-dimensional estimated blurs while maintaining the number of dimensionality of the estimated blurs,
The generating means includes
Correction processing for correcting blur included in information on a signal in the blur estimation region to generate information on a correction signal, and estimation processing for estimating two-dimensional blur based on the information on the signal and the information on the correction signal To generate the plurality of two-dimensional estimated blurs by performing an iterative calculation process that repeats
The plurality of two-dimensional estimated blurs are respectively generated using a plurality of areas acquired as the blur estimation area or using a plurality of arithmetic expressions different from each other in the iterative calculation process. Imaging device. Obtaining at least a portion of a blur estimation area in the blur image and generating a plurality of two-dimensional estimation blurs from the blur estimation area;
Performing a denoising process using the plurality of two-dimensional estimation blurs while maintaining the number of dimensionality of the estimation blurs, and generating two-dimensional noise reduction estimation blurs,
Generating the plurality of two-dimensional estimated blurs;
Correction processing for correcting blur included in information on a signal in the blur estimation region to generate information on a correction signal, and estimation processing for estimating two-dimensional blur based on the information on the signal and the information on the correction signal To generate the plurality of two-dimensional estimated blurs by performing an iterative calculation process that repeats
The plurality of two-dimensional estimated blurs are respectively generated using a plurality of regions determined as the blur estimation regions, or using a plurality of arithmetic expressions different from each other in the iterative calculation processing. Image processing method. Obtaining at least a portion of a blur estimation area in the blur image and generating a plurality of two-dimensional estimation blurs from the blur estimation area;
A step of generating a two-dimensional noise reduction estimated blur by performing a denoising process using the plurality of two-dimensional estimated blurs while maintaining the number of dimensionality of the estimated blurs. An image processing program,
Generating the plurality of two-dimensional estimated blurs;
Correction processing for correcting blur included in information on a signal in the blur estimation region to generate information on a correction signal, and estimation processing for estimating two-dimensional blur based on the information on the signal and the information on the correction signal To generate the plurality of two-dimensional estimated blurs by performing an iterative calculation process that repeats
The plurality of two-dimensional estimated blurs are respectively generated using a plurality of areas acquired as the blur estimation area or using a plurality of arithmetic expressions different from each other in the iterative calculation process. Image processing program.   A storage medium storing the image processing program according to claim 17.