JP2000057339A - Image processing method and apparatus - Google Patents

Abstract

(57) [Summary] The present invention performs image restoration while shifting a deterioration parameter, even if a deterioration parameter constituting a deterioration function necessary for image restoration is unknown, and performs restoration of each restored image. By deriving the degradation parameter having the best image restoration degree, image restoration that does not require an advanced algorithm can be performed. An image for an iterative operation is first generated (step 2-1), and a deteriorated image is restored while changing deterioration parameters constituting a deterioration function (steps 2-2 to 2-4).
A deterioration function is derived by selecting a deterioration parameter having the best image restoration degree (step 2-5), and a restored image is generated by the deterioration function (step 2-6).

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to an image processing method and apparatus for restoring an image accompanied by blurring or blurring deterioration, which is photographed by a photographing device such as a camera, to an image close to an original image.

[0002]

2. Description of the Related Art Conventionally, there are many methods for restoring a deteriorated image (blurred image, blurred image, etc.) such as a Wiener filter, a general inverse filter, and a projection filter. When applying these methods, it is necessary to first determine a deterioration function.
It is most ideal that the deterioration function is analytically obtained from physical factors such as photographing conditions or is estimated from the output of a measuring device (such as an acceleration sensor) provided in the imaging device.

Hereinafter, the deterioration function will be described.

When f (x, y) is an ideal image and g (x, y) is a deteriorated image,

[0005]

(Equation 1)

It is assumed that the following relationship exists. Where h
(X, y, x ′, y ′) represents a deterioration function, and v (x, y) represents random noise in an output image.

[0007] Except for translation, if the degraded image of a point does not depend on the position of that point, the point spread function (PSF: Poi
nt Spread Function) is of the form h (xx ′, yy ′), and equation (1) is

[0008]

(Equation 2)

## EQU1 ## When there is no noise, Fourier transform is performed on both sides of equation (2), and from the convolution theorem,

[0010]

(Equation 3)

## EQU1 ## G (u, v), H (u, v), F
(U, v) are g (x, y), f (x, y), h
Represents the Fourier transform of (x, y). Where H (u, v)
Is an image g (x, y) degraded from the ideal image f (x, y)
Is the transfer function of the system that converts it to

As an example, a deterioration model of deterioration (so-called blurred image) due to relative movement between a camera and a landscape will be described below.

Assume that the images are temporally invariant except for motion. If the relative motion is approximately equal to the motion of the recording film in the plane, the sum of the exposures at points on the film should integrate the instantaneous exposure over the time the shutter is open. Required by Here, it is assumed that the time required for opening and closing the shutter is negligible. If α (t) and β (t) are the components of the displacement in the x and y directions, respectively,

[0014]

(Equation 4)

## EQU1 ## Here, T is the exposure time, which is conveniently between -T / 2 and T / 2.

When both sides of equation (4) are Fourier transformed,

[0017]

(Equation 5)

## EQU1 ## From this equation, it can be seen that the degradation is modeled by equation (3) or equation (2) equivalent thereto. The transfer function H (u, v) of this deterioration is given by the following equation.

[0019]

(Equation 6)

Then, a point response function in the case where the camera shakes for a time T at an angle θ and a constant speed V with respect to the X axis is as follows:

[0021]

(Equation 7)

Is given by here,

[0023]

(Equation 8)

Here, u0 and v0 are the center coordinates of the image. Note that when ω is very small, H (u, v) = T is approximated.

Similarly, the blur deterioration model can be represented by a function. For example, assuming that the blur phenomenon is along the normal distribution law (Gaussian), if the distance from the center pixel is r and any parameter of the normal distribution law is δ2, the deterioration function becomes

[0026]

(Equation 9)

The function is represented by

As described above, the deterioration function is expressed in accordance with the deterioration model. In order to finally perform image restoration using this deterioration function, it is necessary to specify deterioration parameters such as a blur direction and a blur amount. . These parameters may be obtained from a measurement device or the like provided in the imaging device. Also, many techniques have been studied for estimating a deterioration state from image feature amounts (such as an autocorrelation function) and obtaining a deterioration parameter only from a deteriorated image. In addition to the above point spread function, there is a method of estimating a line spread function, an edge spread function, and the like. For example, a method is known in which, if there is a sharp edge in an original image, an edge spread function is used to differentiate the edge to obtain a line spread function, and determine a deterioration function using an image reconstruction technique. .

[0029]

However, when an image is restored using an image restoration algorithm such as a Wiener filter, a very large noise is superimposed on the restored image because only a few percent error is included in the parameters. Is done. For example, in the case of a method of estimating a deterioration state from an image feature amount (such as an autocorrelation function) of a deteriorated image to obtain a deterioration parameter, the restoration effect is weak because the parameter generally includes a large error.
Even when a measurement device (acceleration sensor or the like) provided in an imaging device such as a camera is used, it is technically very difficult to keep the error within several percent, so that the problem of parameter error cannot be ignored.

When the edge spread function is applied,
There is a problem that it is difficult to extract an edge portion because the original image is deteriorated. The present applicant specifies a small area including an edge from the deteriorated image, performs image restoration while changing the parameters of the deterioration function in the small area, obtains an image restoration degree corresponding to the parameter, and obtains an image. Japanese Patent Application No. 5-287564 proposes a technique for performing image restoration of an entire image by using a parameter having the best restoration degree.
However, this proposal is only for the estimation of the edge spreading function and cannot be applied to the estimation of the point spreading function. The edge in the original image corresponds to a high-frequency component in frequency, and is an area on which the largest noise is superimposed when performing image restoration using various filters. That is, even if a parameter having the best image restoration degree is obtained, there is a high possibility that accuracy is poor due to noise. Further, there is a problem that it is troublesome to specify a small area including an edge in the original image every time the restoration operation is performed.

The present invention has been made in view of the above-mentioned conventional example, and when performing image restoration using various deterioration functions accompanied by deterioration parameters, when there is no estimated value of the deterioration parameter or when an analytical estimation is performed. Alternatively, it is an object of the present invention to provide an image processing method and apparatus for performing high-accuracy image restoration while suppressing a calculation load even when an error is included in a deterioration parameter obtained by an output of a measurement device provided in an imaging device. .

[0032]

In order to achieve the above object, the present invention has the following arrangement. That is, an image processing method for restoring a restored image close to the original image from a deteriorated image using a deterioration function, wherein the restored image is restored while changing deterioration parameters constituting the deterioration function, and the image of the restored image is restored. A restoration degree is obtained, a deterioration parameter is selected based on the image restoration degree, a deterioration function is derived, and a restored image is generated using the deterioration function.

Alternatively, there is provided an image processing apparatus for restoring a restored image close to the original image from a deteriorated image using the deterioration function, wherein the deteriorated image is restored while changing the deterioration parameters constituting the deterioration function. An image restoration degree of the image is obtained, a deterioration parameter is selected based on the image restoration degree, a deterioration function is derived, and a restored image is generated using the deterioration function.

Alternatively, the degraded image is restored by a computer while changing the degradation parameters constituting the degradation function, the degree of image restoration of the restored image is obtained, and the degradation parameter is selected based on the degree of image restoration. A computer-readable storage medium storing a computer program for deriving a deterioration function and generating a restored image using the deterioration function.

[0035]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS [First Embodiment] The present invention will be described in detail with reference to the drawings. FIG. 1 is a block circuit configuration diagram of the embodiment. In FIG. 1, an image input device 1 such as a film scanner or a digital camera is connected to a computer 2, and image data is input to the computer 2 by the image input device 1. An input device 3 such as a keyboard, an image display device 4 such as a display, and an image storage device 5 such as a magnetic disk are connected to the computer 2.

The image read into the computer 2 from the image input device 1 is transmitted by an instruction from the input device 3 as shown in FIG.
Are performed and displayed on the image display device 4. The processed image is stored in the image storage device 5 as needed.

Next, the flowchart of FIG. 2 will be described.

First, in step 2-1, the subsequent step 2
An image for an iterative image correction operation (referred to as an iterative operation image) is generated in 2-2 to 2-4. As the iterative calculation image in this step, the input deterioration image itself may be simply used, or an image advantageous for deriving the best deterioration parameter described below or an image which suppresses the calculation load may be generated.

The degradation function derived from the k degradation parameters represented by p1, p2,..., Pk is H (p1, p2,
.., Pk), in step 2-2, step 2
-1 for the iterative operation image generated in -1
Image restoration is performed using (p1, p2,..., Pk). For example, for the blurred image, the blurring angle θ, the blurring angle V, the exposure time T in the above-described equations (7) and (8), and for the blurred image, the degradation obtained by substituting r and δ2 in the equation (9). Based on the function, a restored image is generated using an image restoration algorithm such as a Wiener filter.

Next, in step 2-3, an image restoration degree is calculated from the restored image obtained in the previous step. Focusing on the frequency characteristic of the restored image as an index indicating the degree of image restoration, a case where the power spectrum (intensity distribution) of the most general Fourier transform is used will be described. Generally, in a deteriorated image (blurred image, blurred image, or the like), an edge portion (high-frequency component) having a sharp luminance change becomes a gentle luminance change (low-frequency component) due to deterioration. Since the component dropped to the low frequency is raised to the high frequency by the image restoration algorithm, it is considered that the higher the intensity of the high frequency component of the restored image is, the higher the image restoration degree is on the frequency distribution. Here, the restored image g (x,
Assuming that the real part of the Fourier transform image of y) is R (x, y) and the imaginary part is I (x, y), the power spectrum P (x,
y) is

[0041]

(Equation 10)

Is represented by Here, assuming that the center coordinates of the image P (x, y) are (x0, y0) and the luminance value of a pixel at a distance r from the center coordinates is P (r), the center of gravity C in the frequency distribution is

[0043]

[Equation 11]

Is represented by The larger the calculated center of gravity C, the greater the center of gravity in the frequency distribution.

Further, in the above equation (11), the center of gravity of the power spectrum P (x, y) in all frequency regions is obtained. In some cases, it is more efficient to obtain the center of gravity in a specific frequency band. There is also. For example, in an image restoration algorithm by deconvolution such as a Wiener filter, it is known that noise due to a parameter error or the like appears at a high frequency of the restored image. in this case,
It is better to obtain the center of gravity within a certain specific frequency range except for high frequencies having many noise components and use this as an index of the degree of image restoration.
Here, the range for obtaining the center of gravity on the frequency distribution is a ≦ r ≦ b
Equation (11) gives:

[0046]

(Equation 12)

Thus, it is determined that the larger the center of gravity C 'within the specific frequency range, the higher the degree of image restoration.

Next, while shifting the deterioration parameters in step 2-4, the above-described operations in steps 2-2 to 2-4 are repeatedly performed with all the deterioration parameters. As a method of shifting the deterioration parameter, there is a method of first obtaining an approximate value with a large step width, and calculating with a smaller step width around the approximate value. Further, the shift step width can be specified by the operator. In step 2-5, a deterioration parameter with the best image restoration degree is derived from the result obtained by the operations in steps 2-2 to 2-4. Finally, in step 2-6, a restored image of the input deteriorated image is generated using a deterioration function based on the deterioration parameter obtained in step 2-5 and an image restoration algorithm such as a Wiener filter.

When there are a plurality of deterioration parameters, one parameter is shifted at a time. If the best image restoration degree is obtained for a certain parameter at a certain value, the parameter to be shifted is changed to another parameter, a restored image is generated while shifting the parameter, and the image restoration degree is obtained. Is determined. Such an operation may be repeatedly performed for each parameter. [Second Embodiment] In this embodiment, the image generation block for iterative operation shown in step 2-1 of the flowchart of FIG. 2 will be described in detail.

In the first embodiment, the case where the input degraded image is simply used as the iterative operation image has been described.
As the image size increases, the calculation load becomes very high, and the purpose of obtaining the image restoration degree is redundant. Therefore, in the present embodiment, the input deteriorated image size Sx ×
From Sy, image size sx ×
The image converted to sy is used as an image for iterative operation. There are many existing algorithms for converting the image size Sx × Sy to the image size sx × sy, and for example, a near-list neighbor method, a bilinear method, a bicubic method, or the like can be used. According to this method, it is possible to perform the repetitive calculation with a small amount of calculation, having substantially the same frequency characteristics as the input deteriorated image. Further, the image size for the repetitive operation may be fixed in advance or may be specified by the operator.

Next, a technique is conceivable in which the operator designates a partial area of the input image and uses the cut-out image as an iterative calculation image. For example, consider the case where the image as shown in FIG. 3 has deteriorated (blurred, blurred, etc.). Since the luminance change in the area A is gentle, there is little change in frequency characteristics due to deterioration. However, in the edge portion of the region B, the steep luminance change (high-frequency component) becomes gentle (low-frequency component) due to deterioration. Therefore, when the region B is cut out from the degraded image of FIG. 3 and restored, the restoration effect is considered to be higher as the center of gravity of the restored image on the frequency distribution becomes higher. In other words, it can be said that the region B is more suitable than the region A for determining the quality of the image restoration degree when performing the comparison based on the image restoration degree as shown in the present invention. From the above, if the operator specifies a region including an edge portion and an image obtained by cutting out the region is used as an image for repetitive calculation, the best degradation parameter of the image restoration degree can be obtained with an effective and low calculation load process. Can be derived.

Further, a method for reducing the trouble of specifying the area by the operator in the above-described method will be described. FIG. 4 shows an input deteriorated image having an image size of Sx × Sy. This input degraded image is divided into a finite number of regions (3 × 4 regions in the example of FIG. 4), and each region has an image size sx × sy. The number of divided areas and the image size of the areas may be designated. Then, the operator selects a region that is considered to be most effective for the processing of the present invention from the plurality of divided regions, and sets that region as an image for iterative calculation. It is also possible to obtain the frequency distribution, the class separation of the discriminant analysis method, and the like for each region, estimate a region including many edge portions, and automatically select the region as an image for repetitive calculation. [Third Embodiment] In the first embodiment, it is assumed that the deterioration parameter is unknown in restoring the input deteriorated image. In this case, steps 2-2 to 2-4 in FIG.
, The calculation is performed using all the deterioration parameters, so that the load is very large. It is also conceivable that the error of the obtained deterioration parameter is large depending on the image and the effect of image restoration is low.

In this embodiment, FIG. 2 shown in the first embodiment is used.
A case in which the input or estimation of the deterioration parameter is performed before the processing procedure of will be described. When a certain deterioration parameter is given, the processing shown in the present embodiment makes it possible to restore a higher-speed and higher-definition image than in the first embodiment. FIG. 5 shows a flowchart of this embodiment.

First, in step 5-1: (1) Deterioration parameter input by the operator (2) Deterioration parameter estimation based on output from a measuring device or the like provided in the imaging device (3) Image analysis of the degraded image Degradation parameters are input and estimated using techniques such as estimation of degradation parameters.
In the example of (1), when an extremely small bright spot exists in the image, there is a method in which the operator inputs the deterioration parameter based on the spread of the bright spot. In (2), the deterioration parameter can be estimated from shooting conditions such as the exposure and shutter speed of the imaging device, or the output of a measuring device (such as an acceleration sensor) attached to the imaging device. In the example of (3), there have conventionally been many techniques for estimating a deterioration parameter based on an image feature amount (such as an autocorrelation function) of an input deteriorated image.

In steps 5-2 to 5-4, steps 2-1 to 5-2 in the flowchart (FIG. 2) of the first embodiment are performed.
The same processing as 2-3 is performed. In the deterioration parameter shift block in step 5-5, it is assumed that the best deterioration parameter to be obtained is in the vicinity of the input (estimated) deterioration parameter, and the deterioration parameter is shifted only in the vicinity thereof. Here, since the range of the interposed error differs depending on the method of inputting (estimating) the deterioration parameter (the above (1) to (3), etc.), whether the shift range is variable according to the method of step 5-1 or Alternatively, it can be specified by the operator.

Finally, in steps 5-6 to 5-7, the first
Step 2 in the flowchart of the embodiment (FIG. 2)
Similarly to -5 to 2-6, a deterioration parameter having the best image restoration degree is derived, and the input deterioration image is restored using the deterioration parameter.

[0057]

[Other Embodiments] Even if the present invention is applied to a system including a plurality of devices (for example, a host computer, an interface device, a reader, a printer, etc.), an apparatus (for example, a copying machine) Machine, facsimile machine, etc.).

Another object of the present invention is to supply a storage medium storing the program code of the procedure shown in FIG. 2 or 5 for realizing the functions of the above-described embodiment to a system or an apparatus, and to provide a computer of the system or apparatus. (Or CPU or MPU) by reading and executing the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD
-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

When the computer executes the readout program code, not only the functions of the above-described embodiment are realized, but also the OS (Operating System) running on the computer based on the instruction of the program code. ) Performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, based on the instructions of the program code, The case where the CPU of the function expansion board or the function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing.

[0063]

According to the present invention, the image restoration is performed while shifting the deterioration parameters even if the deterioration parameters constituting the deterioration function necessary for performing the image restoration are unknown, and the image restoration degree of each restored image is reduced. Deriving the best degradation parameters enables image restoration without the need for sophisticated algorithms.

By indicating the degree of image restoration by the frequency characteristic of the restored image, the degree of restoration can be obtained numerically, and the best deterioration parameter can be derived without the intervention of an operator.

Further, with respect to a high calculation load due to repetitive image restoration, the load can be reduced by generating an iterative calculation image that is efficient and low load.

Further, when a deterioration parameter is input / estimated by various methods, and there is a possibility that an error exists in the deterioration parameter, the range of parameter shift is limited, and iterative calculation is performed within a range near the deterioration parameter. This enables faster and higher-resolution image restoration.

[0067]

[Brief description of the drawings]

FIG. 1 is a block diagram of an image processing system according to an embodiment.

FIG. 2 is a flowchart of an image restoration procedure according to the first embodiment.

FIG. 3 is a diagram showing a state in which an image for iterative calculation is cut out.

FIG. 4 is a diagram showing a state in which an image for iterative calculation is cut out.

FIG. 5 is a flowchart of an image restoration procedure according to the third embodiment.

Claims (9)

[Claims] 1. An image processing method for restoring a restored image close to an original image from a deteriorated image using a deterioration function, wherein the restored image is restored while changing a deterioration parameter constituting the deterioration function. An image processing method comprising: obtaining an image restoration degree of an image; selecting a deterioration parameter based on the image restoration degree; deriving a deterioration function; and generating a restored image using the deterioration function. 2. The image processing method according to claim 1, wherein a small area is designated from the deteriorated image, and the small area is used as a deteriorated image restored while changing a deterioration parameter constituting a deterioration function. Method. 3. The method according to claim 1, wherein an image is generated by thinning out the degraded image to a designated size, and the image is used as a degraded image that is restored while changing a deterioration parameter constituting a deterioration function. Image processing method. 4. A degradation parameter is obtained from an analysis process for the degradation image or an output of a measurement device provided in the imaging device.
4. The image processing method according to claim 1, wherein the deterioration parameter is used as a deterioration parameter that is changed when a deteriorated image is restored. 5. The image processing method according to claim 1, wherein the deterioration parameter to be changed when the deteriorated image is restored is changed within a specified range. 6. The image processing method according to claim 1, wherein a frequency characteristic of a restored image is obtained as the degree of image restoration. 7. The image processing according to claim 1, wherein a frequency characteristic of a restored image is derived as the degree of image restoration, and a frequency distribution in a specific frequency body region is obtained from the obtained frequency characteristic. Method. 8. An image processing apparatus for restoring a restored image close to an original image from a deteriorated image by using a deterioration function, wherein the restored image is restored while changing a deterioration parameter constituting the deterioration function. An image processing apparatus comprising: obtaining an image restoration degree of an image; selecting a deterioration parameter based on the image restoration degree; deriving a deterioration function; and generating a restored image using the deterioration function. 9. A computer restores a deteriorated image while changing a deterioration parameter constituting a deterioration function, obtains an image restoration degree of the restored image, and selects a deterioration parameter based on the image restoration degree. A computer-readable storage medium storing a computer program for deriving a deterioration function and generating a restored image using the deterioration function.