JP2007036894A - Image processing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To properly correct a shaking picture by simple processing. <P>SOLUTION: A contour of a shaking area (blurred part) is emphasized by an FIR filter. It is desirable to control the presence/absence of carrying out of correction according to a blur width. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing technique for processing a two-dimensional image, and more particularly to an image processing technique for correcting a blurred image deteriorated by a photographing operation or the like to an image close to an original image.

There is known an image restoration method for restoring (correcting) a blurred image when an image taken by an imaging device such as a camera is shaken due to camera shake or subject blur.
Commonly used image restoration processing is a restoration filter (for example, an inverse filter or a Wiener filter) that restores the camera shake part, or restores the camera shake part from an image point spread function (PSF). There is a process for estimating information for the purpose. For example, in Patent Document 1 below, when a camera shake image is restored using a point spread function generated based on camera shake locus data, the restoration is performed in consideration of luminance values of pixels around pixels on the camera shake locus. Techniques to do are disclosed.
Further, “blur” visually recognized when blurring can be corrected by contour emphasis. For example, Patent Document 2 below discloses a contour correction method including a delay circuit and an amplitude limiting circuit. ing.

Japanese Patent Application Laid-Open No. 11-134481 JP 05-292522 A

By the way, in the technique disclosed in Patent Document 1, the blur image is restored by estimating the luminance values of the peripheral pixels of the blur locus according to the distance from the blur locus. The burden on the calculation unit according to equation (11) is large.
Further, when the contour is emphasized by the technique disclosed in Patent Document 2, unnatural enhancement is performed at a smooth gradation portion like human skin.

  The present invention has been made in view of the above-described viewpoints, and an object thereof is to provide an image processing apparatus that appropriately corrects a blurred image by a simple process.

In order to overcome the above-described problem, the present invention changes the pixel value of a pixel near the center of the pixel area with respect to a pixel area including at least the blur area, with the center pixel of the blur area in the blur image as the center. Filter means for applying an emphasis filter and generating a first correction value of a pixel value corresponding to the central pixel, and the first threshold value so as to fall within a range from a first threshold value to a second threshold value greater than the first threshold value. Limiter means for limiting the one correction value to generate the second correction value;
The image processing apparatus includes a correcting unit that sequentially scans the central pixel in the blurred image and sets a pixel value in each pixel as the second correction value.

Preferably, the first threshold value is a minimum value of pixel values of pixels included in the blur region, and the second threshold value is a maximum value of pixel values of pixels included in the blur region,
The correction means does not correct the blur region on the condition that the difference value between the maximum value and the minimum value is smaller than a predetermined third threshold value.

  Preferably, the correction unit corrects the blur region on the condition that the difference value is further smaller than a predetermined fourth threshold value that is smaller than the third threshold value.

  Preferably, the correction means compares the maximum difference value between the maximum value and the minimum value for each RGB color obtained from the blurred image with the third threshold value or the fourth threshold value.

  Preferably, the correction unit corrects the blur region by limiting the blur direction to angles of 0 degrees, ± 30 degrees, ± 45 degrees, ± 60 degrees, and ± 90 degrees from the horizontal axis.

  Preferably, an angle detection unit or an acceleration detection unit for detecting a blur direction from the horizontal axis of the blur region is provided.

  Preferably, the correction means sets all colors on the condition that the maximum value among the difference values of the maximum value and the minimum value for each RGB color obtained from the blurred image is larger than the third threshold value. to correct.

  According to the present invention, it is possible to appropriately correct a blurred image by a simple process.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

FIG. 1 is a system configuration diagram of a digital still camera 1 which is an embodiment of an image processing apparatus of the present invention. The digital still camera 1 according to the embodiment is a camera having a shake correction function.
As shown in FIG. 1, the digital still camera 1 according to the embodiment processes a photographic lens 21, a lens optical system 2 including an image sensor 22 such as a CCD, an image signal of a subject image acquired by the lens optical system 2, Signal processing system (CDS (correlated double sampling) / AGC (automatic gain control) circuit 3, A / D conversion circuit 4, image processing unit 5) transferred to the CPU 11, motor driver 6 for driving the lens optical system 1, timing A generator (TG) 7, a memory 8, a shake detection sensor 9, a display unit 10, and a CPU 11 that controls the whole are configured.

In the digital still camera 1, first, a subject image incident through the lens optical system 2 is formed on the image sensor 22.
The CPU 11 drives the timing generator 7 and the image sensor 22 is driven by the pulses generated here. Thereby, accumulation and transfer of the subject image are performed.
The image signal output from the image sensor 22 is sampled by the CDS / AGC circuit 3 and amplified at a predetermined amplification factor.
The amplified image signal is converted into digital data (image data) by the A / D conversion circuit 4.

  The image processing unit 5 converts the image signal output from the image sensor into RGB data and outputs it to the CPU 11.

  The blur detection sensor 9 includes, for example, two angular velocity sensors, and the two angular velocity sensors detect the angular velocities around two axes (yaw and pitch) fixed to the digital still camera 1 to thereby detect the above-described 2 at the time of shooting. The camera shake amount and camera shake direction around the axis are calculated.

The CPU 11 performs blur correction processing for each RGB color.
Here, an image that is blurred due to the camera moving due to a force applied to the shutter button during shooting (hereinafter referred to as a blurred image) is more out of focus than a defocused image that is generated when the lens is out of focus. Is different. That is, in the out-of-focus image, the point image has a blurry form spreading in a circular shape, but in the blurred image, the blurring image has a blurring form spreading only in a specific direction.
Therefore, the blur correction process in the present embodiment pays attention to the blur in this specific direction, scans the blur image in units of pixels based on the blur amount and blur direction of the camera calculated by the blur detection sensor 9, Restore the original image.
The blur correction process will be described later.

  The restored image is stored in a frame memory in the memory 8. The CPU 11 sends a display drive signal for displaying an image of the subject to the display unit 10 based on the data in the frame memory.

Hereinafter, the correction principle of the blur correction process in the present embodiment will be described.
FIG. 2 shows a distribution of component values with respect to pixels (pixels) on the image as viewed from a direction perpendicular to the camera shake direction. FIG. 2A shows a case where no camera shake occurs, and FIG. This is when blurring occurs. FIG. 2 shows a change in the component value of any one color of RGB.
Here, it is assumed that the component value for a certain color obtained by correct exposure without camera shake is constant regardless of the pixel position, as shown in FIG. In this case, assuming that one pixel (1 pixel) is moved by one-fifth of the normal exposure time and camera shake for a total of five pixels has occurred, as shown in FIG. The image of the component value of 1 is shifted by one pixel in the right direction, thereby causing blurring areas (blurs) on both sides. That is, in FIG. 2B, the portion with the large component value is reduced by 5 pixels of the camera shake amount (left side), and the small portion is increased by 5 pixels (right side).

  In the blur correction process in the embodiment, in order to correct this blur part, the slope of the component value is made steep at the center part of the blur inclination (stepped part in FIG. 2B), that is, the outline is clarified. Thus, the original image (image without camera shake) is corrected.

  The blur correction processing according to the embodiment includes (1) filter processing and (2) limiter processing. Hereinafter, these processes will be described in order.

(1) Filter process The filter process is a digital filter process for emphasizing the gradient of the component value near the center of the blur region described above.
The filter process will be described with reference to FIG. 3, (a) is a diagram obtained by quantizing a blur region, (b) is a diagram showing filter coefficients of the present filter processing, and (c) is a diagram showing filter results (corrected component values). It is.

In FIG. 3A, when the length of the blur region corresponds to the number of N pixels, the filter coefficient as shown in FIG. 3B is applied to the center pixel (center pixel) of the blur region. Set. In other words, FIR (Finite) is used to multiply the center pixel of the blur region by (2 × E + 1), multiply the N pixels on both sides other than the center pixel by − (E / N), and perform convolution with these filter coefficients. Impulse Response) filter. Thus, the filter emphasizes the degree of change in the component value near the center pixel.
E is a coefficient representing the degree of correction, and 1 is a standard.
As a result of this filtering process, as shown in FIG. 3C, the inclination of the blur region becomes steep. The filter result of the center pixel corresponds to the first correction value of the present invention.

  In the example shown in FIG. 3B, N pixels on both sides from the center pixel are subjected to the convolution calculation, but at least pixels corresponding to the blur region (N / 2 or more on both sides from the center pixel) are subjected to the filtering process. It can be.

(2) Limiter processing As a result of the filter processing described above, the blur region has a steep slope, which improves blurring. However, large undershoots and overshoots occur on both sides of the blur region. Therefore, in order to obtain a natural image quality, the process for removing the undershoot and overshoot is the limiter process.
The limiter process is a process for limiting the first correction value within a range of a predetermined first threshold value to a second threshold value (> first threshold value). The first threshold value and the second threshold value are predetermined values. It may be a value, or may be defined by the range of component values of pixels in the blur region. In the present embodiment, the latter case will be described.
In such a case, first, the maximum value and the minimum value of the component values of each pixel in the blur region are specified. Then, the second correction value is obtained by limiting the first correction value so as to be within the range from the minimum value to the maximum value. As a result, undershoot and overshoot are eliminated.
This limiter process is also referred to as a maximum value filter process and a minimum value filter process.

FIG. 4 shows the second correction value obtained by the limiter process.
As shown in FIG. 4, the second correction value obtained by the limiter process has a smooth component because the length (blur width) of the blur region is small and overshoot and undershoot are suppressed. It turns out that it became a value change.

The correction principle of the blur correction process in the present embodiment has been described above, but the above process is performed for each RGB color.
3 and 4 illustrate the case where the blur width is 5 pixels. However, as the blur width increases to 10, 15 pixels,..., The blur correction effect is sufficiently obtained. Further, the blur correction effect can be increased by increasing the coefficient E (effect coefficient E) representing the correction effect shown in FIG.

Note that if the above-described filter processing is performed on a subject that originally has a small change in component value, an unnatural image may be obtained. For example, if the subject is a human face, there are shadows such as cheeks and nose based on the skin color with a relatively small component value, but if this is recognized as camera shake and filtered, an image different from the original image For example, it becomes an image of an angular nose.
In order to prevent this, it is desirable to perform the filtering process only when a density difference (difference between the maximum value and the minimum value of the component values) exists to some extent.

  Conversely, for subjects with very little change in component values, such as the ground or lawn, the above filter processing is performed to enhance the contour so that the image is not flattened due to camera shake. It is desirable to do so.

From this point of view, two threshold values (third threshold value, fourth threshold value: third threshold value> fourth threshold value) are provided for the difference value between the maximum value and the minimum value, and the difference value is the third threshold value. It is preferable not to perform the correction process on the condition that the value is smaller than the threshold value, and to perform the correction process on the condition that the value is smaller than the fourth threshold value.
In the experimental results, a result with good image quality was obtained by setting the third threshold value to a value of about 30% of the difference value and the fourth threshold value to a value of about 10%.

Further, when the color changes in the color image, it is rare that the three primary colors of RGB change greatly. Therefore, at least one primary color often only changes within the third threshold.
However, when a color change point is included in the blur area, if only some of the primary colors are not filtered at the change point, a color change occurs at the change point and color blur occurs. . In order to avoid this, when the difference value (difference between the maximum value and the minimum value) for at least one primary color among the three primary colors exceeds the third threshold value, it is desirable to perform correction processing for all three primary colors. In other words, when the difference value is less than or equal to the third threshold value for all three primary colors, it is desirable not to perform correction processing for all three primary colors.

In addition, camera shake can actually occur in any direction, but since it is difficult to perform correction processing in all directions of all pixels arranged in a matrix, the correction target angle with respect to the horizontal axis (blur direction) It is desirable from the viewpoint of improving the processing speed.
In the pixel matrix, when the blur direction is an angle of 0 degrees, ± 45 degrees, and ± 90 degrees, the pixels can be easily selected, and the above correction process can be performed. In addition, by selecting a pixel at a position where two pixels are advanced in the X-axis direction and one pixel advanced or retracted in the Y-axis direction of the matrix, correction processing can be performed for a blur direction of ± 30 degrees. By selecting a pixel at a position advanced by one pixel in the X-axis direction and two pixels advanced or retracted in the Y-axis direction of the matrix, correction processing can be performed with respect to a blur direction of ± 60 degrees.
In the experimental results, even when correction processing is performed by limiting the angle from the horizontal axis to 0 degrees, ± 30 degrees, ± 45 degrees, ± 60 degrees, ± 90 degrees (rounded) in this way, The results were almost satisfactory.

Next, with reference to FIG. 5, processing in which the above-described correction processing is applied to an actual image when the angle from the horizontal axis (blurring direction) is +45 degrees will be described.
FIG. 5 is a diagram for explaining a process in which the correction process is applied to an actual image.

This correction process is performed by sequentially scanning the blurred image to be processed in the horizontal direction.
First, when the length of the blur area corresponds to the number of N pixels, (2 × N + 1) registers (memory areas) for storing data of the N pixels are provided in the CPU 11.
Centered on the pixel to be processed (center pixel) and along the +45 degree direction which is the blur direction, data of the center pixel and N pixels on both sides (total (2 × N + 1) pixels) Component value) is stored in a register in the CPU 11.
In FIG. 5, data of pixels (total (2 × N + 1) pixels) indicated by black circles are stored in the register.

Next, a first correction value for the center pixel is calculated based on the pixel data stored in the register. Note that the process of calculating the first correction value is performed for each RGB color.
The first correction value is obtained by a filter process using a convolution operation. That is, the component value of the center pixel is multiplied by (2 × E + 1), the component values of the N pixels on both sides of the center pixel are multiplied by (−E / N), and then all are added together, and the summation result is the center pixel. The first correction value at.

Next, the maximum value and the minimum value are specified from the component values of (2 × N + 1) pixels including the center pixel within the range of (N / 2) pixels in the direction of +45 degrees from the center pixel. The maximum value and the minimum value are specified for each RGB color.
Further, among the RGB colors, the largest difference value between the maximum value and the minimum value, and the above-described two threshold values (third threshold value, fourth threshold value: where third threshold value> fourth threshold value) are set. Compare.
Then, the correction process is not performed on condition that the difference value is between the third threshold value and the fourth threshold value. That is, the first correction value is not used, and the original component value stored in the register is maintained.
The correction process is performed on the condition that the difference value is larger than the third threshold value or smaller than the fourth threshold value. In that case, the first correction value is limited by the specified maximum value and minimum value, and the final correction value is the second correction value. That is, on the condition that the first correction value is larger than the maximum value, the maximum value is set as the second correction value, and on the condition that the first correction value is smaller than the minimum value, the minimum value is set as the second correction value. On the condition that the first correction value is between the maximum value and the minimum value, the first correction value is set as the second correction value.

  Then, the original component value or the calculated second correction value is written in the frame memory in the memory 8 as the final correction value in the center pixel. The center pixel is scanned in the horizontal direction with respect to the blurred image, and the above-described processing is sequentially repeated and written in the frame memory. By performing the process for all the pixels, the same result as that obtained when the blur image is scanned in the +45 degree direction can be obtained.

In the above description, it has been assumed that the direction of camera shake is linear and constant speed, but the correction method described above is not limited to the direction of camera shake being linear and constant speed.
For example, in actual camera shake, the speed may gradually increase or decrease gradually. Therefore, the relationship between the pixel and the component value shown in FIG. 2A is such that the change in the component value between adjacent pixels is reduced or increased, as shown in FIG. 6A. In FIG. 6 (a), the left and right blurs with different component values are different, but the blur width is the same.
As a result of applying the above-described correction processing to the camera shake shown in FIG. 6A, the contour is emphasized as shown in FIG. 6B, and a substantially satisfactory corrected image is obtained.

  In addition, when the camera shake direction changes in the middle, blurring occurs in two or more directions. In such a case, the correction process may be repeated in each direction. In this case, it is less susceptible to noise if the efficacy coefficient E is made smaller.

  As described above in detail, according to the digital still camera according to the embodiment, since the outline of the blur region (blurred portion) is emphasized by the FIR filter, it is not necessary to perform the calculation of the frequency region, and the burden on the calculation of the CPU is reduced. In addition to being reduced, the presence / absence of correction is appropriately controlled according to the blur width, so that unnatural correction can be prevented.

The embodiments of the present invention are not limited to the above-described embodiments, and those skilled in the art can make various modifications without departing from the scope of the present invention.
For example, in the present embodiment, the case where two angular velocity sensors are used as the shake detection sensor 9 has been described, but the shake amount and the shake direction may be calculated using an acceleration sensor. In the present embodiment, the case where the signal processing unit 5 outputs RGB data and corrects each of the RGB color component values has been described. However, the signal processing unit 5 outputs a luminance signal or What is necessary is just to form a color image signal by performing signal processing such as forming a color signal, and luminance data (Y) and color difference data (U, V) may be output. When the luminance data and the color difference data are output, correction can be performed in the same manner using the luminance value and the color difference component value.

1 is a system configuration diagram of a digital still camera according to an embodiment. It is a figure which shows distribution of the component value with respect to the pixel on an image seen from the direction perpendicular | vertical with respect to the camera shake direction. It is a figure for demonstrating a filter process. It is a figure which shows the 2nd correction value obtained by the limiter process. It is a figure for demonstrating the process which applied the correction process to the actual image. It is a figure for demonstrating the aspect of a different camera shake.

Explanation of symbols

1 ... Digital still camera 2 ... Lens optical system
DESCRIPTION OF SYMBOLS 21 ... Shooting lens, 22 ... Image pick-up element 3 ... CDS / AGC circuit 4 ... A / D conversion circuit 5 ... Image processing part 6 ... Motor driver 7 ... Timing generator 8 ... Memory 9 ... Blur detection sensor 10 ... Display part 11 ... CPU

Claims (7)

A filter that emphasizes changes in the pixel values of pixels in the vicinity of the center of the pixel area is applied to the pixel area including at least the blur area centered on the center pixel of the blur area in the blurred image, and corresponds to the center pixel. Filter means for generating a first correction value of the pixel value obtained,
Limiter means for generating a second correction value by limiting the first correction value so as to fall within a range from a first threshold value to a second threshold value greater than the first threshold value;
An image processing apparatus comprising: a correction unit that sequentially scans the central pixel in the blurred image and sets a pixel value in each pixel as the second correction value. The first threshold value is a minimum value of pixel values of pixels included in the blur region, and the second threshold value is a maximum value of pixel values of pixels included in the blur region,
The image processing apparatus according to claim 1, wherein the correction unit does not correct the blur region on condition that a difference value between the maximum value and the minimum value is smaller than a predetermined third threshold value. The image processing apparatus according to claim 2, wherein the correction unit corrects the blur region on condition that the difference value is further smaller than a predetermined fourth threshold value that is smaller than the third threshold value. . The said correction | amendment means compares the largest thing in the difference value of the said maximum value and minimum value for every RGB color obtained from a blurring image with the said 3rd threshold value or a 4th threshold value. 2. The image processing apparatus according to 2 or 3. The correction unit corrects the blur region by limiting the blur direction to angles of 0 degrees, ± 30 degrees, ± 45 degrees, ± 60 degrees, and ± 90 degrees from a horizontal axis. The image processing apparatus according to any one of 1 to 4. The image processing apparatus according to claim 1, further comprising an angle detection unit or an acceleration detection unit for detecting a blur direction from a horizontal axis of the blur region. The correction means corrects all colors on condition that the maximum difference value between the maximum value and the minimum value for each RGB color obtained from a blurred image is larger than the third threshold value. The image processing apparatus according to claim 4, wherein:

Images