JP2010068464A - Image processing apparatus, image processing method, and imaging apparatus - Google Patents

Abstract

An object of the present invention is to prevent a color gradation of a subject from being erroneously determined as color blur and removing an original color component of the subject.
An image processing apparatus includes: a gradient calculating unit that calculates an intensity gradient in a color original image obtained by an imaging apparatus; and color blur in an original image based on information representing a change aspect of the intensity gradient. Color blur reduction processing using the color blur estimated amount for the color blur pixel by determining the color blur estimated amount in the color blur pixel based on the intensity gradient, and the determination units S103 and S104 that determine the pixel and the subject texture pixel And processing units S152 to S155.
[Selection] Figure 1

Description

  The present invention relates to an image processing technique for reducing color blur included in a color image obtained by imaging.

  Colors that do not originally exist may occur as color blurs around the high-luminance portion of the color image obtained by imaging due to chromatic aberration of the imaging optical system (imaging optical system). Color blur is likely to occur in a wavelength region away from the center wavelength of the imaging optical system. For example, in visible light color imaging, blue, red, or a purple artifact that is a mixture of both occurs in a blur shape. The color blur especially for purple is also called purple fringe.

  The chromatic aberration of the imaging optical system can be optically suppressed to some extent by combining a plurality of lenses having different dispersions. However, as the resolution of the image sensor increases and the optical system becomes smaller, it has become difficult to sufficiently suppress chromatic aberration with the optical system alone. For this reason, reduction of color blur by image processing is required.

  Chromatic aberration is roughly classified into lateral chromatic aberration (magnification chromatic aberration) and longitudinal chromatic aberration (axial chromatic aberration). Patent Document 1 discloses an image processing method for reducing color blur by estimating a color blur amount due to longitudinal chromatic aberration and subtracting the estimated amount from an original image. In this image processing method, a spatial gradient of image intensity (luminance) is calculated, and it is determined whether or not color bleeding is caused by the gradient. In this determination, for example, for the R plane,

As described above, the difference between the four pixels in the vertical and horizontal directions is calculated for the two pixels adjacent to the target pixel.

  The color blur estimation method in Patent Document 1 will be briefly described with reference to FIGS. FIG. 6 shows an example of a photographed image including a high brightness light source. The photographed image includes road illumination (high luminance light source) such as a high pressure sodium lamp as a high luminance light source, and a color blur 1001 due to longitudinal chromatic aberration is generated in the vicinity thereof.

  FIG. 7 schematically shows a high-intensity light source (subject) in a captured image, and a luminance saturation region and a color blur region formed around the light source. Since the imaging sensor cannot measure the intensity above the saturation level, a white saturated region in which G and B are saturated, which is slightly larger than a high-luminance subject, is formed. Then, around the white saturated region, a B saturated region where B is saturated and G attenuates, a light blue blur region where both B and G attenuate, and a deep blue blur region where G disappears and only B has intensity And are formed.

  Among these, the blue color blur in the blue saturated region, the light blue blur region, and the pure blue blur region is unnaturally felt.

FIG. 8 shows an image intensity profile in a cross section passing through the center of the subject in FIG. The vertical axis represents intensity, and the horizontal axis represents coordinates (pixel position). In Patent Document 1, the intensity gradient (luminance gradient) in the B plane indicated by ∇B is calculated in the image intensity profile of FIG. 8 to estimate the color blur amount.
JP 2008-147981 A

  However, in the image processing method disclosed in Patent Document 1, for example, when an object having a blue gradation (such as a wall surface of a building or a night sky illuminated in blue) is included in the image, the blue gradation of the object is displayed. There is a possibility of misjudgment as color bleeding. In this case, the blue component of the subject that should be left is removed.

  This will be briefly described with reference to FIGS. FIG. 9 shows an example of a captured image obtained by imaging the night sky. There are blue gradations (subject gradation textures) 1301 to 1306 that gradually darken from the vicinity of the horizon to the top. FIG. 10 shows an image intensity profile in a cross section taken along a broken line in the captured image shown in FIG. The vertical axis represents intensity, and the horizontal axis represents coordinates (pixel position). In FIG. 10, the intensity of G and B has a slope corresponding to the blue gradation of the subject.

  When the intensity gradient wrinkle B is calculated for such an image, it becomes almost the same value as the intensity gradient wrinkle B caused by the blue color blur shown in FIG. 8, and cannot be distinguished from each other. Therefore, the blue component of the subject is estimated as blue color blur and is removed from the captured image.

  The present invention provides an image processing apparatus, an image processing method, and an imaging apparatus that can prevent a color gradation of a subject from being erroneously determined as color blur and removing an original color component of the subject.

  An image processing apparatus according to one aspect of the present invention includes a gradient calculation unit that calculates an intensity gradient in a color original image obtained by an imaging apparatus, and a color in the original image based on information representing a change aspect of the intensity gradient. A discrimination unit that discriminates between a blur pixel and a subject texture pixel, calculates a color blur estimation amount in the color blur pixel based on the intensity gradient, and performs color blur reduction processing using the color blur estimation amount for the color blur pixel. And a processing unit.

  According to another aspect of the present invention, there is provided an image processing method, comprising: calculating an intensity gradient in a color original image obtained by an imaging apparatus; and information representing an aspect of a change in intensity gradient in an original image. The step of discriminating between the color blur pixel and the subject texture pixel, the color blur estimation amount in the color blur pixel is calculated based on the intensity gradient, and the color blur reduction process using the color blur estimation amount is performed on the color blur pixel. And a step.

  Note that an imaging apparatus having the image processing apparatus and an imaging system for obtaining a color original image also constitutes another aspect of the present invention.

  According to the present invention, it is possible to discriminate between color blur pixels and subject texture pixels included in the original image, so that the color blur is reduced while avoiding accidental removal of the original color components of the subject. be able to.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a color imaging apparatus 100 having an image processing apparatus or using an image processing method according to an embodiment of the present invention.

  The color imaging apparatus 100 includes an imaging optical system 110, an image sensor 120, an AD conversion unit 130, a demosaic unit 140, and a color blur removal unit 150. The imaging optical system 110, the image sensor 120, the AD conversion unit 130, and the demosaic unit 140 constitute an imaging system in the imaging apparatus 100.

  In addition, the imaging apparatus 100 includes a visual correction unit 160, a compression unit 170, and a recording unit 180.

  The field 90 (imaging area, subject) and the R (red) light ray 91, the G (green) light ray 92, and the B (blue) light ray 93 shown in FIG. 1 are not components of the color imaging device 100. Illustrated for illustrative purposes.

  In FIG. 1, the captured field 90 is imaged on the image sensor 120 via the imaging optical system 110. In general, an image forming optical system attached to a color imaging apparatus is subjected to a certain chromatic aberration correction. In the imaging optical system 110 of the present embodiment, the longitudinal chromatic aberration in the R and G wavelength regions is well corrected, and the longitudinal chromatic aberration in the B wavelength region remains. Thus, by lowering the correction standard for longitudinal chromatic aberration in the B wavelength region, other aberration corrections can be improved, and the imaging apparatus can be downsized.

  The image sensor 120 is a single plate color image sensor including a general primary color filter. Each primary color filter is composed of three types of color filters having a transmission main wavelength band in the vicinity of 650 nm, 550 nm, and 450 nm, and images color planes corresponding to the R, G, and B bands, respectively.

  In a single-plate color image sensor, these color filters are spatially arranged for each pixel, and the intensity in a single color plane can only be obtained for each pixel. Therefore, a color mosaic image is output from the image sensor 120.

  A three-plate color image sensor that divides incident light into R, G, and B wavelength regions using a color separation prism (not shown) and picks up light of each wavelength with separate image sensors may be used. In this case, the demosaic unit 140 is not necessary.

  The AD conversion unit 130 converts the color mosaic image output as an analog voltage from the image sensor 120 into digital data suitable for subsequent image processing.

  The demosaic unit 140 generates a color image in which all pixels have RGB color information by interpolating the color mosaic image. Note that this interpolation method can be based on simple linear interpolation, E. Chang, S. Cheung, and D. Pan, “Color filter array recovery using a threshold-based variable number of gradients.” Proc. SPIE, vol. 3650, Many methods have been proposed up to the complicated method introduced in pp. 36-43, Jan.1999. The interpolation method in this embodiment may be any of these or other methods.

  In this embodiment, the color filter of the image sensor 120 is a primary color system composed of R, G, and B, but may be a complementary color system color filter. In this case, a color image including R, G, and B color planes is obtained by the color conversion process.

  The resolution of the B plane (blue plane) in the color image generated by the demosaic unit 140 is inferior to the G plane (green plane) and the R plane (red plane) due to the chromatic aberration of the imaging optical system 110. For this reason, as shown in FIG. 8, the image of the B plane is blurred rather than the image of the G plane, and as shown in FIG. 7, a blue color blur (B saturation, light blue blur, and deep blue) is generated around the bright subject. Bleeding) occurs.

  The color blur removal unit 150 removes (reduces) such color blur from the color image. The color blur removal process (color blur reduction process) in the color blur removal unit 150 will be described in detail later.

  The visual correction unit 160 mainly processes a color image from which color blur has been removed (reduced) so as to improve the appearance of the image. For example, image processing such as tone curve (gamma) correction, saturation enhancement, hue correction, and edge enhancement is performed.

  At the end of the processing, the compression unit 170 compresses the corrected color image by a method such as JPEG, and reduces the size at the time of recording.

The compressed color image is recorded on a recording medium such as a hard disk, a DVD, a magnetic tape, or a flash memory by the recording unit 180. The processing units from the image sensor 120 to the recording unit 180 are actually Each may be constituted by a separate device, or may be constituted on a single microprocessor.

  Next, processing (image processing method) in the color blur removal unit 150 as an image processing apparatus will be described with reference to the flowchart of FIG. The processing in the color blur removal unit 150 includes a space calculation step S151, an estimation step S152, an area determination step S153, an overremoval suppression step S154, and a removal step S155. This process is executed by a computer program.

  Here, steps S <b> 151 to S <b> 155 can be considered as the configuration of the color blur removal unit 150. That is, the space calculation step S151 constitutes an inclination calculation unit, a curvature calculation unit, and a determination unit, which will be described later, and the estimation step S152, the region determination step S153, the excessive removal suppression step S154, and the removal step S155 constitute a processing unit.

  In this embodiment, the B plane is a color plane that is a target of color blur removal, and the G plane is a reference plane that is referred to in color blur removal. However, color blur removal can be similarly performed for the G plane and the R plane.

  An input image (color original image: hereinafter simply referred to as an original image) generated as a color image generated when the shutter button of the imaging apparatus 100 is pressed is input to the space calculation step S151.

  As shown in FIG. 3, the space calculation step S151 includes an inclination calculation step S102 for calculating the intensity gradient of the original image and an intensity gradient curvature (hereinafter referred to as intensity), which is information indicating a change mode (how) of the intensity gradient Curvature calculation step S103 for calculating the curvature). Further, in step S151, based on the intensity gradient obtained in the gradient calculation step S102 and the intensity curvature obtained in the curvature calculation step S103, the pixel of interest in the original image is a pixel representing color blur or a pixel representing subject texture. And determining step S104 for determining whether or not.

  Note that steps S102 to S104 can be considered as the configuration of the color blur removal unit 150. In this case, the inclination calculation unit is configured by the inclination calculation step S102, and the curvature calculation unit is configured by the curvature calculation step S103. In addition, a determination unit is configured by the determination step S104.

  In the inclination calculation step S102, intensity gradients (luminance gradients) ∇B and ∇G in the B plane and the G plane in the original image are calculated. Specifically, the first-order differential (gradient) filters 201 and 202 shown in FIG. 4 are applied to the image of interest and two pixels adjacent to this in the x direction (horizontal direction) and the y direction (vertical direction). Apply to calculate ∇B, ∇G. The values of ∇B and ∇G are

Can be expressed as

here,
G (x + 1, y) and B (x + 1, y) are the values of the pixels immediately to the right of the pixel of interest in the G and B planes, respectively.

  G (x-1, y) and B (x-1, y) are the values of the pixels adjacent to the left of the pixel of interest in the G and B planes, respectively.

  G (x, y + 1) and B (x, y + 1) are values of pixels below the pixel of interest in the G and B planes, respectively.

  G (x, y-1) and B (x, y-1) are the values of the pixels adjacent to the target pixel in the G and B planes, respectively.

  On the other hand, in the curvature calculation step S103, the intensity curvature ΔB at the B plane in the original image is calculated. Specifically, the second-order differential (Laplacian) filters 301 and 302 shown in FIG. 5 are applied to the image of interest and two pixels adjacent to this in the x direction and the y direction to calculate ΔB. The value of ΔB is

Can be expressed as

In the determination step S104, the pixel of interest represents a color blur (hereinafter referred to as a color blur pixel) or a pixel representing a subject texture (hereinafter referred to as a color blur pixel) based on ∇B and ΔB calculated as described above and the following logical expression. Hereinafter, it is determined (determined). In addition, _x and _y attached | subjected to を B and (DELTA) B have each shown the x direction and the y direction.

(Logical expression)
(∇B _x ≠ 0 and ΔB _x <0) or (∇B _y ≠ 0 and ΔB _y <0): Color blur pixel Other: Subject texture pixel The discrimination (determination) result in the discrimination step S104 is the following estimation Step S152 is passed along with the intensity gradients ∇B and ∇G.

  Here, the physical meaning of the processing in the determination step S104 will be described with reference to FIGS.

  In FIG. 8, it is empirically understood that the color blur attenuates exponentially in a narrow range centering on the high-luminance subject. On the other hand, as shown in FIG. 10, the gradation texture of the subject has an intensity gradient over a very wide range as compared with color blur. For this reason, in the narrow region of the adjacent three pixels as described above, the gradation texture of the subject attenuates almost linearly, and even if the intensity gradient ∇B is the same as shown in FIGS. 8 and 10, The curvature ΔB differs between color blur and subject texture. Furthermore, since the intensity profile due to color blur has a downwardly convex slope, the curvature (Laplacian) has a negative sign.

  Therefore, the color blur pixel is logically expressed as “the subject texture pixel has an intensity gradient due to color blur (∇B ≠ 0), and the gradient has a downwardly convex curvature (ΔB <0)”. Can be expressed in the formula. Then, in the B plane, the target image corresponding to the logical expression in any of the x direction and the y direction is determined as a color blur pixel.

  In the estimation step S152, a color blur estimation amount that is the intensity of the extra B plane that is color blur is calculated only for the pixels determined as the color blur pixels in the discrimination step S104 among the pixels of the original image. (presume. For pixels determined to be subject texture pixels, for example, the color blur estimation amount is forcibly set to 0 so as not to be a target for subsequent color blur removal.

The estimation method of the color blur estimation amount differs depending on whether B is saturated or not. In preparation for both cases, two types of color blur estimation are performed by estimation step 1 (S152a) and estimation step 2 (S152b). The quantities E ₁ and E ₂ are calculated.

In FIG. 7, in the light blue area and the deep blue area where B is not saturated, the estimated color blur estimated amount E ₁ is calculated in the estimation step 1 (S152a). Estimation step 1, as shown by the following equation, by multiplying the coefficient k ₁ as the estimated amount E ₁ color blur with respect to the absolute value of the intensity gradient ∇B of B.

E ₁ = k ₁ · | ∇B |
Here, the coefficient k ₁ is a positive value, and is preferably around 3.

On the other hand, in the blue saturated region where B is saturated, the intensity gradient of B becomes 0, and the intensity gradient before saturation cannot be obtained. The color blur estimator _{E 2} of such regions is calculated by the estimation step 2 (S152b). In the estimation step 2, as shown by the following equation, the color blur estimation amount E ₂ is obtained using the G intensity gradient ∇G and the coefficient k ₂ .

E ₂ = k ₂ · | ∇G |
Here, the coefficient k ₂ is a positive value, around 3 being preferred.

In the region determination step S153, first, nonlinear conversion is performed on the intensity of the B plane to generate a saturation S. The degree of saturation S indicates whether or not B is saturated, and is 1 in a region where the intensity of B is saturated, and 0 in a region where the intensity of B is small. Then, using this degree of saturation S, E ₁ or E ₂ calculated in the estimation step S152 is selected. That is, the new estimated color blur amount E is

And Note that the saturation S may be a value that continuously changes from 0 to 1, and in this case, a new color blur estimation amount E is set as follows.
E = (1-S) E ₁ + SE ₂
And

  In the excessive removal suppression step S154, the color blur estimation amount E is further corrected to determine a color blur removal amount E ′ that is a color blur amount to be actually removed. The color blur amount estimated in the estimation step S152 follows a certain model and does not necessarily match the actual blur amount. For example, even if the light is detected on the same B plane, the bleeding changes between light having a wavelength of 450 nm and light having a wavelength of 400 nm, but this is not considered in the estimation step S152. If the removal amount is too small, some blueness remains after removal.

  On the other hand, if the removal amount is excessive, B is excessively reduced with respect to the gray background, resulting in a yellowish green color. In particular, the latter is unnatural and gives an observer a great sense of discomfort. Therefore, in the excessive removal suppression step S154, the color blur removal is limited so as to act only within a certain hue range.

  For this reason, first, the chromaticity of the pixel is calculated. For the strength of each plane of R, G, B,

And

  The chromaticity coordinate ab plane is shown in FIG. Blue is in the fourth quadrant, and when the estimated amount E is removed from the B plane intensity, it moves in the upper left direction as indicated by a dotted arrow 321. The starting point of the arrow is the chromaticity before the removal, and the tip is the chromaticity after the estimated amount E is removed.

From this, when the hue range in which color blur removal acts is limited to a> 0 and b <0,
B> 0.22R + 0.68G and B> −1.84R + 3.30G
It becomes.

  Therefore, in step S154, E ′ = 0 is set for a pixel that does not satisfy this condition, and is excluded from the removal target. Accordingly, these pixels are not changed by the removal step S155, and the pixel value is not affected by the color blur removal unit 150. That is, only pixels that satisfy this condition are targeted for removal.

Furthermore, the color blur removal amount E ′ is also applied to pixels that satisfy the conditions.
E '= min (E, B- (0.22R + 0.68G), B-(-1.84R + 3.30G))
Is passed to the removal step S155. The chromaticity change due to the removal of the blur corresponding to the color blur removal amount E ′ remains in the fourth quadrant as indicated by the solid line arrow 322 in FIG. This prevents B from being reduced beyond the hue limit range in the removal step S155.

In the removal step S155, the color blur removal amount E ′ is subtracted from the intensity of the B plane to obtain a new B plane intensity.
B = B−E ′
And

  In this way, color blur removal processing (color blur reduction processing) using the color blur estimation amount is performed. The color image with the B plane corrected in this way is passed to the visual correction unit 160 as the output of the color blur removal unit 150.

  According to the present embodiment, since the color gradation and color blur of the subject included in the original image can be clearly discriminated, it is unnecessary while avoiding accidental removal of the original color component of the subject. Color blur can be reduced.

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment when the present invention is implemented.

  For example, in the above-described embodiment, the color imaging apparatus including all of the imaging optical system 110 to the recording unit 180 has been described. However, an image processing apparatus (such as a personal computer) having at least a color blur removal unit (extraction unit, estimation unit, and removal unit) 150 may be configured as an apparatus different from the color imaging apparatus. In this case, a color image captured by a color imaging apparatus and stored in a recording medium such as a semiconductor memory or a magnetic / optical disk may be input to the image processing apparatus. Alternatively, the color imaging device and the image processing device may be connected by a cable, a wireless LAN, or the like so that an image is transmitted from the color imaging device to the image processing device.

  In the above-described embodiment, the case where the subject texture pixel and the color blur pixel are determined based on the intensity gradient and the intensity curvature in the three adjacent pixels is described. However, for example, in the case where the intensity gradient in more pixels is obtained and the subject texture pixel and the color blur pixel can be distinguished based on the distribution (that is, information indicating a change mode of the intensity gradient), The determination may be performed without using the intensity curvature.

1 is a block diagram illustrating a configuration of a color imaging apparatus that is an embodiment of the present invention. 6 is a flowchart illustrating processing in a color blur removal unit in the embodiment. The flowchart which shows the process in the space calculation step in an Example. The figure which shows the primary differential filter used in an Example. The figure which shows the secondary differential filter used in an Example. The figure which shows the example of the picked-up image containing a high-intensity light source and a color blur. The figure which shows typically the brightness | luminance saturation area | region and color blur area which are formed in the periphery of the high-intensity light source in the picked-up image of FIG. The figure which shows the image intensity profile in the cross section which passes along the center of the to-be-photographed object in FIG. The figure which shows the example of the picked-up image which imaged the night sky. The figure which shows the image intensity profile in the cross section in the broken line in the picked-up image of FIG. The figure explaining the change on a chromaticity diagram by color blur removal.

Explanation of symbols

90 imaging field 102 tilt calculation step 103 curvature calculation step 104 determination step 110 imaging optical system 120 image sensor 130 AD conversion unit 140 demosaic unit 150 color blur removal unit 151 space calculation step 152 estimation step 153 region determination step 154 excessive removal suppression step 155 Removal step 160 Visual correction unit 170 Compression unit 180 Recording unit

Claims (4)

An inclination calculator that calculates an intensity gradient in the color original image obtained by the imaging device;
A discriminating unit that discriminates a color blur pixel and a subject texture pixel in the original image based on information representing the aspect of the change in the intensity gradient;
And a processing unit that calculates a color blur estimated amount in the color blur pixel based on the intensity gradient and performs a color blur reduction process using the color blur estimated amount on the color blur pixel. Image processing device. As the information, it has a curvature calculator that calculates the curvature of the intensity gradient,
The image processing apparatus according to claim 1, wherein the determination unit determines a color blur pixel and a subject texture pixel in the original image based on the intensity gradient and the curvature. Calculating an intensity gradient in the color original image obtained by the imaging device;
Discriminating color blur pixels and subject texture pixels in the original image based on information representing the aspect of change in the intensity gradient;
Calculating an estimated amount of color blur in the color blur pixel based on the intensity gradient and performing a color blur reduction process using the color blur estimated amount on the color blur pixel. Processing method. An imaging system that images a subject and generates an image;
An image pickup apparatus comprising: the image processing apparatus according to claim 1.