JP2012005056A - Image processing device, image processing method and program - Google Patents

Abstract

Even if there is a defocused region in image data, it is possible to correct an optical aberration of an image pickup optical system and to perform a good image restoration that produces a blur according to the defocus or aperture state.
A subject distance calculation unit calculates a subject distance in each distance measurement area of image data generated by an imaging means. The filter selection unit 73 acquires the subject distance of the ranging area corresponding to each divided area as the subject distance of each divided area of the image data. The restoration processing unit 76 corrects the optical aberration of each divided region based on the subject distance of each divided region, and blurs each divided region based on the subject distance of each divided region and the aperture state of the imaging means 62. give.
[Selection] Figure 1

Description

  The present invention relates to a technique for performing blur and optical aberration restoration processing on an image obtained by imaging.

  Blurred (Blur) has occurred in the photographed image, and an image restoration algorithm for restoring the blurred image is conventionally known. The blur includes factors such as aberration, defocus, movement of the imaging device during exposure time (including camera shake), movement of the subject during exposure time, and atmospheric fluctuation.

  As an image restoration algorithm, for example, a technique is known in which the characteristics of the image formation state of an image are represented by a point spread function (PSF) and the image is restored to a blur corrected based on the PSF. For example, Patent Document 1 discloses a technique for correcting blur using a correction function having the inverse characteristics of PSF.

Japanese Patent Laid-Open No. 62-127976

  However, the image actually obtained by shooting has a defocused area, and when restoration processing is performed based on the PSF at the focused subject distance, good restoration with blur corresponding to defocus or aperture state is achieved. There was a problem that images could not be obtained.

  Therefore, an object of the present invention is to correct a blur due to optical aberrations of the imaging optical system even when there is a defocused area in the image, and obtain a good restored image that produces a blur according to the defocus or aperture state. There is.

  The image processing apparatus of the present invention includes a holding unit that holds subject distance information in each ranging area of the imaging unit, subject distance information of each restoration area of the image to be restored, and subject distance information of each ranging area. Based on the defocus information and the diaphragm state of each of the divided areas, and the correction means for acquiring the optical aberration due to the difference in position (image height) and subject distance information in the image of each restoration area Correction means for giving a blur to each of the restoration areas.

  According to the present invention, even when there is a defocused area in the image, the blurring due to the optical aberration of the imaging optical system is corrected, and the defocusing and blurring process according to the aperture state is performed to perform blurring. A good restored image can be obtained.

1 is a diagram illustrating a basic configuration of an image restoration apparatus according to a first embodiment of the present invention. It is a figure which shows the relative positional relationship of A image and B image which are two image signals output from the area sensor for ranging. It is a figure which shows the positional relationship of several AF ranging area and the division area | region which carries out image restoration in the 1st Embodiment of this invention. 6 is a flowchart illustrating a process of performing image restoration by acquiring subject distance information of a plurality of regions in an image in the first embodiment of the present invention. It is a figure which shows the arrangement | sequence of the color mosaic filter on a solid-state image sensor. It is a conceptual diagram of a list of restoration filters. A state in which a point on the focal plane at the in-focus subject distance forms an image on the solid-state image sensor, and a state in which a point not on the focal plane at a subject distance different from the in-focus subject distance is defocused FIG. It is a figure which shows the basic composition of the image restoration apparatus which concerns on the 2nd Embodiment of this invention. FIG. 10 is a diagram showing a positional relationship between a plurality of AF distance measurement areas and a plurality of restoration areas in the second embodiment of the present invention. It is a flowchart which shows the process which acquires the object distance information of the several area | region in an image, and performs image restoration in the 2nd Embodiment of this invention. It is a figure which shows the relationship between the extension position of a lens, and contrast about each AF ranging area.

  First, the principle of image restoration will be described. The z axis is taken as the optical axis of the imaging system, and the x axis and the y axis are taken in the direction perpendicular to the z axis. A faithful image of a subject (object) is called an original image, and the original image is expressed as f (x, y) using position coordinates (x, y). An image obtained by imaging is expressed as g (x, y) using coordinates (x, y) on the image.

The original image f (x, y) can be considered to be composed of an infinite number of point light sources. The two-dimensional delta function δ (x, y) can be considered as a point light source having a luminance of 1, and the original image f (x, y) has a luminance at a position (x ₀ , y ₀ ) on the xy plane. Is expressed as the following equation 1 as a linear sum of point light sources of f (x ₀ , y ₀ ). Note that ₀ is a subscript indicating the subject.

When a point light source is given as an input, a PSF (point spread function) indicating a blurred image as an output through an imaging system is assumed to be h (x, y). This response h (x, y) corresponds to the impulse response of the imaging system L, and is a two-dimensional delta function δ (x−x ₀ , yy) as a point light source at the position (x ₀ , y ₀ ). The output when ₀ ) is given is expressed by the following equation 2.

  Therefore, the relationship shown in the following Expression 3 is established between f (x, y), g (x, y), and h (x, y).

  The value of PSF generally changes depending on the input and output positions. However, in the region where the response of the two-dimensional system is uniform and the response due to the input position does not change, the following expression 4 holds.

Here, * represents a convolution integral.
When this is Fourier-transformed and expressed in the spatial frequency domain, Equation 4 becomes Equation 5 below.

  Here, u and v are spatial frequencies in the x and y directions, respectively, and F (u, v), G (u, v) and H (u, v) are f (x, y) and g (x, x, y) and h (x, y) are two-dimensional Fourier transforms. H (u, v) represents a spatial frequency transfer function indicating the frequency response characteristic of the system, that is, a frequency characteristic of the two-dimensional filter, and is called an optical transfer function (OTF).

  In order to restore an image from the captured image g (x, y), the following Expression 6 obtained by transforming Expression 5 may be used.

  Obtaining the original image f (x, y) by performing inverse Fourier transform on F (u, v) obtained by multiplying 1 / H (u, v), which is an OTF inverse filter, and returning it to the real space. Can do. Further, an image restoration filter obtained by inverse Fourier transform of 1 / H (u, v) is convolved with the captured image g (x, y) as in Equation 7 obtained by inverse Fourier transform of Equation 6. By integrating, the original image f (x, y) can be obtained.

When actually restoring the image, since the captured image g (x, y) includes a noise component, if the image restoration filter F ⁻¹ {1 / H (u, v)} is used as it is, The image is amplified with noise. For this problem, for example, a method considering the presence of a noise component such as a Wiener filter may be used. Note that h (x, y), which is a PSF, varies depending on zoom, aperture, subject position, and image height.

  It is the focal plane that is the surface of the object space that forms a focused image on the image plane. In the imaging apparatus, the focusing lens group of the optical system is extended by AF or manual operation, and a single surface (focal plane) in the object space is focused and photographed. In the case of performing image restoration, an image restoration filter based on a subject distance focused on by extending the focusing lens group (hereinafter referred to as a focused subject distance) is applied.

  However, an image of a defocused region obtained by imaging (hereinafter referred to as a defocused region) is not focused on the focal plane in the object space, and is at a subject distance different from the focused subject distance. The PSF is different from the point on the plane. For this reason, in the defocus area, instead of the PSF at the focused subject distance, the optical arrangement with the focusing lens group extended to the focused subject distance is originally set to the subject distance of the defocused object. An image restoration filter based on the PSF when a certain point is imaged should be applied.

  Actually, a color image including a defocus area is taken, and an image restoration filter based on the PSF of the focused object distance is applied to each of the R, G, and B components in consideration of the PSF for each wavelength band. Restored. As a result, the in-focus area in the image is restored well, but the chromatic aberration due to the aberration of the imaging optical system occurs in the defocus area, and the image before restoration is restored. In comparison, we found that the restored image has a larger color difference and may not be able to be restored. In addition to this example, if the actual PSF of the region to be restored is different from the PSF used to create the image restoration filter, a region in which the restored image is deteriorated more than the image before restoration is generated. There is a case.

  The actual PSF of the imaging optical system includes blur due to defocus and blur due to the aperture state (aperture size and shape), in addition to blur due to aberration according to the subject distance. For this reason, if an image restoration filter based on the PSF that takes into account the defocus state of the area to be restored is applied as it is, a good restored image having a blur corresponding to the defocus or aperture state cannot be obtained.

  As a result of analyzing a conventional defect that cannot be satisfactorily restored in the defocus region of the image, it was found that the aberration generated by the photographing optical system is different depending on the subject distance. The subject distance in the defocus area is different from the in-focus subject distance which is the subject distance focused by the extension of the focusing lens group of the optical system. For this reason, the aberration that occurs when the subject is at the in-focus subject distance is different from the aberration that occurs when the subject is at the subject distance in the defocus area. Nevertheless, the image restoration filter based on the focused subject distance is also applied to the image restoration in the defocus area, so there is a difference from the image restoration that corrects the defocused aberration information, and the difference in the generated aberrations Thus, it was found that the color difference expanded.

  Then, in order to suppress deterioration due to image restoration caused by such a cause, it has been found that the subject distance information may be used for each region having a different focus state. If each subject distance information is used, a good result can be obtained by performing an image restoration with a restoration filter reflecting a difference in aberration due to a different focus state in each region.

  At this time, the subject distance information is required for each region having a different focus state in the image. As a means for acquiring subject distance information within the angle of view for photographing, a method for photographing a plurality of images by changing the amount of extension of the focusing lens group, and creating a depth-of-field map from the plurality of images, and an imaging device In addition to this, a method using a three-dimensional measuring apparatus for measuring the arrangement of a subject is known.

  However, in the method using a plurality of images and a three-dimensional measuring device, when an image is actually captured by an imaging device and restoration processing is performed on the still image or moving image, the subject distance for each region is obtained. This is not practical because it requires time and another measuring device. In order to realize this, it is necessary to acquire object distance information of a plurality of areas in an image and perform image restoration in response to acquisition of a still image or a moving image.

  And it discovered that it was effective to use the distance information of a plurality of AF ranging areas as the subject distance information. In this way, if a PSF that takes into account the defocus state for each area to be restored is obtained from the lens design data and an image restoration filter is created using the PSF, aberrations due to differences in subject distance can be corrected. However, since the image restoration filter is not created from the PSF based on one subject distance (here, the in-focus subject distance) for all the regions to be restored, unlike in the past, it depends on the defocus or aperture state. A good image with blur is not obtained.

  This is because the PSF considering the defocus state includes blur due to the defocus and the aperture state in addition to the blur due to the optical aberration corresponding to the subject distance. Therefore, the PSF of the aberration-free lens corresponding to the designed photographing lens is obtained, and this blur is added to the restoration filter, thereby creating a restored image having a blur corresponding to the defocus and the aperture state.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described in detail with reference to the accompanying drawings. First, a first embodiment of the present invention will be described. FIG. 1 is a diagram showing a basic configuration of an image restoration apparatus according to the first embodiment of the present invention. In the present embodiment, an example will be described in which image restoration processing is performed within a camera that has captured an image, and an image file is stored in a storage medium. Note that the camera according to the present embodiment has a configuration that is an application example of the image processing apparatus of the present invention.

  Reference numeral 1 denotes an optical axis of the optical system. Reference numeral 10 denotes a photographic lens, which includes a focus lens group 12 for adjusting the focus, a lens group 11 other than the focus lens group 12, and a diaphragm 13 for adjusting the amount of light. The photographing lens 10 is a zoom lens, and has a function that allows the zoom position at the time of imaging to be detected through the zoom position control unit 41. The aperture state at the time of shooting has a function that can be detected through the aperture controller 51. Further, the photographing lens 10 is replaceable, and has a function capable of detecting the type of lens used at the time of photographing.

  Reference numeral 21 denotes a main mirror. A light beam from a subject (not shown) is reflected upward by the main mirror 21 via the photographing lens 10 to form an image on the focusing screen 22. The image formed on the focusing screen 22 is reflected by the pentaprism 23 several times, and the subject is visually recognized by the photographer via the eyepiece 24.

  On the other hand, a part of the light beam reaching the main mirror 21 from the imaging lens 10 is transmitted through the half mirror part of the main mirror 21, reflected downward by the sub mirror 31, and guided to the AF detection part 32. At the time of shooting, the main mirror 21 and the sub mirror 31 jump upward and retract outside the shooting beam, so that the beam from the shooting lens 10 forms an image on the solid-state image sensor 62 via the optical LPF 61. The solid-state image sensor 62 is constituted by a CCD, a CMOS sensor, or the like, photoelectrically converts a subject image formed on the light receiving surface, and outputs an image signal. The optical LPF 61 serves to suppress the aliasing component due to spatial sampling of the solid-state image sensor 62. The image signal output from the solid-state image sensor 62 is converted from an analog signal to a digital signal by the A / D converter 63, and RAW image data is stored in the buffer memory 71.

  The AF method in this embodiment is a phase difference detection method having a plurality of AF ranging areas. The AF detection unit 32 includes a pupil division optical system that separates the light beam from the sub-mirror 31 into two light beams, a pair of distance measuring area sensors that capture the two separated images, and the relative positional relationship between the two images. And a phase difference calculation unit that calculates a phase difference that is a positional deviation amount.

  FIG. 2 shows the relative positional relationship between the A and B images, which are two image signals output from the distance measuring area sensor. Since the two images are composed of light beams that have passed through different pupils, the relative positions of the images are changed according to the amount of lens extension, such as the front pin (FIG. 2A), the focus (FIG. 2B), and the rear pin ( It is different in each state of FIG. The relative shift amount between the two images is calculated by performing a correlation operation between the two image signals, and a phase difference δ is obtained. The phase difference δ is zero at the time of focusing, a negative value at the front pin, and a positive value at the rear pin. Since the relationship between the phase difference δ and the defocus of the optical system is uniquely determined, the phase difference δ detected by the AF detection unit 32 is sent to the lens extension control unit 33. The lens extension control unit 33 obtains the lens extension amount necessary for obtaining the focus from the phase difference δ and the position of the focus lens group 12 at the time of AF distance measurement, and outputs the focus lens group 12 by extending the focus lens group 12. Perform focus control.

  FIG. 3 is a diagram illustrating a positional relationship between an image obtained by photographing and a plurality of AF distance measurement areas. Reference numerals 101 to 135 denote AF distance measurement areas. The AF distance measurement area can be selected by manual selection in which the photographer selects an arbitrary AF distance measurement area, or automatic selection in which the image restoration apparatus determines the main subject and focuses. There is. Manual selection includes a method of selecting an AF distance measurement area by button operation, a method of selecting an AF distance measurement area by eye-gaze input, a method of selecting an AF distance measurement area by a liquid crystal display touch panel, and the like.

  Further, when the defocus calculation result for the AF distance measurement area determined according to the photographing intention of the photographer by the selection unit is NG, the selected AF measurement is performed for the AF distance measurement area other than the selected area. Priority is given to an area close to the distance area and close to the center of the shooting screen. For example, switching from manual selection to automatic selection.

  FIG. 4 is a flowchart showing a process of performing image restoration by acquiring subject distance information of a plurality of regions in an image according to the first embodiment of the present invention. An operation for acquiring subject distance information of a plurality of regions in an image and performing image restoration will be described with reference to FIGS. The AF sequence is started when the photographer performs a one-step depression operation of a release button (not shown).

  In step S1, image signals (A image signal and B image signal, which are two image signals) from the distance measuring area sensor of the AF detector 32 are detected for the plurality of AF distance measuring areas 101 to 135 in FIG. . In step S2, as described above, the phase difference calculation unit of the AF detection unit 32 calculates the correlation between the two image signals, thereby obtaining the phase differences δ of the plurality of AF ranging areas 101 to 135. In step S3, the phase differences δ of the plurality of AF ranging areas 101 to 135 calculated in step 2 are sent from the AF detection unit 32 to the ranging information holding unit 34 and held therein. In step S4, an AF ranging area to be focused is selected. When manual selection is set, the AF range area selected by the photographer is selected. When automatic selection is set, the AF range area determined by the image restoration device as the main subject is selected. . In step S5, the phase difference δ of the AF distance measurement area to be focused is sent to the lens extension control unit, and the lens extension amount necessary to obtain the focus is calculated. In step S6, focusing is performed by extending the focusing lens group 12. The position of the focusing lens group 12 at the time of AF distance measurement and in-focus is detected through the lens extension control unit 33, sent to the distance measurement information holding unit 34, and held there.

  In step S7, the subject distance calculation unit 35 calculates the in-focus subject distance from the phase difference δ of the plurality of AF distance measurement areas 101 to 135 and the position of the focus lens group 12 during AF distance measurement and in-focus. The subject distance of each AF distance measurement area is calculated. Since the lens position at the time of focusing is known at the time when the extension amount is calculated in step S5, the calculation in step S7 may be performed during lens extension.

  Next, the imaging sequence is started when the photographer depresses a release button (not shown) one more step. In order to perform imaging, the main mirror 21 and the sub mirror 31 are retracted out of the photographic light beam, and the light beam from the subject forms an image on the solid-state image sensor 62 via the photographic lens 10 and the optical LPF 61. On the solid-state image sensor 62, a color mosaic filter composed of R, G, and B as shown in FIG. 5 is arranged so that signals of R, G, and B color components can be acquired. In step S8, the subject image formed on the solid-state image sensor 62 is photoelectrically converted and output as an image signal, converted into a digital signal by the A / D converter 63, and stored in the buffer memory 71 as RAW image data. Stored.

  Next, the image restoration process will be described. In FIG. 1, reference numeral 74 denotes a restoration filter DB, which stores a restoration filter used for image restoration. In this embodiment, the restoration filter is created based on the PSF obtained from the design data of the imaging system.

  The imaging system design data includes the design data of the photographing lens 10, the size and shape of the diaphragm 13, the spatial frequency characteristics of the optical LFP 61, the pixel pitch and pixel aperture shape of the solid-state image sensor 62, and the color mosaic filter on the solid-state image sensor 62. And the like. Also, respective restoration filters are prepared for the R, G, and B color components. For this reason, it is necessary to consider the transmittance characteristics for each wavelength band in order to create a restoration filter. This corresponds to the PSF weighting coefficient for each wavelength band, and includes the spectral transmittance characteristic of the imaging lens 10, the spectral transmission characteristic of the color mosaic filter on the solid-state imaging element 62, the spectral sensitivity characteristic of the solid-state imaging element 62, and the like. Further, when an infrared cut filter (not shown) is used, it is necessary to consider its spectral transmission characteristics.

  Actually, the image restoration filter stored in the restoration filter DB 74 has a shape corresponding to each type of the imaging lens 10 by dividing the range of each parameter of the zoom position and the aperture. FIG. 6 shows a conceptual diagram of the restoration filter list. Since the photographing lens 10 is replaceable, a restoration filter list is prepared for each type of the imaging lens 10 being used, the zoom position at the time of photographing, and the aperture parameter (FIG. 6A). Further, a restoration filter table corresponding to the focused subject distance df at the time of shooting is prepared (FIG. 6B), and restoration processing is performed on the captured RAW image data using the corresponding restoration filter. Is called.

  Since the PSF changes according to the angle of view (image height) of the imaging optical system even in one shooting state, the image is divided into a plurality of restoration areas as shown in FIG. A restoration process is performed using a restoration filter corresponding to H.235.

  The image restoration processing of the RAW image data acquired in step S8 will be described with reference to FIGS. A CPU 70 includes a filter selection unit 73, a restoration processing unit 76, and a post-processing unit 77 as its functional configuration. The filter selection unit 73 detects the type of the photographic lens 10, receives the zoom position at the time of imaging RAW image data from the zoom position control unit 41, and the aperture state at the time of imaging from the aperture control unit 51, and receives the drawing from the restoration filter DB 74. The restoration filter list of 6 (a) is selected. Next, the filter selection unit 73 selects a corresponding restoration filter table from FIG. 6B based on the focused subject distance df calculated by the subject distance calculation unit 35. The restoration filter interpolation generation unit 75 restores a filter corresponding to an intermediate value of parameters (zoom position, aperture state, focused subject distance, subject distance in each restoration area) that are not in the restoration filter list in the restoration filter DB 74. Is necessary, an interpolation operation is performed from the restoration filter of the adjacent condition to generate a necessary restoration filter. In addition, the restoration filter interpolation generation unit 75 stores a reduced number of divisions for each parameter in order to reduce the data capacity of the restoration filter table in the restoration filter DB 74, and generates a restoration filter with necessary conditions. Is also used.

  In step S9, the subject distance corresponding to each restoration area 201-235 is acquired. In the present embodiment, as shown in FIG. 3, the division positions of the AF ranging areas 101 to 135 and the restoration areas 201 to 235 are matched. For this reason, as the subject distance corresponding to each restoration area, the subject distance of each AF distance measurement area corresponding thereto may be used. This subject distance is held in the ranging information holding unit 34 as a phase difference δ during AF, and the subject distance d of each AF ranging area calculated by the subject distance calculating unit 35 in step S7 is used. In step S10, a restoration filter for each restoration area is acquired. Specifically, from FIG. 6B, a subject distance in the AF distance measurement area selected for focusing in step S4, that is, an image signal for AF is detected, and a focusing lens group is detected. The restoration filter table corresponding to the in-focus subject distance df, which is the in-focus subject distance, is selected. In FIG. 7, the point on the focal plane at the focused subject distance df is not on the focal plane at the subject distance d different from the focused subject distance df when the image is formed on the solid-state imaging device 62. A dot indicates a defocused state. The selected restoration filter table is the lens arrangement when the focusing lens group 10 is extended so as to focus on the focal plane at the focused subject distance df, and as shown in FIG. 7, the focused subject distance df, It consists of a restoration filter based on the inverse function of PSF at each subject distance d in the defocused area on the image. The restoration processing unit 76 selects a restoration filter corresponding to the subject distance d of the corresponding AF distance measurement area from the selected restoration filter table for each of the restoration areas 201 to 235 shown in FIG. In this way, the restoration area based on the PSF of the in-focus subject distance df is selected as the in-focus area, and the restoration filter based on the PSF of the de-focused object distance d is selected as the defocus area. It becomes possible to correct the optical aberration due to the difference.

  In step S11, a blur correction filter for obtaining blur according to the defocus and the aperture state is acquired from the blur correction filter DB 72 of FIG. As the aperture state, the F value at the time of shooting is used. This blur correction filter is created based on the PSF of an aberration-free lens corresponding to the photographing lens 10, and blur correction that divides the range of each parameter of the zoom position and aperture state for each type of the imaging lens 10. A filter list is provided. In the filter list of the restoration filter, the restoration filter table corresponding to the focused subject distance df is selected, and then the restoration filter corresponding to the restoration area is selected. However, since the PSF of the non-aberration lens does not change depending on the image height, the blur correction filter corresponding to the defocused state of each restoration area, that is, the subject distance d, is obtained from the filter table corresponding to the focused subject distance df of the blur correction filter. Select.

  In step S12, the restoration processing unit 76 multiplies the restoration filter selected in step S10 by the blur correction filter selected in step S11 to generate a restoration filter that produces blur according to defocus and aperture state. To do. Then, the restoration processing unit 76 performs image restoration by convolving and integrating the filter with respect to the RAW image data for each restoration region.

  The post-processing unit 77 performs image processing called so-called development processing such as demosaicing processing, white balance processing, noise reduction processing, and γ processing on the RAW image data that has been restored. Because it is not, it is omitted. The restored image is stored in a predetermined image format in a recording medium 78 including a removable memory card.

  As in the present embodiment, even when there is a defocused area in the image, the subject distance information of each area is retained, and the defocus area is defocused based on the PSF of the focused object distance df. Aberration correction is performed based on the PSF of the subject distance d. The blur according to the defocus or the diaphragm state gives the blur based on the aperture state at the time of shooting and the PSF of the non-aberration lens corresponding to the subject distance d for each restoration area. As a result, even when there is a defocused area in the image, it is possible to correct the optical aberration of the imaging optical system and obtain a good restored image that produces a blur according to the defocus or aperture state.

  In the present embodiment, the color mosaic filter is arranged on the solid-state image sensor 62 and the signals of the R, G, and B color components are acquired. However, the R, G, and B colors are obtained by the color separation optical system. A light beam may be separated into components, and an R, G, and B solid-state imaging device is arranged for each light beam to obtain an image signal.

  In this embodiment, the image restoration filter is created based on the PSF obtained from the design data of the imaging system. The PSF is an inverse Fourier transform of the system optical transfer function (OTF). Therefore, an image restoration filter may be created based on the OTF obtained from the design data of the imaging system. Since the blur correction filter is created based on the PSF of the aberration-free lens, similarly, the blur correction filter may be created based on the OTF of the aberration-free lens.

  Next, a second embodiment of the present invention will be described. In the present embodiment, a case will be described in which image restoration is performed to generate blur according to the aperture state designated by the user.

  FIG. 8 is a diagram showing a basic configuration of an image restoration apparatus according to the second embodiment of the present invention. In this embodiment, the image restoration process is not performed in the camera that captured the image, but the image file is stored in a storage medium, and the image restoration process is performed in the PC. In the present embodiment, the number of AF ranging areas is smaller than the number of restoration areas.

  FIG. 10 is a flowchart showing processing for acquiring subject distance information of a plurality of regions in an image and performing image restoration in the second embodiment of the present invention. Below, the description of the same part as 1st Embodiment is abbreviate | omitted, and only a different part from 1st Embodiment is demonstrated.

  In the second embodiment, as shown in FIG. 8, the camera 80 does not perform image restoration processing, but forms an image on the solid-state image sensor 62 and stores it in the buffer memory 71 via the A / D converter 63. Information necessary for image restoration is added to the image data in the recording unit 79 and the image file is stored in the recording medium 78. Here, the information necessary for image restoration is the type of the taking lens 10, the zoom position, the aperture state, the focused subject distance df, and the subject distance d of each AF distance measurement area. In addition, the camera body ID for identifying the camera 80, the lens No. ID, ISO sensitivity at the time of photographing, and the like may be added.

  The recording unit 79 detects the type of the photographing lens 10, and controls the zoom position at the time of imaging of the RAW image data from the subject distance calculation unit 35 based on the in-focus subject distance df and the subject distance d in each AF ranging area. The aperture state at the time of shooting is received from the aperture control unit 51 from the unit 41. Then, the recording unit 79 adds the information necessary for the above-described image restoration to the header portion of the image file that stores the RAW image data, and stores the information in the recording medium 78 that is a detachable card memory. Note that the additional information for image restoration may be stored in the recording medium 78 as a separate file instead of the header portion of the image file.

  FIG. 9 is a diagram illustrating a positional relationship among an image obtained by shooting, a plurality of AF distance measurement areas, and a plurality of restoration areas. The AF ranging areas 109 to 113, 116 to 120, 123 to 127, and the restoration areas 201 to 235 are arranged as shown in FIG. 9, and the number of AF ranging areas is smaller than the number of restoration areas.

  Next, the image restoration process in the PC 90 in the present embodiment will be described with reference to FIGS. Steps S1 to S7 in FIG. 10 are the same processes as steps S1 to S7 in FIG. 4 in the first embodiment, and thus description thereof is omitted.

  The user removes the recording medium 78 from the camera 80 and attaches it to the PC 90. The recording medium 78 stores an image file in which information necessary for image restoration is added to the header portion. The restoration processing unit 76 acquires RAW image data from an image file in which information necessary for image restoration stored in the recording medium 78 is added to the header portion (step S21 in FIG. 10). The filter selection unit 73 reads out the type of the photographing lens 10, the zoom position at the time of photographing, and the aperture state at the time of photographing from the image file in which information necessary for image restoration stored in the recording medium 78 is added to the header portion. The restoration filter list shown in FIG. 6A is selected from the restoration filter DB 74. In step S22, the subject distances of the AF ranging areas 109 to 113, 116 to 120, and 123 to 127 are obtained from the image file in which information necessary for image restoration stored in the recording medium 78 is added to the header portion. The focal subject distance df and the subject distance d of each AF distance measurement area are read and acquired. In the present embodiment, as shown in FIG. 9, the number of AF ranging areas 109 to 113, 116 to 120, and 123 to 127 is smaller than the number of restoration areas 201 to 235, and they do not match. For this reason, for the restoration areas 201 to 208, 214 to 215, 221 to 222, and 228 to 235 that have no matching AF distance measurement area, the subject distance of the nearest AF distance measurement area is set to the subject distance of the restoration area. And Thereafter, as in the first embodiment, a restoration filter is acquired in step S23. At this time, the restoration filter is selected using the F value at the time of shooting as the aperture state parameter.

  In step S <b> 24, a blur correction filter for generating blur according to the defocus and aperture state is acquired from the blur correction filter DB 72. At this time, in the present embodiment, the F value specified by the user is used instead of the F value at the time of shooting as the aperture state as in the first embodiment. Thereby, the blur correction filter corresponding to the F value designated by the user is selected. In step S25, as in the first embodiment, the restoration processing unit 76 performs an image restoration calculation using the restoration filter acquired in step S23 and the blur correction filter acquired in step S24. Subsequent processing is the same as that of the first embodiment, and thus description thereof is omitted.

  As described above, even when a defocused region exists in the image, the optical aberration of the imaging optical system is corrected, and a restored image having a blur according to the defocus and the aperture F value designated by the user is obtained.

  For the restoration area that does not have a matching AF distance measurement area, the subject distance information of the nearest AF distance measurement area is used as the subject distance information of the restoration area. The subject distance information of the AF area having the closest aberration characteristic may be used. In the case of an axisymmetric photographic lens, the AF distance measurement area where the image height from the optical axis is closest is the area where the aberration characteristics are closest.

  As described above, the phase difference detection type AF is described. However, as another embodiment, a contrast type AF may be used. In this case, the focusing lens group 12 is moved in the range from the closest distance to infinity, and the contrast is detected with respect to the lens extension position in each AF distance measurement area as shown in FIG. In each AF distance measurement area, the lens extension position corresponding to the peak position of the contrast is the position where each subject is in focus. The subject distance d in each AF distance measurement area can be obtained by calculation from the lens extension position where the contrast reaches a peak. Further, when the image is actually picked up, the subject distance in the AF ranging area selected for focusing may be set as the focused subject distance df.

  When there is subject distance information of a plurality of AF distance measurement areas in one restoration area, the subject distance information may be selected from the information. In addition, although the embodiment has been described in which the focus group of the optical system is extended and focused, the relative distance between the imaging optical system and the solid-state imaging device may be moved as a focusing means. In this case, position information obtained by moving the relative distance is used instead of the position of the focusing lens group.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  1: optical axis, 10: imaging lens, 11: lens group other than the focusing lens group, 12: focusing lens group, 13: stop, 21: main mirror, 22: focusing plate, 23: pentaprism, 24: eyepiece Lens: 31: Sub mirror, 32: AF detection unit, 33: Lens extension control unit, 34: Distance measurement information holding unit, 35: Subject distance calculation unit, 41: Zoom position control unit, 51: Aperture control unit, 61: Optical LPF, 62: imaging device, 63: A / D, 71: buffer memory, 72: blur correction filter DB, 73: filter selection unit, 74: restoration filter DB, 75: restoration filter interpolation generation unit, 76: restoration processing unit , 77: post-processing unit, 78: recording medium, 79: recording unit

Claims (8)

Holding means for holding subject distance information in each ranging area of the imaging means;
Acquisition means for acquiring subject distance information of each restoration area of image data for performing image restoration based on subject distance information of each distance measurement area;
Correction means for correcting optical aberration due to a difference in position and subject distance information in the image of each restoration area and giving blur to each restoration area based on defocus and aperture state of each restoration area; An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the optical aberration is corrected based on aberration information obtained from design data of the imaging system. As the blur based on the defocus information and the aperture state of each restoration area, the blur of the point spread function (PSF) or the optical transfer function (OTF) of the aberration-free optical system in the aperture state at the time of the defocus and imaging is used. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. As the blur based on the defocus information and the aperture state of each restoration area, the blur of the point spread function (PSF) or the optical transfer function (OTF) of the aberration-free optical system in the defocus and the aperture state specified by the user The image processing apparatus according to claim 1, wherein the image processing apparatus is used.   The said acquisition means acquires the subject distance information of the ranging area of the position corresponding to the said restoration area as the subject distance information of the said restoration area, The any one of Claim 1 thru | or 4 characterized by the above-mentioned. Image processing device.   The acquisition unit acquires subject distance information of a distance measurement area at a position near the restoration area as subject distance information of the restoration area when there is no distance measurement area corresponding to the restoration area. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. An image processing method executed by an image processing apparatus,
Holding step for holding subject distance information in each ranging area of the imaging means;
An acquisition step of acquiring subject distance information of each restoration area of image data for performing image restoration based on the subject distance information of each distance measurement area;
Correcting the optical aberration due to the difference in position and subject distance information in the image of each restoration area, and correcting the defocusing and the diaphragm state of each restoration area to the restoration area. An image processing method comprising: A calculation step of calculating subject distance information in each ranging area of the imaging means;
An acquisition step of acquiring subject distance information of each restoration area of image data for performing image restoration based on the subject distance information of each distance measurement area;
Correcting the optical aberration due to the difference in position and subject distance information in the image of each restoration area, and correcting the defocusing and the diaphragm state of each restoration area to the restoration area. A program that causes a computer to execute.

Images