JP2016001782A - Image processing apparatus, imaging apparatus, image processing method, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system that is advantageous to image restoration processing to an image including an overexposed area.SOLUTION: The image processing system includes a generation means (132) that generates an image restoration filter based on an imaging condition of an imaging means that outputs first image data imaged with first exposure, a detection means (133) that detects a region that satisfies a certain condition of the first image data based on the shape of the image restoration filter, a combination means (134) that generates combined image data so as to replace a signal value of the pixel of the region detected by the detection means with a value based on the signal value of second image data imaged with second exposure that is smaller than the first exposure by the imaging means, and a processing means (135) that executes image restoration processing for the combined image data by the image restoration filter.

Description

  The present invention relates to an image processing apparatus that performs image restoration processing on a deteriorated image.

  Patent Document 1 proposes a method of converting characteristics of an imaging optical system into data in advance and correcting a deteriorated image based on the characteristics data. As a method for correcting such a deteriorated image, for example, a method for correcting a deteriorated image using an image restoration filter generated from an optical transfer function (OTF) of an imaging optical system (hereinafter, image restoration processing) is known. Yes.

  In image restoration processing using OTF, a large spatial filter (image restoration filter) is often applied. In the image restoration process by applying the image restoration filter, the overexposed region becomes a problem. The whiteout region is an image region in which the signal value that should originally be obtained cannot be obtained due to saturation of the image sensor, and the gradation is white. Therefore, even if the image restoration filter is applied, the image cannot be correctly restored around the overexposed region. As a result, a phenomenon occurs such that an image is folded around a whiteout region, or a color that does not exist in the subject (hereinafter referred to as “coloring”) occurs.

  In order to solve this problem, Patent Document 2 proposes a method of reducing the degree of correction in the vicinity of a whiteout region and suppressing the occurrence of defects.

JP-A-4-087655 JP 2007-181170 A

  However, in the prior art disclosed in the above-mentioned patent document, the correction effect is suppressed for the overexposed region and its peripheral region, and thus the image cannot be restored effectively.

  In view of the above problems, the present invention provides an image processing device, an imaging device, an image processing method, a program, and a storage medium that are advantageous for image restoration processing for an image including an overexposed region.

  An image processing apparatus according to one aspect of the present invention includes: a generation unit that generates an image restoration filter based on a shooting condition of an imaging unit that outputs first image data captured with a first exposure amount; Detecting means for detecting a region satisfying a predetermined condition of the image data, and a signal value of a pixel in the region detected by the detecting means is set to a second exposure amount smaller than the first exposure amount by the imaging means. Combining means for generating composite image data so as to replace the value based on the signal value of the second image data imaged in step 1, and processing means for performing image recovery processing on the composite image data by the image recovery filter It is characterized by having.

  Other objects and features of the present invention are illustrated in the following examples.

  According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, a program, and a storage medium that are advantageous for image restoration processing to an image including a whiteout region.

It is a block diagram which shows the structure of the imaging device which concerns on the Example of this invention. It is a block diagram which shows the structure of the image processing apparatus which concerns on the Example of this invention. It is a flowchart which shows the flow of the image processing method which concerns on 1st embodiment of this invention. It is a flowchart which shows the flow of a detection process of the saturation area | region which concerns on 1st embodiment of this invention. It is a flowchart which shows the flow of the synthetic | combination process of the saturation area | region which concerns on 1st embodiment of this invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  First, a method for correcting blur (deterioration) of an image obtained by the imaging apparatus of the present invention using information on an optical transfer function (OTF) of the imaging optical system will be described below. This method is called “image restoration” or “image restoration”. Hereinafter, processing for correcting image degradation using information on the optical transfer function of the imaging optical system (imaging system) will be referred to as image restoration processing.

  The imaging optical system has a different refractive index depending on the wavelength of light, and the image is shifted (blurred) depending on the wavelength of light, resulting in deterioration of the image. This image degradation is called aberration. This aberration includes spherical aberration, coma aberration, field curvature, astigmatism, and the like of the imaging optical system. These aberrations can be expressed by a point spread function (PSF). The point spread function (hereinafter referred to as PSF) indicates how wide a point of the subject is mapped. For example, the two-dimensional distribution of light on the imaging sensor surface when a light emitter having a very small volume is photographed in the dark corresponds to the PSF of the imaging optical system used for imaging. In order to obtain the PSF, it is not always necessary to photograph a subject such as a point light source. For example, a method is known in which a chart having black and white edges is photographed and the PSF is obtained from the photographed image by a calculation method corresponding to the chart. Yes. It is also possible to calculate the PSF from the design data of the imaging optical system. An optical transfer function (hereinafter referred to as OTF) that can be obtained by Fourier transforming a point spread function is information in the frequency space of aberration. This OTF can be expressed by a complex number, and the absolute value of the OTF, that is, the amplitude component is called MTF (Modulation Transfer Function) and the phase component is called PTF (Phase Transfer Function).

  Since the OTF of the imaging optical system affects (deteriorates) the amplitude component (hereinafter referred to as MTF) and phase component (hereinafter referred to as PTF) of the image, each point of the image of the subject acquired via the imaging optical system is coma. It becomes an asymmetrically blurred image (degraded image) like aberration. Furthermore, since the PSF is different for each color component (for example, red, blue, green, etc.) included in the image, different blur occurs for each color component, resulting in an image (degraded image) in which the color is blurred.

  The outline of the image restoration process is shown below.

The point spread function (PSF) of the imaging system used to acquire the degraded image g (x, y), the original image f (x, y), and g (x, y) is represented by h (x, y). When y), the following equation holds. Here, * indicates convolution (convolution integration, sum of products), and (x, y) indicates image coordinates in real space.
g (x, y) = h (x, y) * f (x, y) (Formula 1)
When Formula 1 is Fourier-transformed and converted into a display format in the frequency space, it can be expressed as Formula 2.
G (u, v) = H (u, v) .F (u, v) (Formula 2)
Here, H (u, v) is an optical transfer function (OTF) obtained by Fourier transform of the point spread function h (x, y). G (u, v) and F (u, v) are obtained by Fourier transform of g (x, y) and f (x, y), respectively. (U, v) indicates a frequency (coordinate) in a two-dimensional frequency space.

In order to obtain the original image (original image) from the deteriorated image, both sides of Equation 2 may be divided by H (u, v).
G (u, v) / H (u, v) = F (u, v) (Formula 3)
This F (u, v), that is, G (u, v) / H (u, v) is inverse Fourier transformed and returned to the real space, whereby the original image f (x, y) can be obtained as a restored image. .

When both sides of Equation 3 are inverse Fourier transformed, Equation 3 is expressed by Equation 4.
g (x, y) * R (x, y) = f (x, y) (Formula 4)
Here, the inverse Fourier transform of 1 / H (u, v) is represented as R (x, y). This R (x, y) is an image restoration filter.

  Since this image restoration filter is based on the optical transfer function (OTF), it is possible to correct the deterioration of the amplitude component and the phase component.

  Next, the configuration of the image pickup apparatus according to the embodiment of the present invention will be described with reference to FIG.

  In FIG. 1, reference numeral 100 denotes a digital camera (imaging device). 400 is a photographing lens (imaging optical system), 410 is a diaphragm of the photographing lens, 110 is an imaging device that converts an optical image into an electrical signal, and 120 is an A / D converter that converts an analog signal output of the imaging device 110 into a digital signal. It is. An image processing unit 130 performs predetermined demosaic processing and color conversion processing on the data from the A / D converter 120 or the data recorded in the RAM 190.

  The image pickup apparatus of the present invention guides light from a subject to an image pickup device such as a charge coupled device (CCD) or a CMOS sensor by an image pickup optical system including a photographic lens and the like, and an electric signal corresponding to the subject image. Is getting. An imaging unit is configured by the imaging optical system and the imaging device. The electrical signal is converted from analog to digital (AD) and demosaiced to obtain image data of the subject.

  Reference numeral 140 denotes a signal value of a pixel to be repaired by using a signal value of a peripheral pixel with respect to a signal value of a missing pixel or a pixel having low reliability in the imaging device 110, or noise removal or contouring of image data. It is a camera signal processing unit that performs signal processing such as enhancement.

  Data from the A / D converter 120 is written into the RAM 190 via the image processing unit 130 and the camera signal processing unit 140, or data from the A / D converter 120 is directly written through the camera signal processing unit 140.

  Reference numeral 220 denotes an image display unit composed of a TFT LCD or the like, which displays display image data written in the RAM 190.

  A live view function can be realized by sequentially displaying image data captured using the image display unit 220.

  The RAM 190 is a memory for storing captured still images and moving images, and has a sufficient storage capacity for storing a predetermined number of still images and moving images for a predetermined time.

  This makes it possible to write a large amount of images to the RAM 190 at high speed even in continuous shooting where a plurality of still images are continuously shot.

  The RAM 190 can also be used as a work area for the camera signal processing unit 140.

  Reference numeral 170 denotes a control unit composed of, for example, a CPU, which reads the operation program for each block included in the digital camera 100 from the ROM 180, develops it in the RAM 190, and executes it to control the operation of each block included in the digital camera 100. The ROM 180 is a rewritable nonvolatile memory, and stores parameters necessary for the operation of each block in addition to the operation program for each block provided in the digital camera 100.

  Reference numeral 210 denotes an operation member such as a shutter switch. During the operation of a shutter button (not shown), SW1 is turned ON, and AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (flash pre-flash) are performed via the imaging system control unit 200. ) Instruct the start of operations such as processing. Further, when the shutter button (not shown) is completely pressed, SW2 is turned ON, and a signal read from the imaging device 110 is used for development processing using arithmetic operations in the A / D converter 120, the image processing unit 130, and the camera signal processing unit 140. Instruct the start of operation. Further, after the development process, the image data is read from the RAM 190, the compression process is performed, and an instruction to start the recording process for writing the image data to the compact flash (registered trademark) card or other medium 160 as a recording medium by the media interface 150 is given. To do.

  Reference numeral 300 denotes a photometric unit (photometric means) that measures the illuminance of the subject. The light metering unit 300 divides the photographing scene into regions and measures the light, and calculates an exposure amount for photographing the subject with an appropriate exposure by performing an average or a weighted average of the photometric results for all regions. The method for calculating the appropriate exposure can be specified by the photometry mode.

  Reference numeral 310 denotes a corrected exposure calculation for calculating a corrected exposure amount for suppressing an overexposure area included in image data when an image is captured with an exposure amount (appropriate exposure amount, first exposure amount) determined by the photometric unit 300. Part.

  Hereinafter, with reference to FIG. 2, the inside of the image processing unit 130 according to the first embodiment of the present invention is schematically shown.

  FIG. 2 is a block diagram showing a configuration example of the image processing apparatus according to the first implementation form of the present invention processed in the image processing unit 130. The image processing apparatus according to the present embodiment includes an input unit 131, a recovery filter generation processing unit 132, a saturated region detection processing unit 133, a saturated region synthesis processing unit 134, an image recovery processing unit 135, and an output unit 136.

  FIG. 3 is a flowchart showing an image processing method according to the first embodiment of the present invention. In FIG. 3, first, in step S <b> 301, the input unit 131 captures imaging data (image data) obtained by converting a signal of a subject image captured by the imaging device 110 into a digital signal via the A / D converter 120 and imaging time information (imaging conditions). ). Note that the shooting time information includes information such as lens identification information for identifying a lens attached to the imaging apparatus when the image was shot, shooting focal length information for specifying a zoom position, aperture value, and the like. is there. In S301, the input unit (acquisition unit) sets the first image data captured by the imaging unit under the set condition (first exposure amount) and the shooting condition (set) when the first image data is captured. Condition).

  In S303, OTF data is read from the imaging conditions input in S301 as preparation for generating an image restoration filter for restoring image degradation caused by the characteristics of the imaging optical system for the image data input in S301. The OTF data is data for generating an image restoration filter for restoring an image degradation component caused by the optical system based on an optical transfer function (OTF) obtained by Fourier transforming the point spread function (PSF). .

  In S305, the recovery filter generation processing unit 132 generates an image recovery filter based on the OTF data read in S303. That is, the recovery filter generation processing unit (generation unit) generates an image recovery filter from the OTF data based on the imaging condition acquired in S301. The image restoration filter is a filter that is applied to the target pixel and surrounding pixels, as in a general filter process such as a conventional sharpness or median filter. The image restoration filter has a two-dimensional data array having a width and a height for each target pixel, and obtains each value of the filter for each pixel in a region including the target pixel. The filter has different parameters for each region of the image, and the size varies depending on the case. Therefore, the size of the filter also varies depending on the target pixel.

  In step S <b> 307, first image data captured under the conditions set in the imaging apparatus 100 is read from the RAM 190. The object of the present embodiment is to effectively restore the entire image data while suppressing the influence of the saturated region included in the first image data.

  In S309, the saturated region detection processing unit (detection unit) 133 detects a region (saturated region) that satisfies a predetermined condition included in the first image data and its affected range based on the shape of the image restoration filter generated in S305. Then, it is recorded in the RAM 190 as distribution information. Details of the saturation region detection processing will be described later.

  In step S311, the corrected exposure calculation unit 310 reads the second image data captured successively after the first image data with the corrected exposure amount calculated to suppress overexposure included in the first image data. The exposure amount (second exposure amount) when obtaining the second image data is calculated by the corrected exposure calculation unit 310. As a second exposure amount calculation method in the corrected exposure calculation unit 310, a method of determining the corrected exposure with a preset correction amount is conceivable. For example, when the exposure amount (first exposure amount) at the time of obtaining the first image data is corrected by −1 step, the signal value of the pixel obtained as the RAW data is a half value at the time of proper exposure. . That is, the second exposure amount at the time of obtaining the second image data is an exposure amount obtained by reducing the first exposure amount at the time of obtaining the first image data by a predetermined magnification (for example, 1/2). To do. In other words, it is possible to record up to twice the luminance range as compared with normal shooting, and it is possible to effectively suppress overexposure. Note that, unlike the first exposure amount, the second exposure amount is an exposure amount that is smaller than the first exposure amount. Further, as a method for calculating the second exposure amount in the corrected exposure calculation unit 310, a method of dynamically determining the correction amount using the measurement result of the photometry unit 300 is conceivable. A general photometric sensor of a digital camera is divided into a plurality of regions, and it is possible to obtain measurement results in individual regions. Of these, the difference exposure between the exposure amount calculated from the result measured as the brightest part of the subject and the exposure amount when obtaining the first image data may be used as the correction amount. That is, the second exposure amount at the time of obtaining the second image data is an exposure amount calculated based on the first exposure amount at the time of obtaining the first image data and the photometric result of the photometry unit 300. In order to restore the image correctly, it is important to suppress overexposure, and in order to suppress overexposure, it is possible to always shoot with a lower exposure. There is a problem that noise increase due to adjustment is conspicuous.

  In S313, based on the distribution information of the saturation region included in the first image data generated by the saturation region synthesis processing unit 134 in S309, the signal value of the pixel in the saturation region included in the first image data is overexposed. It is replaced with the signal value of the pixel of the second image data that has been suppressed and synthesized. That is, the saturation region composition processing unit (combining means) is a value based on the signal value of the second image data obtained by capturing the detected saturation region with the second exposure amount smaller than the first exposure amount by the imaging device 110. The composite image data is generated so as to be replaced by. Then, composite image data in which overexposure of the first image data is suppressed is output. Details of the saturation region synthesis processing will be described later.

  In S315, the image recovery processing unit 135 performs image recovery processing for recovering the deterioration caused by the photographing optical system on the composite image data output in S313. In other words, the image restoration processing unit (processing unit) performs image restoration processing on the composite image data using the image restoration filter generated in S305. Then, the output unit 136 outputs the result to the camera signal processing unit 140.

  Next, the flow of the saturation region detection process performed in S309 will be described using FIG. In FIG. 4, first, a memory area for holding distribution information, which is a result of detecting saturated pixels included in the first image data to be subjected to image restoration processing and its affected range in S401, is secured in the RAM 190.

  In S403, the memory area secured in S401 is initialized with 0 as a preparation for recording distribution information.

  In step S <b> 405, first image data that is a target for detecting a saturated region is read from the RAM 190.

  In step S407, control variables for scanning all the pixels of the first image data are initialized. All pixel values are substituted for the variable M, and 0 is substituted for the loop variable m.

  S409 to S415 are a loop process for detecting a saturation region.

  In step S409, the mth pixel of the image data is acquired.

  In S411, it is determined whether the signal value of the pixel acquired in S409 is saturated. Whether or not it is saturated is determined by whether or not the signal value has reached a predetermined value. Specifically, the saturation is determined as a saturated pixel when the same value as the maximum quantization bit precision of the image data is recorded, and 1 is set to the corresponding pixel of the distribution information. If the target pixel is determined as a saturated pixel in S411, the processing from S413 to S415 is performed to update the distribution information of the saturated region.

  In S413, an image restoration filter corresponding to the image height of the mth pixel is read. The image restoration filter is generated from OTF data based on the PSF. Since the PSF generally has a property that the amount of point spread increases as the image height increases, the shape of the image restoration filter based on the PSF is also different for each image height. Different.

  In S415, the distribution of the influence range in which a failure such as aliasing or coloring occurs in the recovery result due to the mth pixel determined to be a saturated pixel is based on the tap size of the image recovery filter read in S413. Can be detected. Therefore, a saturated area is recorded by setting 1 for the pixels included in this range. As described above, the saturation region of this embodiment is detected based on the signal value of the pixel of the first image data and the characteristics (shape) of the image restoration filter read in S413. Specifically, in S411, it is determined whether or not the signal value of the pixel of the first image data is saturated. In S415, the range corresponding to the area of the image restoration filter centering on the saturated pixel is saturated. Detect as a region. Here, assuming that the mth pixel is a saturated pixel, 1 is set to the pixels corresponding to the area of the image restoration filter with the mth pixel as the center, and this is set as the saturated region.

  If it is determined in S411 that the target pixel is not a saturated pixel, the loop variable m and the variable M representing the total number of pixels are compared in S417. From the comparison result of S417, when the process is performed for all the pixels, the process ends. If there are remaining pixels, the loop variable is incremented by 1 in S419, and the processing from S409 to S415 is repeated until the entire image is processed.

  Next, the flow of synthesis processing performed by the saturated region synthesis processing unit 134 will be described with reference to FIG.

  In S501, the distribution information of the saturated region included in the first image data created by the saturated region detection processing unit 133 is read.

  In S503 and S505, the first image data and the second image data photographed with the overexposure suppressed by the corrected exposure amount are read.

  As preparatory processing for composition processing, it is necessary to perform gain processing for the second image data by the difference in exposure amount from the first image data.

  In S507, the corrected exposure amount calculated by the corrected exposure calculation unit 310 is acquired.

  In step S509, the gain processing for the corrected exposure is applied to the second image data. That is, gain processing is performed on the second image data based on the difference between the first exposure amount and the second exposure amount. At this time, a pixel having a signal value exceeding the maximum value of quantization bit accuracy is generated in the gain-processed second image data, but is not clipped here. Since the signal value exceeding the maximum value of the quantization bit accuracy can be regarded as the original signal value of the saturated pixel, the signal value of these pixels is used to perform overexposed gradation information in the first image data. The signal value of the pixel in the portion where no is recorded is supplemented by subsequent synthesis processing.

  In step S511, a control variable for scanning all the pixels of the first image data is initialized. All pixel values are substituted for the variable M, and 0 is substituted for the loop variable m.

  S513 to S517 are a loop process for detecting the saturation region.

  In S513, the m-th pixel of the distribution information of the saturation region is acquired.

  In S515, if the value acquired in S513 is 1, which means a saturation region, the corresponding pixel of the first image data is replaced with the signal value of the pixel of the second image data in S517.

  In S515, if the value acquired in S513 is 0, which means that it is not the saturation region, the loop variable m is compared with the variable M representing the total number of pixels in S519. From the comparison result of S519, when the process is performed for all the pixels, the process ends. If there are remaining pixels, the loop variable is incremented by 1 in S521, and the processing of S513 to S517 is repeated until the entire image is processed, and the resultant synthesized image data is output to the image restoration processing unit 135.

  In the image restoration processing unit 135, the image restoration filter generated by the restoration filter generation processing unit 132 is applied to the composite image data output from the saturated region synthesis processing unit 134, and image quality degradation caused by the optical system is restored. . The restoration process performed here is a general restoration process using an image restoration filter. However, in this embodiment, gain processing is performed on images captured with different exposures in saturated pixels in which the image data is overexposure and the true pixel signal value cannot be obtained in the previous stage of the restoration processing and in the affected area. The pixel signal value is replaced. Since the replaced portion exceeds the maximum output bit level, it is clipped at the maximum output level after the image restoration processing.

  As described above, according to the present embodiment as described above, since the problem caused by applying the image restoration filter to the overexposed pixel and its peripheral region occurs, the effect is suppressed and the deterioration remains. Degradation can also be restored for the region. That is, it is possible to reduce image quality deterioration caused by the imaging optical system without suppressing the restoration effect for the overexposed region and the surrounding region.

  Although the present invention has been described in detail based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the present invention are also included in the present invention. included.

  The present invention is also applicable to a case where a software program that realizes the functions of the above-described embodiments is supplied directly from a recording medium to a system or apparatus having a computer that can execute the program using wired / wireless communication, and the program is executed. Included in the invention.

  Accordingly, the program code itself supplied and installed in the computer in order to implement the functional processing of the present invention by the computer also realizes the present invention. That is, the present invention includes a computer program itself in which a procedure for realizing the functional processing of the present invention is described.

  In this case, the program may be in any form as long as it has a program function, such as an object code, a program executed by an interpreter, or script data supplied to the OS. As a recording medium for supplying the program, for example, a magnetic recording medium such as a hard disk or a magnetic tape, an optical / magneto-optical storage medium, or a nonvolatile semiconductor memory may be used.

  As a program supply method, a computer program that forms the present invention is stored in a server on a computer network, and a connected client computer downloads and programs the computer program.

  The present invention can be suitably used for an imaging apparatus such as a compact digital camera, a single-lens reflex camera, and a video camera.

132 Recovery filter generation processing unit 133 Saturation region detection processing unit 134 Saturation region synthesis processing unit 135 Image recovery processing unit

Claims (14)

Generating means for generating an image restoration filter based on imaging conditions of an imaging means for outputting first image data imaged with a first exposure amount;
Detecting means for detecting a region satisfying a predetermined condition of the first image data;
The signal value of the pixel in the region detected by the detection unit is replaced with a value based on the signal value of the second image data captured by the imaging unit with a second exposure amount smaller than the first exposure amount. A compositing means for generating composite image data,
Processing means for performing image restoration processing on the composite image data by the image restoration filter;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein the second exposure amount is an exposure amount calculated based on the first exposure amount and a predetermined magnification. A photometric means for measuring the illuminance of the subject,
The image processing apparatus according to claim 1, wherein the second exposure amount is an exposure amount calculated based on the first exposure amount and a photometric result of the photometric unit.   4. The image processing according to claim 1, wherein the detection unit detects the region based on a shape of the image restoration filter and a signal value of the first image data. 5. apparatus.   The detection means determines whether or not the signal value of the first image data is saturated, and a range corresponding to the area of the image restoration filter centering on a pixel of the saturated signal value, The image processing apparatus according to claim 4, wherein the image processing apparatus is detected as a region.   The image processing apparatus according to claim 1, wherein the detection unit includes a pixel having a saturated signal value in the first image data in the region.   The detection means detects the region based on a pixel of the saturated signal value in the first image data and a shape of the image restoration filter corresponding to the position of the pixel. The image processing apparatus according to claim 6.   The image processing according to any one of claims 5 to 7, wherein the detection unit determines that a pixel having a signal value reaching a predetermined value is a pixel having the saturated signal value. apparatus.   9. The method according to claim 1, wherein the synthesizing unit performs a gain process on the second image data based on a difference between the first exposure amount and the second exposure amount. The image processing apparatus according to item 1.   10. The image processing apparatus according to claim 1, further comprising an acquisition unit configured to acquire image data captured by the imaging unit and a shooting condition when the image data is captured. Imaging means;
An image processing apparatus comprising: the image processing apparatus according to claim 1. A generation step of generating an image restoration filter based on the imaging condition of the imaging means for outputting the first image data captured with the first exposure amount;
A detection step of detecting a region satisfying a predetermined condition of the first image data;
The signal value of the pixel in the region detected by the detection step is replaced with a value based on the signal value of the second image data captured by the imaging unit with the second exposure amount smaller than the first exposure amount. A compositing step for generating composite image data,
A processing step of applying an image restoration process to the composite image data by the image restoration filter;
An image processing method comprising:   A computer-executable program in which the procedure of the image processing method according to claim 12 is described.   A computer-readable storage medium storing a program for causing a computer to execute each step of the image processing method according to claim 12.