JP2019092215A - Image processing apparatus, imaging apparatus, image processing method, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device for improving the quality of a photographed image. An image processing device (image processing circuit 125) is set by setting means 125a for setting weighting coefficients corresponding to a plurality of regions including a first region of each of a plurality of parallax images, and the setting means. It has an image generation means 125b that synthesizes the plurality of parallax images based on a weighting coefficient to generate a composite image, and the setting means has the plurality of parallax images according to the effective aperture values of the plurality of parallax images. A weighting coefficient corresponding to the first region in each parallax image is set. [Selection diagram] Fig. 1

Description

  The present invention relates to an image processing apparatus that combines a plurality of parallax images and outputs an image.

  2. Description of the Related Art Conventionally, there has been known an imaging apparatus capable of dividing an exit pupil of a photographing lens into a plurality of regions and simultaneously photographing a plurality of parallax images according to the divided pupil regions.

  Patent Document 1 discloses an imaging device using a two-dimensional imaging element in which one microlens and a photoelectric conversion unit divided into a plurality of parts are formed for one pixel. The divided photoelectric conversion unit is configured to receive different partial pupil regions of the exit pupil of the photographing lens via one microlens, and performs pupil division. A plurality of parallax images according to the divided pupil partial regions can be generated based on the light reception signals of the divided photoelectric conversion units. Patent Document 2 discloses an imaging device that generates a captured image by adding all light reception signals of each of the divided photoelectric conversion units.

U.S. Pat. No. 4,410,804 Japanese Patent Application Publication No. 2001-083407

  By the way, in the imaging apparatus as disclosed in Patent Documents 1 and 2, while focusing on the main subject, the foreground and the background are largely blurred, so that photographing which effectively highlights the main subject is performed. is there. However, depending on the shooting scene, when the foreground (petals) located in front of the main subject (for example, a bird) is largely blurred, a blur may occur before the main subject is hidden and the quality of the shot image may be degraded. is there.

  Therefore, the present invention provides an image processing device, an imaging device, an image processing method, a program, and a storage medium that improve the quality of a captured image.

  An image processing apparatus according to one aspect of the present invention is based on setting means for setting weighting factors corresponding to a plurality of areas including the first area of each of a plurality of parallax images, and the weighting factors set by the setting means. Image generation means for combining the plurality of parallax images to generate a composite image, wherein the setting means is configured to set each of the plurality of parallax images according to an effective aperture value of the plurality of parallax images. A weighting factor corresponding to the first area is set.

  According to another aspect of the present invention, an imaging device includes: an imaging element in which a plurality of pixels formed of a plurality of photoelectric conversion units that receive light beams passing through different pupil partial regions of an imaging optical system; Setting means for setting weight coefficients corresponding to a plurality of areas including the first area of each of the plurality of parallax images obtained from the photoelectric conversion unit; and the plurality of parallaxes based on the weight coefficients set by the setting means An image generation unit that combines the images to generate a composite image, and the setting unit corresponds to the first region in each of the plurality of parallax images according to the effective aperture value of the plurality of parallax images Set the weighting factor.

  In an image processing method as another aspect of the present invention, a weighting factor corresponding to a plurality of areas including a first area of each of a plurality of parallax images is set, and a weighting factor set by the setting is set. And combining the plurality of parallax images to generate a combined image, wherein the setting step includes the step of setting each of the plurality of parallax images according to an effective aperture value of the plurality of parallax images. A weighting factor corresponding to the first area is set.

  A program according to another aspect of the present invention is configured to cause a computer to execute the image processing method.

  A storage medium as another aspect of the present invention stores the program.

  Other objects and features of the present invention are described in the following examples.

  According to the present invention, it is possible to provide an image processing device, an imaging device, an image processing method, a program, and a storage medium which improve the quality of a captured image.

It is a block diagram of an imaging device in each example. FIG. 2 is a diagram showing a pixel arrangement in Embodiment 1. FIG. 2 is a diagram showing a pixel structure in Embodiment 1. It is explanatory drawing of the image pick-up element in each Example, and a pupil division function. It is explanatory drawing of the image pick-up element in each Example, and a pupil division function. It is a related figure with the defocusing amount and the image shift amount in each Example. It is an example of the front blur covering image to the main subject. It is an explanatory view of image processing (blur adjustment processing) in each example. It is an explanatory view of an effect of image processing (blur adjustment processing) in each example. It is explanatory drawing of the effective aperture value by the pupil shift in each Example. FIG. 6 is a diagram showing a pixel arrangement in Embodiment 2. FIG. 7 is a diagram showing a pixel structure in Example 2; It is a schematic explanatory drawing of the refocusing process in each Example.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  First, with reference to FIG. 1, a schematic configuration of an imaging apparatus according to a first embodiment of the present invention will be described. FIG. 1 is a block diagram of an imaging apparatus 100 (camera) in the present embodiment. The imaging device 100 is a digital camera system provided with a camera body and an interchangeable lens (imaging optical system or imaging optical system) that is detachable from the camera body. However, the present embodiment is not limited to this, and is also applicable to an imaging device in which a camera body and a lens are integrally configured.

  The first lens group 101 is disposed at the front (object side) of the plurality of lens groups constituting the photographing lens (imaging optical system), and can advance and retract in the direction of the optical axis OA (optical axis direction) It is held by the lens barrel in the state. The diaphragm / shutter 102 (diaphragm) adjusts the light amount at the time of shooting by adjusting the aperture diameter, and functions as an exposure time adjustment shutter at the time of still image shooting. The second lens group 103 advances and retracts integrally with the aperture-shutter 102 in the optical axis direction, and has a zoom function of performing a zooming operation in conjunction with the advancing and retracting operation of the first lens group 101. The third lens group 105 is a focusing lens group that performs focusing (focusing operation) by advancing and retracting in the optical axis direction. The optical low pass filter 106 is an optical element for reducing false color and moiré of a captured image.

  The imaging element 107 performs photoelectric conversion of an object image (optical image) through an imaging optical system, and is configured of, for example, a CMOS sensor or a CCD sensor, and a peripheral circuit thereof. As the image sensor 107, for example, a two-dimensional single-plate color sensor in which a primary color mosaic filter of Bayer arrangement is formed on a chip on a light receiving pixel having m pixels in the horizontal direction and n pixels in the vertical direction is used. Be

  The zoom actuator 111 performs a variable power operation by moving the first lens group 101 and the second lens group 103 along the optical axis direction by rotating (driving) a cam barrel (not shown). The aperture shutter actuator 112 controls the aperture diameter of the aperture / shutter 102 to adjust the light amount (shooting light amount), and controls the exposure time at the time of still image shooting. The focus actuator 114 moves the third lens group 105 in the optical axis direction to perform focusing.

  The electronic flash 115 is a lighting device used to light an object. As the electronic flash 115, a flash illumination device provided with a xenon tube or an illumination device provided with a continuously emitting LED (Light Emitting Diode) is used. The AF auxiliary light means 116 projects an image of a mask having a predetermined aperture pattern onto a subject via a light projection lens. Thereby, it is possible to improve the focus detection capability for a dark subject or a low contrast subject.

  The CPU 121 is a control device (control means) that controls various controls of the imaging device 100. The CPU 121 includes an arithmetic unit, a ROM, a RAM, an A / D converter, a D / A converter, a communication interface circuit, and the like. The CPU 121 reads out and executes a predetermined program stored in the ROM to drive various circuits of the imaging apparatus 100 to control a series of operations such as focus detection (AF), photographing, image processing, or recording. Do.

  The electronic flash control circuit 122 performs lighting control of the electronic flash 115 in synchronization with the photographing operation. The auxiliary light drive circuit 123 performs lighting control of the AF auxiliary light means 116 in synchronization with the focus detection operation. The imaging device drive circuit 124 controls the imaging operation of the imaging device 107, A / D converts the acquired image signal, and transmits it to the CPU 121.

  The image processing circuit 125 (image processing apparatus) performs processing such as γ (gamma) conversion, color interpolation, or JPEG (Joint Photographic Experts Group) compression of image data output from the image sensor 107. In the present embodiment, the image processing circuit 125 includes a determination unit 125a and an image generation unit 125b. The determination unit 125a determines a weighting factor that changes according to the position of each of the plurality of parallax images. The image generation unit 125 b combines a plurality of parallax images based on the weight coefficient to generate an image.

  The focus drive circuit 126 (focus drive means) drives the focus actuator 114 based on the focus detection result, and moves the third lens group 105 along the optical axis direction to perform focus adjustment. The diaphragm shutter drive circuit 128 drives the diaphragm shutter actuator 112 to control the aperture diameter of the diaphragm shutter 102. The zoom drive circuit 129 (zoom drive means) drives the zoom actuator 111 according to the zoom operation of the photographer.

  The display 131 (display means) includes, for example, an LCD (Liquid Crystal Display). The display 131 displays information on the shooting mode of the imaging apparatus 100, a preview image before shooting, a confirmation image after shooting, an in-focus state display image at the time of focus detection, and the like. The operation unit 132 (operation switch group) includes a power switch, a release (shooting trigger) switch, a zoom operation switch, a shooting mode selection switch, and the like. The release switch has a two-step switch of a half-pressed state (SW1 is on) and a full-press state (SW2 is on). The recording medium 133 is, for example, a flash memory that is removable from the imaging apparatus 100, and records a photographed image (image data).

  Subsequently, the pixel array and the pixel structure of the image sensor 107 in the present embodiment will be described with reference to FIGS. 2 and 3. FIG. 2 is a diagram showing a pixel array of the imaging element 107. As shown in FIG. FIG. 3 is a view showing a pixel structure of the image sensor 107, and FIG. 3 (a) is a plan view of the pixel 200G of the image sensor 107 (view from the + z direction), and FIG. ) Is a cross-sectional view (a view from the −y direction) of line aa in FIG.

  FIG. 2 illustrates a pixel array (array of imaging pixels) of the image sensor 107 (two-dimensional CMOS sensor) in a range of 4 columns × 4 rows. In the present embodiment, each imaging pixel (pixels 200R, 200G, and 200B) is configured of two sub-pixels 201 and 202. For this reason, in FIG. 2, the arrangement of the sub-pixels is shown in the range of 8 columns × 4 rows.

  As shown in FIG. 2, in the pixel group 200 of 2 columns × 2 rows, pixels 200R, 200G, and 200B are arranged in a Bayer arrangement. That is, in the pixel group 200, the pixel 200R having a spectral sensitivity of R (red) is at the upper left, the pixel 200G having a spectral sensitivity of G (green) is at the upper right and lower left, and the pixel 200B having a spectral sensitivity of B (blue). Are respectively located at the lower right. Each pixel 200R, 200G, and 200B (each imaging pixel) is comprised by the sub pixel 201, 202 arranged in 2 row x 1 row. The sub-pixel 201 is a pixel that receives the light flux that has passed through the first pupil region of the imaging optical system. The sub-pixel 202 is a pixel that receives the light flux that has passed through the second pupil region of the imaging optical system.

As shown in FIG. 2, the imaging element 107 is configured by arranging a large number of 4 × 4 imaging pixels (8 columns × 4 rows of sub-pixels) on the surface, and an imaging signal (sub-pixel signal) Output). The imaging element 107 of the present embodiment has a cycle P of pixels (imaging pixels) of 4 μm, and the number N of pixels (imaging pixels) is 5575 horizontal rows × 3725 vertical rows = about 207.5 million pixels. Further, in the imaging element 107, the period P _SUB of the sub-pixel in the column direction is 2 μm, and the number N _{SUB of} sub-pixels is 11150 columns wide × 3725 rows high = approximately 41.5 million pixels.

As shown in FIG. 3B, the pixel 200G of this embodiment is provided with a micro lens 305 for condensing incident light on the light receiving surface side of the pixel. A plurality of microlenses 305 are two-dimensionally arrayed, and are arranged at a position separated from the light receiving surface by a predetermined distance in the z-axis direction (the direction of the optical axis OA). Further, in the pixel 200G, a photoelectric conversion portion 301 and a photoelectric conversion portion 302 which are divided into N _H in the x direction (two divisions) and divided into N _{V in} the y direction (one division) are formed. The photoelectric conversion unit 301 and the photoelectric conversion unit 302 correspond to the sub pixel 201 and the sub pixel 202, respectively.

  Each of the photoelectric conversion unit 301 and the photoelectric conversion unit 302 is configured as a photodiode having a pin structure in which an intrinsic layer is sandwiched between a p-type layer and an n-type layer. If necessary, the intrinsic layer may be omitted and configured as a photodiode of a pn junction. In the pixel 200 </ b> G (each pixel), a color filter 306 is provided between the microlens 305 and the photoelectric conversion unit 301 and the photoelectric conversion unit 302. If necessary, the spectral transmittance of the color filter 306 can be changed for each sub-pixel, or the color filter may be omitted.

  As shown in FIG. 3, light incident on the pixel 200 G is collected by the micro lens 305, separated by the color filter 306, and then received by the photoelectric conversion unit 301 and the photoelectric conversion unit 302. In the photoelectric conversion unit 301 and the photoelectric conversion unit 302, pairs of electrons and holes are generated according to the amount of light received, and after they are separated by the depletion layer, the electrons of negative charge are stored in the n-type layer. On the other hand, holes are discharged to the outside of the imaging element 107 through a p-type layer connected to a constant voltage source (not shown). The electrons accumulated in the n-type layers of the photoelectric conversion unit 301 and the photoelectric conversion unit 302 are transferred to the electrostatic capacitance unit (FD) through the transfer gate and converted into a voltage signal.

  Subsequently, the pupil division function of the image sensor 107 will be described with reference to FIG. FIG. 4 is an explanatory diagram of a pupil division function of the image sensor 107, and shows a state of pupil division in one pixel unit. FIG. 4 is a cross-sectional view of the aa cross section of the pixel structure shown in FIG. 3A when viewed from the + y side, and shows an exit pupil plane of the imaging optical system. In FIG. 4, in order to correspond to the coordinate axis of the exit pupil plane, the x-axis and y-axis of the cross-sectional view are respectively inverted with respect to the x-axis and y-axis of FIG.

  In FIG. 4, the pupil partial area 501 (first pupil partial area) of the sub-pixel 201 (first sub-pixel) has the light receiving surface of the photoelectric conversion unit 301 whose center of gravity is decentered in the −x direction, and the micro lens 305. Therefore, they are in a substantially conjugate relationship. Therefore, the pupil partial region 501 represents a pupil region that can be received by the sub-pixel 201. The center of gravity of the pupil partial region 501 of the sub-pixel 201 is decentered to the + x side on the pupil plane. In addition, the pupil partial region 502 (second pupil partial region) of the sub-pixel 202 (second sub-pixel) is substantially conjugate with the light receiving surface of the photoelectric conversion unit 302 whose center of gravity is decentered in the + x direction It is in a relationship. Therefore, the pupil partial region 502 represents a pupil region that can be received by the sub-pixel 202. The center of gravity of the pupil partial region 502 of the sub-pixel 202 is decentered to the −x side on the pupil plane. The pupil area 500 is a pupil area that can be received by the entire pixel 200G when all the photoelectric conversion units 301 and 302 (sub-pixels 201 and 202) are combined.

  Incident light is collected by the micro lens 305 at a focal position. However, due to the influence of diffraction due to the wave nature of light, the diameter of the focused spot can not be made smaller than the diffraction limit Δ, and has a finite size. While the light receiving surface size of the photoelectric conversion units 301 and 302 is about 1 to 2 μm, the focusing spot of the microlens 305 is about 1 μm. For this reason, the pupil partial regions 501 and 502 in FIG. 4 that are in a conjugate relationship with the light receiving surfaces of the photoelectric conversion units 301 and 302 via the micro lens 305 are not clearly divided into pupils because of diffraction blurring, and the light reception ratio A distribution (pupil intensity distribution) is obtained.

  FIG. 5 is an explanatory diagram of the image sensor 107 and the pupil division function. The light beams having passed through different pupil partial regions 501 and 502 in the pupil region of the imaging optical system are incident on the imaging surface 600 of the imaging device 107 at different angles to each pixel of the imaging device 107 and divided into 2 × 1. Light is received by the subpixels 201 and 202. In the present embodiment, an example in which the pupil region is divided into two in the horizontal direction is described, but the present invention is not limited thereto, and pupil division may be performed in the vertical direction as necessary. .

  In this embodiment, a plurality of imaging elements 107 share one microlens and pass different areas (first and second pupil partial areas) of the pupil of the imaging optical system (shooting lens) And a plurality of sub-pixels for receiving the luminous flux of The imaging device 107 includes, as a plurality of sub-pixels, a first sub-pixel (a plurality of sub-pixels 201) and a second sub-pixel (a plurality of sub-pixels 202). In the present embodiment, the light reception signals of the plurality of sub-pixels 201 are collected to generate a first parallax image, and the light reception signals of the plurality of sub-pixels 202 are collected to generate a second parallax image. As described above, in the present embodiment, light reception signals of a plurality of sub-pixels are collected for each different pupil partial area to generate respective parallax images.

  In the present embodiment, each of the first parallax image and the second parallax image is an image of a Bayer array. Demosaicing processing may be performed on each of the first parallax image and the second parallax image as needed. Further, in the present embodiment, by adding and reading the signals of the sub-pixels 201 and 202 for each pixel of the image sensor 107, a captured image having a resolution of N effective pixels can be generated. As described above, in the present embodiment, a captured image is generated from a plurality of parallax images (a first parallax image and a second parallax image).

  Next, with reference to FIG. 6, the relationship between the defocus amount and the image shift amount of the first parallax image acquired from the sub pixel 201 of the image sensor 107 and the second parallax image acquired from the sub pixel 202 will be described. Do. FIG. 6 is a diagram showing the relationship between the defocus amount and the image shift amount. In FIG. 6, the imaging element 107 is disposed on the imaging surface 600, and as in FIGS. 4 and 5, the exit pupil of the imaging optical system is divided into two pupil partial regions 501 and 502. ing.

  The defocus amount d is the distance from the imaging position of the object to the imaging surface 600 as | d |, and the front pin state where the imaging position is closer to the object than the imaging surface 600 is negative sign (d <0), The back pin state whose position is on the opposite side of the object with respect to the imaging plane 600 is defined as a positive code (d> 0). In the in-focus state where the imaging position of the subject is on the imaging surface 600 (in-focus position), the defocus amount d = 0 is established. In FIG. 6, the subject 601 in the in-focus state (d = 0) and the subject 602 in the front pin state (d <0) are respectively shown. The front pin state (d <0) and the rear pin state (d> 0) are collectively referred to as a defocus state (| d |> 0).

  In the front pinning state (d <0), among the light flux from the subject 602, the light flux that has passed through the pupil partial region 501 (or the pupil partial region 502) is once condensed. Thereafter, the light beam spreads to a width Γ1 (Γ2) centered on the gravity center position G1 (G2) of the light beam, and becomes an image blurred on the imaging surface 600. The blurred image is received by the sub-pixels 201 (sub-pixels 202) constituting each pixel arranged in the image sensor 107, and a first parallax image (second parallax image) is generated. Therefore, the first parallax image (second parallax image) is recorded at the gravity center position G1 (G2) on the imaging surface 600 as an object image in which the object 602 is blurred to a width Γ1 (Γ2). The blur width Γ1 (Γ2) of the subject image generally increases in proportion to the increase of the magnitude | d | of the defocus amount d. Similarly, the size | p | of the image shift amount p of the subject image between the first parallax image and the second parallax image (= difference G1−G2 of the barycentric position of the light flux) is also the size of the defocus amount d. As the | d | increases, they generally increase proportionately. The same applies to the back-pin state (d> 0), but the direction of the image deviation of the object image between the first parallax image and the second parallax image is opposite to that in the front-pin state.

  As described above, in the present embodiment, as the magnitude of the defocus amount of the imaging signal obtained by adding the first parallax image and the second parallax image or the first parallax image and the second parallax image increases, The magnitude of the image shift amount between the one parallax image and the second parallax image increases.

  Next, image processing of blur adjustment (pre-blurring reduction on a main subject) in the present embodiment will be described. The image processing of the blur adjustment (pre-blurring reduction on the main subject) is executed by the image processing circuit 125 (determination unit 125a, image generation unit 125b) based on an instruction of the CPU 121. The image processing circuit 125 inputs a plurality of parallax images (first parallax image and second parallax image) acquired by the imaging element 107, and executes the image processing of the present embodiment.

  FIG. 7 shows an example of a front blurred image on a main subject. In the area 700 of FIG. 7, the foreground (petals) as the second subject located in front of (the closest side to) the main subject (bird) as the first subject is largely blurred to cause a front blur coverage in which the main subject is hidden. .

  Here, j and i are integers, and the jth position in the row direction and the ith position in the column direction of the first parallax image (and the second parallax image) are represented as (j, i), and the pixel at position (j, i) Of the first parallax image is A (j, i), and the second parallax image is B (j, i).

  As a first step, the CPU 121 (image processing circuit 125) sets a predetermined area R = [j1, j2] × [i1, i2] and a boundary width σ of the predetermined area to reduce front blur on the main subject. Do. Then, a table function T (j, i) according to the predetermined region R and the boundary width σ of the predetermined region is calculated by the following equation (1).

  The table function T (j, i) is 1 inside the predetermined region R and 0 outside the predetermined region R, and changes continuously from approximately 1 to 0 at the boundary width σ of the predetermined region R. The predetermined region R may be circular or any other shape as needed. Also, a plurality of predetermined areas and boundary widths may be set as necessary.

As a second step, the CPU 121 (image processing circuit 125) sets the actual coefficient w (-1 ≦ w ≦ 1), and the first weighting coefficient of the first parallax image A (j, i) according to the following equation (2A) Calculate W _a (j, i). Similarly, the CPU 121 (image processing circuit 125) calculates a second weighting factor W _b (j, i) of the second parallax image B (j, i) according to the following equation (2B).

As a third step, the image processing circuit 125, first parallax image A (j, i), the second parallax image B (j, i), the first weighting coefficient _W a (j, i), and, second weight An output image I (j, i) is generated from the coefficient W _b (j, i) by the following equation (3).

As necessary, the image processing circuit 125 may generate the output image I _s (j, i) according to the following equation (4A) or equation (4B) in combination with the refocusing process based on the shift amount s.

  Here, the refocusing process will be described with reference to FIG. FIG. 13 shows the one-dimensional direction (column direction, horizontal direction) by the first signal (light reception signal of the first sub-pixel) and the second signal (light reception signal of the second sub-pixel) acquired by the image sensor 107 of this embodiment. ) Is an explanatory view of the refocusing process of FIG. In FIG. 13, i is an integer, and the first signal of the i-th pixel in the column direction of the imaging element 107 disposed on the imaging surface 600 is schematically represented as Ai and the second signal is Bi. The first signal Ai is a light reception signal of the light beam incident on the i-th pixel at the chief ray angle θa (corresponding to the pupil partial region 501 in FIG. 5). The second signal Bi is a light reception signal of the luminous flux that has entered the i-th pixel at the chief ray angle θb (corresponding to the pupil partial region 502 in FIG. 5).

  The first signal Ai and the second signal Bi include not only light intensity distribution information but also incident angle information. Therefore, the first signal Ai is moved in parallel along the angle θa to the virtual imaging surface 610, the second signal Bi is moved in parallel along the angle θb to the virtual imaging surface 610, and these are added. A refocus signal at the virtual imaging plane 610 can be generated. Parallel translation of the first signal Ai to the virtual imaging plane 610 along the angle θa corresponds to a +0.5 pixel shift in the column direction, and the second signal Bi to the virtual imaging plane 610 along the angle θb. The translating corresponds to a -0.5 pixel shift in the column direction. Therefore, the first signal Ai and the second signal Bi are relatively shifted by +1 pixel relative to each other, and the first signal Ai and the second signal (Bi + 1) are correspondingly added and added, so that A focus signal can be generated. Similarly, shift addition signals (in each virtual imaging plane according to the integer shift amount) are obtained by performing integer shift on the first signal Ai and the second signal Bi (shifting by an integer number of pixels) and adding them. Refocus signal can be generated.

  In this embodiment, a plurality of parallax images obtained by an imaging element in which a plurality of pixels provided with a plurality of sub-pixels for receiving light beams passing through different pupil partial regions of the imaging optical system are input. Each parallax image is multiplied by a weighting factor and synthesized to generate an output image. Preferably, in the present embodiment, the weighting factor for each of the plurality of parallax images changes continuously according to the area of the output image. Preferably, in the present embodiment, the output image is generated by multiplying each of the plurality of parallax images by weighting factors and adding or shifting and adding.

  Next, with reference to FIG. 8, the principle of image processing of blur adjustment for reducing front blur on a main subject will be described. FIG. 8 is an explanatory diagram of image processing for blur adjustment. In FIG. 8, the imaging element 107 of this embodiment is disposed on the imaging surface 600, and the exit pupil of the imaging optical system is divided into two pupil partial regions 501 and 502 as in FIG. 5.

  FIG. 8A is an example of a front-blurred image of the main subject, in which the image p1 (focused image) of the main subject q1 (first subject) is blurred with the subject q2 (second subject) in front. (Γ1 + Γ2) shows a state of being overlapped and photographed. FIGS. 8B and 8C show the example of FIG. 8A divided into a light beam passing through the pupil partial region 501 of the imaging optical system and a light beam passing through the pupil partial region 502, respectively. ing.

  In FIG. 8B, the light flux from the main subject q1 passes through the pupil partial area 501 to form an image p1 in an in-focus state, and the light flux from the subject q2 in front passes the pupil partial area 501. It spreads to the blurred image 1 in the defocus state, and is received by the sub-pixel 201 of each pixel of the image sensor 107. A first parallax image is generated from the light reception signal of the sub pixel 201. In the first parallax image, the image p1 of the main subject q1 and the blurred image Γ1 of the subject q2 in front are captured without overlapping each other. In a predetermined area (near the image p1 of the subject q1), among the plurality of parallax images (the first parallax image and the second parallax image), the closest object (the blurred image Γ1 of the subject q2) is photographed in the narrowest range 1 is an example of a parallax image. Further, in a predetermined area (near the image p1 of the subject q1), among the plurality of parallax images (the first parallax image and the second parallax image), the blurred image Γ1 of the subject q2 is small and the contrast evaluation value is the largest. It is an example of a parallax image.

  On the other hand, in FIG. 8C, the light flux from the main subject q1 passes through the pupil partial area 502 and forms an image on the image p1 in focus, and the light flux from the subject q2 in front passes the pupil partial area 502 It spreads to the blurred image 2 in the defocus state, and is received by the sub-pixel 202 of each pixel of the image sensor 107. A second parallax image is generated from the light reception signal of the sub pixel 202. In the second parallax image, the image p1 of the main subject q1 and the blurred image 2 of the subject q2 in front of each other are captured overlapping each other. This is an example of a parallax image in which a subject on the near side is photographed in the narrowest range among a plurality of parallax images in a predetermined area. Further, this is an example of a parallax image in which a blurred image is small and the contrast evaluation value is the largest among a plurality of parallax images in a predetermined region. In a predetermined area (near the image p1 of the subject q1), among the plurality of parallax images (first and second parallax images), the closest subject (blurred image 2 of the subject q2) is photographed in the widest range 1 is an example of a parallax image. Further, in a predetermined area (near the image p1 of the subject q1), among the plurality of parallax images (first parallax image and second parallax image), the blurred image 2 of the subject q2 is large and the contrast evaluation value is the smallest. It is an example of a parallax image.

In the present embodiment, in the predetermined area (near the image p1), the first weighting factor W _a of the first parallax image with less overlap between the image p1 and the blurred image Γ1 is larger, and the overlap between the image p1 and the blurred image Γ2 is more It is set larger than the second weight coefficient W _b of the two parallax images, and an output image is generated by equation (3). As a result, it is possible to generate an image in which the front blur on the main subject is reduced.

  Preferably, in the present embodiment, in a predetermined area of the output image, the weight coefficient of the parallax image in which the closest subject is photographed in the widest range among the plurality of parallax images is the smallest or the closest side The weighting factor of the parallax image in which the subject is photographed in the narrowest range is the largest. Also preferably, in the present embodiment, in a predetermined region of the output image, the weight coefficient of the parallax image with the smallest contrast evaluation value is the smallest among the plurality of parallax images, or the parallax image with the largest contrast evaluation value The weighting factor of is the largest.

  Next, with reference to FIG. 9, the effect of the image processing (blur adjustment processing) in the present embodiment will be described. FIG. 9 is an explanatory diagram of the effect of the image processing (blur adjustment processing). FIG. 9A shows an image before the blur adjustment processing for reducing the front blur covering of the petals on the main subject (bird) in the predetermined area 700 of FIG. 7. FIG. 9B shows an image after the blur adjustment processing. In the image of FIG. 9 (a) (image before the blur adjustment process), the bird is covered with beaks, eyes, and feathers and white of the petals in front of the blur (circled by a broken line in FIG. 9 (a) Inside). On the other hand, in the image of FIG. 9B (image after blur adjustment processing), such front blur is reduced. In this embodiment, weighting coefficients for each of a plurality of parallax images (a first weighting coefficient, a second weighting coefficient, a second weighting coefficient, and a second weighting coefficient) are used in order to not change the blur shape of the imaging optical system except in a predetermined area where blur adjustment processing is not performed. Preferably, the weighting factors are added approximately equally to generate an output image.

  Next, with reference to FIG. 10, pupil shift at the peripheral image height of the image sensor 107 in the present embodiment will be described. FIG. 10 is an explanatory diagram of an effective aperture value due to pupil shift, and pupil partial regions 501 and 502 in which the subpixels 201 and 202 of each pixel arranged at the peripheral image height of the imaging device 107 receive light and an imaging optical system The relationship with the exit pupil 400 of FIG.

  FIG. 10A shows the case where the exit pupil distance D1 of the imaging optical system (the distance between the exit pupil 400 and the imaging surface of the imaging device 107) and the set pupil distance Ds of the imaging device 107 are substantially equal. . In this case, similarly to the central image height, the exit pupil 400 of the imaging optical system is divided substantially equally by the pupil partial regions 501 and 502 even at the peripheral image height.

  On the other hand, as shown in FIG. 10B, when the exit pupil distance Dl of the imaging optical system is shorter than the set pupil distance Ds of the imaging element 107, the peripheral image height of the imaging element 107 A pupil shift occurs between the exit pupil 400 and the entrance pupil of the image sensor 107. For this reason, the exit pupil 400 of the imaging optical system is divided into non-uniform pupils. In the case of FIG. 10B, the effective aperture value of the first parallax image corresponding to the pupil partial region 501 is smaller (brighter) than the effective aperture value of the second parallax image corresponding to the pupil partial region 502. On the other hand, at the image height on the opposite side, the effective aperture value of the first parallax image corresponding to the pupil partial region 501 is larger (darker) than the effective aperture value of the second parallax image corresponding to the pupil partial region 502 .

  As shown in FIG. 10C, when the exit pupil distance D1 of the imaging optical system is longer than the set pupil distance Ds of the imaging element 107, the exit pupil of the imaging optical system at the peripheral image height of the imaging element 107 A pupil shift occurs between the point 400 and the entrance pupil of the image sensor 107. For this reason, the exit pupil 400 of the imaging optical system is divided into non-uniform pupils. In the case of FIG. 10C, the effective aperture value of the first parallax image corresponding to the pupil partial region 501 is larger (darker) than the effective aperture value of the second parallax image corresponding to the pupil partial region 502. On the other hand, at the image height on the opposite side, the effective aperture value of the first parallax image corresponding to the pupil partial region 501 is smaller (brighter) than the effective aperture value of the second parallax image corresponding to the pupil partial region 502 . As the pupil division becomes uneven at the peripheral image height due to the pupil shift, the effective F-numbers of the first parallax image and the second parallax image also become uneven. For this reason, the spread of blurring of one of the first parallax image and the second parallax image is increased, and the spread of the other blurring is reduced. For this reason, in the present embodiment, the weight coefficient of the parallax image having the smallest effective aperture value is made the smallest among a plurality of parallax images in a predetermined region of the output image, as needed. It is preferable to make the weight coefficient of the largest parallax image the largest.

  According to the above configuration, it is possible to reduce front blur on a main subject after shooting and to improve the quality as a captured image.

  Next, an imaging apparatus according to a second embodiment of the present invention will be described with reference to FIGS. 11 and 12. The present embodiment is different from the first embodiment in which a photographed image is generated from the first parallax image and the second parallax image in that the photographed images are generated from the first to fourth parallax images as the plurality of parallax images.

  FIG. 11 is a diagram showing a pixel array of the image sensor 107 in the present embodiment. FIG. 12 is a diagram showing the pixel structure of the image sensor 107, and FIG. 12 (a) is a plan view of the pixel 200G of the image sensor 107 (view from the + z direction), and FIG. ) Is a cross-sectional view (a view from the −y direction) of line aa in FIG.

  FIG. 11 illustrates a pixel array (array of imaging pixels) of the image sensor 107 (two-dimensional CMOS sensor) in a range of 4 columns × 4 rows. In the present embodiment, each imaging pixel (pixels 200R, 200G, and 200B) is configured by four sub-pixels 201, 202, 203, and 204. Therefore, in FIG. 11, the arrangement of the sub-pixels is shown in the range of 8 columns × 8 rows.

  As shown in FIG. 11, in the pixel group 200 of 2 columns × 2 rows, the pixels 200R, 200G, and 200B are arranged in a Bayer arrangement. That is, in the pixel group 200, the pixel 200R having a spectral sensitivity of R (red) is at the upper left, the pixel 200G having a spectral sensitivity of G (green) is at the upper right and lower left, and the pixel 200B having a spectral sensitivity of B (blue). Are respectively located at the lower right. Each pixel 200R, 200G, and 200B (each imaging pixel) is comprised by the sub pixel 201, 202, 203, 204 arranged in 2 column x 2 row. The sub-pixel 201 is a pixel that receives the light flux that has passed through the first pupil region of the imaging optical system. The sub-pixel 202 is a pixel that receives the light flux that has passed through the second pupil region of the imaging optical system. The sub-pixel 203 is a pixel that receives the light flux that has passed through the third pupil region of the imaging optical system. The sub-pixel 204 is a pixel that receives the light flux that has passed through the fourth pupil region of the imaging optical system.

As shown in FIG. 11, the imaging element 107 is configured by arranging a large number of imaging pixels (4 columns × 4 rows of imaging pixels (8 columns × 8 rows of subpixels) on the surface, and imaging signals (subpixel signals) Output). The imaging element 107 of the present embodiment has a cycle P of pixels (imaging pixels) of 4 μm, and the number N of pixels (imaging pixels) is 5575 horizontal rows × 3725 vertical rows = about 207.5 million pixels. Further, in the imaging element 107, the period P _{SUB in} the column direction of the sub-pixels is 2 μm, and the number N _{SUB of} sub-pixels is 11150 columns wide × 7450 rows high = approximately 83 million pixels.

As shown in FIG. 12B, the pixel 200G of the present embodiment is provided with a micro lens 305 for condensing incident light on the light receiving surface side of the pixel. The micro lens 305 is disposed at a predetermined distance from the light receiving surface in the z-axis direction (the direction of the optical axis OA). Further, in the pixel 200G, photoelectric conversion units 301, 302, 303, and 304 which are N _H division (two division) in the x direction and N _V division (two division) in the y direction are formed. The photoelectric conversion units 301 to 304 correspond to the sub pixels 201 to 204, respectively.

  In this embodiment, the imaging element 107 shares one microlens, and a plurality of light fluxes passing through mutually different areas (first to fourth pupil partial areas) of the pupil of the imaging optical system (photographing lens) It has a plurality of sub-pixels that receive light. The imaging device 107 includes, as a plurality of sub-pixels, a first sub-pixel (a plurality of sub-pixels 201), a second sub-pixel (a plurality of sub-pixels 202), a third sub-pixel (a plurality of sub-pixels 203), and It includes four sub-pixels (plural sub-pixels 204). In the present embodiment, light reception signals of a plurality of sub-pixels 201 are collected to generate a first parallax image. Similarly, the light reception signals of the plurality of sub-pixels 202 are collected to collect the second parallax image, and the light reception signals of the plurality of sub-pixels 203 are collected to collect the third parallax image, the light reception signals of the plurality of sub-pixels 204 are collected, and a fourth parallax image Generate each Further, in the present embodiment, the first to fourth parallax images are images of the Bayer arrangement. Demosaicing processing may be performed on each of the first to fourth parallax images as needed.

Here, j and i are integers, and the jth position in the row direction and the ith position in the column direction of each of the first to 42nd parallax images are represented as (j, i). Then, the first parallax image of the pixel at the position (j, i) is A (j, i), the second parallax image is B (j, i), the third parallax image is C (j, i), the fourth parallax Let the image be D (j, i). Also, the first weight coefficient of the first parallax image is W _a (j, i), the second weight coefficient of the second parallax image is W _b (j, i), and the third weight coefficient of the third parallax image is W _c (J, i) _Let W _d (j, i) be the fourth weighting factor of the fourth parallax image.

  As a first step, the CPU 121 (image processing circuit 125) sets a predetermined area R = [j1, j2] × [i1, i2] and a boundary width σ of the predetermined area to reduce front blur on the main subject. Do. Then, a table function T (j, i) according to the predetermined area R and the boundary width σ of the predetermined area is calculated by the equation (1).

  The table function T (j, i) is 1 inside the predetermined region R and 0 outside the predetermined region R, and changes continuously from approximately 1 to 0 at the boundary width σ of the predetermined region R. The predetermined region R may be circular or any other shape as needed. Also, a plurality of predetermined areas and boundary widths may be set as necessary.

As a second step, the CPU 121 (image processing circuit 125) sets the real numbers wa, wb, wc, wd (wa + wb + wc + wd = 0), and the first of the first parallax images A (j, i) according to the following equation (5A). The weighting factor W _a (j, i) is calculated. Similarly, the CPU 121 causes the second weight coefficient W _b (j, i) of the second parallax image B (j, i) and the third parallax image C (j, i) to be expressed by the following equations (5B) to (5D). third weighting factor _W c (j, in) i), and calculates a fourth parallax image D (j, fourth weighting factor _W d (j of i), i), respectively.

  As a third step, the image processing circuit 125 generates an output image I (j, i) according to the following equation (6).

If necessary, the image processing circuit 125 may generate the output image I _st (j, i) by the following equation (7A) or equation (7B) in combination with the refocusing process using the shift amounts s and t. Good.

  In the present embodiment, the configuration other than the above is the same as that of the first embodiment, and thus the description thereof is omitted. According to the above configuration, it is possible to reduce front blur on a main subject after shooting and to improve the quality as a captured image.

  As described above, in each embodiment, the image processing apparatus (image processing circuit 125) includes the determination unit 125a and the image generation unit 125b. The determination unit 125a determines a weighting factor that changes according to the position of each of the plurality of parallax images. The image generation unit 125 b combines the plurality of parallax images based on the weighting factor to generate an image. The image processing apparatus may be configured to have an acquisition unit capable of acquiring a weighting factor determined by an apparatus such as the CPU 121 (an apparatus having the same function as the determination unit 125a) instead of the determination unit 125a. .

  Preferably, the image generation unit generates an image by multiplying each of the plurality of parallax images by weighting factors and adding (combining). Further preferably, the image generation unit generates an image by multiplying each of the plurality of parallax images by weighting factors and performing shift addition (combining using a refocusing process).

Preferably, the weighting factor changes continuously according to the position of each of the plurality of parallax images. This can be realized, for example, by using a table function T (j, i). Also preferably, the sum (sum) of the weighting factors of each of the plurality of parallax images is constant at all positions in the plurality of parallax images. For example, as in the first embodiment, the first weight coefficient W _a of the first parallax image and the second weight coefficient W _{b of} the second parallax image obtained from the imaging device having two sub-pixels for one microlens, and Is constant at any position in the image. Alternatively, as in the second embodiment, the sum of first to fourth weight coefficients W _{a to} W _d of first to fourth parallax images obtained from an imaging device having four sub-pixels for one microlens as in Example 2 The (sum) is constant at any position in the image.

  Preferably, the plurality of parallax images are acquired by an imaging device in which a plurality of pixels configured of a plurality of photoelectric conversion units (a plurality of sub-pixels) that receive light beams passing through different pupil partial regions of the imaging optical system are arrayed Be done. That is, the plurality of parallax images are generated from light reception signals of the sub-pixels for each pupil partial region.

  Preferably, the image generation unit combines a plurality of parallax images based on the weighting factor in a first region (for example, a predetermined region R and a boundary width σ) of the image. More preferably, the first area is an area set to reduce blurring of the second subject (subject q2) closer to the first subject (subject q1). Further preferably, the image generation unit generates an image by multiplying and adding a plurality of parallax images with equal weighting factors in a second area different from the first area of the image (for example, outside the predetermined area R). Do.

  Preferably, in the first area of the image, the determination means is a weight of the parallax image in which the second object closer to the first object than the first object is photographed in the widest range among the plurality of weighting coefficients of the plurality of parallax images. Make the coefficient the smallest. Further preferably, the determination unit is configured to use a parallax image in which a second subject closer to the first subject than the first subject is photographed in the narrowest range among the plurality of weighting coefficients of the plurality of parallax images in the first region of the image. Make the weighting factor the largest.

  Preferably, the determination means makes the weight coefficient of the parallax image having the smallest contrast evaluation value the smallest among the plurality of weight coefficients of the plurality of parallax images in the first region of the image. Still preferably, the determination means maximizes the weight coefficient of the parallax image having the largest contrast evaluation value among the plurality of weight coefficients of the plurality of parallax images in the first region of the image.

  Preferably, the determination means makes the weight coefficient of the parallax image with the smallest effective aperture value the smallest among the plurality of weight coefficients of the plurality of parallax images in the first region of the image. Further preferably, the determination means maximizes the weight coefficient of the parallax image having the largest effective aperture value among the plurality of weight coefficients of the plurality of parallax images in the first region of the image.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  According to each embodiment, it is possible to provide an image processing device, an imaging device, an image processing method, a program, and a storage medium that improve the quality of a captured image.

  Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes can be made within the scope of the present invention.

125 Image processing circuit (image processing device)
125a determination means 125b image generation means

Claims (29)

Setting means for setting weighting coefficients corresponding to a plurality of areas including the first area of each of the plurality of parallax images;
And image generation means for synthesizing the plurality of parallax images based on the weighting factor set by the setting means to generate a synthesized image,
The image processing apparatus, wherein the setting unit sets a weighting factor corresponding to the first region in each of the plurality of parallax images in accordance with the effective aperture value of the plurality of parallax images.   The setting unit sets a weighting factor corresponding to the first region in each of the plurality of parallax images as a weighting factor having a smaller weighting factor corresponding to a parallax image having a smaller effective aperture value among the plurality of parallax images. The image processing apparatus according to claim 1, wherein the image processing apparatus is set as follows.   The setting unit sets a weighting factor corresponding to the first region in each of the plurality of parallax images as a weighting factor having the smallest weighting factor corresponding to the parallax image having the smallest effective aperture value among the plurality of parallax images. The image processing apparatus according to claim 1, wherein the image processing apparatus is set to   The setting means sets a weighting factor corresponding to the first region in each of the plurality of parallax images to a weighting factor having the largest weighting factor corresponding to the parallax image having the largest effective aperture value among the plurality of parallax images. The image processing apparatus according to any one of claims 1 to 3, wherein the image processing apparatus is set as follows. Setting means for setting weighting coefficients corresponding to a plurality of areas including the first area of each of the plurality of parallax images;
And image generation means for synthesizing the plurality of parallax images based on the weighting factor set by the setting means to generate a synthesized image,
The image processing apparatus, wherein the setting unit sets a weighting factor corresponding to the first region in each of the plurality of parallax images in accordance with a contrast evaluation value of the plurality of parallax images.   The setting means may set a weighting factor corresponding to the first region in each of the plurality of parallax images, and a weighting factor having a smaller weighting coefficient corresponding to a parallax image having a smaller contrast evaluation value among the plurality of parallax images. The image processing apparatus according to claim 5, wherein the image processing apparatus is set to be   The setting means sets a weighting factor corresponding to the first region in each of the plurality of parallax images as a weighting factor having the smallest weighting factor corresponding to the parallax image having the smallest contrast evaluation value among the plurality of parallax images. The image processing apparatus according to claim 5 or 6, wherein the image processing apparatus is set as follows.   The setting means sets a weighting factor corresponding to the first region in each of the plurality of parallax images to a weighting factor having the largest weighting factor corresponding to the parallax image having the largest contrast evaluation value among the plurality of parallax images. The image processing apparatus according to any one of claims 5 to 7, wherein the setting is performed as follows. Setting means for setting weighting coefficients corresponding to a plurality of areas including the first area of each of the plurality of parallax images;
And image generation means for synthesizing the plurality of parallax images based on the weighting factor set by the setting means to generate a synthesized image,
The setting means sets a weighting factor corresponding to the first area in each of the plurality of parallax images according to the range in which the subject is photographed in the first area in each of the plurality of parallax images. Image processing device.   The setting unit is configured to set a weighting factor corresponding to the first region in each of the plurality of parallax images, and a weighting factor corresponding to a parallax image in which the subject is photographed in a wider range among the plurality of parallax images. 10. The image processing apparatus according to claim 9, wherein the image processing apparatus is set to have a larger weight coefficient.   The setting unit is configured to set a weighting factor corresponding to the first region in each of the plurality of parallax images and a weighting factor corresponding to a parallax image in which the subject is photographed in the widest range among the plurality of parallax images. 11. The image processing apparatus according to claim 9, wherein the image processing apparatus is set to have the largest weight coefficient.   The setting unit is configured to set a weighting factor corresponding to the first region in each of the plurality of parallax images, and a weighting factor corresponding to a parallax image in which the subject is photographed in the narrowest range among the plurality of parallax images. The image processing apparatus according to any one of claims 9 to 11, wherein the image processing apparatus is set to have the smallest weight coefficient.   The image processing apparatus according to any one of claims 1 to 12, wherein the first area is an area set to reduce blurring of an object in the first area.   The image according to any one of claims 1 to 13, wherein the image generation unit generates the image by multiplying and adding the corresponding weight coefficient in each of the plurality of parallax images. Processing unit.   The image processing apparatus according to any one of claims 1 to 14, wherein the weighting factor continuously changes in each of the plurality of parallax images in accordance with a function based on a position with respect to the plurality of regions.   The image processing apparatus according to any one of claims 1 to 15, wherein a sum of the weighting coefficients corresponding to the plurality of regions in each of the plurality of parallax images is constant.   The plurality of parallax images are obtained by an imaging device in which a plurality of pixels configured of a plurality of photoelectric conversion units that receive light fluxes that have passed through different pupil partial regions of an imaging optical system are obtained. Item 17. The image processing device according to any one of items 1 to 16.   The image generation means may generate an image of an area different from the first area of the combined image by multiplying the plurality of parallax images by an equal weight coefficient and adding them. The image processing device according to any one of 17. An image pickup element in which a plurality of pixels configured of a plurality of photoelectric conversion units that receive light beams passing through different pupil partial regions of an imaging optical system;
Setting means for setting weighting coefficients corresponding to a plurality of areas including the first area of each of the plurality of parallax images obtained from the plurality of photoelectric conversion units;
And image generation means for synthesizing the plurality of parallax images based on the weighting factor set by the setting means to generate a synthesized image,
The imaging device, wherein the setting unit sets a weighting factor corresponding to the first region in each of the plurality of parallax images in accordance with an effective aperture value of the plurality of parallax images. An image pickup element in which a plurality of pixels configured of a plurality of photoelectric conversion units that receive light beams passing through different pupil partial regions of an imaging optical system;
Setting means for setting weighting coefficients corresponding to a plurality of areas including the first area of each of the plurality of parallax images obtained from the plurality of photoelectric conversion units;
And image generation means for synthesizing the plurality of parallax images based on the weighting factor set by the setting means to generate a synthesized image,
The setting device sets a weighting factor corresponding to the first region in each of the plurality of parallax images in accordance with a contrast evaluation value of the plurality of parallax images. An image pickup element in which a plurality of pixels configured of a plurality of photoelectric conversion units that receive light beams passing through different pupil partial regions of an imaging optical system;
Setting means for setting weighting coefficients corresponding to a plurality of areas including the first area of each of the plurality of parallax images obtained from the plurality of photoelectric conversion units;
And image generation means for synthesizing the plurality of parallax images based on the weighting factor set by the setting means to generate a synthesized image,
The setting means sets a weighting factor corresponding to the first area in each of the plurality of parallax images according to the range in which the subject is photographed in the first area in each of the plurality of parallax images. An imaging device.   22. The image pickup device according to any one of claims 19 to 21, wherein the image pickup device includes the plurality of photoelectric conversion units for one microlens, and the microlenses are two-dimensionally arranged. The imaging device of description. Setting a weighting factor corresponding to a plurality of areas including the first area of each of the plurality of parallax images;
Combining the plurality of parallax images based on the weighting factor set by the setting step to generate a combined image;
The image processing method characterized in that the setting step sets a weight coefficient corresponding to the first region in each of the plurality of parallax images in accordance with the effective aperture value of the plurality of parallax images. Setting a weighting factor corresponding to a plurality of areas including the first area of each of the plurality of parallax images;
Combining the plurality of parallax images based on the weighting factor set by the setting step to generate a combined image;
The image processing method characterized in that the setting step sets a weighting factor corresponding to the first region in each of the plurality of parallax images in accordance with a contrast evaluation value of the plurality of parallax images. Setting a weighting factor corresponding to a plurality of areas including the first area of each of the plurality of parallax images;
Combining the plurality of parallax images based on the weighting factor set by the setting step to generate a combined image;
The setting may include setting a weighting factor corresponding to the first area in each of the plurality of parallax images according to a range in which the subject is photographed in the first area in each of the plurality of parallax images. Characteristic image processing method. Setting a weighting factor corresponding to a plurality of areas including the first area of each of the plurality of parallax images;
A program configured to cause a computer to execute combining the plurality of parallax images based on the weighting factor set in the setting step to generate a combined image.
The program is characterized in that, in the setting step, a weighting factor corresponding to the first region in each of the plurality of parallax images is set according to an effective aperture value of the plurality of parallax images. Setting a weighting factor corresponding to a plurality of areas including the first area of each of the plurality of parallax images;
A program configured to cause a computer to execute combining the plurality of parallax images based on the weighting factor set in the setting step to generate a combined image.
The program is characterized in that, in the setting step, a weighting factor corresponding to the first region in each of the plurality of parallax images is set according to the contrast evaluation value of the plurality of parallax images. Setting a weighting factor corresponding to a plurality of areas including the first area of each of the plurality of parallax images;
A program configured to cause a computer to execute combining the plurality of parallax images based on the weighting factor set in the setting step to generate a combined image.
The setting may include setting a weighting factor corresponding to the first area in each of the plurality of parallax images according to a range in which the subject is photographed in the first area in each of the plurality of parallax images. Program to feature.   A computer readable storage medium storing the program according to any one of claims 26 to 28.