JP2018061231A - Imaging apparatus, imaging method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a blurred region in a composite image due to camera shake or the like when synthesizing a plurality of images captured by changing the focus position. An image pickup apparatus includes an image pickup means, a setting means for setting a plurality of target focus positions, and a focus position calculation means, and the image pickup means has a plurality of means according to the target focus position. The image is captured, the calculation means calculates the focus position with respect to the subject when each of the plurality of images is captured, and the setting means compares the target focus position with the calculated focus position. It is characterized in that at least a part of the target focus position is reset according to the result. [Selection diagram] Fig. 5

Description

  The present invention relates to an imaging apparatus that performs image composition.

  When shooting multiple subjects with different distances from the camera, or when shooting subjects that are long in the depth direction, the depth of field in the imaging optical system is insufficient. It may not be possible.

  Therefore, a plurality of images are picked up by changing the focus position, only a focused area is extracted from each image, and a single image is synthesized to generate a composite image focused on the entire imaging area. Technology (hereinafter referred to as depth synthesis) is known (see Patent Document 1). By using this depth synthesis technique, it is possible to obtain an image in which the entire intended subject is in focus.

JP 2002-84444 A

  However, when imaging is performed using the technique of depth synthesis, the following problems may occur when camera shake or the like occurs.

  FIG. 13 is a diagram for explaining camera shake in depth synthesis. FIG. 13A is a diagram for explaining imaging using ideal depth synthesis. In FIG. 13A, the user places the digital camera on 1301, and performs imaging a plurality of times while moving the focus position by a predetermined distance 1302.

  FIG. 13B shows a case where camera shake has occurred when the third image is captured in the imaging in FIG. Since the imaging position 1301 is shifted rearward in the optical axis direction, the focus position comes closer to the distance that should be originally captured. Further, as shown in FIG. 13C, when a fourth image is captured, a case where camera shake occurs in a direction opposite to the direction of camera shake in FIG. 13B is shown. In this case, the fourth image is farther from the focus position that should be captured. When combining processing is performed using a plurality of images captured in FIG. 13C, there is no focused image in the region 1303, and a blurred portion remains in the combined image.

  The present invention has been made in view of the above problems, and provides an imaging device capable of generating a composite image that reduces blurring due to camera shake or the like when combining using a plurality of images captured by changing the focus position. The purpose.

  The present invention includes an imaging unit, a setting unit that sets a plurality of target focus positions, and a calculation unit for the focus position, and the imaging unit captures a plurality of images according to the target focus position. The calculation unit calculates a focus position with respect to the subject when each of the plurality of images is captured, and the setting unit determines whether the target focus position is compared with the calculated focus position. An imaging apparatus is provided, wherein at least a part of the target focus position is reset.

  According to the present invention, when combining using a plurality of images picked up by changing the focus position, it is possible to generate a combined image that reduces blur due to camera shake or the like.

1 is an external view of a digital camera as an example of an imaging apparatus according to a first embodiment of the present invention. It is a block diagram which shows the structure of the digital camera which concerns on embodiment of this invention. It is a figure for demonstrating the array of the sensor of 1st Embodiment. It is a figure for demonstrating incidence | injection to the pixel of the optical signal in 1st Embodiment. It is a flowchart for demonstrating the process of the depth synthetic | combination in 1st Embodiment. It is a flowchart explaining the captured image evaluation process in 1st Embodiment. 5 is a flowchart for explaining correction of a focus position in the first embodiment. It is a figure for demonstrating the variation | change_quantity correction | amendment of a focus position in 1st Embodiment. It is a figure for demonstrating the structure of the digital single-lens reflex camera in 2nd Embodiment. It is a flowchart for demonstrating the process of the depth synthetic | combination in 2nd Embodiment. It is a flowchart for demonstrating the process of the depth synthetic | combination in 3rd Embodiment. It is a figure for demonstrating the analysis of the contrast value in 3rd Embodiment. It is a figure for demonstrating the camera shake in depth composition.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, a digital camera will be described as an example, but the present invention can be applied if an imaging apparatus capable of adjusting the focus position is used.

(First embodiment)
FIG. 1 is an external view of a digital camera as an example of the imaging apparatus of the present embodiment. The monitor 101 displays images and various information. The shutter button 102 is provided for performing an imaging instruction. A mode switch 103 is provided for switching various modes. The connector 104 connects the connection cable 105 and the digital camera. The controller wheel 106 is an operation member that can be rotated. A switch 107 is a power switch, and switches between power on and power off. The memory card 108 is a recording medium. The slot 109 is a slot for storing the memory card 108. The memory card 108 stored in the slot 109 can communicate with the main body of the digital camera. The lid 110 is a lid of the slot 109.

  FIG. 2 is a block diagram showing the structure of the digital camera according to this embodiment. The digital camera 200 can capture a still image, records information on a focus position, and can calculate a contrast value and synthesize an image. Furthermore, the digital camera 200 can perform an enlargement process or a reduction process on an image captured and stored or an image input from the outside.

  The control unit 201 is a signal processor such as a CPU or MPU, for example, and controls each part of the digital camera 200 while reading a program built in the ROM 205 described later in advance. For example, as will be described later, the control unit 201 instructs the imaging unit 204 described later to start and end imaging. Alternatively, an image processing command is issued to an image processing unit 207, which will be described later, based on settings prepared in advance. A command from the user is input to the digital camera 200 by an operation unit 210 described later, and reaches each part of the digital camera 200 through the control unit 201.

  The drive unit 202 is configured by a motor or the like, and mechanically operates an optical system 203 described later under a command from the control unit 201. For example, based on a command from the control unit 201, the driving unit 202 moves the position of the focus lens included in the optical system 203 and adjusts the focal length of the optical system 203.

  The optical system 203 includes a zoom lens, a focus lens, a diaphragm, and the like. The diaphragm is a mechanism that adjusts the amount of transmitted light. The focus position can be changed by changing the lens position. However, the “focus position” here is defined based on the subject unless otherwise specified.

  The imaging unit 204 is a photoelectric conversion element, and performs photoelectric conversion that converts an incident optical signal into an electrical signal. For example, a CCD or a CMOS sensor can be applied to the imaging unit 204. Details of the structure of the imaging unit will be described later. The imaging unit 204 is provided with a moving image capturing mode, and can capture a plurality of temporally continuous images as each frame of the moving image.

  The ROM 205 is a read-only nonvolatile memory as a recording medium, and stores parameters necessary for the operation of each block in addition to the operation program for each block provided in the digital camera 200. The RAM 206 is a rewritable volatile memory, and is used as a temporary storage area for data output in the operation of each block included in the digital camera 200.

  The image processing unit 207 performs various image processing, such as white balance adjustment, color interpolation, and filtering, on the image output from the imaging unit 204 or image signal data recorded in the internal memory 209 described later. . In addition, compression processing is performed on the data of the image signal captured by the imaging unit 204 according to a standard such as JPEG.

  The image processing unit 207 includes an integrated circuit (ASIC) that collects circuits for performing specific processing. Alternatively, the control unit 201 may perform a part of or all of the functions of the image processing unit 207 by performing processing according to a program read from the ROM 205. When the control unit 201 has all the functions of the image processing unit 207, it is not necessary to have the image processing unit 207 as hardware.

  The display unit 208 is a liquid crystal display or an organic EL display for displaying an image temporarily stored in the RAM 206, an image stored in an internal memory 209 described later, a setting screen of the digital camera 200, or the like. is there. The monitor 101 in FIG. 1 corresponds to the display unit 208.

  The built-in memory 209 is a place where an image captured by the image capturing unit 204, an image obtained by the processing of the image processing unit 207, information on a focus position at the time of image capturing, and the like are recorded. Further, instead of the built-in memory, a memory card 108 or the like may be used as shown in FIG.

  The operation unit 210 is, for example, a button, switch, key, mode dial, or the like attached to the digital camera 200 or a touch panel that is also used as the display unit 208. A command from the user reaches the control unit 201 via the operation unit 210. The shutter button 102, the mode switch 103, the controller wheel 106, and the switch 107 in FIG.

  FIG. 3 is a diagram for explaining an array of sensors constituting the imaging unit 204 in the present embodiment. FIG. 3A shows that the pixel 300 includes two photoelectric conversion units 301 and 302 that can read an optical signal independently of each other. However, each pixel may have a structure having three or more photoelectric conversion units. For example, FIG. 3B illustrates a structure in which the pixel 310 includes four photoelectric conversion units 311 to 314. In the following description, a description will be given based on a structure in which one pixel has two photoelectric conversion units.

  FIG. 4 is a diagram for explaining the incidence of an optical signal on a pixel in the present embodiment.

  In FIG. 4, the pixel array 401 includes a microlens 402, a color filter 403, and photoelectric conversion units 404 and 405. The photoelectric conversion units 404 and 405 belong to the same pixel and correspond to the microlens 402 and the color filter 403. FIG. 4 is a top view of the digital camera, and shows that two photoelectric conversion units 404 and 405 corresponding to one pixel are arranged side by side. Out of the luminous flux emitted from the exit pupil 406, the upper luminous flux (corresponding to the luminous flux from the region 407) enters the photoelectric conversion unit 405 with the optical axis 409 as a boundary, and the lower luminous flux (corresponding to the luminous flux from the region 408). ) Enters the photoelectric conversion unit 404. That is, the photoelectric conversion units 404 and 405 receive light in different areas of the exit pupil of the imaging lens. Here, when the signal received by the photoelectric conversion unit 404 is an A image and the signal received by the photoelectric conversion unit 405 is a B image, the control unit 201 can calculate the amount of defocus based on the phase difference between the A image and the B image. , Distance information can be acquired.

  FIG. 5 is a flowchart for explaining the process of depth synthesis in this embodiment.

  When a user command reaches the control unit 201 via the operation unit 210, the process of depth synthesis starts. In step S501, the control unit 201 changes the focus position when the first image is captured based on the subject depth of the digital camera 200 and the amount of change in the focus position between images when the second and subsequent images are captured. , And the number of images to be captured. A plurality of focus positions are calculated and set based on these values. Next, in step S502, the control unit 201 moves the lens constituting the optical system 203 to change the focus position of the digital camera 200. If the first image, the lens is moved to a preset focus position, and if it is the second image or later, the closest focus position that has not been imaged among the preset focus positions. Or move to the focus position on the infinity side. In step S503, the imaging unit 204 is controlled to perform imaging. In step S504, the control unit 201 evaluates the image captured in step S503. Details of the captured image evaluation will be described later.

  In step S505, the control unit 201 determines whether or not the captured image satisfies the criterion. If the criterion is satisfied, the control unit 201 proceeds to step S506 and determines whether or not a set number of images have been captured.

  If the control unit 201 determines in step S505 that the captured image does not satisfy the standard, the focus position set for the next imaging is corrected in setting step S507, and the process returns to step S502. Details of the correction of the set focus position in step S507 will be described later.

  If it is determined in step S506 that a set number of images have been captured, in step S508, the control unit 201 executes a combining process to generate a combined image. If it is determined in step S506 that the set number of images has not been captured, the process returns to step S502.

  In the image composition in step S508, a known method may be used, and an example thereof will be described below. First, for alignment, the absolute value sum (SAD: Sum Of Absolute Difference) of the output differences of the pixels of the plurality of images is obtained while shifting the relative positions of the two images. The relative moving amount and moving direction of the two images when the absolute value sum is the smallest are obtained. Then, after calculating the conversion coefficient of the affine transformation or projective transformation according to the obtained movement amount and the movement direction, minimize the error between the movement amount by the conversion coefficient and the movement amount calculated from the absolute value sum. Optimize transform coefficients using the square method. Based on the optimized conversion coefficient, a deformation process is performed on the image to be aligned. The image processing unit 207 performs the above-described alignment and deformation processing on all the images captured by the imaging unit 204 in step S503, and then gives a composition ratio to each region of each image. As an example, the image processing unit 207 gives a composition ratio of 100% to pixels included in a corresponding area of an image having an in-focus area among a plurality of images corresponding to the same area. A synthesis ratio of 0% is given to the pixels included in the corresponding area. Alternatively, the composition ratio may be assigned to the corresponding area of each image in accordance with the degree of focusing between the images in each area. In order to prevent unnaturalness at the synthesis boundary, the image processing unit 207 causes the synthesis ratio to change stepwise between adjacent pixels. Finally, a composite image is generated based on the composite ratio of each pixel.

  Next, the captured image evaluation in step S504 will be described.

  FIG. 6 is a flowchart illustrating the captured image evaluation process according to this embodiment. When the captured image evaluation process starts, the control unit 201 calculates a focus position when the image is captured in the latest step S503 in step S601. In the present embodiment, the focus position at the time of imaging is calculated from the phase difference between the A image and the B image at each pixel of the image sensor when capturing a still image.

  In step S602, the control unit 201 calculates a shift amount of the focus position. A focus position shift amount is calculated from the focus position at the time of imaging acquired in step S601 and each focus position set in step S501. For example, if the focus position acquired in step S601 is the focus position of the Nth imaged image, the focus position of the focus position set in step S501 is compared with the Nth focus position. The amount of deviation is calculated.

  In step S603, the control unit 201 determines whether the shift amount of the focus position is within a predetermined value.

  If the deviation amount of the focal length exceeds the predetermined amount in step S603, the control unit 201 proceeds to step S604, determines that the criterion is not satisfied, and ends the process.

  Conversely, if the focal length deviation amount is within a predetermined value in step S603, the control unit 201 proceeds to step S605, determines that the criterion is satisfied, and ends the process. Based on the determination result of step S604 or 605, the determination of step S505 in FIG. 5 is performed.

  This is the description of the captured image evaluation process in step S504.

  Next, the focus position correction process in step S507 will be described. FIG. 7 is a flowchart for explaining the correction of the focus position in the present embodiment.

  In step S701, the control unit 201 calculates the distance between the focus position in the previous imaging and the focus position in the current imaging. In step S702, the control unit 201 determines whether the distance of the focus position of the image calculated in step S701 exceeds a predetermined value. This predetermined value is a value determined based on the focal length and the permissible circle of confusion of the image sensor. If this predetermined value is exceeded, no focus is achieved between these two images. An area will be created.

  If the control unit 201 determines in step S702 that the distance between the focus positions exceeds a predetermined value, the control unit 201 proceeds to step S703 and adds a new focus position between the previous imaging position and the current imaging position. . At the same time, the control unit 201 adds one to the planned number in step S506 described above. Then, the control unit 201 proceeds to step S704. When the control unit 201 adds a new focus position in step S704, it is desirable to capture an image at the added focus position in step S603 immediately after that. You may take an image at the added focus position after taking an image at another focus position, but if camera shake occurs during that time, correct the added focus position again according to the camera shake. This is because a need arises. Conversely, if the control unit 201 determines in step S702 that the distance between the focus positions is within a predetermined value, the control unit 201 proceeds to step S704 without adding a new focus position.

  In step S704, the focus position set in step S501 or the focus position updated when the previous step S704 was processed is updated with a new focus position. That is, the reference focus position when calculating the focus position shift amount in step S602 in FIG. 6 is updated. Specifically, when the focus position is set in order from the closest side and the image is captured, the change in the focus position set in step S501 with reference to the focus position on the most infinite side among the captured focus positions. The focus position shifted by the amount is set for the remaining number of images to be captured. That is, if the focus position on the most infinite side among the focus positions that have been imaged is shifted to the infinity side from the focus position initially set in step S501, a plurality of focus positions set at the time of the remaining imaging are also obtained. Each is reset to the position shifted to the infinity side by the same amount. By doing this, even if the focus position is shifted in the middle, it is possible to keep the difference in focus position between images captured thereafter.

  In this way, the control unit 201 corrects the focus position in step S704, and ends the flowchart of FIG.

  FIG. 8 is a diagram for explaining the correction of the focus position in the present embodiment. FIG. 8A shows the focus position set in step S501 of FIG.

  FIG. 8B shows a state in which camera shake occurs on the near side in the optical axis direction when the third image is captured. A focus position 801, a focus position 802, and a focus position 803 indicate the focus positions when the first, second, and third images are captured, respectively. The focus position 803 when the third image is actually captured is shifted from the focus position when the original third image is captured by the amount indicated by the arrow 821. This state corresponds to a state in which NO is determined in step S603 in FIG. 6 and YES is determined in step S702 in FIG. Here, although the focus position is shifted, since the focus position of the third image is only close to the focus position of the second image, the focus position is not added here.

  FIG. 8C shows a focus position 804 when the fourth image is captured. A new focus position 804 is set at a position shifted from the focus position 803 of the third image by the amount of change set in step S501 in FIG.

  FIG. 8D shows a state in which camera shake occurs on the infinity side in the optical axis direction when the fifth image is captured. The focus position 805 when the fifth image is actually captured is shifted from the focus position when the fifth image is captured by the amount indicated by the arrow 822. In FIG. 9D, the distance between the focus position 804 when the fourth image is captured and the focus position 805 when the fifth image is captured exceeds a predetermined value. Therefore, there is a region that is not included in the subject depth in any of the fourth and fifth images between these focus positions. This state corresponds to a state where NO is determined in step S603 in FIG. 6 and NO is determined in step S702 in FIG.

  Therefore, as shown in FIG. 8E, a new focus position 811 is set between the focus position 804 and the focus position 805, and an image is taken at this focus position 811. The sled that sets the focus position 811 corresponds to step S703 in FIG. If the distance between the focus position 804 and the focus position 805 is wide and it is not sufficient to add one focus position, two or more focus points may be added. By adding a new focus position, the distance of the focus position between images with adjacent focus positions can be kept within a predetermined value.

  FIG. 8F shows a state where a new focus position 806 is set at a position shifted from the focus position 805 by the amount of change set in step S501 in FIG. 5, and an image is captured.

  As described above, according to the first embodiment, when combining a plurality of images picked up by changing the focus position, the focus position when the image is actually picked up, the planned focus position, and By calculating the deviation, the focus position of the next imaging can be adjusted, and blurring when generating a composite image can be reduced.

  Note that if the amount of camera shake toward the front side is large, even if the planned number of images has been captured, there is a possibility that the imaging will be completed before the schedule. Therefore, even when the planned number of sheets is completed in step S506 in FIG. 5, the control unit 201 has a case where the closest focus position and the most infinite focus position are narrower than the initially set range. Alternatively, imaging may be performed by further adding a focus position. For example, if the focus position of the last image actually captured is close to the focus position of the first image compared to the focus position of the last image derived from the value set in step S501, the focus position distribution range is narrow. It can be judged. Conversely, if the amount of camera shake at infinity is large, and as a result of adding a focus position on the way, imaging at the originally planned focus position is completed before taking the image for the scheduled number In this case, the imaging may be terminated at that time.

(Second Embodiment)
In the second embodiment, an example in which an imaging device provided with a focus detection sensor is used separately from the imaging unit 204 will be described. Below, it demonstrates in detail, referring drawings. In addition, the description of the same place as 1st Embodiment is abbreviate | omitted.

  FIG. 9 is a diagram for explaining the structure of a digital single-lens reflex camera. The following description is based on a digital single-lens reflex camera. However, the present invention is not limited to this, and any imaging device provided with an imaging unit 204 and a focus detection sensor may be used. A lens barrel 901 is attached to the digital single-lens reflex camera 900, and when taking an image, the mirror 903 and the sub-mirror 904 escape from the imaging optical path, and the light passing through the lens barrel 901 is guided to the imaging sensor 902. . Before and after the image is captured, a part of the light reflected by the mirror 903 arranged in the imaging optical path is refracted by the prism 905 and then guided to the finder 907 as shown in FIG. Part of the light that reaches the mirror 903 passes through the mirror 903, is reflected by the sub-mirror 904 connected to the mirror 903, and is guided to the AF sensor 906. The AF sensor 906 can calculate the shift amount and direction of the focus position.

  FIG. 10 is a flowchart for explaining the depth synthesis processing in this embodiment. Steps S1001 to S1008 are the same as steps S501 to S508 in FIG. 5 of the first embodiment.

  When the initial focus position, the focus position change amount, and the number of times of imaging are set in step S1001, the control unit 201 inserts the mirror 903 into the imaging optical path in step S1002. Then, the control unit 201 retracts the mirror 903 from the imaging optical path in step S1013 before imaging in step S1003.

  In step S1012, the AF sensor 906 acquires distance measurement information, and the acquired distance information is used for captured image evaluation in step S1004 performed by the control unit 201. Actually, there is a difference between the timing at which ranging information is acquired in step S1012 and the timing at which imaging is performed in step S1003, but there is no problem because the difference is slight. Therefore, the captured image evaluation can be performed in step S1004 using the distance information acquired in step S1002.

  According to the second embodiment, even when an imaging apparatus provided with a focus detection sensor in addition to the imaging unit 204 is used, when a plurality of images captured by changing the focus position are combined, camera shake occurs. Even if it occurs, a composite image that reduces blur can be generated.

(Third embodiment)
In the third embodiment, instead of directly calculating the shift amount and direction of the focus position, an imaging apparatus that estimates the focus position at the time of imaging based on the analysis result of the contrast value and performs the captured image evaluation or the like. The used example will be described. Hereinafter, the third embodiment will be described in detail with reference to the drawings. Note that portions similar to those of the first embodiment are omitted.

  FIG. 11 is a flowchart for explaining the depth synthesis processing in this embodiment.

  In step S1101, the control unit 201 sets an initial focus position, a change amount of the focus position, and the number of captured images based on user settings. In step S1102, the control unit 201 moves the focus position of the imaging apparatus to the focus position closest to the infinity among the focus positions set in step S1101. In step S1103, the imaging apparatus performs imaging at each set focus position.

  In step S1104, the image processing unit 207 sets a point of interest and analyzes the contrast value of the point of interest in each image. It is preferable that the image processing unit 207 extracts an edge portion from the image and sets as many attention points as possible on the edge portion. FIG. 12 is a diagram for explaining the contrast value of the target point in the present embodiment. FIG. 12A shows a subject, and areas 1201 to 1204 are areas arbitrarily designated by the image processing unit 207 (hereinafter referred to as attention points). By calculating and plotting the contrast value of each point of interest in each image captured at the set focus position, the relationship between the contrast value and the set focus position can be obtained. FIG. 12B shows the relationship between the contrast value and the set focus position with curves 1211 to 1214 at the respective attention points 1201 to 1204 in FIG. FIG. 12B shows that when the focused point is in focus, the contrast value increases, and the contrast value decreases as the focus position shifts. If camera shake does not occur, the relationship between the contrast value and the set focus position is as shown in FIG.

  Next, the analysis of the contrast value of the attention point in step S1104 will be described with an example. For simplicity, an example in which only the attention points 1201 and 1202 are analyzed will be described below. FIG. 12C shows the contrast values at the respective focus positions of the attention points 1201 and 1202 in actual imaging. In FIG. 12C, since the imaging unit 204 picks up images at discrete focus positions, discrete contrast values are obtained. The approximate curves of these contrast values are the curves 1211 and 1212 in FIG. is there. Therefore, if the image processing unit 207 can calculate the contrast value as shown in FIG. 12C, the control unit 201 does not generate camera shake during imaging, and it is not necessary to add a focus position in step S1105. to decide.

  FIG. 12D shows a contrast value when the camera shakes to the near side. In FIG. 12D, the contrast value has a maximum value and is smoothly changing, but only the contrast value at the focus position 1220 is greatly deviated from the other values. Approximate curves of contrast values other than the focus position 1220 are indicated by curves 1221 and 1222. It can be seen that the contrast value 1223 of the attention point 1201 is higher than the value at the focus position 1220 of the curve 1221, and the contrast value 1224 of the attention point 1202 is lower than the value at the focus position 1220 of the curve 1222. That is, the contrast value of the attention point of the image captured at the focus position 1220 is closer to the value of the nearer focus position on the curves 1221 and 1222. Thereby, the control part 201 can judge that it shakes to the near side.

  FIG. 12E shows the contrast value when the camera shakes toward the infinity side. In FIG. 12E, as in FIG. 12D, only the contrast value at the focus position 1230 is greatly deviated from the other values. Approximate curves of contrast values other than the focus position 1220 are indicated by curves 1231 and 1232. Specifically, it can be seen that the contrast value 1233 of the attention point 1201 is lower than the value at the focus position 1230 of the curve 1231, and the contrast value 1234 of the attention point 1202 is higher than the value at the focus position 1230 of the curve 1232. That is, the contrast value of the attention point of the image captured at the focus position 1230 is closer to the focus position value on the infinity side on the curves 1231 and 1232. As a result, the control unit 201 can determine that the camera shakes to infinity.

  When the image processing unit 207 sets as many attention points as possible in step S1101, the accuracy of the determination of the direction of camera shake can be improved. Further, the image processing unit 207 does not use a point of interest having a contrast value that cannot calculate an approximate curve for the determination of addition of the focus position in step S1105.

  When camera shake occurs, an image usable for composition could not be captured at the focus position 1220 or 1230, and therefore the control unit 201 determines in step S1105 that the focus position needs to be added. In step S1106, a focus position is added, and in step S1107, the imaging unit 204 captures an image at the added focus position. The focus position to be added is preferably located at the focus position 1220 or 1230 described above, but may be any position near the focus position 1220 or 1230 as long as an unfocused section does not occur.

  On the other hand, when the control unit 201 determines in step S1106 that the addition of the focus position is not necessary, the process directly proceeds to step S1108. In step S1108, the image processing unit 207 performs composition processing.

  According to the third embodiment, when synthesizing using a plurality of images picked up by changing the focus position, the focus position of the next image pickup can be determined without directly calculating the shift amount and direction due to camera shake. A composite image that can be adjusted to reduce blur can be generated.

(Other embodiments)
The above embodiment has been described based on the implementation with a digital camera, but is not limited to a digital camera. For example, it may be implemented by a portable device with a built-in image sensor or a network camera capable of capturing an image.

  Note that the present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus execute the program. It can also be realized by a process of reading and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

200 Digital Camera 201 Control Unit 202 Drive Unit 203 Optical System 204 Imaging Unit 205 ROM
206 RAM
207 Image processing unit 208 Display unit 209 Built-in memory 210 Operation unit

Claims (11)

Imaging means;
Setting means for setting a plurality of target focus positions;
A focus position calculation means,
The imaging means captures a plurality of images according to the target focus position,
The calculating means calculates a focus position with respect to a subject when each of the plurality of images is captured;
The setting device resets at least a part of the target focus position according to a comparison result between the target focus position and the calculated focus position.   The focus position calculated when the first image is captured and the focus position calculated when the second image is captured in the first image and the second image sequentially captured by the imaging unit. The setting means resets a part of a target focus position for an image to be captured later when the difference between and a predetermined threshold value is larger than a predetermined threshold value. Imaging device.   The imaging apparatus according to claim 1, further comprising a combining unit that combines the plurality of images.   The imaging apparatus according to claim 3, wherein at least some of the plurality of images overlap at least part of the angle of view.   5. The imaging apparatus according to claim 3, wherein the synthesizing unit generates the synthesized image using a focused area of each of the plurality of images.   The imaging apparatus according to claim 1, wherein the imaging unit includes a plurality of photoelectric conversion units for one microlens.   The imaging apparatus according to claim 6, wherein the calculation unit calculates a focus position with respect to a subject when each of the plurality of photoelectric conversion units is imaged based on outputs of the plurality of photoelectric conversion units. Contrast value detection means for detecting the contrast value of the image,
The imaging according to any one of claims 1 to 7, wherein the calculating means calculates a focus position with respect to a subject when each of the plurality of images is captured based on the contrast value. apparatus. Having a sensor for focus detection,
9. The calculation unit according to claim 1, wherein the calculation unit calculates a focus position with respect to a subject when each of the plurality of images is captured based on the focus detection sensor. 10. The imaging device described. Imaging step;
A setting step for setting a plurality of target focus positions;
A focus position calculating step, and
In the imaging step, a plurality of images corresponding to the target focus position are captured,
In the calculating step, a focus position with respect to the subject when each of the plurality of images is captured is calculated;
In the setting step, at least a part of the target focus position is reset according to a comparison result between the target focus position and the calculated focus position. A program for causing a computer of an imaging apparatus to operate,
Imaging step;
A setting step for setting a plurality of target focus positions;
And a step of calculating a focus position,
In the imaging step, a plurality of images corresponding to the target focus position are captured,
In the calculating step, a focus position with respect to the subject when each of the plurality of images is captured is calculated;
In the setting step, at least a part of the target focus position is reset according to a comparison result between the target focus position and the calculated focus position.

Images