JP2010277033A - Imaging apparatus and shake correction method - Google Patents

Abstract

Even when optical deterioration occurs due to shake correction, a captured image in which the optical deterioration is corrected can be obtained.
An image sensor 21 photoelectrically converts an optical image formed by a lens unit 11 with an image sensor to generate an image signal of a captured image. The shake detection unit 42 detects shake and generates a shake detection signal indicating the detection result. The driver 12 displaces the position of the optical image formed on the imaging surface of the imaging device 11 on the imaging surface by displacing the relative positional relationship between the lens unit 11 and the imaging device 21 with respect to the optical axis. Let The control unit 50 displaces the relative positional relationship between the lens unit 11 and the image sensor 21 according to the shake detection signal, thereby causing the shake of the optical image on the imaging surface caused by the shake detected by the shake detection unit 42. Correct. The optical degradation correction unit 25 corrects optical degradation for a captured image that has undergone optical degradation due to shake correction using a captured image with less optical degradation than the captured image.
[Selection] Figure 1

Description

  The present invention relates to an imaging apparatus and a shake correction method. Specifically, even if shake correction that causes optical deterioration is performed, a captured image in which the optical deterioration is corrected can be obtained.

  In recent years, downsizing and weight reduction of image pickup apparatuses and high magnification of optical zooms have progressed, and their usability has been greatly improved. However, as a result of the reduction in size and weight, even a user who is not familiar with the handling of the imaging apparatus can easily operate, so there are many opportunities for imaging without considering camera shake. In addition, when imaging is performed at a high magnification, the influence of camera shake appears remarkably in the captured image, and a stable image cannot be obtained.

  For this reason, an imaging apparatus equipped with a camera shake correction apparatus that can prevent the influence of camera shake has been put into practical use. For example, the invention of Patent Document 1 proposes a method of obtaining a stable image by correcting camera shake by a user by moving a correction lens in two directions orthogonal to the optical axis. In shake correction, a method of moving the image sensor in a direction orthogonal to the optical axis according to the shake is also used.

  In such shake correction, the correction angle of shake correction is limited so that the degradation of the image quality of the captured image is not noticeable due to the effect of optical deterioration. For example, the resolution of the captured image depends on the MTF (Modulation Transfer Function) characteristic of the lens. Also, due to the characteristics of the lens, the MTF deteriorates on the edge side of the screen from the center of the image. FIG. 23 illustrates the relationship between shake correction and MTF degradation. For example, when there is no shake and the correction angle for shake correction is “0 °”, as shown in FIG. 23A, the MTF is in the best state at the center of the image. Further, the lens is designed so that the MTF does not deteriorate more than a predetermined ratio (for example, 95%) with respect to the central portion even at the end of the captured image. Further, when the correction angle is increased and the correction lens and the image sensor are moved, the position where the MTF is the best moves from the center of the image as shown in FIG. Therefore, the correction angle limit range is set with respect to the correction angle for shake correction so that the degradation of the MTF can be suppressed within an allowable range, for example.

Japanese Patent Laid-Open No. 4-18515

  By the way, when the correction angle limit range is provided for the correction angle of shake correction, if the shake is large, the correction angle is limited and the shake correction cannot be performed appropriately. For this reason, for example, when imaging is performed while walking, if the shake is large, shake correction is not performed, and a good captured image without shake cannot be obtained. Therefore, if the correction angle is set larger than the correction angle limit range when the shake is large, the shake correction can be performed even when imaging is performed while walking. However, if the correction angle for shake correction is larger than the correction angle limit range, as shown in FIG. 23C, the degradation of the MTF exceeds the allowable range, and the captured image becomes an image with degraded image quality. End up.

  Therefore, an object of the present invention is to provide an image pickup apparatus and a shake correction method capable of obtaining a picked-up image in which the optical deterioration is corrected even when the optical deterioration is caused by the shake correction.

  According to a first aspect of the present invention, an image sensor that photoelectrically converts an optical image formed by a lens unit to generate an image signal of a captured image, and shake detection are performed, and a shake detection signal that indicates a detection result is generated. On the imaging surface of the image sensor that is caused by the shake detected by the shake detection unit by displacing the relative positional relationship between the shake detection unit, the lens unit, and the image sensor according to the shake detection signal. A shake correction control unit that corrects the shake of the optical image and an image that has undergone optical degradation due to the shake correction, and corrects the optical degradation using a captured image that has less optical degradation than the captured image. And an optical degradation correction unit that performs the imaging.

  According to a second aspect of the present invention, a shake detection unit detects a shake and generates a shake detection signal indicating a detection result, and a shake correction control unit uses a relative relationship between the lens unit and the image sensor. Correcting the shake of the optical image on the imaging surface of the image sensor caused by the shake detected by the shake detection unit by displacing the positional relationship according to the shake detection signal; and optical deterioration correction And a step of correcting the optical deterioration using a captured image with less optical deterioration than the captured image, with respect to the captured image in which the optical deterioration is caused by the shake correction.

  In this invention, when correcting the shake of the optical image on the image pickup surface by displacing the relative positional relationship between the lens unit and the image pickup device in accordance with the shake detection signal, the picked-up image is the object of correction of the optical degradation. It is determined whether the image is an image or an image that does not require correction. For example, when the relative positional relationship between the lens unit and the image sensor exceeds the limit range during the exposure period of the captured image, it is determined that the optical deterioration correction target is to be determined. Alternatively, an evaluation value is obtained based on a period and an excess amount in which the relative positional relationship exceeds a preset positional relationship, and when the evaluation value exceeds a threshold value, it is determined that the optical degradation correction target is to be obtained.

  In addition, a correction image is extracted from a captured image that is determined not to require correction of optical deterioration. For example, in the exposure period of the captured image that is the correction target, when the maximum time in which the displacement of the relative positional relationship is the maximum is the first half of the exposure period, no correction is required that is captured before the captured image that is the correction target. A correction image is extracted from the captured image. Further, when the maximum time is the second half of the exposure period, a correction image is extracted from a non-corrected captured image captured after the captured image to be corrected. When a captured image that is not required to be corrected after the captured image that is the correction target or a captured image that is not required to be corrected before or after the captured image that is the correction target cannot be extracted, the correction stored in the correction unnecessary image storage unit A correction image is extracted from an unnecessary captured image.

  For the correction image, for example, a correction area of a captured image determined to be a correction target is determined based on a shake correction operation, and an image of an area corresponding to the correction area is extracted as a correction image. For example, when the time during which the relative positional relationship exceeds the limit range is longer than a preset time during the exposure period of the captured image determined to be the correction target, the correction region is determined from the maximum displacement of the relative positional relationship. A correction area is determined. Further, when the time when the relative positional relationship exceeds the limit range is longer than a preset time, the correction region is determined from the displacement of the relative positional relationship that becomes a predetermined time interval. In this way, the correction image is extracted, and optical deterioration in the captured image determined as the correction target is corrected using the correction image.

  In addition, when a captured image generated by the image sensor in the still image mode is determined as an optical degradation correction target, a captured image is continuously generated, and the generated captured image does not require correction. Is determined, a correction image is extracted from the captured image.

  According to the present invention, the shake of the optical image formed on the imaging surface of the image sensor by the lens unit is corrected by displacing the relative positional relationship between the lens unit and the image sensor in accordance with the shake detection result. The In addition, the optical degradation correction is performed on the captured image in which the optical degradation has occurred due to the shake correction, using the captured image with less optical degradation than the captured image. For this reason, even when optical deterioration occurs due to shake correction, a captured image in which this optical deterioration is corrected can be obtained.

It is a figure which shows the structure of an imaging device. It is a figure which shows the structure of an optical degradation correction | amendment part. It is the figure which illustrated optical degradation with respect to the displacement amount of a correction lens part. It is the figure which illustrated the relationship between a focal distance and the limit value of displacement. It is a flowchart which shows shake correction operation | movement. It is a flowchart which shows determination of a limiting condition. It is a flowchart which shows a pan / tilt determination process. It is the figure which showed the other example of the relationship between a focal distance and a displacement range. It is the figure which illustrated the operation | movement performed by a control part according to the detection signal obtained from the shake detection part. It is a flowchart which shows optical degradation correction | amendment operation | movement. It is a flowchart which shows the 1st operation | movement of a correction | amendment object discrimination | determination process. It is a figure for demonstrating the 1st operation | movement of a correction | amendment object discrimination | determination process. It is a flowchart which shows the 2nd operation | movement of a correction | amendment object discrimination | determination process. It is a figure for demonstrating the 2nd operation | movement of a correction | amendment object discrimination | determination process. It is a flowchart which shows an image extraction process operation. It is a figure for demonstrating an image extraction process operation. It is a flowchart which shows correction processing operation. It is a figure for demonstrating correction | amendment process operation | movement. It is a figure which shows the correction | amendment process operation | movement when the correction angle in shake correction is small. It is a figure which shows the correction | amendment process operation | movement when the correction angle in shake correction is large. It is a figure which shows the 1st degradation correction operation | movement at the time of still image imaging. It is a figure which shows the 2nd degradation correction operation | movement at the time of still image imaging. It is a figure for demonstrating the relationship between shake correction and deterioration of MTF.

Hereinafter, modes for carrying out the invention will be described. In the present invention, the optical image formed by the lens unit is photoelectrically converted by the image sensor to generate an image signal of the captured image. Further, shake detection is performed, and a shake detection signal indicating a detection result is generated. In accordance with the shake detection signal, the relative positional relationship between the lens unit and the image sensor is displaced with respect to the optical axis, thereby correcting the shake of the optical image on the imaging surface caused by the shake. Further, the optical degradation is corrected using a captured image with less optical degradation than the captured image with respect to the captured image that has undergone optical degradation due to shake correction. The description will be given in the following order, and after explaining the shake correction that causes the optical deterioration, the correction of the optical deterioration caused by the shake correction will be described.
1. 1. Configuration of imaging apparatus 2. Configuration of optical deterioration correction unit 3. Shake correction operation 4. Optical degradation correction operation 5. Optical degradation correction operation in moving image capturing Optical degradation correction in still image capture

<1. Configuration of Imaging Device>
FIG. 1 shows the configuration of the imaging apparatus. The imaging apparatus 10 includes a lens unit 11, a driver 12, a position sensor 13, an imaging device 21, a timing signal generation (TG) unit 22, an analog front end (AFE) unit 23, a signal processing unit 24, and a detection unit 31. . Furthermore, the imaging apparatus 10 includes an image output unit 32, a display unit 33, a recording / playback processing unit 34, an operation unit 41, a shake detection unit 42, and a control unit 50.

  The lens unit 11 includes a zoom lens 111 that performs zooming, a focus lens 112 that performs focusing, and a correction lens unit 113 that moves the position of an optical image formed on the imaging surface of the imaging device 21 described later on the imaging surface. ing. Further, the lens unit 11 is provided with a diaphragm mechanism 114 for adjusting the light amount.

  The correction lens unit 113 includes, for example, a correction lens provided so that the optical axis coincides with the optical axis of the imaging optical system, and an actuator that displaces the correction lens in a direction orthogonal to the optical axis of the imaging optical system. It is configured. The correction lens unit having such a configuration displaces the correction lens in a direction orthogonal to the optical axis of the imaging optical system, and moves the position of the optical image formed on the imaging surface on the imaging surface.

  The correction lens unit 113 may use a variable apex angle prism unit. The variable apex angle prism unit is provided with a translucent entrance end plate and an exit end plate on the end face of a bendable cylinder such as a bellows, and a translucent liquid having a desired refractive index is enclosed in the cylinder. Is. When the variable apex angle prism unit is used, one of the entrance end plate and the exit end plate is fixed, and the other is driven by an actuator to form an optical wedge. The correction lens unit having such a configuration moves the position of the optical image formed on the imaging surface on the imaging surface by, for example, displacing the inclination angle of the exit end plate with respect to the incident end plate.

  In this way, the correction lens unit 113 is provided to displace the relative positional relationship between the lens unit 11 and the image sensor 21 with respect to the optical axis. Alternatively, the image sensor 21 may be displaced in a direction orthogonal to the optical axis so that the relative positional relationship between the lens unit 11 and the image sensor 21 is displaced with respect to the optical axis. In this case, an actuator that displaces the image sensor 21 in a direction orthogonal to the optical axis is provided.

  The driver 12 drives actuators of the zoom lens 111, the focus lens 112, and the correction lens unit 113 based on a lens control signal from the control unit 50 described later. The driver 12 drives the diaphragm mechanism 114 based on the diaphragm control signal from the control unit 50. When the image sensor 21 is displaced in a direction orthogonal to the optical axis, the driver 12 drives the actuator of the image sensor 21 based on a control signal from the control unit 50.

  The position sensor 13 detects the lens positions of the zoom lens 111, the focus lens 112 and the correction lens unit 113 and the set position of the diaphragm mechanism 114 and supplies a position signal to the control unit 50. When the image sensor 21 is displaced in a direction orthogonal to the optical axis, the position sensor 13 detects the position of the image sensor 21 and supplies a position signal to the control unit 50.

  For example, an image sensor such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) image sensor is used as the image sensor 21. The imaging device 21 converts the subject image formed on the imaging surface by the lens unit 11 into an electrical signal and outputs the electrical signal to the AFE unit 23.

  The TG unit 22 generates various drive pulses necessary for outputting an electrical signal indicating a captured image by the image sensor 21 and an electronic shutter pulse for controlling the charge accumulation time of the image sensor 21.

  The AFE unit 23 performs noise removal processing such as CDS (Correlated Double Sampling) processing on the electrical signal (image signal) output from the image sensor 21, and AGC (Automatic Gain Control) using the imaging signal as a desired signal level. Process. Further, the AFE unit 23 converts an analog image pickup signal subjected to noise removal processing and gain control into a digital signal and outputs the digital signal to the signal processing unit 24.

  The signal processing unit 24 performs camera signal preprocessing, camera signal processing, optical deterioration correction processing, resolution conversion processing, compression / decompression processing, and the like.

  In the camera signal preprocessing, the image signal supplied from the AFE unit 23 is subjected to a defect correction process for correcting a signal of a defective pixel in the image sensor 21, a shading correction process for correcting a peripheral light loss of the lens, and the like. In camera signal processing, processing such as white balance adjustment and brightness correction is performed. In addition, a digital camera or the like may be configured to obtain red, green, and blue signals with a single image sensor by providing a color filter array in front of the image sensor. is there. In such a case, in the camera signal processing, demosaic processing is performed, and a signal of a color that is missing in each pixel is generated by interpolation or the like using signals of surrounding pixels.

  In the optical deterioration correction process, optical deterioration is corrected so that a captured image with good image quality can be obtained even if the shake to be corrected is large. The optical deterioration correction process is performed by the optical deterioration correction unit 25, and the configuration of the optical deterioration correction unit 25 will be described later. In the resolution conversion process, the image signal that has been subjected to optical deterioration correction or the image signal that has been subjected to decompression decoding is converted to a predetermined resolution.

  In the compression / decompression process, the image signal after the optical deterioration correction process or the image signal subjected to the resolution conversion process is compression-encoded to generate, for example, a JPEG encoded signal. In the compression / decompression process, a JPEG encoded signal is decompressed and decoded. Note that the compression / decompression process may be performed by compressing and encoding the image signal of a still image by a method different from the JPEG method. In the compression / decompression process, a moving image signal may be compressed and encoded by a moving image compression method.

  The detection unit 31 detects the brightness level and focus state of the captured image using the imaging signal supplied to the signal processing unit 24, generates a detection signal indicating the brightness level and focus state, and generates the detection signal. To supply.

  The image output unit 32 converts the image signal processed by the signal processing unit 24 into an image signal having a format corresponding to an external device connected to the imaging apparatus 10 and outputs the image signal.

  The display unit 33 displays an image captured by the imaging apparatus 10 and a captured image reproduced by the recording / reproduction processing unit 34. The display unit 33 also displays a menu for setting the imaging device 10 and the like.

  In the recording / reproducing processor 34, for example, a recording medium such as a flash memory, an optical disk, or a magnetic tape is used. The recording / playback processing unit 34 records the image signal or the encoded signal of the captured image output from the signal processing unit 24 on a recording medium. The recording / playback processing unit 34 reads the image signal recorded on the recording medium and supplies the image signal to the image output unit 32 and the display unit 33, and reads the encoded signal recorded on the recording medium to perform signal processing. Processing to be supplied to the unit 24 is performed. Note that the recording / reproduction processing unit 34 is not limited to the configuration in which the recording medium is detachable. For example, a hard disk device or the like may be incorporated as the recording / playback processing unit 34.

  The operation unit 41 includes operation buttons and a touch panel provided on the screen of the display unit 33. The operation unit 41 generates an operation signal corresponding to a user operation and supplies the operation signal to the control unit 50.

  The shake detection unit 42 is configured using a shake detection sensor, for example, a gyro sensor, that detects the shake of the imaging apparatus 10. The shake detection sensor includes a yawing angular velocity sensor that detects, for example, an angular velocity according to shake in the yawing direction, and a pitching angular velocity sensor that detects, for example, an angular velocity according to shake in the pitching direction. The shake detection signal output from the yawing angular velocity sensor or the pitching angular velocity sensor is, for example, the reference value VL0 when the angular velocity is not given. Also, when rotating in one direction (forward direction), the signal level of the shake detection signal becomes higher than the reference value VL0 according to the angular velocity, and when rotating in the other direction (reverse direction), the shake depends on the angular velocity. The signal level of the detection signal is lower than the reference value VL0.

  The shake detection unit 42 is provided with a processing circuit that performs signal processing of the shake detection signal. In the processing circuit, for example, the shake detection signal is filtered, and unnecessary signal components such as a noise component, a frequency component higher than the signal component of the angular velocity value, and a resonance frequency component are removed. Further, the processing circuit performs correction of drift caused by temperature change and time change, processing to convert the shake detection signal into a digital signal, and supply it to the control unit 50. When the shake detection signal from the shake detection unit 42 is output as an analog signal, the control unit 50 may be configured to convert the shake detection signal into a digital signal and use it.

  Note that the shake detection unit 42 is not limited to a configuration using an angular velocity sensor. For example, shake detection may be performed using an acceleration sensor or the like. When an acceleration sensor is used, the speed can be calculated by integrating the output of the acceleration sensor. Furthermore, since the moving distance can be calculated by integrating the speed, the magnitude of the shake can be determined based on the output of the acceleration sensor.

  The control unit 50 includes a CPU (Central Processing Unit), a memory, and the like. The memory stores programs executed by the CPU and various data. As this memory, for example, a nonvolatile memory such as an EEPROM (Electronically Erasable and Programmable ROM) or a flash memory is used. The CPU of the control unit 50 executes a program stored in the memory, and the operation of the imaging device 10 corresponds to a user operation based on various data stored in the memory and operation signals supplied from the operation unit 41. Each part is controlled so that it becomes operation. For example, when the shutter operation is performed in the still image mode, the control unit 50 controls the operation of the TG unit 22 and the like, and records and reproduces the encoded signal of the still image captured at a desired shutter speed. Recorded on a recording medium. When a recording start operation is performed in the moving image mode, a moving image encoded signal or the like is recorded on the recording medium of the recording / reproducing processor 34.

  Further, the control unit 50 generates a lens control signal and an aperture control signal based on the position signal supplied from the position sensor 13 and the detection signal supplied from the detection unit 31 and supplies the lens control signal and the aperture control signal to the driver 12. Accordingly, the focus lens 112 and the aperture mechanism 114 are driven by the driver 12 so that a captured image with a desired brightness can be obtained. Further, when the user performs a zoom operation, the control unit 50 generates a lens control signal and supplies it to the driver 12 to drive the zoom lens 111 so that a captured image with a desired zoom ratio is obtained.

  In the imaging apparatus 10 configured as described above, the control unit 50 generates a control signal based on the shake detection signal from the shake detection unit 42 or the position signal supplied from the position sensor 13 and supplies the control signal to the driver 12. The driver 12 displaces the relative positional relationship between the lens unit 11 and the imaging element 21 with respect to the optical axis based on the control signal, and the optical image on the imaging surface caused by the shake detected by the shake detection unit 42 is detected. Correct the shake.

  The optical degradation correction unit of the signal processing unit 24 performs optical degradation correction processing, and corrects optical degradation so that a captured image with good image quality can be obtained even if the shake to be corrected is large.

<2. Configuration of optical deterioration correction unit>
FIG. 2 shows a configuration of an optical deterioration correction unit 25 that is provided in the signal processing unit 24 and corrects optical deterioration. The optical degradation correction unit 25 includes a memory unit 251, a correction target determination unit 255, a correction unnecessary image extraction unit 256, a correction unnecessary image storage unit 257, and a correction processing unit 258.

  The memory unit 251 includes, for example, five memories 251n-2, 251n-1, 251n, 251n + 1, and 251n + 2. For example, FIFO (First-In / First-Out) is used for the five memories, and image signals are written and read by a first-in first-out method, and the image signals are temporarily stored.

  The memory 251n + 2 temporarily stores the image signal supplied to the optical deterioration correction unit 25. The memory 251n + 2 reads out the stored image signal and outputs it to the memory 251n + 1 and the correction unnecessary image extraction unit 256. The memory 251n + 1 temporarily stores the image signal supplied from the memory 251n + 2. In addition, the memory 251n + 1 reads out the stored image signal and outputs it to the memory 251n and the correction unnecessary image extraction unit 256. The memory 251n temporarily stores the image signal supplied from the memory 251n + 1 or the correction processing unit 258. The memory 251n reads out the stored image signal and outputs it to the memory 251n-1 or the correction processing unit 258. The memory 251n-1 temporarily stores the image signal supplied from the memory 251n. The memory 251n-1 reads out the stored image signal and outputs it to the memory 251n-2 and the correction unnecessary image extraction unit 256. The memory 251n-2 temporarily stores the image signal supplied from the memory 251n-1. Also, the memory 251n-2 reads out the stored image signal and outputs it to the correction unnecessary image extraction unit 256. Further, the memory 251n-2 reads out the stored image signal and outputs it from the optical deterioration correction unit 25.

  As described above, when the memory unit 251 is configured by the five memories 251n-2, 251n-1, 251n, 251n + 1, and 251n + 2, the memory unit 251 is two frames before the frame stored in the memory 251n. The image signals from the second frame to the second frame are temporarily stored.

  The correction target determination unit 255 determines whether the captured image is an image to be corrected for optical degradation or an image that does not require correction based on the relative positional relationship between the lens unit 11 and the image sensor 21. For example, the displacement amount determined based on the correction angle when shake correction is performed or the position signal from the position sensor 13 is information indicating the relative positional relationship between the lens unit 11 and the image sensor 21. Therefore, the correction target determination unit 255 detects the relative positional relationship between the lens unit 11 and the image sensor 21 based on these pieces of information, and the captured image is corrected for optical deterioration based on the detected relative positional relationship. It is discriminated whether the image is a non-correction image or an image that does not require correction.

  In addition, when the correction target determination unit 255 determines that the captured image is the correction target, the maximum displacement of the relative positional relationship during the exposure period and the time information when the maximum displacement is reached are corrected in relation to the captured image. Record it in the information table as information. Further, in the information table, a time index indicating the time during the exposure period is recorded in relation to the relative positional relation so that the change in the relative positional relation during the exposure period can be grasped. The correction target determination unit 255 sequentially deletes unnecessary information from the information table. For example, information about the captured image after the optical deterioration correction process is deleted from the information table.

  When the correction target determination unit 255 determines that the captured image stored in the memory 251n is a correction target, the correction unnecessary image extraction unit 256 selects a captured image with less optical deterioration than the captured image stored in the memory 251n. Extracted from the memory unit 251 and output to the correction processing unit 258.

  Further, the correction unnecessary image extraction unit 256 causes the correction unnecessary image storage unit 257 to store the image signal of the captured image determined to be correction unnecessary by the correction target determination unit 255. The correction unnecessary image extraction unit 256 sequentially updates the image signal stored in the correction unnecessary image storage unit 257 with the image signal of a new captured image determined to be correction unnecessary.

  Furthermore, when the unneeded image extraction unit 256 cannot extract from the memory unit 251 a captured image with less optical deterioration than the captured image stored in the memory 251n, the image of the captured image stored in the unneeded image storage unit 257 is extracted. The signal is read and output to the correction processing unit 258.

  The correction processing unit 258 extracts a correction image from the correction-unnecessary captured image extracted by the correction-unnecessary image extraction unit 256, and uses the correction image to correct the optical image in the correction target captured image stored in the memory 251n. Compensate for deterioration.

<3. Shake correction operation>
In the imaging apparatus, it is desirable that the optical degradation is within an allowable range over the entire zoom region even if the correction lens unit and the imaging element are displaced to the maximum displacement range. However, it is difficult to suppress optical degradation within an allowable range due to various restrictions. On the other hand, since the necessity of shake correction is more important on the tele side than on the wide side, it is common to reduce optical degradation on the tele side in the design of the imaging optical system. Optical degradation increases.

  Next, a shake correction operation when the correction lens unit 113 is driven to perform shake correction will be described. FIG. 3 is a diagram illustrating optical deterioration with respect to the displacement amount of the correction lens unit. In FIG. 3, the horizontal axis indicates the absolute value Ad of the displacement amount, and the vertical axis indicates the image deterioration amount Ae.

  Here, the relationship between the displacement amount at the telephoto end and the image deterioration amount is, for example, a characteristic indicated by a solid line. Further, the relationship between the displacement amount at the wide end and the image deterioration amount has a characteristic indicated by a broken line, for example. That is, in order to suppress the image deterioration amount Ae to “RL”, it is necessary to limit the absolute value of the displacement amount at the tele end to the displacement range of “dt”. In addition, the absolute value of the displacement amount at the wide end needs to be limited to a displacement range of “dw” that is narrower than “dt” at the tele end.

  FIG. 4 illustrates the relationship between the focal length f and the displacement limit value Adm when the image degradation amount Ae is suppressed to “RL”. In FIG. 4, the horizontal axis represents the focal length f, and the vertical axis represents the displacement limit value Adm.

  As shown in FIG. 4, the displacement limit value Adm with respect to the focal distance f is “dt” when the zoom position is on the telephoto side. Further, when the focal length is shortened and moved to the wide side, the displacement limit value Adm decreases. Further, when the focal length becomes shorter and the lens moves to the wide end, the displacement limit value Adm becomes “dw”. Therefore, in order to suppress the image degradation amount Ae to “RL”, it is necessary to set the maximum displacement range of the correction lens unit 113 to “dt” and limit the displacement range according to the zoom position as shown by a solid line in FIG. is there. In addition, when the displacement range is limited, for example, when the shake is large when the zoom position is on the wide side, the displacement range is limited and shake correction cannot be performed. Therefore, when the shake is large, for example, as shown by the broken line in FIG. 4, the restriction on the displacement range is canceled and the displacement range is set to the maximum displacement range, thereby improving the shake correction performance on the wide side and the shake correction. To obtain a picked-up image.

  FIG. 5 is a flowchart showing the shake correction operation. Note that the shake detection unit 42 is configured using an angular velocity sensor, and the following description will be made assuming that a shake detection signal that is angular velocity data is converted into a digital signal and supplied to the control unit 50.

  In step ST1, the control unit 50 performs high-pass filter processing of the shake detection signal. The controller 50 removes the DC component from the shake detection signal by performing high-pass filter processing, and proceeds to step ST2.

  In step ST2, the control unit 50 performs an integration process of the shake detection signal. The controller 50 integrates the shake signal, converts the angular velocity into an angle, and proceeds to step ST3.

  In step ST3, the control unit 50 calculates a correction amount. The controller 50 determines the focal length based on the position signal supplied from the position sensor 13. Further, the control unit 50 calculates the correction amount of the correction lens unit 113 from the determined focal length and the angle obtained in step ST2, and proceeds to step ST4.

  In step ST4, the control unit 50 determines the limiting condition. FIG. 6 is a flowchart showing determination of the limiting condition. In step ST11, the control unit 50 updates the determination angular velocity data history. In order to determine the limiting condition, the control unit 50 retains the angular velocity data within a predetermined time as the determination angular velocity data history, and proceeds to step ST12.

  In step ST12, the control unit 50 performs pan / tilt determination processing. FIG. 7 is a flowchart showing the pan / tilt determination process. In step ST31, the control unit 50 determines whether the angular velocity data is equal to or higher than a predetermined level. The control unit 50 proceeds to step ST32 when the angular velocity data is equal to or higher than a predetermined level set in advance, and proceeds to step ST34 when it is not equal to or higher than the predetermined level.

  In step ST32, the control unit 50 determines whether or not the duration time is equal to or longer than a predetermined time. The control unit 50 proceeds to step ST33 when the state where the angular velocity data is equal to or higher than the predetermined level continues for a predetermined time or longer, and proceeds to step ST33 when the duration of the state where the angular velocity data is higher than the predetermined level is not equal to or longer than the predetermined time. Proceed to

  In step ST33, the control unit 50 determines that the pan / tilt operation is being performed, and ends the pan / tilt determination process. The control unit 50 determines that the panning operation is performed when the imaging direction is changed to the right direction or the left direction for a predetermined time or more at an angular velocity equal to or higher than a predetermined level. In addition, the control unit 50 determines that the tilting operation is being performed when the imaging direction is changed to an upward direction or a downward direction at an angular velocity equal to or higher than a predetermined level for a predetermined time or more, and ends the process.

  In step ST34, the control unit 50 determines that the pan / tilt operation has not been performed, and ends the pan / tilt determination process. The control unit 50 determines that the panning operation or the tilting operation has not been performed since the imaging direction has not been continuously changed at an angular velocity equal to or higher than a predetermined level for a predetermined time, and ends the processing.

  Returning to FIG. 6, in step ST <b> 13, the control unit 50 determines whether a pan / tilt operation is being performed. When it is determined that the pan / tilt operation is being performed in the pan / tilt determination process of step ST12, the control unit 50 proceeds to step ST14. Further, when it is determined that the pan / tilt operation is not performed in the pan / tilt determination process of step ST12, the control unit 50 proceeds to step ST15.

  In step ST14, the control unit 50 clears the determination angular velocity data history and proceeds to step ST17.

  In step ST15, the control unit 50 determines whether or not a ratio equal to or higher than the threshold value of the angular velocity data is equal to or greater than a predetermined value. The control unit 50 proceeds to step ST16 when the ratio of the angular velocity data equal to or greater than the threshold is equal to or greater than a predetermined value. Moreover, the control part 50 progresses to step ST17, when the ratio more than the threshold value of angular velocity data is not more than predetermined value.

  In step ST16, the control unit 50 sets the displacement range in the shake correction performance priority mode. Here, when the process of step ST13 and step ST15 is performed and the process proceeds to step ST16, the shake is large. Therefore, the control unit 50 sets the displacement range in which the restriction on the displacement range is canceled as the displacement range of the correction lens unit 113 as shown by the broken line in FIG.

  In step ST17, the control unit 50 sets the displacement range in the image quality priority mode. That is, the control unit 50 determines that it is not necessary to prioritize the shake correction performance because the shake is small or the pan / tilt operation is performed. Therefore, the control unit 50 performs an operation that prioritizes image quality so that a good captured image with little optical deterioration can be obtained. Therefore, the limited displacement range as shown by the solid line in FIG. Set the displacement range.

  Returning to FIG. 5, in step ST <b> 5, the control unit 50 performs determination result reflection processing. When the correction amount calculated in step ST3 exceeds the displacement range set by the determination of the restriction condition in step ST4, the control unit 50 limits the correction amount to the set displacement range. Further, when the correction amount calculated in step ST3 does not exceed the displacement range set by the determination of the limiting condition in step ST4, the control unit 50 uses the correction amount without limitation.

  In step ST6, the control unit 50 outputs the image stabilization target position. The control unit 50 sets the position moved by the correction amount determined in step ST5 in the direction of correcting the shake of the optical image formed on the imaging surface as the vibration correction target position of the correction lens in the correction lens unit 113. . Further, the control unit 50 generates a lens control signal so that the correction lens of the correction lens unit 113 is at the image stabilization target position, and outputs the lens control signal to the driver 12. The driver 12 drives the actuator of the correction lens unit 113 based on the lens control signal supplied from the control unit 50 to move the correction lens to the image stabilization target position.

  When such processing is performed, when the control unit 50 detects that the shake is large, the shake correction performance priority mode is set, the restriction on the displacement range is released, and the displacement range is expanded. Therefore, for example, when imaging is performed while walking with the zoom position set to the wide side, an imaging operation in which shake is corrected can be performed.

  When imaging while walking, if the zoom position is set to the tele side, it is difficult to hold the imaging apparatus so that a desired subject is always at the center position of the screen. For this reason, when taking an image while walking, the zoom position is generally set to the wide side. If the displacement range of the correction lens unit 113 is switched from the solid line characteristic shown in FIG. 4 to the broken line characteristic when the shake is large, the displacement range of the correction lens unit 113 is widened on the wide side. Therefore, when imaging is performed, an imaging operation in which shake is corrected can be performed.

  In order to improve the shake correction performance even when the zoom position is on the tele side, the maximum displacement range is set to “du” wider than “dt” as shown in FIG. As shown by a broken line in FIG. 8, the displacement range of the correction lens unit 113 may be switched so as to expand the displacement range from the wide end to the tele end.

  Further, when the control unit 50 does not detect that the shake is large, the image quality priority mode is set, and the displacement range is limited according to the zoom position. Therefore, for example, when imaging is performed with the zoom position set to the wide side, it is possible to suppress deterioration in image quality due to shake correction. Further, when the control unit 50 detects that a panning operation or a tilting operation is being performed, the image quality priority mode is set, and the displacement range is limited according to the zoom position. Therefore, it is possible to prevent the shake from being corrected during the panning operation or the tilting operation and the subject from moving smoothly according to the panning operation or the tilting operation.

  In other words, since the displacement range of the correction lens unit 113 is switched based on the magnitude of the shake, it is possible to perform shake correction that achieves both mitigation of image degradation and improvement of shake correction performance. Further, even if a panning operation or a tilting operation is performed, it is possible to prevent an adverse effect due to shake correction.

  FIG. 9 is a diagram illustrating an operation performed by the control unit 50 in accordance with the shake detection signal obtained from the shake detection unit.

  FIG. 9A shows a change in the signal level of the shake detection signal in correspondence with the passage of time. In this figure, the signal level of the shake detection signal when no shake is detected is the reference value “VL0” as described above. The threshold level “VLth1” is a threshold used for switching the displacement range of the correction lens unit 113 to the displacement range of the shake correction performance priority mode or the image quality priority mode. The threshold level “VLth2” is a threshold for determining whether a panning operation or a tilting operation is being performed.

  FIG. 9B shows a determination result of whether or not a panning operation or a tilting operation is being performed. When it is determined that the panning operation or the tilting operation is being performed, “YES” is indicated. When it is not determined that the panning operation or the tilting operation is being performed, “NO” is indicated.

  FIG. 9C shows a ratio at which the shake detection signal exceeds the threshold level “VLth1”. In this figure, the threshold level “VRth” is a threshold used for switching the displacement range of the correction lens unit 113 to the displacement range of the shake correction performance priority mode or the image quality priority mode.

  FIG. 9D shows the displacement range of the correction lens unit 113, and shows whether the displacement range is the image quality priority mode or the shake correction performance priority mode.

  For example, the control unit 50 starts the operation by setting “NO” as a determination result of whether or not the panning operation or the tilting operation is performed, and setting the displacement range of the correction lens unit 113 as the displacement range of the image quality priority mode.

  For example, when the user starts walking at time t1 and the shake of the imaging device 10 increases, the shake detection signal increases in the amount of change in the signal level with respect to the reference value “VL0” as shown in FIG. Further, when the ratio of the shake detection signal exceeding the threshold level “VLth1” increases due to the large shake, the ratio of the shake detection signal exceeding the threshold level “VLth1” increases as shown in FIG. To do.

  When the ratio exceeds the threshold level “VRth” at time t2, as shown in FIG. 9B, the determination result as to whether or not the panning operation or the tilting operation is being performed is “NO”. ing. Therefore, since the ratio exceeds the threshold level “VRth” when the panning operation and the tilting operation are not performed, the control unit 50 moves the displacement of the correction lens unit 113 as illustrated in FIG. Switch the range to the displacement range of shake correction performance priority mode.

  The displacement range in the shake correction performance priority mode is a displacement range in which the restriction is released as indicated by a broken line in FIG. Therefore, since the shake correction range is widened compared to the case where the displacement range is set in the image quality priority mode, even if the shake of the optical image formed on the image pickup surface of the image pickup device increases due to, for example, shake during walking. It becomes possible to correct. For this reason, it is possible to perform an imaging operation in which shake is corrected while walking.

  Thereafter, at time t3, for example, the user stops walking and starts a panning operation or a tilting operation. When the panning operation or the tilting operation is performed and the signal level of the shake detection signal exceeds the threshold level “VLth2” as shown in FIG. 9A, the control unit 50 starts measuring the duration time. The controller 50 starts measurement for the duration of the state in which the signal level of the shake detection signal exceeds the threshold level “VLth2” from the time t4 when the signal level exceeds the threshold level “VLth2”.

  When the duration time becomes equal to or longer than a predetermined time at time t5, the control unit 50 determines that the panning operation or the tilting operation is performed, and as shown in FIG. The determination result as to whether or not the tilting operation is being performed is “YES”. Furthermore, the control unit 50 clears the determination angular velocity data history. When the determination angular velocity data history is cleared, the ratio at which the shake detection signal exceeds the threshold level “VLth1” is “0” as shown in FIG. Accordingly, since the ratio is equal to or lower than the threshold level “VRth”, the control unit 50 switches the displacement range of the correction lens unit 113 to the displacement range of the image quality priority mode, as shown in FIG.

  After that, when the panning operation and the tilting operation are finished, as shown in FIG. 9A, the signal level of the shake detection signal becomes, for example, a threshold level “VLth2” or less at time t6. At this time, the control unit 50 determines that the panning operation or the tilting operation is finished, and determines whether or not the panning operation or the tilting operation is performed as shown in FIG. “NO”.

  As described above, according to the shake detection signal, the magnitude of the shake, whether the panning operation or the tilting operation is performed are determined, and the displacement range of the correction lens unit 113 is switched based on the determination result. Is done. Therefore, shake correction performance can be improved.

  In FIG. 9A, the signal level of the shake detection signal is compared with the threshold levels “VLth1” and “VLth2”, but the absolute value of the amount of change from the reference value “VL0” of the shake detection signal is calculated. The absolute value may be compared with a threshold value. As described above, by using the absolute value, it is possible to determine the magnitude of the shake, whether or not the panning operation or the tilting operation is performed regardless of the direction of movement.

  The displacement range may be switched to the displacement range of the shake correction performance priority mode when the shake is large, and is limited to the case where the shake detection signal is performed based on the ratio exceeding the threshold level “VLth1”. is not. For example, when the number of times that the signal level of the shake detection signal exceeds the threshold value exceeds a predetermined number of times within the unit time, the displacement range of the shake correction performance priority mode may be switched.

  Further, the shake correction operation mode may be selected by the user. For example, an off mode, a standard mode, and an active mode are provided as operation modes. The off mode is an operation mode in which no shake correction is performed. The standard mode is an operation mode in which shake correction is performed by limiting the displacement range of the correction lens unit to a range narrower than the maximum displacement range regardless of the shake. The active mode is an operation mode in which shake correction is performed by switching the displacement range of the correction lens unit according to shake. In this way, for example, when imaging using a tripod or the like, the user can perform imaging by stopping the shake correction function by selecting the off mode. In addition, when performing imaging while holding the imaging apparatus with a hand in a stationary state, the user can perform imaging that prioritizes image quality by correcting camera shake and the like by selecting the standard mode. Furthermore, when taking an image in a state where a shake larger than the hand shake occurs, for example, when taking an image while walking or taking an image in a moving car, the user may select the active mode. When the active mode is selected, shake correction with improved shake correction performance is performed. Therefore, shake that cannot be corrected in the standard mode can be corrected.

<4. Optical degradation correction operation>
When the shake correction is performed as described above, when the control unit 50 detects that the shake is large, the shake correction performance priority mode is set, the restriction on the displacement range is released, and the displacement range is expanded. When the user selects the active mode, the restriction on the displacement range is released and the displacement range is expanded. As described above, when the displacement range is enlarged, the relative positional relationship between the lens unit 11 and the image pickup device 21 may exceed the optical deterioration allowable range, resulting in optical deterioration of the captured image. For example, the degradation of the MTF exceeds an allowable range, and the captured image becomes an image with degraded image quality. Therefore, when shake correction is performed such that the optical deterioration exceeds the allowable range, for example, when the operation mode is set to the shake correction performance priority mode or the active mode, the optical deterioration correction operation is performed.

  FIG. 10 is a flowchart showing the optical deterioration correction operation. In step ST41, the optical deterioration correction unit 25 performs a correction target determination process. Based on the relative positional relationship between the lens unit 11 and the image sensor 21, the optical degradation correction unit 25 determines a captured image to be corrected that needs to be corrected for optical degradation, and proceeds to step ST42.

  In step ST42, the optical deterioration correction unit 25 performs a correction-unnecessary image extraction process. The optical deterioration correction unit 25 extracts a correction-free captured image that has less optical deterioration than the correction target captured image, and proceeds to step ST43.

  In step ST43, the optical deterioration correction unit 25 performs a correction process. The optical degradation correction unit 25 extracts, from the image signal extracted in step ST42, an image signal of a correction image corresponding to an image region (image degradation region) in which optical degradation is noticeable in the captured image determined as the correction target. . In addition, the optical degradation correction unit 25 corrects optical degradation in the captured image determined to be the correction target using the extracted image signal of the correction image, and generates an image signal in which the optical degradation is corrected. Return to ST41.

<5. Optical degradation correction operation for video imaging>
Next, an optical deterioration correction operation in moving image capturing will be described. In the following description, for example, a correction angle for shake correction is used as information indicating the relative positional relationship between the lens unit 11 and the image sensor 21.

[Correction target discrimination processing operation]
FIG. 11 is a flowchart showing a first operation of the correction target discrimination process. In step ST51, the correction target determination unit 255 determines whether it is a frame start. The correction target determination unit 255 returns to step ST51 when the frame is not started, and proceeds to step ST52 when the frame is started.

  In step ST52, the correction target determination unit 255 performs initialization. The correction target determination unit 255 initializes parameters and the like used in the correction target determination process. For example, the correction target determination unit 255 clears a correction flag indicating whether the frame is a correction target frame or a frame that does not require correction. Further, the correction target determination unit 255 corrects a maximum correction angle value that indicates the maximum value of the shake correction angle within the exposure period of the captured image, and a correction angle maximum value time that indicates the time when the correction angle reaches the maximum value within the exposure period Is advanced to step ST53.

  In step ST53, the correction target determining unit 255 detects a correction angle. The correction target determination unit 255 detects a correction angle for shake correction according to the position signal from the position sensor 13 and the shake detection signal from the shake detection unit 42, and proceeds to step ST54. Note that the correction target determination unit 255 may acquire a correction angle for shake correction from the control unit 50.

  In step ST54, the correction target determination unit 255 determines whether or not the correction angle exceeds the threshold angle. The correction target determination unit 255 proceeds to step ST55 when the correction angle exceeds the threshold angle, and proceeds to step ST60 when the correction angle does not exceed the threshold angle. The threshold angle is set in advance to an angle at which optical deterioration does not exceed the allowable range, for example.

  In step ST55, the correction target determination unit 255 sets a correction flag. Since the correction angle exceeds the threshold angle, the correction target determination unit 255 determines the captured image as a correction target, sets a correction flag, and proceeds to step ST56.

  In step ST56, the correction target determining unit 255 stores the frame number. The correction target determination unit 255 stores the frame number of the frame whose correction angle exceeds the threshold angle, that is, the frame for which the correction flag is set, and proceeds to step ST57.

  In step ST57, the correction target determination unit 255 determines whether the correction angle is larger than the maximum correction angle value. The correction target determination unit 255 proceeds to step ST58 when the correction angle is larger than the maximum correction angle value, and proceeds to step ST60 when the correction angle is not larger than the maximum correction angle value.

  In step ST58, the correction target determination unit 255 sets a correction angle maximum value. The correction target discriminating unit 255 sets the correction angle detected in step ST53 to the maximum correction angle value, and proceeds to step ST59.

  In step ST59, the correction target determination unit 255 sets a correction angle maximum value time. The correction target determination unit 255 sets the current time within the exposure period to the correction angle maximum value time, and proceeds to step ST60.

  In step ST60, the correction target determination unit 255 determines whether or not it is the end of the frame. The correction target determination unit 255 returns to step ST53 when the frame has not ended, and proceeds to step ST61 when the frame has ended.

  In step ST61, the correction target determination unit 255 records the correction information in the information table. The correction target determination unit 255 records the correction flag, frame number, correction angle maximum value, and correction angle maximum value time as correction information in the information table. Further, when the correction target determination unit 255 determines that the frame is a frame to be corrected, the correction target determination unit 255 associates the detected correction angle with the time index and records it as correction information in the information table. The correction information may be recorded in the information table when the correction angle maximum value or the correction angle maximum value time is updated, regardless of whether the frame is a correction handling frame or a frame that does not require correction. Further, when the correction angle is detected, the time index and the correction angle may be related and sequentially recorded in the information table.

  When such processing is performed for each frame, a frame whose correction angle exceeds the threshold angle can be determined as a frame to be interpolated. Further, for each correction target frame, the maximum correction angle value, the maximum correction angle value time, and the like in the frame can be recorded in the information table. Furthermore, when it is determined that the frame is to be corrected, the detected correction angle can be related to the time index and recorded in the information table as correction information. The information table enables at least the correction information of the frame stored in the memory unit 251 to be recorded. In the information table, the oldest correction information in the information table may be sequentially updated with new information so that the latest information is recorded.

  FIG. 12 is a diagram for explaining a first operation of the correction target determination process. In addition, (A) of FIG. 12 has illustrated the time change of the correction angle at the time of imaging. FIG. 12B illustrates a captured image for three frames obtained by performing the imaging operation. Here, since the captured images for three frames are subjected to shake correction, the images having the same imaging range are obtained. Further, the hatched portion in the captured image shows a simplified image portion in which optical degradation has occurred.

  The correction target determination unit 255 detects the correction angle during the exposure period based on the sensor signal from the position sensor 13 and the shake detection signal from the shake detection unit 42. The correction target determination unit 255 sets this captured image as a correction target when the correction angle exceeds the threshold angle during the exposure period, that is, when the correction angle exceeds the correction angle limit range. Here, frames F1 and F3 shown in FIG. 12 are frames that do not require correction because the correction angle does not exceed the correction angle limit range during the exposure period. The frame F2 is a correction target because the correction angle exceeds the correction angle limit range during the exposure period. The correction target determination unit 255 sets a correction flag for the frame F2. Further, the correction target determination unit 255 records the correction angle maximum value and the correction angle maximum value time in the information table. Further, the correction target determination unit 255 records the correction target frame F2 in the information table by associating the correction index with the time index.

  FIG. 13 is a flowchart illustrating a second operation of the correction target determination process. In step ST71, the correction target determination unit 255 determines whether it is a frame start. The correction target determination unit 255 proceeds to step ST72 when the frame is started, and returns to step ST71 when the frame is not started.

  In step ST72, the correction target determination unit 255 performs initialization. The correction target determination unit 255 initializes parameters and the like used in the correction target determination process. For example, the correction target determination unit 255 clears a correction flag indicating whether the frame is a correction target frame or a frame that does not require correction. Further, the correction target determination unit 255 corrects a maximum correction angle value that indicates the maximum value of the shake correction angle within the exposure period of the captured image, and a correction angle maximum value time that indicates the time when the correction angle reaches the maximum value within the exposure period. To clear. Further, the correction target determination unit 255 clears the correction angle integral value obtained by integrating the correction angle, and proceeds to step ST73.

  In step ST73, the correction target determination unit 255 detects a correction angle. The correction target determination unit 255 detects a correction angle for shake correction according to the position signal from the position sensor 13 and the shake detection signal from the shake detection unit 42, and proceeds to step ST74. Note that the correction target determination unit 255 may acquire a correction angle for shake correction from the control unit 50.

  In step ST74, the correction target determination unit 255 determines whether or not the correction angle exceeds the threshold angle. The correction target determination unit 255 proceeds to step ST75 when the correction angle exceeds the threshold angle, and proceeds to step ST83 when the correction angle does not exceed the threshold angle.

  In step ST75, the correction target determination unit 255 adds a portion of the correction angle exceeding the threshold angle to the correction integrated value, and proceeds to step ST76.

  In step ST76, the correction target determination unit 255 stores the correction angle and the current time, and proceeds to step ST77.

  In step ST77, the correction target determination unit 255 determines whether or not the correction integrated value exceeds the integral value threshold. The correction target determination unit 255 proceeds to step ST78 when the corrected integral value exceeds the integral value threshold value, and proceeds to step ST80 when the corrected integral value does not exceed the integral value threshold value.

  In step ST78, the correction target determining unit 255 sets a correction flag. Since the correction integrated value exceeds the integral value threshold, the correction target determination unit 255 determines that the correction target is a correction target, sets a correction flag, and proceeds to step ST79.

  In step ST79, the correction target determination unit 255 stores the frame number. The correction target determination unit 255 stores the frame number of the frame whose correction angle exceeds the threshold angle, that is, the frame for which the correction flag is set, and proceeds to step ST80.

  In step ST80, the correction target determination unit 255 determines whether or not the correction angle is larger than the maximum correction angle value. The correction target determination unit 255 proceeds to step ST81 when the correction angle is larger than the maximum correction angle value, and proceeds to step ST83 when the correction angle is not larger than the maximum correction angle value.

  In step ST81, the correction target determination unit 255 sets a correction angle maximum value. The correction target determination unit 255 sets the correction angle detected in step ST73 to the maximum correction angle value, and proceeds to step ST82.

  In step ST82, the correction target determination unit 255 sets the correction angle maximum value time. The correction target determination unit 255 sets the current time within the exposure period to the correction angle maximum value time, and proceeds to step ST83.

  In step ST83, the correction target determination unit 255 determines whether or not it is the end of the frame. The correction target determination unit 255 returns to step ST73 when the frame has not ended, and proceeds to step ST84 when the frame has ended.

  In step ST84, the correction target determining unit 255 records the correction information in the information table. The correction target determination unit 255 records the correction flag, frame number, correction angle maximum value, and correction angle maximum value time as correction information in the information table. Further, when the correction target determination unit 255 determines that the frame is a correction target frame, the correction target determination unit 255 associates the detected correction angle with the time index and records it as correction information in the information table. The correction information may be recorded in the information table when the correction angle maximum value or the correction angle maximum value time is updated regardless of whether the frame is a correction target frame or a frame that does not require correction. Further, when the correction angle is detected, the time index and the correction angle may be related and sequentially recorded in the information table.

  When such processing is performed for each frame, a frame in which the integral value of the portion exceeding the threshold value of the correction angle exceeds the integral value threshold value can be determined as a frame to be interpolated. Further, for each frame to be interpolated, the correction angle maximum value, the correction angle maximum value time, and the like in the frame can be recorded in the information table as correction information. Further, when it is determined that the frame is a correction handling frame, the detected correction angle can be related to the time index and recorded in the information table as correction information.

  FIG. 14 is a diagram for explaining a second operation of the correction target determination process. FIG. 14A illustrates the change in correction angle with time during imaging. FIG. 14B illustrates a captured image for three frames obtained by performing the imaging operation. Further, the hatched portion in the captured image shows a simplified image portion in which optical degradation has occurred.

  The correction target determination unit 255 detects the correction angle during the exposure period based on the sensor signal from the position sensor 13 and the shake detection signal from the shake detection unit 42. The correction target determination unit 255 adds a portion of the correction angle that exceeds the threshold angle to the correction integral value, and sets a captured image whose correction integration value exceeds the integral value threshold as a correction target. Here, when the corrected integral value obtained by sequentially adding the portions of the correction angle exceeding the threshold angle “θth” within the frame does not exceed the integral value threshold in the frames F11 and F12, this frame is determined as a frame that does not require correction. . In addition, in the frame F13, when the corrected integral value obtained by sequentially adding the portions exceeding the threshold angle “θth” of the correction angle within the frame exceeds the integral value threshold value in the frame F13, this captured image is set as a correction target. The correction target determination unit 255 sets a correction flag for the frame F13 that is a captured image that has been determined as a correction target. Further, the correction target determination unit 255 records the correction angle maximum value and the correction angle maximum value time in the information table. Further, the correction target determination unit 255 records the frame F13 in the information table in association with the correction angle and the time index.

[Image extraction processing operation]
FIG. 15 is a flowchart showing the image extraction processing operation. In step ST91, the correction unnecessary image extraction unit 256 determines whether a correction flag is set for the captured image. The correction unnecessary image extraction unit 256 proceeds to step ST92 when the correction flag is set in the correction information regarding the captured image, and ends the process when the correction flag is not set.

  In step ST92, the unnecessary image extraction unit 256 determines whether or not the correction flags of the previous and subsequent frames are clear. The correction unnecessary image extraction unit 256 searches the correction information of the previous and subsequent frames based on the frame number indicated by the correction information regarding the captured image. Here, in the correction information regarding the previous and subsequent frames, the process proceeds to step ST93 when the correction flag is clear, and when the correction flag for the previous and subsequent frames is set, or when the correction flag is cleared only for one frame, step. Proceed to ST94.

  In step ST93, the correction unnecessary image extracting unit 256 determines whether or not the maximum correction angle time is delayed from the center of the exposure period. The correction unnecessary image extraction unit 256 proceeds to step ST96 when the maximum correction angle time is delayed from the center of the exposure period, and proceeds to step ST97 when the maximum correction angle time is not delayed from the center of the exposure period.

  In step ST94, the correction unnecessary image extraction unit 256 determines whether the correction flags of the previous and subsequent frames are set. The correction unnecessary image extraction unit 256 proceeds to step ST95 when any one of the preceding and succeeding frames is clear, and proceeds to step ST98 when both the correction flags of the preceding and following frames are set.

  In step ST95, the unnecessary image extraction unit 256 determines whether the correction flag of the subsequent frame is set. The correction unnecessary image extraction unit 256 proceeds to step ST96 when the correction flag of the previous frame is set, and proceeds to step ST97 when the correction flag of the subsequent frame is set.

  In step ST96, the correction unnecessary image extraction unit 256 extracts the captured image of the subsequent frame, and ends the image extraction process.

  In step ST97, the correction unnecessary image extraction unit 256 extracts the captured image of the previous frame and ends the image extraction process.

  In step ST98, the correction unnecessary image extraction unit 256 extracts the captured image stored in the correction unnecessary image storage unit, and ends the image extraction process.

  When such image extraction processing is performed for each frame, when the captured image is a correction target, the captured image of the previous frame or the subsequent frame with less optical deterioration than the captured image or the captured image in the correction unnecessary image storage unit is corrected. Unnecessary image extraction unit 256 extracts the image.

  FIG. 16 is a diagram for explaining the image extraction processing operation. Note that FIG. 16A illustrates a change in correction angle with time during imaging. FIG. 16B illustrates a captured image for three frames obtained by performing the imaging operation. Further, the hatched portion in the captured image shows a simplified image portion in which optical degradation has occurred.

  When the correction flag is set in the correction information related to the captured image of the frame F2 stored in the memory 251n, the correction unnecessary image extraction unit 256 extracts the captured image with less optical deterioration than the captured image of the frame F2. I do. As shown in FIGS. 16A and 16B, the correction-unnecessary image extraction unit 256 determines that the correction angle is within the correction angle limit range in the frames F1 and F3 before and after the frame F2, and that correction is unnecessary. When it is, it is determined whether the correction angle maximum value time of the correction information related to the captured image of the frame F2 is close to the previous frame or the rear frame. Here, when the maximum correction angle time in the frame F2 is close to the previous frame as shown in FIG. 16A, the captured image of the frame F1, which is the previous frame, is extracted from the memory 251n-1, and the correction processing unit 258 is performed. Output to.

  FIG. 16C shows a case where the captured image of the frame F1, which is the previous frame, is determined as a correction target, and the captured image of the frame F3, which is the subsequent frame, is determined to be unnecessary. . In this case, the correction unnecessary image extraction unit 256 extracts the captured image of the frame F3 from the memory 251n + 1 and outputs it to the correction processing unit 258.

  Further, (D) of FIG. 16 illustrates a case where the captured image stored in the memory unit 251 is determined as a correction target. In this case, since the correction unnecessary image extraction unit 256 cannot extract a captured image with little optical deterioration from the memory unit 251, for example, the captured image of the frame F-1 stored in the correction unnecessary image storage unit 257. Is extracted and output to the correction processing unit 258.

[Correction processing operation]
FIG. 17 is a flowchart showing the correction processing operation. In step ST101, the correction processing unit 258 determines whether a correction flag is set for the captured image. The correction processing unit 258 proceeds to step ST102 when the correction flag is set, and proceeds to step ST103 when the correction flag is not set.

  In step ST102, the correction processing unit 258 clears the time index and proceeds to step ST103.

  In step ST103, the correction processing unit 258 determines whether the correction processing execution condition is satisfied. The correction processing unit 258 performs correction processing when the correction angle of the time index is equal to the correction angle of the time index obtained by adding the threshold time to the time index. The threshold time is set to a time shorter than the exposure time of one frame, for example, about 10% to 50% of the exposure time of one frame. The correction processing unit 258 proceeds to step ST104 when the correction angle of the time index and the correction angle of the time index obtained by adding the threshold time to the time index are not equal, and proceeds to step ST106 when they are equal.

  In step ST104, the correction processing unit 258 determines whether or not the time index is the last time index. When the time index is not the last time index in the exposure period, the correction processing unit 258 proceeds to step ST105, and when the time index indicates that it is the last time index in the exposure period, the correction processing operation ends. .

  In step ST105, the correction processing unit 258 increments the time index and returns to step ST103.

  In step ST106, the correction processing unit 258 sets a correction angle for the correction processing. The correction processing unit 258 sets the correction angle of the time index when the correction process execution condition is satisfied in step ST103 as the correction angle of the correction process.

  When the processing from step ST103 to step ST105 is performed in this way, the correction angle for the correction processing is set when the period until the correction angle becomes the next equal correction angle is equal to the threshold time, and the process proceeds to step ST107.

  By the way, if the period during which the correction angle exceeds the correction angle limit range is short in the exposure period of one frame, the optical deterioration is small. Therefore, even if the correction angle exceeds the correction angle limit range, it is not necessary to correct optical deterioration when the period during which the correction angle limit range is exceeded is short. Here, if the processing from step ST103 to step ST105 is performed, the correction angle of the correction processing is not set when the period exceeding the correction angle limit range is shorter than the threshold time. Therefore, unnecessary correction of optical deterioration can be prevented.

  In step ST107, the correction processing unit 258 acquires a correction image. The correction processing unit 258 determines a correction area according to the set correction angle. Further, the correction processing unit 258 extracts the image of the correction area as a correction image from the captured image extracted by the correction unnecessary image extraction unit 256, and proceeds to step ST108. In addition, if the relationship between the correction angle and the optical deterioration is measured and the optical deterioration correction area corresponding to the correction angle is determined and stored in advance, the correction area can be easily set according to the set correction angle. Can be determined.

  In step ST108, the correction processing unit 258 performs image correction processing. The correction processing unit 258 corrects optical deterioration in the captured image to be corrected using the correction image in the correction area acquired in step ST107. The correction processing unit 258 corrects optical deterioration by pasting the extracted correction image, for example, on the correction region in the captured image to be corrected. Further, the optical deterioration may be corrected by blending the correction image and the image of the correction area in the correction target captured image. Further, when the correction image and the captured image to be corrected are blended, the ratio of the correction image is reduced in the area where the optical deterioration is small, and the ratio of the correction image is increased in the area where the optical deterioration is large. By controlling the blend ratio in this way, the boundary between the area where the optical deterioration is corrected and the area where the optical deterioration is not corrected can be made inconspicuous.

  When such correction processing is performed for each frame, an image in a region where the optical degradation is significant in the captured image to be corrected is extracted from the captured image with less optical degradation than the captured image to be corrected. Thus, optical deterioration can be corrected.

  Although not shown in the flowchart, when a threshold time shorter than the exposure period is set and the time during which the correction angle exceeds the correction angle limit range is longer than the threshold time, the correction angle maximum value is corrected. The correction of the optical deterioration may be performed by setting the correction angle. In this case, since the correction angle of the correction process is larger than that of the correction operation shown in FIG. 17, the correction area becomes wide.

  FIG. 18 is a diagram for explaining the correction processing operation. Note that FIG. 18A illustrates the change in the correction angle with time during imaging. FIG. 18B illustrates a captured image for three frames obtained by performing the imaging operation. FIG. 18C shows a correction image obtained by extracting the image of the correction area from the captured image extracted by the correction unnecessary image extraction unit. FIG. 18D shows a captured image after the correction process. Note that the shaded portion in the captured image shows a simplified image portion where optical degradation has occurred.

  When the correction flag is set in the correction information regarding the captured image of the frame F2 stored in the memory 251n, the correction unnecessary image extraction unit 256 determines whether or not the correction processing execution condition is satisfied based on the correction information. To do. Here, when both the correction angle of the time index Tid-a and the correction angle of the time index Tid-b obtained by adding the threshold time “Tth” to the time index Tid-a are the correction angle “α”, this correction angle “Α” is an optical correction angle of the optical correction process.

  Further, the correction processing unit 258 determines a correction region according to the set correction angle “α”. The correction area is determined so as to be an area including an optically deteriorated portion in the captured image of the frame F2. The correction processing unit 258 extracts an image of the correction area as a correction image from the captured image extracted by the correction unnecessary image extraction unit 256. For example, by pasting the correction image on the correction area of the captured image of the frame F2, it is possible to generate an image signal of the captured image in which the optical deterioration is corrected.

  FIG. 19 shows a correction processing operation when the correction angle in shake correction is small. Note that (A) in FIG. 19 exemplifies a change in the correction angle with time during imaging. FIG. 19B illustrates a captured image obtained by performing the imaging operation.

  As shown in FIG. 19, if the correction angle for shake correction is small, the correction angle of the time index and the correction angle of the time index obtained by adding the threshold time “Tth” to this time index do not occur. Therefore, the correction angle of the correction process is not set and the correction process is not performed. That is, it is possible to prevent correction processing from being performed on a captured image with little optical deterioration.

  FIG. 20 shows the correction processing operation when the correction angle in shake correction is large. Note that FIG. 20A illustrates a change in the correction angle with time during imaging. FIG. 20B illustrates a captured image obtained by performing the imaging operation. FIG. 20C shows a correction image obtained by extracting an image of the correction area from the captured image extracted by the correction unnecessary image extraction unit. FIG. 20D shows a captured image after the correction process. Further, the hatched portion in the captured image shows a simplified image portion in which optical degradation has occurred.

  As shown in FIG. 20, when the correction angle of shake correction is larger than that shown in FIG. 18, the correction angle of the time index Tid-c and the time obtained by adding the threshold time “Tth” to this time index Tid-c The correction angle of the index Tid-d is equal to the correction angle “β”. Further, the correction angle “β” is larger than the correction angle “α”. The correction processing unit 258 uses the correction angle “β” as the correction angle of the correction process, and extracts a correction image in the correction area corresponding to the correction angle “β” from the captured image with less optical deterioration than the captured image to be corrected. . For this reason, even when the correction angle in shake correction is large and the range in which optical degradation has occurred is wide, a correction image corresponding to the range in which optical degradation has occurred is extracted, and optical degradation is corrected. An image signal of the captured image can be generated.

  As described above, when performing the optical degradation correction operation when capturing a moving image, when a captured image to be corrected is determined, a captured image with less optical degradation than the captured image to be corrected is extracted from the consecutive captured images. . Further, an image of a correction area determined according to the relative positional relationship between the lens unit and the image sensor is extracted from the extracted image as a correction image, and the correction image is used in the captured image to be corrected. Optical degradation is corrected. For this reason, for example, even when the moving image is captured while walking, even if the displacement range of the relative positional relationship between the lens unit and the image sensor is expanded, the optical generated by expanding the displacement range Degradation can be corrected. That is, even if optical deterioration occurs when the shake correction characteristic is improved, a captured image in which this optical deterioration is corrected can be obtained.

<6. Optical degradation correction operation for still image capture>
Next, a case where a still image capturing operation is performed will be described. In still image capturing, a still image signal of one frame is generated according to a shutter operation. For this reason, there is no captured image before and after the still image to be corrected as in the case of moving image capturing, and thus it is not possible to extract a captured image with less optical deterioration than the still image to be corrected. Therefore, in the first optical deterioration correction operation at the time of still image capturing, when the still image is determined as the correction target, the image capturing operation is continuously performed, and the captured image that is continuous with the still image generated according to the shutter operation (hereinafter referred to as the captured image). For example, one “continuous image” is generated. Further, when it is determined that the generated continuous image does not require correction, a correction image is extracted from the continuous image.

  FIG. 21 is a diagram for explaining a first deterioration correction operation in still image capturing. Note that (A) in FIG. 21 exemplifies a change in the correction angle with time during imaging. FIG. 21B illustrates still images and continuous images obtained by performing the imaging operation. FIG. 21C shows a correction image obtained by extracting an image of the correction area from the continuous image extracted by the correction unnecessary image extraction unit. FIG. 21D shows a still image after the correction process.

  The correction target determination unit 255 determines whether the still image generated in response to the shutter operation is a correction target and notifies the control unit 50 of the determination result. When it is determined that the still image is a correction target, the control unit 50 performs imaging after the still image frame FS, and generates a continuous image signal that is the frame FCa, as shown in FIG. To do. The correction unnecessary image extraction unit 256 supplies the continuous image to the correction processing unit 258 when it is determined that the continuous image does not need correction. As shown in FIG. 21C, the correction processing unit 258 extracts a correction image from the continuous image, and corrects optical deterioration in the still image determined to be a correction object using the correction image. Then, the image signal of the still image in which the optical deterioration shown in (D) of FIG. 21 is corrected is generated.

  In this way, when the still image is determined as the correction target, the image capturing operation is performed continuously, and the correction image is extracted from the continuous image generated in response to the shutter operation and the correction process is performed. Even in the still image capturing operation, optical degradation can be corrected.

  Further, when the continuous image frame FCa is determined as the correction target, the optical degradation in the continuous image is large, and therefore the optical degradation in the still image cannot be corrected using the continuous image. Therefore, the optical deterioration correction unit 25 does not correct optical deterioration. In addition, the control unit 50 performs an identification display indicating that the optical deterioration correction is not performed on the display unit 33, for example, “correction failure”. Further, when recording of a still image in which optical deterioration has occurred is not performed, a display such as “imaging failure” may be performed.

  Next, the second optical deterioration correction operation when capturing a still image will be described. In the second optical deterioration correction operation, a continuous image is generated so as to reduce optical deterioration, and a correction image is extracted from the continuous image and correction processing is performed. Specifically, when a continuous image is generated, correction angle offset processing is performed so that the correction angle is within the correction angle limit range. For example, imaging is performed after the correction angle is offset to the center position of the correction angle limit range. In this case, since the correction angle is set to the center position of the correction angle limit range at the start of continuous image exposure, the correction angle is less likely to exceed the correction angle limit range during the continuous image exposure period. That is, optical degradation of continuous images can be reduced. Further, the imaging range of the continuous image is different from the still image generated according to the shutter operation because the correction angle offset processing is performed. Therefore, when extracting the correction image from the continuous image, the extraction range corresponding to the still image correction area is moved according to the offset amount of the correction angle, and the image is extracted from the moved extraction range.

  FIG. 22 is a diagram for explaining a second deterioration correction operation at the time of still image capturing. Note that (A) in FIG. 22 exemplifies the change over time of the correction angle during imaging. FIG. 22B illustrates still images and continuous images obtained by performing the imaging operation. FIG. 22C shows a correction image obtained by extracting an image of the correction area from the continuous image extracted by the correction unnecessary image extraction unit. FIG. 22D shows a still image after correction processing.

  The correction target determination unit 255 determines whether the still image generated in response to the shutter operation is a correction target and notifies the control unit 50 of the determination result. When it is determined that the still image is a correction target, the control unit 50 performs imaging after the still image frame FS to generate a continuous image signal as the frame FCb. In addition, when capturing a continuous image, the control unit 50 starts exposure after offsetting the correction angle to the center position of the correction angle limit range, for example. Therefore, as illustrated in FIG. 22B, the continuous image of the frame FCb has a different imaging range from the still image of the frame FS.

  The correction unnecessary image extraction unit 256 supplies the continuous image to the correction processing unit 258 when it is determined that the continuous image does not need correction. The correction processing unit 258 extracts a correction image from the continuous image. Here, the still image and the continuous image have different imaging ranges as shown in FIG. Therefore, when the correction image is extracted from the continuous image, the extraction range corresponding to the still image correction area is moved according to the offset amount of the correction angle as shown in FIG. An image is extracted from the extraction range. The correction unnecessary image extraction unit 256 corrects the optical deterioration in the still image determined as the correction target using the correction image extracted from the continuous image, and the optical deterioration is corrected as shown in FIG. An image signal of the still image thus generated is generated.

  As described above, when a still image is determined as a correction target, an image capturing operation is continuously performed, and a continuous image continuous with the still image generated according to the shutter operation is generated. Further, by performing correction angle offset at the start of continuous image capturing, optical deterioration of the continuous image can be reduced, so that optical deterioration can be corrected even in the still image capturing operation.

  It should be noted that the present invention should not be construed as being limited to the above-described embodiments of the invention. For example, the shake correction mechanism displaces the relative positional relationship between the lens unit 11 and the image pickup device 21 with respect to the optical axis, so that the position of the optical image formed on the image pickup surface of the image pickup device 21 is changed on the image pickup surface. If it is the structure displaced by this, it is not restricted to the above-mentioned structure.

  Further, in the optical degradation correction unit 25 shown in FIG. 2, the configuration is illustrated in which the memory unit 251 stores the captured images for two frames before and after the correction target captured image. It is good also as a structure which memorize | stores a captured image. In addition, the captured image of the previous frame that does not require correction is stored in the correction-unnecessary image storage unit 257, and the captured image of the subsequent frame is stored in the memory unit 251, so It is good also as a structure etc. which extract.

  The embodiment of the present invention discloses the present invention in the form of exemplification, and it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the gist of the present invention. That is, in order to determine the gist of the present invention, the claims should be taken into consideration.

  In the image pickup apparatus and the shake correction method of the present invention, the optical image formed on the image pickup surface of the image pickup device by the lens portion is displaced by displacing the relative positional relationship between the lens portion and the image pickup device in accordance with the shake detection result. Correct the shake. In addition, the optical degradation is corrected using a captured image having optical degradation that is less than that of the captured image with respect to the captured image that has undergone optical degradation due to shake correction. For this reason, even when optical deterioration occurs due to shake correction, a captured image in which this optical deterioration is corrected can be obtained. Therefore, it is suitable for digital cameras and video cameras.

  DESCRIPTION OF SYMBOLS 10 ... Imaging device, 11 ... Lens part, 12 ... Driver, 13 ... Position sensor, 21 ... Image sensor, 22 ... TG part, 23 ... AFE part, 24. ..Signal processing unit 25... Optical deterioration correction unit 31. Detection unit 32. Image output unit 33. Display unit 34. Operation unit 42 ... shake detection unit 50 ... control unit 111 ... zoom lens 112 ... focus lens 113 ... correction lens unit 114 ... stop mechanism 251, ... Memory unit 251n-2, 251n-1, 251n, 251n + 1, 251n + 2 ... Memory (FIFO), 255 ... Correction target discriminating unit, 256 ... Uncorrected image extraction unit, 257 ..Unnecessary image storage unit, 258... Correction processing unit

Claims (15)

An image sensor that photoelectrically converts an optical image formed by the lens unit to generate an image signal of a captured image;
A shake detection unit that detects shake and generates a shake detection signal indicating a detection result;
By displacing the relative positional relationship between the lens unit and the image sensor according to the shake detection signal, the optical image on the imaging surface of the image sensor caused by the shake detected by the shake detection unit. A shake correction control unit for correcting shake;
An image pickup apparatus comprising: an optical deterioration correction unit configured to correct the optical deterioration with respect to a picked-up image in which optical deterioration is caused by the shake correction using a picked-up image having less optical deterioration than the picked-up image. The optical deterioration correction unit is
A correction target determination unit that determines whether the captured image is an image to be corrected for optical degradation or an image that does not need to be corrected based on a relative positional relationship between the lens unit and the image sensor;
A correction-unnecessary image extracting unit that extracts a captured image that has been determined to be unnecessary for correction by the correction target determining unit;
A correction processing unit that extracts a correction image from the captured image extracted by the correction-unnecessary image extraction unit and corrects optical degradation of the captured image that is determined as a correction target by the correction target determination unit using the correction image. The imaging apparatus according to claim 1, further comprising: The correction target determining unit determines that the optical degradation is to be corrected when a relative positional relationship between the lens unit and the image sensor exceeds a limit range during an exposure period of the captured image. The imaging device described. The correction target determination unit obtains an evaluation value based on a period during which the relative positional relationship between the lens unit and the imaging element exceeds a preset positional relationship and an excess amount during the exposure period of the captured image. The imaging apparatus according to claim 2, wherein when the evaluation value exceeds a threshold value, the imaging apparatus determines that the optical degradation correction target.   The correction-unnecessary image extraction unit extracts a correction-unnecessary captured image captured before the correction target captured image or a correction-unnecessary captured image captured after the correction target captured image. Item 3. The imaging device according to Item 2. The correction-unnecessary image extraction unit stores a correction-unnecessary captured image in a correction-unnecessary image storage unit, and sequentially updates the captured image stored in the correction-unnecessary image storage unit with a correction-unnecessary captured image, so that the correction is unnecessary. The imaging apparatus according to claim 5, wherein when the captured image cannot be extracted, the captured image stored in the correction-unnecessary image storage unit is extracted. The correction target determination unit detects a maximum time during which the displacement of the relative positional relationship between the lens unit and the imaging element is maximum in an exposure period of a captured image that is a correction target of the optical degradation,
When the maximum time is the first half of the exposure period, the correction-unnecessary image extraction unit extracts a non-correction-captured image captured before the image to be corrected, and the maximum time The imaging apparatus according to claim 5, wherein when it is the second half of the exposure period, an uncorrected captured image captured after the captured image to be corrected is extracted. The correction processing unit determines a correction area of the captured image determined to be the correction target based on the shake correction operation, and corresponds to the correction area from the captured image extracted by the correction unnecessary image extraction unit. The imaging apparatus according to claim 2, wherein an image of a region is extracted as the correction image. The correction processing unit is longer than a preset time during which the relative positional relationship between the lens unit and the imaging element exceeds a limit range during the exposure period of the captured image determined as the correction target. The imaging device according to claim 8, wherein the correction region is determined from a maximum displacement of a relative positional relationship between the lens unit and the imaging device. The correction processing unit is longer than a preset time during which the relative positional relationship between the lens unit and the imaging element exceeds a limit range during the exposure period of the captured image determined as the correction target. The image pickup apparatus according to claim 8, wherein the correction region is determined from a displacement of a relative positional relationship between the lens unit and the image pickup device at a predetermined time interval. When the captured image generated by the image sensor in the still image mode is determined as the optical degradation correction target by the correction target determination unit, the captured image is continuously generated by the image sensor,
The correction-unnecessary image extracting unit extracts a correction image from the continuously generated captured image when the continuously generated captured image is determined to be a corrected unnecessary captured image. Imaging device. When a captured image generated by the image sensor in the still image mode is determined as an optical degradation correction target by the correction target determination unit, a relative positional relationship between the lens unit and the image sensor is determined to be a predetermined positional relationship. And then generate images continuously with the image sensor,
12. The correction-unnecessary image extracting unit extracts a correction image from the continuously generated captured image when the continuously generated captured image is determined to be a corrected unnecessary captured image. Imaging device. A display unit that indicates that the optical degradation of the captured image generated in the still image mode is not corrected when the continuously generated captured image is determined as a captured image to be corrected. 12. The imaging device according to 12. The shake correction control unit enlarges a displacement range when the magnitude of shake indicated by the shake detection signal is larger than a predetermined level, and relative to the lens unit and the image sensor in the enlarged displacement range. The target positional relationship is displaced according to the shake detection signal,
The imaging apparatus according to claim 1, wherein the optical deterioration correction unit corrects the optical deterioration when the displacement range is expanded. In the shake detection unit, detecting a shake and generating a shake detection signal indicating a detection result;
In the shake correction control unit, the relative positional relationship between the lens unit and the image sensor is displaced in accordance with the shake detection signal, thereby causing the image pickup surface of the image sensor to be generated by the shake detected by the shake detection unit. Correcting the shake of the optical image of
A shake correction method comprising: a step of correcting the optical degradation using a captured image with less optical degradation than the captured image for a captured image that has undergone optical degradation due to the shake correction in an optical degradation correction unit.

Images