JP2002305682A - Electronic camera and image processing system - Google Patents

Abstract

[PROBLEMS] To provide an electronic camera and an image processing system capable of efficiently correcting a blur generated during exposure as needed and generating a natural blur corrected image data even in a panning shot. provide. SOLUTION: When an image of a subject is taken by an image taking means, first image data taken with a first exposure time with relatively little image blur and a second exposure time longer than the first time are taken. The second image data is generated, and the difference between the amplitude ratio and the phase of the high-frequency component in the two image data is calculated. The presence / absence of image blur and the direction of panning are determined based on the calculated amplitude ratio and phase difference of the high-frequency component, and image blur image correction processing and recording of image data are performed according to the presence / absence of image blur. The content of the image blur correction processing is changed according to the shooting direction.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an electronic camera and an image processing system, and more particularly to an electronic camera and an image processing system for generating an image in which blur is corrected.

[0002]

2. Description of the Related Art Conventionally, the following method has been proposed as a measure against camera shake. A first method is to incorporate an image stabilizing optical system in a photographing optical system of a camera, detect a camera shake with a camera shake sensor, and move the image stabilizing optical system during imaging according to the detected camera shake to obtain an image associated with the camera shake. This is to prevent blurring.

In the second method, a camera shake during image pickup is detected and recorded by a shake sensor, and a shake image is restored by performing image processing on an image taken based on the camera shake information. The third method is to continuously capture a plurality of images, detect a movement between the plurality of images by a method such as pattern matching, and combine the plurality of images based on the detected movement between the images. This is to generate an image in which the blur has been corrected.

[0004]

However, in the above-mentioned first and second methods, it is necessary to incorporate a blur sensor in the camera, which causes a problem that the size of the camera is increased and the cost is increased. Was.

Further, the first conventional method requires a mechanical movement mechanism for moving the shake correction optical system to correct the shake, which results in an increase in the size of the camera and an increase in cost. However, there is a problem that battery consumption is severe. In the first and second methods of the related art, since the blur is detected by the blur sensor, there is a problem that the subject blur caused by the movement of the subject during exposure cannot be corrected.

In the third conventional method, camera shake occurring between a plurality of images can be corrected.
There is a disadvantage that blurring occurring during exposure of one image cannot be corrected. In the second and third methods of the related art, image processing for blur correction is uniformly performed regardless of the presence or absence of blur. Therefore, even if there is no blur, other processing is performed until the image processing is completed. There was a disadvantage that it could not be performed.

In the above-described conventional second and third methods, image processing for blur correction is uniformly performed.
There is a disadvantage that the background which should be an originally flowing image is subjected to blur correction with respect to the panned image data, resulting in unnatural image data.

Accordingly, the present invention eliminates the need for a blur sensor and a blur correction optical system, can efficiently correct image blur and subject blur generated during exposure as needed, and can provide a natural image even when panning is performed. An object of the present invention is to provide an electronic camera and an image processing system that can generate blur-corrected image data.

[0009]

In order to achieve the above object, in an electronic camera according to the present invention, there is provided an image pickup means for picking up an image of a subject, a recording means for recording image data, and Means for continuously generating first image data and second image data by means;
A blur determining unit configured to determine whether the first image data or the second image data is blurred based on the first image data and the second image data; Based on the second image data,
Image processing means for generating third image data in which blur has been corrected by image processing, and when the blur determining means determines that the first image data or the second image data is not blurred, The image processing means does not perform image processing for blur correction, and the recording means records one of the first image data and the second image data, and the first If it is determined that the image data or the second image data is blurred, the image processing means performs image processing for blur correction, and the recording means records the third image data. And

In an electronic camera according to a second aspect of the present invention, an image pickup means for picking up a subject image, an exposure control means for controlling an exposure time for picking up the subject image by the image pickup means, and an image control means for recording image data. Recording means, first image data captured at a first exposure time by the exposure control means and the imaging means, and second image data captured at a second exposure time longer than the first exposure time. Image generation means for continuously generating, and correcting a spatial frequency component included in the second image data based on the first image data and the second image data generated by the image generation means Image processing means for generating third image data whose blur has been corrected by the method, and whether or not the second image data is blurred based on the first image data and the second image data When the second image data is determined not to be blurred by the blur determination means, the image processing does not perform the processing for correcting the blur, and the recording means In addition to recording the second image data, if the blur determination unit determines that the second image data is not blurred, the image processing performs a process of correcting blur, and the recording unit performs blur correction. The corrected third image data is recorded.

In the electronic camera according to the third aspect of the present invention, an image pickup means for picking up an object image, an exposure control means for controlling an exposure time of the image pickup of the object image by the image pickup means, the exposure control means and the image pickup means Means for continuously generating first image data picked up by means for a first exposure time, and second image data picked up for a second exposure time longer than the first exposure time; A panning direction setting unit that sets a panning direction when capturing the second image data, based on the first image data and the second image data generated by the image generation unit,
Image processing means for generating third image data by correcting a spatial frequency component included in the second image data, wherein the image processing means comprises a panning set by the panning direction setting means The method is characterized in that the spatial frequency component included in the second image data is corrected to change the content of the image processing when the third image data is generated according to the direction.

According to a fourth aspect of the present invention, in the electronic camera according to the third aspect, the image processing means is arranged so that the panning direction set by the panning direction setting means is horizontal. The third image data is generated by mainly correcting the vertical spatial frequency component included in the second image data, and when the panning direction set by the panning direction setting means is the vertical direction, The third image data is generated by mainly correcting horizontal spatial frequency components included in the second image data.

In an electronic camera according to a fifth aspect of the present invention, an image pickup means for picking up a subject image, an exposure control means for controlling an exposure time for picking up the subject image by the image pickup means, the exposure control means and the image pickup means Means for continuously generating first image data picked up by means for a first exposure time, and second image data picked up for a second exposure time longer than the first exposure time; A panning direction detection unit that detects a panning direction when the second image data is captured, based on the first image data and the second image data, and the second image generated by the image generation unit. An image processing unit that generates third image data by correcting spatial frequency components included in the second image data based on the first image data and the second image data; The image processing means corrects a spatial frequency component included in the second image data according to the panning direction detected by the panning direction detection means, and performs image processing when generating third image data. Is characterized by changing the content of

According to a sixth aspect of the present invention, in the electronic camera according to the fifth aspect, the image processing means is arranged so that the panning direction detected by the panning direction detecting means is horizontal. The third image data is generated by mainly correcting the vertical spatial frequency component included in the second image data, and the panning direction detected by the panning direction detecting means is vertical. The third image data is generated by mainly correcting horizontal spatial frequency components included in the second image data.

According to a seventh aspect of the present invention, in the electronic camera according to the second, third, or fifth aspect, the second exposure time is such that the brightness level of the image data is appropriate. An electronic camera characterized in that the first exposure time is not more than about 1/2 of the second exposure time, as well as an exposure time that gives a proper exposure amount.

In the electronic camera according to the present invention, an image pickup means for picking up an image of a subject and generating image data, and processing the image data to reduce the influence of blur contained in the image data. Image processing means for correcting, and determining means for determining whether or not the image data is affected by blur, wherein the image processing means determines that the image data is affected by blur by the determining means. Only in the case where image processing is performed, image processing for correcting the influence of blur included in the image data is performed.

In an image processing system according to a ninth aspect of the present invention, there is provided an image pickup means for picking up a subject image, an exposure control means for controlling an exposure time for picking up the subject image by the image pickup means, the exposure control means, A first image data captured by a first exposure time by an imaging unit; and an image generation unit configured to continuously generate second image data captured by a second exposure time longer than the first exposure time. An electronic camera provided, based on the first image data and the second image data generated by the electronic camera, detects a direction of a panning shot when the second image data is captured, and An image processor that corrects a spatial frequency component included in the second image data according to a follow-up shooting direction based on the first image data and the second image data, and generates third image data. Characterized in that comprising the device.

[0018]

Embodiments of the present invention will be described below with reference to the drawings. 1 and 2 are perspective views showing the configuration of an embodiment of an electronic camera 1 to which the present invention has been applied. FIG. 1 is a view of the electronic camera 1 as viewed from the front. A photographing lens 2 for forming a subject image, a finder 4 used for confirming a photographing range of the subject, and a flashlight for illuminating the subject at the time of photographing are provided in front of the electronic camera 1. , A colorimetric element 6 for measuring the color of the subject, and a photometric element 7 for measuring the brightness of the subject. On the upper surface of the electronic camera 1, there is provided a shutter button 3 which is operated at the time of photographing a subject.

FIG. 2 is a view of the electronic camera 1 as viewed from the rear. The eyepiece of the finder 4, a display unit (display LCD) 8 for displaying a captured image, and the direction of panning during panning shooting are set. Correction setting member 41 for selecting whether or not to perform shake correction
Is provided. On the side of the electronic camera 1, a memory card slot 9 for mounting a removable storage medium (memory card) for recording captured image information (image data) is provided.

Next, an electric configuration inside the electronic camera 1 will be described with reference to a block diagram shown in FIG. CPU
10 is a means for controlling the operation of the entire electronic camera, based on a control program stored in the ROM 26,
Each unit connected to the CPU control bus 11 is controlled.

The CCD 20 used as the image pickup means has a plurality of pixels, and photoelectrically converts a light image formed on each pixel into an image signal (electric signal).
The digital signal processor (DSP) 21 is a CCD
20 and a CCD horizontal drive pulse.
The D drive circuit 19 is controlled to supply a CCD vertical drive pulse to the CCD 20.

The image adjusting unit 22 is controlled by the CPU 10 to sample the image signal photoelectrically converted by the CCD 20 at a predetermined timing, and
It is designed to amplify to a predetermined level. The analog / digital conversion circuit (AD conversion circuit) 23 quantizes (digitizes) the image signal sampled by the image adjustment unit 22 with a predetermined number of bits, and converts the image signal into DSP 21 as image data.
To be supplied.

The DSP 21 controls the data bus 24 connected to the buffer memory 30 and the memory card 25. The DSP 21 compresses the image data supplied from the AD conversion circuit 23 and temporarily stores the image data in the buffer memory 30. The image data stored in the memory 30 is read out, and the image data is read from the memory card 2 used as a recording unit.
5 is recorded.

The DSP 21 reads out image data from the buffer memory 3 or the memory card 25, decompresses the image data, and stores the decompressed image data in the frame memory 2.
7 and displayed on the display LCD 8. The DSP 21 manages the timing of data input / output in recording on the memory card 25, storing image data in the buffer memory 30, and the like.

The DSP 21 performs image processing for correcting a blurred image, which will be described later, as image processing means. The buffer memory 30 is used to reduce the difference between the data input / output speed with respect to the memory card 25 and the processing speed of the CPU 10, the DSP 21, and the like.

The shutter button 3 is an operation member operated by a user to give a photographing instruction, and outputs signals to the CPU 10 according to three operation states: a non-operation state, a half-press state, and a full-press state. When a photographing instruction (full press) is given by operating the shutter button 3, the CPU 10
A photographing instruction command is sent to the CPU 21 so that the above-described photographing operation is executed.

The photometric element 7 measures the light quantity of the object and its surroundings, and outputs the measurement result to the photometric circuit 34. The photometric circuit 34 performs predetermined processing on an analog signal that is a photometric result supplied from the photometric element 7, converts the analog signal into a digital signal, generates photometric data, and outputs the photometric data to the CPU 10. ing.

The colorimetric element 6 measures the color temperature of the object and its surroundings, and outputs the measurement result to the colorimetric circuit 33. The colorimetric circuit 33 performs predetermined processing on the analog signal that is the colorimetric result supplied from the colorimetric element 6, converts the analog signal into a digital signal to obtain colorimetric data, and sends the colorimetric data to the CPU 30. The output has been made.

The timer 28 has a built-in clock circuit and outputs time data corresponding to the current time to the CPU 10. The aperture driving circuit 16 sets the aperture diameter of the aperture 18 to a predetermined value by a step motor 17.

The diaphragm 18 is arranged between the photographing lens 2 and the CCD 20 to change the aperture diameter of light incident on the CCD 20 from the photographing lens 2. The shutter drive circuit 13, the step motor 14, and the shutter 15 are used as exposure control means.
The shutter 15 is operated by the stepping motor 14 to control the exposure time of the CCD 20.

The shutter 15 includes the photographing lens 2 and the CCD 2
0, and controls the blocking and transmission of light incident on the CCD 20 from the photographing lens 2. C
The PU 10 controls the photometric circuit 34 and the colorimetric circuit 33, and receives the photometric data of the photometric element 7 and the colorimetric data of the colorimetric element 6.

The CPU 10 refers to the photometric data and a predetermined table so that the aperture value data of the aperture 18 and the shutter speed (exposure) of the shutter 15 are adjusted so that the luminance value of the image data picked up by the CCD 20 becomes an appropriate level. Time) data is determined, and the data is supplied to an aperture driving circuit and a shutter driving circuit.

The CPU 10 refers to a predetermined table to measure the colorimetric data (color temperature) supplied from the colorimetric circuit 33.
Is calculated, and the white balance adjustment value is supplied to the image adjustment unit 22. When the brightness of the subject is equal to or less than a predetermined value according to the photometric data, the CPU 10 controls the strobe driving circuit 35 at the time of photographing so that the strobe 5 emits light as appropriate.

In accordance with the time data supplied from the timer 28, the CPU 10 uses the information of the shooting date and time and the file name as header information of the image data,
5 is recorded in the photographed image recording area.
The CPU 10 controls the lens driving circuit 12 to move the photographing lens 2 to perform an autofocus operation.

The CPU 10 reads out data of the focal length set by the photographing lens 2 via the focal length detecting circuit 44. The CPU 10 controls the display circuit 31 in the finder to display settings in various operations on the LCD 32 in the finder.

The CPU 10 exchanges predetermined data with a predetermined external device (not shown) via the interface 29. The CPU 10 receives signals from the various changeover switches and the operation buttons 40, and performs appropriate processing. The various changeover switches and the operation buttons 40 include changeover switches operated by a user to switch the camera operation to an operation mode (macro mode) optimized for close-up shooting. CPU 10
When the mode is switched to the macro mode, by controlling the lens driving circuit 12 and moving the taking lens 2,
The optical system arrangement is suitable for close-up photography.

The CPU 10 has a CCD as an image generating means.
20, controlling the shutter drive circuit 13 to generate images having different exposure times as described later. The shake correction selection member 42 is an operation member operated by a user to instruct whether or not to perform shake correction, and outputs a signal corresponding to an operation state to the CPU 10. The CPU 10 performs a later-described blur correction process on image information (image data) stored in the buffer memory in response to an operation of the blur correction selection member 42.

The panning direction setting member 41 is an operation member operated by the user to instruct the panning direction in performing panning, and outputs a signal to the CPU 10 according to the operation state. The CPU 10 changes the processing at the time of executing the blur correction processing to be described later on the image information (image data) stored in the buffer memory in accordance with the operation of the panning direction setting member 41. In addition, panning is a method of shooting the main subject moving against the background while shaking the camera according to the movement of the subject during exposure to prevent the main subject from blurring due to movement during exposure It is. As the setting state of the panning direction setting member 41, there are three states of not panning, panning in the horizontal direction of the screen, and panning in the vertical direction of the screen.

The strobe mode setting member 43 is an operation member operated by a user to set an operation mode (strobe mode) when shooting with the strobe 5 emitting light, and outputs a signal corresponding to the operation state. Output to CPU10. The flash mode includes a normal mode and a slow synchro mode. In the normal mode, when the photometric data output from the photometric element 7 indicates a luminance lower than a predetermined value, the CPU 10 automatically causes the flash 5 to emit light at the time of shooting. At this time, the high-speed side of the shutter speed is the shutter speed (for example, 1/60 second) at which the shutter is fully opened.
On the low-speed side, the shutter speed is limited to a shutter speed that does not cause blurring (for example, 1 / f when the focal length of the photographing lens 2 is f (mm), but the focal length is 35 mm in terms of a silver halide camera).

On the other hand, the slow synchro mode is a mode in which the shutter speed is not restricted to a low speed side during the flash shooting, in order to shoot with the background atmosphere and the strobe lighting. FIG. 4 is a flowchart showing a basic sequence of the electronic camera 1. This sequence is executed by the CPU 10, the DSP 21 and the like in FIG. In this sequence, the shake correction selection member 42 is set to perform shake correction, and the panning direction setting member 41 is set.
The strobe mode setting member 43 is appropriately set by the photographer.

The sequence starts when the power is turned on,
In S100, it is detected whether or not the shutter button 3 has been half-pressed. If half-pressing is not performed, S100 is repeated.
If it is determined that the half-press has been performed, the process proceeds to S101, in which the luminance of the subject is measured by the photometric element 7, and based on the photometric data, when the subject is imaged by the CCD 20, the luminance level of the image data becomes appropriate. An aperture value and an exposure time (shutter speed) T for obtaining an appropriate exposure amount are calculated.

In S102, it is detected whether or not the shutter button 3 has been fully pressed. If the full press is not performed, S101
To S102 are repeated. If it is determined in step S102 that the shutter button 3 has been fully pressed, the process proceeds to step S103, in which the step motor 17 is driven to control the aperture 18 to the aperture value determined in step S101.

In S104, it is detected whether or not the macro mode is set. When the macro mode is set, the effect of blurring is greater during close-up shooting than in normal shooting.
Go to 109. If the macro mode has not been set, the process proceeds to S105.

In S105, it is detected whether or not the focal length of the photographing lens 2 is telephoto. For example, if the focal length is 100 mm or more, it is determined to be a telephoto lens. In the case of a telephoto lens, the effect of blurring is greater than that of a lens with a shorter focal length. Therefore, the process advances to step S109 to execute shooting for correcting blurring. If the lens is not a telephoto lens, the process proceeds to S106.

In step S106, it is detected whether or not the shutter speed T set in step S101 is sufficiently high with respect to the focal length of the photographing lens 2. For example, assuming that the focal length is f, when the shutter speed T is higher than 1 / f, it is determined that the shutter speed T is sufficiently higher than the focal length of the photographing lens 2. If the shutter speed is low,
Since the effect of blurring is greater than in the case where the shutter speed is high, it is necessary to execute S109 in order to execute shooting for correcting blurring.
Proceed to. If the shutter speed is high, the process proceeds to S107.

In S107, it is detected whether or not the slow sync mode has been set. If the slow synchro mode is set, the shutter speed is not limited on the low speed side, and the effect of blurring is large. Therefore, the process proceeds to S109 in order to execute shooting for blur correction. If the slow sync mode has not been set, the process proceeds to S108, and the mode shifts to the normal shooting mode. Note that the operation in the normal shooting mode is not related to the present invention, and will not be described.

In S109, the shutter speed is changed to S10
The step motor 14 is driven to operate the shutter 15 as the exposure time T / 2, which is half of the exposure time T at which the appropriate exposure amount determined in 1 is obtained, and the CCD 20 exposes the subject image to take an image. The image obtained at this time is referred to as image 1.

The electric charges stored in the CCD 20 are transferred to the DSP 21 via the image adjustment unit 22 and the AD conversion circuit 23.
In step S110, the image 1 is compressed. Here, 1/8 JPEG compression with little deterioration of the high frequency component of the spatial frequency is performed. Then, in S111, in order to prohibit the reproduction of the image 1 having a small exposure amount, the reproduction prohibition flag is set to 1
Is recorded. As a result, the underexposed dark image 1
Can be prevented from being displayed. And S112
Thus, the image 1 is recorded in the buffer memory 30.

In step S113, the image is exposed for an appropriate exposure time T using the same aperture value as in the shooting in step S109. The image obtained at this time is referred to as image 2. Then, in S114, the image 2 captured with the appropriate exposure amount is displayed on the display LCD 8. As a result, the image 2 having an appropriate luminance level is displayed instead of the dark image 1, so that the user can confirm that the exposure has been performed with the appropriate exposure amount.

In step S115, the image 2 is reduced to a 1/16 JP.
EG compression is performed. This is because the image 2 loses the high frequency component of the spatial frequency of the image due to camera shake, and therefore, even if the image 2 is compressed at 1/16, the deterioration of the image quality is small.
As described above, the image 1 containing many high-frequency components of the spatial frequency is compressed at a low compression ratio, and the image 2 containing little high-frequency components is compressed at a high compression ratio, so that the buffer memory can be used efficiently. ing.

At S 116, the image 2 is recorded in the buffer memory 30. In S117, it is detected whether or not there is a difference between the spatial frequency components of the images 1 and 2 obtained by different exposure times. For example, the MTF in the spatial frequency domain of the image 1 and the image 2 obtained by performing the Fourier transform
(Modulation transfer function), the MTF of image 2 in the high frequency component
If it has decreased by a predetermined amount from F, it is determined that the high-frequency component has decreased due to the blur, and the blur image correction processing in S119 and thereafter is performed. In the high frequency component, the MTF of the image 2 is M
It is determined that there is no difference if it is not lower than the TF by a predetermined amount,
The flow proceeds to S118 as the blurred image correction processing is unnecessary.

In S118, the image 1 recorded in the buffer 30 is deleted, and the image 2 is recorded in the memory card 25.
In step S119, the image 1 and the image 2 are
An image 3 in which the blur is corrected is created. The details of the image processing in step S119 will be described later with reference to FIGS.
This will be described with reference to the flowchart of FIG.

After the images 1 and 2 are erased from the buffer memory 30 in S120, the image 3 is recorded on the memory card 25 in S121, and the basic sequence of the camera ends. Next, a flowchart (corresponding to S119 in FIG. 4) for performing the blurred image correction processing in FIGS. 5 and 6 will be described. Note that data D1 (x, y) of image 1 and data D2 of image 2
(X, y) is matrix data as shown in FIG. 7, where the maximum value of x is Xmax and the maximum value of y is Ymax. The image 1 and the image 2 are stored in the buffer memory 30.
Are expanded to return to the data of the space coordinates. Image 1 and Image 2
Although color information is included in addition to luminance information, for simplicity, the following description will be made on the assumption that it is luminance information. Further, the following description will be made on the assumption that the image 1 and the image 2 are 8-bit quantized data, the black level is 0, and the white level is 255.

First, in S200 of FIG. 5, coordinate parameters x and y indicating the position of each pixel are initialized to 1, and
Horizontal panning flag H, vertical panning flag V
Is initialized to 0. In S201, image data D1 (x, y) to D1 of 8 × 8 pixel data blocks from image 1
(X + 7, y + 7) is read out and doubled in S202. This is because the exposure amount of the image 1 was half of the appropriate value. Even if the data becomes 256 or more, the information is held as it is. This is to refer to the data of the image 1 having a short exposure time at the portion where the overexposure occurs.

In step S203, it is checked whether or not each image data of the image 1 is overexposed (200 or more) or blackened (50 or less). S2 if it is determined that there is no overexposure or black spots
Move to 04. If it is determined that there is overexposure or underexposure, the process proceeds to S209. In S204, 8 from image 2
Image data D2 (x, x) of × 8 pixel data blocks
y) to D2 (x + 7, y + 7) are read. And S2
At 05, as in S203, a check is made for overexposure and underexposure.

This is for avoiding portions with overexposure and black spots when performing image processing from images 1 and 2. If it is determined in step S205 that there is no whiteout or blackout, the process proceeds to step S206. If it is determined that there is overexposure or underexposure, the process proceeds to S209.

In step S206, the image data D1 (x, y) shown in FIG.
For eight pixel data blocks, four types of spatial frequencies fx and fy having a period of 8 pixels to 2 pixels in both the vertical and horizontal directions
(Fx = 1 to 4, fy = 1 to 4) are subjected to a spatial frequency analysis by a method such as Fourier transform, and as shown in FIG. 8B, the amplitude A1 (f
x, fy) and the phase P1 (fx, fy) are calculated.

In S207, the amplitude A2 (fx, fy) and the phase P2 (fx, fy) are obtained for the 8.times.8 pixel data blocks of the image data D2 (x, y) as in S206. In S208, an amplitude ratio cA (f), which is a ratio obtained by dividing A1 (fx, fy) by A2 (fx, fy) for each frequency.
x, fy) and phases P2 (fx, fy) and P1 (f
The phase difference dP (fx, fy), which is the difference between x, fy), is obtained.

The processing from S201 to S208 is repeated until the horizontal direction x is equal to or more than the maximum value Xmax of the number of pixels in the horizontal direction of the image, and the vertical direction y is the maximum value Y of the number of pixels in the vertical direction of the image.
S209, S210, S210
In steps 211 and S212, the loop is completed to complete.

When this loop is completed, 8 ×
The amplitude ratio cA (fx, f
y) and the phase difference dP (fx, fy) are determined. Generally, in a blurred image, since the waves of various spatial frequency components are shifted and overlap with each other, the amplitude of the spatial frequency (particularly, the high-frequency component) is reduced and the phase is changed as compared with a subject image without blur. I do. That is, since the image 1 with the higher shutter speed is photographed more sharply than the image 2 with the lower shutter speed, the frequency component (especially the high frequency component) of the image 2 is larger than the image 1 in the spatial frequency region. I'm affected by blur.

Accordingly, the spatial frequency component (especially the high frequency component) of the image 2 is converted into the amplitude ratio cA (fx, fy) and the phase difference d.
By correcting to the level of the spatial frequency of the image 1 based on P (fx, fy), it is possible to generate an image in which the blur has been corrected. However, since the exposure time of image 1 is short, the signal-to-noise ratio (SN ratio) is poor and contains a lot of noise components. Therefore, the amplitude ratio cA (fx, fy) and the phase difference calculated based on image 1 are calculated. dP (fx,
fy) also includes a lot of noise, and even if the image 2 is corrected based on the noise, the image becomes a noisy image.

The influence of noise is obtained by averaging the amplitude ratio cA (fx, fy) and the phase difference dP (fx, fy) obtained for each block of 8.times.8 pixel data over all blocks. Is removed. Then, the noise-free amplitude ratio cA (fx, fy) and the phase difference dP (fx, f
By correcting the image 2 using y), it is possible to generate a blur-free image without noise.

In S213, the amplitude ratio cA (fx, fy)
And the phase difference dP (fx, fy) is averaged over all blocks to obtain an average amplitude ratio Aav (fx, fy) and an average phase difference Pav (fx, fy), and the process proceeds to S214 in FIG. In S214 of FIG. 6, the average amplitude ratio Aav (fx, f
Of the components in y), the sum of components satisfying fx> fy is taken as Axsum, and the sum of components satisfying fy> fx is taken as Aysum. Where Axsum, Ay
sum corresponds to the magnitude of the blur in the x and y directions, respectively, and the larger the value of Axsum and Aysum, the larger x
This indicates that the shake in the directions y and y is large.

In S215, Axsum> Aysum ×
4 is detected. In other words, if the blur in the x direction (horizontal direction) is four times or more greater than the blur in the y direction (vertical direction), the camera is intentionally shaken in the x direction (horizontal panning), and the x direction is used. Is determined to be increasing, and the horizontal follow shot flag H is set to 1.

In S216, Aysum> Axsum ×
4 is detected. In other words, when the blur in the y direction (vertical direction) is four times or more greater than the blur in the x direction (horizontal direction), the camera is intentionally shaken in the y direction (vertical follow shot), so that the y direction is used. Is determined to be increasing, and the vertical follow shot flag V is set to 1.

After initialization with x = 9 and y = 9 in S217, image data D2 (x, y) to D2 (x + x) of 8 × 8 pixel data blocks of the blurred image 2 in S218.
7, y + 7) to four types of spatial frequencies fx, fy (fx
= 1 to 4, fy = 1 to 4), the amplitude A1 (fx, fy) and the phase P1 for each spatial frequency
(Fx, fy) is calculated.

In step S219, the horizontal follow shot flag H is set.
Is 1, a frequency component in which the frequency component fx in the x direction is 1 (low frequency component) and each frequency component fy in the y direction is 1 to 4 in order to mainly correct vertical blurring of the image 2. , The amplitude A2 (fx, fy) is changed to Aav (f
x, fy) times amplitude A3 (fx, fy), phase P2
A phase P3 (fx, fy) obtained by advancing (fx, fy) by Pav (fx, fy) is calculated.

In step S220, the vertical panning flag V
Is 1, if the frequency component fy in the y direction is 1 (low frequency component) and each frequency component fx in the x direction is 1 to 4, in order to mainly correct horizontal blurring of the image 2. , The amplitude A2 (fx, fy) is changed to Aav (f
x, fy) times amplitude A3 (fx, fy), phase P2
A phase P3 (fx, fy) obtained by advancing (fx, fy) by Pav (fx, fy) is calculated.

In S221, the panning flag H in the horizontal direction is set.
Is 0 and the vertical panning flag V remains 0,
In order to correct the horizontal and vertical blurring of the image 2 assuming that the panning is not a panning, the frequency components fy,
The amplitude A2 (fx, fy) is converted to Aav (fx, fx,
fy) times the amplitude A3 (fx, fy) and the phase P2 (f
x, fy) by Pav (fx, fy)
Calculate (fx, fy).

In step S222, the amplitude A3 (fx, fy)
And the image data D3 (x, y) to D3 (x + 7, y + 7) of the image 3 based on the phase P3 (fx, fy). X> Xmax−
Until 8, y> Ymax−8, the 8 × 8 pixel data blocks are sequentially moved by 8 pixels, and S223 and S223
At 224, S225, and S226, the process is completed by turning the loop. When this loop is completed, if panning is performed, image data D3 (x, y) of the image 3 that has been blur-corrected according to the panning direction (x = 9 to Xmax-8, y = 9 to
Ymax-8) is obtained, and when the panning is not performed, the direction-independent image blur correction 3 is performed.
Image data D3 (x, y) (x = 9 to Xmax-8,
y = 9 to Ymax-8).

Next, in the blocks around the entire periphery of image 3 (8 pixels in width), there is a possibility that a background not included in image 1 may enter image 2 due to a difference between image 1 and image 2.
There is a high possibility that blurring cannot be correctly reproduced. So S2
At 27, the peripheral blocks are shifted from image 2 to image 3. This completes the blur correction and ends.

As described above, in the blurred image correction processing shown in FIGS. 5 and 6, the image 1 and the image 2 are divided into 8 × 8 pixel data blocks, and the blocks are shifted by 8 pixels to 4 × 4 pixel data blocks. Calculate the amplitude data and the phase data, calculate the difference between the ratio of the amplitude data of the image 1 and the image 2 and the phase data, calculate the average of them, detect the panning direction based on the image 1 and the image 2, The image 2 is formed by the average amplitude ratio data and the average phase difference data according to the detected panning direction.
Is corrected, and the data of the image 3 in which the blur is corrected is reconstructed from the corrected amplitude data and phase data.

Next, a flowchart (corresponding to S119 in FIG. 4) of a blurred image correction process different from FIGS. 5 and 6 will be described with reference to FIG. The blur image correction processing shown in FIG.
6 is different from FIG. 6 in the method of detecting the panning direction.
In FIG. 9, the panning direction setting member 4 indicates the panning direction.
That is, the detection is performed according to the setting state of 1.

The operation of FIG. 9 corresponds to steps S214 to S216 of FIG.
By replacing S314 with S316. In S314 of FIG. 9, the panning direction setting member 4
1 is read. In S315, if the setting state is the vertical panning shot, the vertical panning flag V
Is set to 1.

In step S316, if the setting state is horizontal panning shooting, the horizontal panning flag H is set to 1.
(Explanation of Modifications) The present invention is not limited to the embodiments described above, and various modifications and changes are possible.

In the above embodiment, image processing for blur correction is performed in the electronic camera 1. However, an image processing system as shown in FIG. 10 is configured so that the electronic camera 1 captures images 1 and 2. The image 1 and the image 2 are fetched into the image processing device 50 including a personal computer or the like via the memory card 25 or the like on which the image 1 and the image 2 are recorded, Based on this, the image 3 with the blur corrected may be generated. In this case, in the flowchart of FIG.
The electronic camera 1 performs the processing up to and including S117 to S12.
The processing up to 1 is performed by the image processing apparatus 50. According to such an image processing system, the correction processing portion of a blurred image having a large calculation scale and a large load can be left to the image processing apparatus 50 having a high processing capability outside the electronic camera 1, so that the electronic camera can take a photograph. It is possible to concentrate on the operation, and it is possible to perform an operation such as high-speed continuous shooting, and it is possible to perform a more advanced blurred image correction process on the image processing device 50 side at a high speed.

In the above embodiment, the acquisition of the first image is performed before the acquisition of the second image. However, the acquisition of the second image may be performed first. In the above embodiment, the blurred image correction process is performed by analyzing, in the spatial frequency domain, the second image captured with the appropriate exposure time and the first image captured with half the appropriate exposure time. However, the first image may have an exposure time shorter than the exposure time of the second image. However, the exposure time of the first image is preferably not more than about 1/2 of the exposure time of the second image so as not to cause image blurring. Also, the exposure time of the second image does not necessarily have to be exactly the proper exposure time,
Any exposure time may be used as long as the effect of blurring appears.

In the above embodiment, the blurred image is corrected by analyzing two images captured at two different exposure times in the spatial frequency domain. Using more than one image,
The blurred image may be corrected. This makes it possible to further reduce the influence of image noise.

In the above-described embodiment, the first image and the second image are recorded, and then the blurred image correction process is performed immediately to generate the third image. After the recording of the second image and the second image, the blurred image correction process may be performed after a certain period of time to generate the third image. This makes it possible to postpone the blurred image correction processing that places a heavy burden on the electronic camera, so that it is possible to continuously capture and record images at high speed. For example, a blurred image correction process is performed on captured image data according to a specific operation by a user.

In the above embodiment, the exposure time is controlled by a mechanical shutter. However, the exposure time may be controlled by an electronic shutter function provided in the image sensor. In the above embodiment,
The presence / absence of image blur was determined by comparing the spatial frequency components of the first image data and the second image data. However, the presence / absence of image blur was determined using the spatial frequency component of a single image data. You can also. If camera shake, subject shake, or out-of-focus occurs in the normal image data,
The image data is smoothed, and the high-frequency spatial frequency components are damaged. Therefore, the presence or absence of image blur may be determined by comparing the high frequency component in the spatial frequency region with a predetermined value.

Alternatively, it is possible to determine the presence or absence of image blurring from the spatial frequency component of the image data from the amount of compression code. In general, it can be determined that the higher the compression code amount, the more high frequency components in the spatial frequency domain. Therefore, the presence or absence of image blur may be determined by comparing the compression code amount with a predetermined value. In addition, since such a value of the compression code amount is obtained from the result of the image compression processing, it is not necessary to add any special processing.

Alternatively, a known spatial frequency filter (for example, a high-pass filter for obtaining a difference between adjacent pixels)
It is of course possible to simply calculate the spatial frequency component using, for example, image detection and contrast detection to determine the presence or absence of image blur.
Further, in the above embodiment, based on two image data obtained at different exposure times, the spatial frequency component of the image data obtained at the long exposure time is corrected by the amplitude ratio data and the phase difference data to reduce the blur. Although the corrected image has been generated, the image processing for generating the image in which the blur has been corrected is not limited to this. For example, an image with less blur can be obtained by simply amplifying the high-frequency component of blurred image data.

In the above embodiment, the blur correction processing is performed on the spatial frequency component in the direction perpendicular to the panning direction according to the panning direction to obtain more natural panning image data. However, the image processing is not limited to this method, and may be any image processing that performs blur correction while leaving the feel of a panning shot. For example, image processing in which image data is subjected to high-pass filtering only in a direction corresponding to a panning direction may be used.

[0084]

As described above, in the electronic camera and the image processing system according to the present invention, since a blurred image can be corrected without using a blur correction optical system or a blur sensor, the size of the electronic camera and the image processing system can be reduced. Can be achieved, and an increase in cost can be suppressed.

In the electronic camera and the image processing system according to the present invention, blurring is performed in the spatial frequency domain on the basis of two image data obtained at different exposure times, so that image blurring during exposure can be corrected. it can. Further, in the electronic camera and the image processing system according to the present invention, it is determined whether or not there is a blur based on image data, and image processing for blur correction is performed only when it is determined that there is a blur. Correction can be made efficiently as needed, and if it is determined that there is no blur, it is possible to quickly shift to the next operation (for example, a shooting operation).

In the electronic camera and the image processing system according to the present invention, since the image processing for blur correction is optimized according to the panning direction, a flow image having a natural image processing result can be generated.

[Brief description of the drawings]

FIG. 1 is a front perspective view showing a configuration of an embodiment of an electronic camera according to the present invention.

FIG. 2 is a rear perspective view showing a configuration of an embodiment of an electronic camera according to the present invention.

FIG. 3 is a block diagram showing an electrical configuration inside the electronic camera according to the present invention.

FIG. 4 is a flowchart showing a basic sequence of the electronic camera.

FIG. 5 is a flowchart illustrating an example of a shake image correction process.

FIG. 6 is a flowchart illustrating an example of a shake image correction process.

FIG. 7 is a diagram illustrating a configuration of image data.

FIG. 8 is a diagram showing a block configuration of image data, amplitude data, and phase data.

FIG. 9 is a flowchart illustrating another example of a blurred image correction process.

FIG. 10 is a diagram showing a configuration of an embodiment of an image processing system according to the present invention.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 electronic camera 2 shooting lens 3 shutter button 8 display LCD 10 CPU 13 shutter drive circuit 15 shutter 20 CCD 21 DSP 25 memory card 30 buffer memory 41 panning direction setting member 50 image processing device

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 5/235 H04N 5/235 5/91 5/91 J

Claims (9)

[Claims] 1. An image capturing means for capturing an image of a subject, a recording means for recording image data, and an image generating means for continuously generating first image data and second image data by the image capturing means. A blur determining unit configured to determine whether the first image data or the second image data is blurred based on the first image data and the second image data; and And image processing means for generating third image data in which blur has been corrected by image processing based on the second image data, wherein the first image data or the second image data is If it is determined that the image is not blurred, the image processing means does not perform image processing for blur correction, and the recording means does not perform image processing of one of the first image data or the second image data. If the first image data or the second image data is determined to be blurred by the blur determination unit, the image processing unit performs image processing for blur correction, and An electronic camera for recording the third image data. 2. An image pickup means for picking up a subject image, an exposure control means for controlling an exposure time for picking up the subject image by the image pickup means, a recording means for recording image data,
A first image data imaged at a first exposure time by the exposure control means and the imaging means and a second image data imaged at a second exposure time longer than the first exposure time are continuously generated. An image generating means for correcting a blur by correcting a spatial frequency component included in the second image data based on the first image data and the second image data generated by the image generating means Image processing means for generating the third image data, and blur determination means for determining whether or not the second image data is blurred based on the first image data and the second image data. When the blur determination unit determines that the second image data is not blurred, the image processing does not perform blur correction processing, and the recording unit performs the second image data processing.
When the image data is recorded, and when the second image data is determined not to be blurred by the blur determining unit, the image processing performs a process of correcting blur, and the recording unit corrects the blur. An electronic camera for recording the third image data. 3. An image pickup means for picking up a subject image, an exposure control means for controlling an exposure time for picking up a subject image by the image pickup means, and an image picked up at a first exposure time by the exposure control means and the image pickup means. First image data and the first image data;
Image generating means for continuously generating second image data picked up at a second exposure time longer than the second exposure time;
A panning direction setting unit that sets a panning direction when capturing the image data of the second image data, and the second image data based on the first image data and the second image data generated by the image generating unit. Image processing means for generating third image data by correcting a spatial frequency component included in the image data, wherein the image processing means responds to the panning direction set by the panning direction setting means. An electronic camera that corrects a spatial frequency component included in the second image data to change the content of image processing when generating third image data. 4. The electronic camera according to claim 3, wherein
The image processing means may include, if the panning direction set by the panning direction setting means is horizontal, the second
Generating the third image data by mainly correcting the vertical spatial frequency component included in the image data of
When the panning direction set by the panning direction setting means is a vertical direction, the third image data is generated by mainly correcting a horizontal spatial frequency component included in the second image data. An electronic camera characterized by the following. 5. An image pickup means for picking up a subject image, an exposure control means for controlling an exposure time for picking up a subject image by said image pickup means, and an image picked up at a first exposure time by said exposure control means and said image pickup means. First image data and the first image data;
Image generating means for continuously generating second image data captured at a second exposure time longer than the first exposure time;
Panning direction detecting means for detecting a panning direction when the second image data is captured, based on the image data and the second image data, and the first image generated by the image generating means. Image processing means for generating third image data by correcting a spatial frequency component included in the second image data based on the image data and the second image data, the image processing means comprising: In accordance with the panning direction detected by the panning direction detecting means, the content of image processing for generating third image data by correcting a spatial frequency component included in the second image data is changed. An electronic camera, characterized in that: 6. The electronic camera according to claim 5, wherein
The image processing means may include, if the panning direction detected by the panning direction detection means is horizontal, the second
Generating third image data by mainly correcting vertical spatial frequency components included in the image data of
When the panning direction detected by the panning direction detecting means is the vertical direction, the third image data is generated by mainly correcting horizontal spatial frequency components included in the second image data. An electronic camera characterized by the following. 7. A method according to claim 2, wherein the second and third steps are different from each other.
In the electronic camera described in the above, the second exposure time is an exposure time that gives an appropriate exposure amount such that the luminance level of the image data is appropriate, and the first exposure time is the second exposure time. An electronic camera characterized by being about 1/2 or less. 8. An image pickup means for picking up an image of a subject and generating image data, an image processing means for performing image processing on the image data to correct the influence of blur contained in the image data, and the image data Discriminating means for discriminating whether or not the image data is affected by blur. The image processing means is included in the image data only when the discriminating means determines that the image data is affected by blur An electronic camera that performs image processing for correcting the effect of blur. 9. An image pickup means for picking up a subject image, an exposure control means for controlling an exposure time for picking up a subject image by said image pickup means, and an image picked up by said exposure control means and said image pickup means at a first exposure time. First image data and the first image data;
An electronic camera including image generation means for continuously generating second image data captured at a second exposure time longer than the first exposure time; and the first camera generated by the electronic camera.
Based on the image data and the second image data, while detecting a panning direction when the second image data is captured, based on the first image data and the second image data, An image processing system comprising: an image processing device that corrects a spatial frequency component included in second image data according to a panning direction and generates third image data.