JP5289105B2 - Image processing apparatus, control method therefor, and imaging apparatus - Google Patents

Description

The present invention, images processing device and a control method thereof, and relates to an imaging apparatus, particularly to an image processing apparatus and image processing method performs image processing on the image captured, and relates to an imaging apparatus.

  2. Description of the Related Art Conventionally, there has been known an imaging apparatus that detects a defocus amount distribution from a photographed image and performs image processing based on the defocus amount distribution in a camera having a plurality of ranging areas. In such an imaging apparatus, a defocus amount is detected in each of a plurality of ranging areas, and the defocus amount is classified into a plurality of defocus amount groups based on a histogram of the detected defocus amount. Then, image processing corresponding to the defocus amount of each classified grape is performed on the captured image.

  For example, Patent Document 1 describes an imaging device that is classified into a plurality of defocus amount groups based on a defocus amount with a low appearance frequency based on a defocus amount histogram.

JP 2008-15754 A

  However, in the conventional technique, it is not always possible to extract a defocus amount group to which only a main subject that is a subject intended by the photographer belongs from among the defocus amount groups classified based on a histogram of defocus amounts. There was no problem.

  This is because the defocus amount group is formed by classifying the defocus amount into groups depending on whether it is within or outside the wide defocus range, so other subjects may be included in the group to which the main subject belongs. Because it is expensive. It is generally difficult to set a defocus range that separates the main subject and other subjects at a stage where the defocus amounts are unknown.

  As described above, according to the conventional method of classifying the defocus amount into a plurality of defocus amount groups based on the histogram, even if image processing is performed only on the main subject, another subject belonging to the same defocus amount group is used. The image processing is performed up to. Therefore, conventionally, there has been a problem in that it is difficult to selectively perform image processing that is relatively more effective than the other subjects on the main subject.

Accordingly, an object of the present invention, an image processing apparatus and a control method thereof capable of obtaining the defocus amount of each object to be Utsushitai, and to provide an image pickup apparatus.

The above-described object is an image sensor that photoelectrically converts a subject image to generate and output an image signal, and an image signal obtained by independently receiving light beams that have passed through different areas of the exit pupil of the imaging optical system. capable of outputting the image pickup device and a focus detection means for detecting the defocus amount of the object based on the phase difference of the output image signal from the imaging device, the subject detection processing on the image signal output from the imaging device performed, and a subject detecting means for identifying a detected region on the imaging element corresponding to a region of the photographic material were the focus detection means, the imaging element corresponding to the realm of the subject identified by the object detecting means based on the phase difference of the image signals output from the area of the upper, Ru is achieved by an image processing apparatus characterized by detecting the defocus amount of the detected object.

According to the present invention , since the defocus amount for each subject can be obtained, for example , image processing can be selectively performed on the region of the subject intended by the photographer.

It is a block diagram which shows the structure of an example of the imaging device which can apply this invention. It is a figure for demonstrating schematically the imaging optical system containing the image pick-up element based on the pixel for imaging. It is a figure for demonstrating schematically the imaging optical system containing the image pick-up element based on the pixel for focus detection. It is a figure which shows arrangement | positioning of an example of the pixel for imaging on an image sensor, and the pixel for focus detection. It is a flowchart which shows an example imaging process applicable to embodiment of this invention. 2 is a flowchart illustrating an example of image processing according to the present embodiment. It is a figure for demonstrating the image processing by this embodiment. It is a figure for demonstrating the image processing by this embodiment. It is a figure for demonstrating the setting method of the image processing control parameter f.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows an exemplary configuration of an imaging apparatus 100 to which the present invention can be applied. The image sensor 107 is composed of a CCD or CMOS sensor, forms an object optical image that has passed through the photographing lens 101, and outputs an image signal obtained by converting the image into electric charges according to the amount of light of the imaged object optical image. To do. In addition, the imaging element 107 includes a focus detection pixel that outputs a subject image signal, which will be described later, in addition to the imaging pixel that outputs an image signal. The imaging signal output from the imaging element 107 includes these image signal and subject image signal.

  The main mirror 102 having the semi-transmissive portion is retracted out of the photographing light beam at the time of photographing, and is obliquely installed in the photographing optical path at the time of detecting the focus. FIG. 1 shows a state where the main mirror 102 is inserted in the photographing light beam (mirror down). The main mirror 102 guides a part of the light beam that has passed through the photographing lens 101 to a finder optical system including the focusing plate 103, the pentaprism 104, and the eyepiece lens 105 while being obliquely provided in the photographing optical path.

  The sub mirror 106 can be folded and unfolded with respect to the main mirror 102 in synchronization with the operation of the main mirror 102. A part of the light beam that has passed through the semi-transmissive portion of the main mirror 102 is reflected downward by the sub mirror 106 and is incident on the phase difference type focus detection unit 108 to detect the in-focus state of the photographing lens 101.

  On the other hand, in contrast-type AF distance measurement, the main mirror 102 is always retracted out of the photographing light flux. Then, in synchronization with this operation, the subject optical image that has passed through the photographing lens 101 is formed on the image sensor 107 in a state in which the sub mirror 106 is folded with respect to the main mirror 102 (mirror up), and the photographing lens 101 is focused. A state is detected.

  The system controller 112 includes, for example, a CPU, a ROM, a RAM, and the like. The CPU appropriately controls the operation of each unit of the imaging apparatus 100 using the RAM as a work memory in accordance with a program stored in advance in the ROM.

  The lens driving unit 113 is connected to the system controller 112, communicates with the photographing lens 101, a lens driving mechanism that drives the photographing lens 101 to perform focus adjustment, and a driving circuit that controls the lens driving mechanism. Is provided. The mirror driving unit 114 is connected to the system controller 112 and drives the main mirror 102 out of the imaging light flux. The image sensor driving unit 115 is connected to the system controller 112 and drives the image sensor 107.

  The analog signal processing unit 109 performs signal processing such as gain adjustment and noise removal on the imaging signal output from the imaging element 107, and outputs it as an image signal. The A / D converter 110 converts the image signal based on the analog signal output from the analog signal processing unit 109 into an image signal based on a digital signal (hereinafter referred to as image data).

  The digital signal processing unit 111 is connected to the system controller 112 and performs image processing such as shading correction and gamma correction on the image data output from the A / D converter 110. The digital signal processing unit 111 further includes a detection unit that detects the in-focus state of the photographing lens 101 based on image data based on the subject image signal output from the focus detection pixels of the image sensor 107.

  The buffer memory 116 is connected to the digital signal processing unit 111 and can store image data for a plurality of frames captured by the image sensor 107. For example, the image data A / D converted by the A / D converter 110 is temporarily stored in the buffer memory 116. The digital signal processing unit 111 reads the image data stored in the buffer memory 116 and performs each process described above, and the processed image data is stored in the buffer memory 116 again.

  A recording / playback signal processing unit 117 serving as a recording unit is connected to the digital signal processing unit 111 and controls recording and playback of image data on the external storage medium 118. The image data that has been subjected to various types of processing by the digital signal processing unit 111 is temporarily stored in the buffer memory 116, supplied to the recording / reproducing signal processing unit 117, and stored in the external storage medium 118. As the external storage medium 118 as a recording medium, for example, a removable nonvolatile memory can be used.

  The recording / playback signal processing unit 117 can perform compression coding on the image data and can perform decompression processing on the compressed image data that has been compression-coded. As a compression encoding method, for example, a JPEG method can be applied. For example, the recording / playback signal processing unit 117 performs compression coding on the image data and stores the image data in the external storage medium 118. The recording / playback signal processing unit 117 expands the compressed image data read from the external storage medium 118.

  The display unit 120 includes a display device such as an LCD and a drive circuit that drives the display device, and reproduces and displays captured images and image data stored in the storage medium 118. When displaying an image on the display unit 120, the image data stored in the buffer memory 116 is read out, and a digital image signal that is image data is converted into an analog image signal by the D / A converter 119. Then, an image is displayed on the display unit 120 using the analog image signal.

  Images captured by the image sensor 107 are displayed on the display unit 120 in two types. The first form is a display form when the release operation is not performed, and is a display form called a through image in which images repeatedly picked up by the image sensor 107 are sequentially updated and displayed. The second form is a display form called a freeze picture that displays an image picked up by the image pickup element 107 for a predetermined time after the release operation of the camera.

  The subject detection unit 121 is connected to the digital signal processing unit 111 and detects whether or not a subject exists in the image data subjected to various processes by the digital signal processing unit 111. In the subject detection process, for example, at least one of color information, saturation information, contrast information, and contour information is detected from the image data, and the region of the subject is detected based on the detected information.

  Note that a human face may be detected by face detection, and subject detection may be performed based on the detected face. For example, it is conceivable that the contour of the face detected by the face detection is acquired, the contour that can be considered continuous with respect to this contour is further detected, and the contour of the entire subject is obtained. Not limited to this, it is conceivable to measure the distance to the face detected by the face detection, and to set a region focused on the measured distance within a predetermined range as the subject region.

  Many methods for face detection have already been proposed and put into practical use. For example, Japanese Patent Laid-Open No. 8-63597 proposes a method of determining a face candidate area corresponding to the shape of a human face and determining the face area from the feature amount of the face candidate area. In addition, a method for detecting a face candidate area by extracting the outline of a human face from an image and a correlation value with a plurality of templates having various shapes of the face are calculated, and the face candidate area is calculated based on the correlation value. A method has been proposed.

  The operation unit 122 is connected to the system controller 112 and receives a user operation on the imaging apparatus 100. The operation unit 122 includes various operators for operating the imaging apparatus 100, such as a power switch for turning on / off the camera, a release button, and a setting button for selecting a shooting mode such as a person shooting mode. Is provided. When these operators are operated by the user, a signal corresponding to the operation is input to the system controller 112. The release button includes a first switch (SW) that is turned on by a first stroke operation (half-press operation) of the release button operated by the photographer and a second stroke operation (full-press operation) of the release button. A second SW that is turned on is connected.

  Next, the structure of an example of the image sensor 107 including the imaging pixels and the focus detection pixels will be described with reference to FIGS.

  FIG. 2 schematically illustrates an imaging optical system including the imaging element 107 related to the imaging pixels. Of the four pixels of 2 rows × 2 columns shown in FIG. 2A, pixels having G (green) spectral sensitivity are arranged in the diagonal two pixels, and R (red) and B (blue) are arranged in the other two pixels. The Bayer arrangement is employed in which one pixel having a spectral sensitivity of 1) is arranged. In addition, focus detection pixels having a structure described later are distributed and arranged in a predetermined rule between the Bayer arrays.

FIG.2 (b) shows the AA sectional drawing in Fig.2 (a). The photodiode PD is a photoelectric conversion element of a CMOS image sensor and constitutes a pixel. The contact layer CL is a wiring layer that forms signal lines for transmitting various signals in the CMOS image sensor. For each pixel, a red color filter CF _R , a green color filter CF _G and a blue color filter CF _B are arranged in a Bayer array as shown in FIG. On-chip microlenses ML are arranged on the forefront of each pixel.

Light from the subject is incident on the on-chip microlens ML via an optical photographing lens TL (Taking Lens) and is condensed. Condensed light is incident on the photodiode PD, for example, the extracted red wavelength components by the color filter CF _R. The photodiode PD converts incident light into electric charge by photoelectric conversion.

  Here, the on-chip microlens ML and the photodiode PD of the imaging pixel are configured to capture the light beam that has passed through the optical photographing lens TL as effectively as possible. In other words, the exit pupil EP (Exit Pupil) of the optical photographing lens TL and the photodiode PD are conjugated with each other by the on-chip microlens ML, and the effective area of the photodiode PD is designed to be large.

The pixels corresponding to the color filters CF _R , CF _G, and CF _B (hereinafter referred to as R pixel, G pixel, and B pixel, respectively) have the same structure. Accordingly, the exit pupil EP corresponding to each of the R, G, and B pixels for imaging has a large diameter, and light from the subject can be efficiently taken into the photodiode PD, and S of the imaging signal output from the imaging element can be obtained. / N is improved.

  Focus detection pixels for performing focus detection by the pupil division phase difference method are arranged between the Bayer arrays. More specifically, among the 2 rows × 2 columns of pixels constituting the Bayer array, the G pixel is left as an imaging pixel, and the R pixel and the B pixel are replaced with focus detection pixels.

  When obtaining an image signal for recording or display, the main component of luminance information is acquired by G pixels. Since human image recognition characteristics are sensitive to luminance information, if this G pixel is missing, image quality degradation is likely to be perceived. On the other hand, the R pixel or the B pixel mainly acquires color information (color difference information). Here, human image recognition characteristics are less sensitive to color information than luminance information. For this reason, it is difficult for the R pixel and B pixel for obtaining the color information to recognize the image quality deterioration even if some loss occurs.

  An image pickup optical system including the image pickup element 107 related to the focus detection pixel will be schematically described with reference to FIG. FIG. 3A shows a plan view of an example of pixels of 2 rows × 2 columns including focus detection pixels. FIG. 3B is a cross-sectional view of an example of the imaging optical system taken along plane AA in FIG. In FIG. 3B, the on-chip microlens ML and the photodiode PD have the same structure as that shown in FIG.

In FIG. 3A, a pixel S _HA and a pixel S _HB are pixels for performing focus detection. Hereinafter, the pixel S _HA and the pixel S _HB are collectively referred to as a focus detection pixel. In FIG. 3A, the filled portions in the pixels S _HA and S _HB are openings of the wiring layer CL. That is, the light irradiated to the focus detection pixels is incident on the photodiode PD through the opening of the wiring layer CL. The signal due to the focus detection pixels, is not used for image generation, to the pixel S _HA and the pixel S _HB, filter CF _W by the transparent film is arranged.

In the example of FIG. 3, the opening _{OP HA} corresponding to the pixel _{S HA,} and, the opening _{OP HB} corresponding to the pixel _{S HB,} is provided offset in one direction from the center line of the on-chip microlens ML . Thereby, pupil division by an image sensor is performed.

Specifically, since the opening OP _HA corresponding to the pixel S _HA is biased to the right side in FIG. 3, it receives the light beam that has passed through the left exit pupil EP _HA of the optical photographing lens TL. Similarly, since the opening OP _HB corresponding to the pixel S _HB is biased to the left in FIG. 3, it receives the light beam that has passed through the right exit pupil EP _HB of the optical photographing lens TL.

Pixels _SHA are regularly arranged in the horizontal direction, and a subject image acquired by these pixel groups is defined as an image A. Similarly, the pixels _SHB are regularly arranged in the horizontal direction, and a subject image acquired by these pixel groups is defined as an image B. By detecting the relative position between the image A and the image B, the amount of defocus (defocus amount) of the subject image can be detected.

  FIG. 4 shows an exemplary arrangement of imaging pixels and focus detection pixels on the image sensor 107. For example, a square area of 10 rows × 10 columns = 100 pixels is defined as one block BLK. The screen based on the image signal is divided into blocks BLK. With this block BLK as a focus detection area, focus detection pixels are provided for each focus detection area.

In the upper left block BLKh (1, 1), the R pixel and the B pixel at the lower left corner are respectively replaced with a set of focus detection pixels S _HA and S _HB that perform pupil division in the horizontal direction. In the block BLKv (1, 2) on the right side of the block BLKh (1, 1), a pair of focus detection pixels S _VC and S _VD that perform pupil division in the vertical direction on the R pixel and B pixel in the lower left corner. Replace each one. The focus detection pixels S _VC and S _VD have the same structure as the focus detection pixels S _HA and S _HB , respectively. The pixel arrangement of the block BLKv (2, 1) adjacent below the block BLKh (1, 1) is the same as that of the block BLKv (1, 2). The pixel arrangement of the block BLKh (2, 2) on the right side of the block BLKv (2, 1) is the same as that of the block BLKh (1, 1).

When this arrangement rule is generalized, in block BLK (i, j), if (i + j) is an even number, focus detection pixels S _HA and S _HB for horizontal pupil division are arranged. On the other hand, if (i + j) is an odd number, focus detection pixels S _VC and S _VD for vertical pupil division are arranged. Thus, the focus detection pixels are arranged over the entire screen based on the imaging signal.

The block BLKh (i, j) indicates the block BLK in which the focus detection pixels S _HA and S _HB for horizontal pupil division are arranged. The block BLKv (i, j) represents the block BLK in which the focus detection pixels S _VC and S _VD for vertical pupil division are arranged.

Here, a signal processing method for detecting the defocus amount from the A image and the B image that are subject images acquired from the focus detection pixels of the image sensor 107 will be described. When the number of pixels constituting the focus detection pixel is L and the pixel number is i (i = 0,..., L), the A image signal is the signal A (i), and the B image signal is the signal B ( i). At this time, the following formula (1) or formula (2) is calculated for k ₁ ≦ k ≦ k ₂ .

The value M is the number of operation pixels represented by (M = L− | k |), the value k is called a relative shift amount, and the value k ₁ and the value k ₂ are generally − In many cases, it is considered as L / 2 or L / 2. Further, the operator max {a, b} indicates that the larger one is extracted from the value a and the value b, and the operator smaller than the min {a, b}, the value a and the value b is extracted. Represents what to do. Therefore, X ₁ (k), X ₂ (k), Y ₁ (k), and Y ₂ (k) in the above formulas (1) and (2) can be considered as broad correlation amounts.

Further, when the above formulas (1) and (2) are examined in detail, the value X ₁ (k) and the value Y ₁ (k) actually indicate the correlation amount in the displacement of the value (k−1). , Value X ₂ (k) and value Y ₂ (k) represent the correlation amounts in the displacement of value (k + 1), respectively. Therefore, the evaluation amount X (k), which is the difference between the correlation amounts X ₁ (k) and X ₂ (k), is the amount of change in the correlation amount of the subject image signals A (i) and B (i) at the relative displacement amount k. Means.

The correlation amounts X ₁ (k) and X ₂ (k) are minimum when the correlation between the A image and the B image is the highest from the above definition. Therefore, the evaluation amount X (k) that is the amount of change should have a value of 0 and a slope of negative when the correlation is highest. However, since the evaluation amount X (k) is discrete data, the following equation (3) is actually obtained.
X (kp) ≧ 0, X (kp + 1) <0 (3)

Further, assuming that there is a peak of the correlation amount in the relative displacement interval [kp, kp + 1] where the value {X (kp) −X (kp + 1)} is maximum, an interpolation calculation is performed according to the following equation (4): It is possible to detect a focus shift amount PR equal to or less than a pixel unit.
PR = kp + X (kp) / {X (kp) -X (kp + 1)} (4)

On the other hand, the correlation amounts Y ₁ (k) and Y ₂ (k) are opposite to the correlation amounts X ₁ (k) and X ₂ (k) when the correlation between the A image and the B image is highest from the above definition. To the maximum. Therefore, the evaluation amount Y (k) that is the amount of change should have a value of 0 and a positive slope when the correlation between the A and B images is the highest. Since the evaluation amount Y (k) is also discrete data like the above-described evaluation amount X (k), the following equation (5) is actually obtained.
Y (kp) ≦ 0, Y (kp + 1)> 0 (5)

Further, when the value {Y (kp) −Y (kp + 1)} is the maximum, an interpolation calculation according to the following equation (6) can be performed to detect a defocus amount PR equal to or less than a pixel unit.
PR = kp + | Y (kp) / {Y (kp) −Y (kp + 1)} | (6)

The defocus amount DEF of the subject image plane with respect to the planned image formation plane can be obtained from the following equation (7) from the defocus amount PR obtained in the above equation (4) or equation (6). The value is a conversion coefficient determined by the size of the opening angle of the center of gravity of the light beam passing through the pair of distance measuring pupils.
DEF = K × PR (7)

<Embodiment>
Next, an embodiment of the present invention will be described using the flowchart of FIG. FIG. 5 is a flowchart showing an example of imaging processing applicable to the embodiment of the present invention. In the imaging processing according to the present embodiment, a subject area is detected from a captured image, and a defocus amount DEF is detected for each subject area. By performing defocus amount detection in this way, a distribution of the defocus amount DEF focused on the subject is obtained. Based on the obtained defocus amount distribution, image processing is performed only on a subject region (subject region) intended by the photographer. In the following description, as an image process, a blur addition process for blurring the outline of an image will be described as an example.

  In first step S <b> 101, the system controller 112 waits for the operation of the first SW in the release switch provided in the operation unit 122. If the first SW is operated, the process proceeds to step S102.

  In step S102, an AF process for controlling the driving of the photographing lens 101 so as to focus on the subject is executed. The AF distance measurement method may be a phase difference method using the focus detection unit 108. In addition, a contrast method may be used in which a subject image formed by the photographing lens 101 is captured by the image sensor 107 and an in-focus position is determined using an image signal in the focus area. Further, a phase difference method using focus detection pixels arranged in the image sensor 107 may be used.

  When the subject is focused by the AF process in step S102, the process proceeds to step S103. In step S <b> 103, the system controller 112 waits for the second SW in the release switch provided in the operation unit 122 to be operated. If the second SW is operated, the process proceeds to step S104.

  In step S <b> 104, the system controller 112 controls the mirror driving unit 114 to drive the main mirror 102 out of the imaging light beam, and controls the image sensor driving unit 115 to drive the image sensor 107. The subject optical image passes through the imaging lens and is formed on the imaging element 107, and an imaging signal is output from the imaging element 107. This imaging signal includes an image signal from the imaging pixels and a subject image signal from the focus detection pixels. The imaging signal output from the imaging element 107 is converted into image data via the analog signal processing unit 109 and the A / D converter 110 and supplied to the digital signal processing unit 111. The digital signal processing unit 111 acquires, from the image data, image data obtained by the imaging pixels of the imaging device 107 and image data obtained by the focus detection pixels, and temporarily stores them in the buffer memory 116.

  In the next step S105, the digital signal processing unit 111 performs image processing on the image data of the imaging pixels based on the image data of the imaging pixels acquired in step S104 and the image data of the focus detection pixels. Is done. Although details will be described later, in the present embodiment, this image processing is assumed to be blurring addition processing.

  When the process of step S105 ends, the process proceeds to step S106. In step S106, the image data subjected to the image processing in step S105 is stored in, for example, the external storage medium 118.

  Next, the image processing in step S105 described above will be described with reference to FIGS. FIG. 6 is a flowchart illustrating an example of image processing according to the present embodiment. FIG. 7 shows an example of an image obtained by imaging, and FIG. 8 shows an example of a distance measurement area in the image of FIG. 7 and 8, the squares indicate the blocks BLK (i, j) where the pair of focus detection pixels described with reference to FIG. 4 are arranged.

  First, in step S <b> 201, the digital signal processing unit 111 reads out image data at the position of the imaging pixel and image data at the position of the focus detection pixel from the image data stored in the buffer memory 116. Hereinafter, the image data at the position of the imaging pixel is referred to as captured image data, and the image data at the position of the focus detection pixel is referred to as subject image image data. These captured image data and subject image data are supplied to the subject detection unit 121.

  In the next step S202, a subject region is detected by the subject detection unit 121 as a subject detection unit based on, for example, color information, saturation information, contrast information, and contour information detected from captured image data. In the example of FIG. 7, the subject detection unit 121 detects the subjects a to d from the captured image data, and specifies a region in the captured image data of the detected subject. Here, it is assumed that the subjects a, c, and d are people and the subject b is a ball. In addition, it is assumed that the subject a is a main subject that is a main subject.

  In the next step S203, the digital signal processing unit 111 determines whether or not the subject has been detected in step S202. If it is determined that the subject has not been detected, the series of processes in the flowchart of FIG. 6 is terminated, and the process proceeds to step S106 in the flowchart of FIG.

  On the other hand, if it is determined in step S203 that the subject has been detected, the process proceeds to step S204. In step S204, the defocus amount is detected for each block BLK of the image sensor 107 corresponding to the subject area detected by step S202 by the digital signal processing unit 111 serving as a defocus amount detecting unit. The Then, the digital signal processing unit 111 serving as a representative value calculating unit calculates a representative defocus amount that is a representative value representing the defocus amount of the subject area.

  In the example of FIGS. 7 and 8, the block BLK including, for example, the region of the subject a illustrated in FIG. 7 forms a distance measurement area A illustrated in FIG. Similarly, the blocks BLK including the areas of the subjects b to d in FIG. 7 form distance measuring areas C to D illustrated in FIG. 8 respectively.

  When a plurality of different defocus amounts are detected in the detected one subject region, one defocus amount (representative defocus amount) is associated with the one subject region. For example, an average value of a plurality of defocus amounts detected from one subject region is used as a representative defocus amount, and is associated with the one subject region.

  However, the present invention is not limited to this, and the defocus amount detected at the foremost side among the plurality of defocus amounts by the plurality of blocks BLK included in one subject region can be selected as the representative defocus amount. Further, the defocus amount having the highest S / N of the image image among the defocus amounts by the plurality of blocks BLK included in one subject area may be used as the representative defocus amount.

  In the photographed image example shown in FIG. 7, the representative defocus amounts are detected in the ranging areas A to D shown in FIG. 8 corresponding to the areas a to d of the subject, respectively. Thereby, the distribution of the defocus amount in the screen based on the captured image data can be detected in units of subjects.

  When the representative defocus amount of each distance measurement area is detected in step S204, the process proceeds to step S205. In step S205, for each region of the subject detected by the subject detection unit 121 by the digital signal processing unit 111 as an image processing unit, a representative defocus amount corresponding to the region and an image processing control parameter set in advance are set. Based on this, image processing is performed. That is, the digital signal processing unit 111 applies image processing common to each subject with an intensity corresponding to the representative defocus amount with respect to the subject region. Further, in the image processing, the image processing control parameter is added to the processing intensity. When the image processing is completed, a series of processes according to the flowchart of FIG. 6 is terminated, and the process proceeds to step S106 of the flowchart of FIG.

  The image processing in step S205 described above will be described in more detail. In this embodiment, as an example of image processing, a blur addition process for adding blur to an image will be described. In the following description, the area of the subject detected by the subject detection unit 121 is represented as a partial image Pn. However, the subscript n is a variable representing the area of the subject. In the example of FIG. 7, the partial image Pn becomes the partial images Pa, Pb, Pc and Pd for the subjects a to d.

For each of the partial images Pn (x, y), a normal distribution function N (i, j) is subjected to a convolution operation (filtering operation) as shown in the following equation (8), and a processed partial image On ( x, y) are generated respectively. That is, this convolution calculation is performed for each subject area. In Expression (8), the operator “*” represents a two-dimensional convolution operation.
On (x, y) = Pn (x, y) * N (i, j) (8)

The normal distribution function N (i, j) is expressed by the following equation (9).
N (i, j) = 1 / sqrt (2πσ ² ) × exp (− (i ² + j ² ) / (2σ ² )) (9)
However, in equation (9), the variable σ represents the standard deviation, and when σ = 0, the value of the normal distribution function N (i, j) is 1.

Now, the standard deviation σ is defined as in the following equation (10).
σ = | dn | / f (10)
However, in Expression (10), the value f is an image processing control parameter, and the value dn is a representative defocus amount of each partial image Pn.

  Here, a method of setting the image processing control parameter f will be described with reference to FIG. FIG. 9 shows an example of defocus amounts dn of the partial images Pa, Pb, Pc and Pd by the subjects a to d in the photographed image example shown in FIG. Further, the image processing control parameter f is conceptually shown in FIG. In FIG. 9, the defocus amounts corresponding to the partial images Pa, Pb, Pc and Pd (subjects a to d) in the photographed image example shown in FIG. 7 are defocus amounts da, db, dc and dd, respectively. Represents. An example of the depth of field when the aperture of the attached photographing lens 101 is opened is indicated by a depth of field 301.

  According to FIG. 9, the partial image Pa is in focus and the defocus amount da is zero. The partial image Pb is defocused by the defocus amount db on the closest side. The partial images Pc and Pd are defocused by defocus amounts dc and dd on the infinity side, respectively. Further, it is shown that the partial images Pa, Pb and Pc are within the depth of field 301.

  As the image processing control parameter f, for example, a value of Fδ is used by using the F value of the attached lens. Where δ is an allowable circle of confusion. Not limited to this, the image processing control parameter f may be manually set by operating the operation unit 122. With the image processing parameter f set in this way, the main subject is classified within the range of ± f / 2 around the defocus amount of the focused distance measuring point, and the other subject is classified outside this range.

  The value of the standard deviation σ changes according to the defocus amount dn and the image processing control parameter f according to the above-described equation (10). Therefore, the amount of image processing applied to each partial image Pn also varies with the defocus amount dn and the image processing control parameter f. When the defocus amount dn is 0, the standard deviation σ = 0, and the value of the normal distribution function N (i, j) is 1. Therefore, in the convolution calculation of Expression (8), partial image Pn (x, y) = processed partial image On (x, y), and partial image Pn (x, y) does not change.

  On the other hand, as the absolute value of the defocus amount dn increases, the intensity of the image processing increases, and the blur intensity of the partial image Pn having a defocus amount other than 0 after the image processing increases. If the absolute value of the defocus amount dn is not 0, the smaller the value of the image processing control parameter f, the greater the intensity of image processing according to the normal distribution function N (i, j), and the partial image Pn after image processing. The bokeh intensity increases.

  In other words, the image processing control parameter f can be said to be a parameter representing the depth of field by the image processing in the digital signal processing unit 111. The image processing control parameter f is set so that only the main subject a falls within the depth of field indicated by the image processing control parameter f when an object other than the main subject a in focus is to be subjected to blurring processing. Is set as follows. Accordingly, it is possible to perform the blur addition process on the subjects b and c that are subjects other than the main subject a that are within the depth of field 301 when the attached photographing lens 101 is opened. It can be made to stand out relative to the other subjects b to d.

  Depending on the setting of the image processing control parameter f, it is possible to obtain a blur effect photographed with a photographing lens having a larger aperture than that of the mounted photographing lens by post-processing. Also, when shooting a high-brightness subject, the blurring effect when shooting with a large aperture opening is reduced even when adjusting the exposure amount by reducing the aperture opening to exceed the limit on the high-speed side of the shutter speed. It can be obtained by processing.

  Further, the function for performing the convolution operation on the partial image Pn is not limited to the normal distribution function shown in Expression (9), and the amount of blur to be added can be controlled by other distribution functions. Further, the image processing applied to the partial image Pn is not limited to the blur addition processing. For example, sharpness processing corresponding to the defocus amount dn may be performed on the partial image Pn. Further, for example, it is conceivable to change the contrast, lightness, and saturation for each partial image Pn according to the defocus amount dn.

  As described above, according to the present invention, a defocus amount distribution focused on a subject by detecting a plurality of subject areas from pre-captured image data and detecting a defocus amount for each subject area. Get. As a result, image processing can be performed only on the area of the subject intended by the photographer.

Claims (9)

An imaging device that photoelectrically converts a subject image to generate and output an image signal, and that can output image signals obtained by independently receiving light beams that have passed through different areas of the exit pupil of the imaging optical system Elements,
Focus detection means for detecting the defocus amount of the object based on the phase difference between the image signal output from the imaging element,
Wherein performs subject detection processing on the image signal output from the imaging device, and an object detecting means for specifying a region on the imaging element corresponding to the sensed area of the photographic material was,
Said focus detecting means, based on the phase difference of the image signal output from the region on the imaging element corresponding to the realm of the subject identified by the subject detecting means, the defocusing of the sensed object An image processing apparatus for detecting an amount . Image processing means for applying predetermined image processing to the image signal;
  The image processing means performs the predetermined image processing for each area of the subject specified by the subject detection means according to the defocus amount detected by the focus detection means. The image processing apparatus according to 1. The image processing means includes
A blur addition process for adding blur to the image signal is applied as the predetermined image processing, and the image of the subject area is reduced so that the blur intensity to be added becomes smaller as the subject area has a smaller defocus amount. The image processing apparatus according to claim 2 , wherein blur addition processing is applied to the signal . The focus detection means includes
Mean value of the defocus amount detected based on a phase difference of the image signal output from the region on the imaging element corresponding to a region of the object identified by the object detecting means, of the defocus amount defocus amount cormorants Chi-focus position is detected in the forefront, or any of the higher the defocus amount of S / N of the most image signals of the defocus amount, corresponding to a region of the subject the image processing apparatus according to any one of claims 1 to 3, characterized in that <br/> detected as the defocus amount. The subject detection means includes
Color information from the image signal, chroma information, detects at least one of the contrast information and contour information, any claims 1 to 4, characterized in that for detecting the object based on the detected information The image processing apparatus according to claim 1. The subject detection means includes
Wherein detecting a face from an image signal, the image processing apparatus according to any one of claims 1 to 4, characterized in that for detecting the object based on the detection face. The imaging device includes an imaging pixels for outputting the image signal, and a focus detection pixels for outputting the image signal
The image processing apparatus according to any one of claims 1 to 6, wherein the this. An imaging device that photoelectrically converts a subject image to generate and output an image signal, and that can output image signals obtained by independently receiving light beams that have passed through different areas of the exit pupil of the imaging optical system An image processing apparatus control method comprising an element,
Focus detection means, and the focus detection step of detecting a defocus amount of the object based on the phase difference between the image signal output from the imaging element,
Yes object detection means, performs subject detection processing on the image signal output from the imaging device, and a subject detection step of identifying a detected region on the imaging element corresponding to a region of the photographic material was And
Wherein said focus detection means in focus detection step, based on said phase difference of said image signal output from the region on the imaging element corresponding to the realm of the subject identified by the object detection step is the detection A method for controlling an image processing apparatus, comprising: detecting a defocus amount of a subject . An imaging optical system;
An imaging device that photoelectrically converts a subject image to generate and output an image signal, and that can output image signals obtained by independently receiving light beams that have passed through different areas of the exit pupil of the imaging optical system Elements,
Focus detection means for detect the defocus amount of the object based on the phase difference before Kizo signal outputted from the imaging element,
Wherein performs subject detection processing on the image signal output from the imaging device, and an object detecting means for specifying a region on the imaging element corresponding to the sensed area of the photographic material was,
Said focus detecting means, based on the phase difference of the image signal output from the region on the imaging element corresponding to the realm of the subject identified by the subject detecting means, the defocusing of the sensed object An imaging apparatus characterized by detecting an amount .

Images