JP2003189171A - Image processing apparatus and method, control program, and recording medium - Google Patents

Abstract

(57) [Summary] [PROBLEMS] To provide an image method capable of generating an image having a blurring effect by a large-diameter lens by image processing and obtaining a higher-resolution image. Kind Code: A1 An image shift amount in a plurality of regions in a plurality of images having input parallax is calculated (S101), and at least two image shift amounts different from each other in the plurality of regions according to the calculated image shift amount. Image processing (S104, S10
Perform 5) by switching.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and method, a control program, and a recording medium that add blur to an image and generate an image with blur.

[0002]

2. Description of the Related Art Generally, it is said that the image quality of a camera or a digital camera depends largely on the performance of an image pickup lens. However, a high-end camera lens such as a single lens reflex system has a large aperture. You can get a bright image.

Even when the subject brightness is the same, the depth of field can be changed by selecting various combinations of the aperture and the shutter speed, and as a result, the degree of blurring of subjects other than the main subject can be freely set. Can be controlled.
It is for this reason that attractive photos can be obtained with high-end models.

However, a camera having a large-diameter lens has a drawback that the entire camera is large and heavy because the image pickup lens is generally large. On the other hand, a lens shutter type camera has the advantage of being small and lightweight, but since the image pickup lens has a small aperture and a large depth of field, it was taken with a camera equipped with a large aperture lens. It is difficult to take such a blurred image.

Therefore, Japanese Laid-Open Patent Publication No. 9-181966 discloses a method of adding blur by image processing from an image picked up by a pair of image pickup lenses having a parallax. FIG. 19 is a diagram showing the concept of the image processing method in this conventional example.

First, a pair of images A and B having parallax are input. Next, based on the images A and B, a known correlation calculation or the like is used to calculate the image shift amount of each subject in the image, and the distance of each subject is calculated based on the image shift amount. Then, the blur of each subject is calculated based on the distance information, and image processing for reproducing the blur is performed on the image A or the image B. When calculating the blur, the operator inputs the focal length of the photographing lens, the F number, the focus position, and the like in advance, and the blur is calculated by combining the distance information. As described above, it is possible to generate an image having a blur due to a large-diameter lens from an image taken with a small-diameter lens.

[0007]

However, in the above-mentioned conventional example, the two images A and B are used only for calculating the image shift amount, and the image generated through the image processing for adding the blur is the image. Only one of A and image B is present. For example, taking a digital camera as an example, assuming that the number of pixels of the image A or the image B is 640 × 480 pixels (about 300,000 pixels), the two images A and B are
In order to obtain the above, at least an image pickup device of 600,000 pixels or two image pickup devices of 300,000 pixels are required. However, the image generated by adding blur is 300,000 pixels. That is, in order to obtain an image with an arbitrary number of pixels with blur added by image processing, at least twice the number of pixels is required, which causes a problem that a high-resolution image cannot be obtained due to an increase in cost and space. there were.

Further, when the image shift amount (distance) of each subject is calculated, if the image contrast is low, the image shift amount by the correlation calculation cannot be calculated in many cases. However, in the above-mentioned conventional example, no consideration is given to the processing operation in the case where the calculation cannot be performed in this way. Blurred images are output, and the output image is unnatural and unintended, and has low image quality.

Therefore, the present invention has been made in view of the above-mentioned problems, and an object of the present invention is to generate an image having a blur effect as if by a lens having a large aperture by image processing and to obtain a higher resolution image. An object is to provide an image processing apparatus and method, a control program, and a recording medium that can be obtained.

Another object of the present invention is to provide an image processing apparatus and method, a control program, and a recording medium which perform appropriate processing even when the image shift amount cannot be calculated and do not deteriorate the image quality.

[0011]

In order to achieve the above object, image shift amount calculation means for calculating the image shift amount in a plurality of regions in a plurality of images having input parallax,
The image processing apparatus includes an image processing unit that performs at least two different image processes for the images of the plurality of regions according to the image displacement amount calculated by the image displacement amount calculation unit. .

Further, according to the present invention, an image shift amount in a plurality of regions in a plurality of images having an input parallax is calculated, and an image shift amount for the plurality of regions is calculated according to the calculated image shift amount. This is an image processing method in which at least two different image processes are switched.

The present invention also provides a control program for causing a computer to execute the above image processing method.

The present invention also provides a medium recording the above control program.

Further, according to the present invention, an image shift amount calculating means for calculating an image shift amount in a plurality of regions in a plurality of images having input parallax, and the image shift calculated by the image shift amount calculating means. Blurred image processing means for performing image processing for adding blur to the images of the plurality of regions according to the amount, and image processing for adding blur to the image of the region where the image shift amount cannot be calculated. The image processing apparatus has a prohibiting control unit.

Further, according to the present invention, the image shift amount in a plurality of regions in the plurality of images having the input parallax is calculated, and the image of the plurality of regions is calculated according to the calculated image shift amount. An image processing method for performing a blurred image process for adding blur, which prohibits image processing for adding blur to an image of an area in which the image shift amount cannot be calculated.

The present invention also provides a control program for causing a computer to execute the above image processing method.

The present invention also provides a medium recording the above control program.

Further, according to the present invention, the first image, an input means for inputting a second image having a parallax with respect to the first image, and the first and second input by the input means. Image processing means for performing different image processing on a corresponding first area of the image and a second area different from the first area, and combining the first image and the second image. The image processing apparatus has the following.

Further, according to the present invention, the first image, an input means for inputting a second image having a parallax with respect to the first image, and an image obtained by the parallax of the first and second images. A predetermined blur is added to at least the first image based on the shift amount of the image, and when the shift amount of the image due to the parallax of the first and second images is equal to or less than a predetermined detection level, the predetermined blur is generated. The image processing apparatus does not add the.

Further, according to the present invention, a first image and a second image having a parallax with respect to the first image are input, and the corresponding first and second images corresponding to each other are input. An image processing method is provided in which different image processing is performed on a first area and a second area different from the first area, and the first image and the second image are combined.

Further, according to the present invention, a first image and a second image having a parallax with respect to the first image are input, and an image shift amount due to the parallax between the first and second images is input. Based on the above, at least a predetermined blur is added to the first image, and when the image shift amount due to the parallax of the first and second images is equal to or less than a predetermined detection level, the predetermined blur is added. The image processing method is not performed.

The present invention also provides a control program for causing a computer to execute each of the above image processing methods.

The present invention also provides a recording medium on which the above control program is recorded.

[0025]

Preferred embodiments of the present invention will be described below with reference to the accompanying drawings.

(First Embodiment) FIG. 1 is a view showing the concept of an image processing method according to the first embodiment of the present invention. In the figure,
The image C, the image D, the image E, ... Are a plurality of images having parallax.

First, in step S101, these plural images are input, and the image shift amounts in the plural regions in the image are calculated by a known correlation calculation. Next, in step S102, it is determined whether or not the image shift amount cannot be calculated in each area. Then, if the calculation is not possible, the process proceeds to step S103. On the other hand, if the calculation is not possible, the process proceeds to step S106. Next, in step S103, the main subject and the background are separated based on the image shift amount, the process proceeds to step S104 in the area determined as the main subject in the shooting screen, and in the case of the area determined as the background. It proceeds to step S105. In step S104, shift correction image processing (details will be described later) is performed to return the image shift amount to a predetermined amount so that the image quality is not deteriorated due to the shift of the images during the image synthesis in step S106. On the other hand, in step S105, blurred image processing for adding blurring different from that of the photographing optical system is performed. Finally, in step S106
Then, a plurality of images are combined to create one image with higher resolution. The above is the general flow of the image processing method in the present embodiment. In addition, in 107 surrounded by a dotted line in the figure,
It can be said that at least two different image processes are performed.

Next, a specific example for realizing the above image processing method will be described.

2 to 4 are views showing an image pickup section of an image pickup apparatus capable of picking up a plurality of images having parallax. These have the same configuration as the image pickup apparatus disclosed in Japanese Patent Laid-Open No. 2001-78213, have an optical system provided for each color of the color filter, and combine the respective images to obtain a single-panel image pickup with a Bayer array. An image equivalent to the image captured by the element can be formed. Each image has parallax because it has an independent optical system, and the position of each optical image, that is, the image shift amount, changes depending on the distance of the subject. Further, this image shift amount is designed to be 1/2 pixel at a certain predetermined distance, and a method called pixel shift is used which can obtain a high-resolution image with a small number of pixels. In the present embodiment, image processing is performed on an image obtained by an image pickup apparatus using this image pickup unit.

First, the detailed structure of the image pickup section will be described.

FIG. 2 is a side view of an image pickup section for picking up a plurality of images having parallax. The imaging unit 201 includes an imaging lens 202, a housing 203, and an imaging element 204 from the incident surface side, and has a structure in which the housing 203 is sandwiched between the imaging lens 202 and the imaging element 204. Also, FIG.
As shown in FIG. 3, a diaphragm 205 having openings 205a and 205c and color filters 206a and 206c are formed on the incident surface side of the imaging lens 202 by means of printing or the like.

FIGS. 3 and 4 are views of the image pickup lens 202 as seen from the incident surface side or the exit surface side.
a, 202b, 202c, 202d and four openings 2
05aA 205b, 205c, 205d.
Then, the four openings 205a, 205b, 205
4 color filters 20 corresponding to c and 205d
6a, 206b, 206c, 206d are formed. Therefore, four optical images are also formed on the image sensor 204. The lens units 202a, 20a
2b, 202c, and 202d are formed with infrared cut filter films having low transmittance in the wavelength range of 670 nm and thereafter. Then, the lens units 202a and 202
The distance b, the distance between the lens portions 202a and 202c, and the distance between the lens portions 202b, 202c and 202d are set equal to each other.

FIG. 5 is a view of the image pickup element 204 as seen from the incident surface side. The four lens portions 202a, 202b, 202c,
Four imaging regions 204a, 204 corresponding to 202d
b, 204c, and 204d. It should be noted that the image sensor 204 has four imaging regions 204a, 204b, 204.
Pixels are also provided as additional pixels in areas other than c and 204d, and these four imaging areas are cut out and used during imaging.

Here, the color filters 206a and 206d have a spectral transmittance characteristic of mainly transmitting green color,
The optical filter 206b has a spectral transmittance characteristic of mainly transmitting red color, and the optical filter 206c has a spectral transmittance characteristic of mainly transmitting blue color. That is, these are primary color filters.

Next, a pixel shift configuration, that is, an image shift due to parallax will be described. FIG. 6 is a diagram of the image sensor 202 of FIG. 2 viewed from the incident surface. In the figure, a point So is a center point of the image sensor pixel formation portion, and points Sa, Sb, Sc, and Sd.
Represents the center point of each of the imaging regions 204a, 204b, 204c, and 204d.

On the other hand, FIG. 7 is a cross-sectional view of the optical system in which only the taking lens 202 and the light receiving surface of the image pickup element 204 are indicated by thick lines in the image pickup section shown in FIG. In the figure, the center line Lo is a line that passes through the center point So and is orthogonal to the light receiving surface of the image sensor 204, and consider the point light source K that is on the center line Lo and is separated by a distance d. Of the light rays emitted from the point light source K and passing through the lens portions 202a and 202c of the image pickup lens 202, the principal rays are La and Lc, and the intersections with the light receiving surface of the image pickup element 204 are Ka and Kc. The lens unit 202
Although b and 202d are not displayed, since they are symmetrical, they can be considered in the same way, and a description thereof will be omitted. Then, in FIG. 6, the lens portions 202a, 2 related to the point light source K
02b, 202c, 202d chief rays and image sensor 204
The intersections of are shown as Ka, Kb, Kc and Kd.
Therefore, if the four optical images formed on the image pickup element 204 when a subject located at the distance d is photographed are 207a, 207b, 207c, and 207d in FIG. 6, these optical images are points Ka, Kb, and Kc. , Kd, respectively.

At this time, points Ka, Kb, Kc, and Kd are points S.
1/4 from a, Sb, Sc, Sd toward the center point So
It is set so as to shift the pixels. That is, the respective imaging regions 204a, 204b, 204c, 204
Since d receives the optical images which are shifted by 1/2 pixel from each other, it is nothing but a pixel shift. Then, the optical image 20
7a, 207b, 207c, 207d to points Sa, Sb,
By combining so that Sc and Sd overlap, a high resolution image can be obtained by shifting the pixels. Furthermore, by providing the color filter as described above, the image to be combined becomes a known Bayer array image that is optimal for color images. The above is the configuration of pixel shifting in the present embodiment.

By the way, as shown in FIG.
Considering the case where the distance d of the (object) changes and moves to the point light source Kx, the chief rays of the lens units 202a and 202c change as shown by the dotted line in the figure, and the intersection of the chief ray and the light receiving surface of the image sensor 204 is determined. Ka and Kc also move to the positions of points Kxa and Kxc. That is, the position of the optical image changes depending on the distance to the subject, that is, the optical image received in the imaging regions 202a and 202c in the state where the 1/2 pixel shift is established, in contrast to the optical image in such a case. This indicates that there is a gap. In the present embodiment, this degree of deviation is called the image deviation amount. For example, 1 /
The absolute value of the image shift amount when shifting two pixels is 0.5 [p
image].

Next, the relationship between subject distance and parallax in this embodiment will be described.

In FIG. 7, the point light source K as just described
Is moved toward the image pickup lens 202 side, the interval between the points Ka and Kc is widened, that is, the image shift amount is increased. On the other hand, when the point light source K moves away from the imaging lens 202, the distance between the points Ka and Kc becomes narrower, that is, the image shift amount becomes smaller. Therefore, by setting the change amount of the image shift amount and the pixel pitch depending on the distance for convenience and detecting the image shift amounts at a plurality of positions in the shooting screen, the subject and the background can be easily separated.

Now, let us actually calculate the image shift amount in this embodiment. In FIG. 7, a point light source at an arbitrary position on the center line Lo is Kx, and a distance from the light receiving surface at that time is dx. Also, as shown in the figure, a, c,
xa and xc are defined, the distance between the lens units 202a and 202c is 1, and the distance between the image pickup lens 202 and the light receiving surface of the image pickup element 204 is f. At this time, “xa + xc” represents the image shift amount at the distance dx when the image shift amount at the distance d is used as a reference, and this is calculated. Then, the following equations (1) and (2) are established from the similarity condition of the triangles. dx-f: l = f: a + c + xa + xc (1) df: l = f: a + c (2)

Now, the lens section 202 of the image pickup lens 202
Since a is a green color filter and lens part 202c is a red color filter, there is a difference in parallax change sensitivity depending on wavelength, but the difference is negligible, so when ignored, a, c, and xa And xc are equal. Therefore, a, c, xa, and xc are defined as follows. a = c = P / 2, xa = xc = X / 2 (3) Substituting the above equation (3) into equations (1) and (2) and solving for X, the image shift amount X (at an arbitrary distance dx That is, the value of “xa + xc”) is calculated, and the following equation (4) is obtained. X = lf / (dx−f) −lf / (d−f) (4) Then, when the pixel pitch of the image sensor 204 is Sp and the image shift amount X is divided by Sp, the image shift amount in pixel units is obtained. It can be calculated.

Then, based on the equation (4), the value of each parameter is set to find the relationship between the image shift amount and the distance. The following equation (5) is the value of each parameter used in this embodiment. f = 1.45 mm, Sp = 3.9 μm d = 2380 mm, l = 1.40112 mm (5) where d is the distance when 1/2 pixel shift is established, and is assumed to be the angle of view of the imaging lens 202. It was decided based on the size of the subject.

FIG. 8 is a graph in which the relationship between the image shift amount and the distance is calculated based on the equations (4) and (5). The horizontal axis represents the distance to the subject, the vertical axis represents the image shift amount at that distance, and the unit is pixel [pixel]. A curve 208 represents the image shift amount calculated from the above equations (4) and (5). Here, since the relative parallax amount is calculated based on the image shift amount at the distance d, the image shift amount is 0 near the distance 2380 [mm], but the absolute amount at this distance is actually the absolute amount. The image shift amount as
It is −0.5 pixel due to the effect of shifting by 2 pixels.

From the curve 208, 0 to 1000 m
In the range of m, the change of the image shift amount is large, 1000 mm
After that, the change in the image shift amount is small. So 0-1
There is a main subject at a point 209 in the range of 000 mm, and
Assuming that there is a background at a point 210 after 0 mm, it is possible to easily separate the main subject and the background if it is possible to detect a plurality of image shift amounts in the shooting screen and determine the difference X1. .

By the way, the image shift amount at an infinite distance becomes a value when dx of the equation (4) is infinite, and when calculated using the parameter of the equation (5), it is -0.2.
The image shift amount is 19 [pixels], and with such an image shift amount, there is almost no deterioration in image quality due to pixel shift.

The change in the image shift amount has been described by focusing on the lens portions 202a and 202c. However, since the intervals between the lens portions are set equal, the lens portion 202a is set.
And 202b, lens sections 202b and 202d, lens section 2
The image shift amount similarly changes in 02c and 202d. Further, the diagonal lens portions 202a and 202a
The sensitivity for d, 202b, and 202c is also √2 times, but the pixel pitch is also √2 times, so the change in the image shift amount is the same as in FIG.

Next, an image processing method using an image obtained by the image pickup section as described above will be described. Since the rough flow of the image processing has been described with reference to FIG. 1, a more specific processing method will be described here.

FIG. 9 shows four parallax images obtained by the above-mentioned image pickup section, and images 211a, 211b, and 2 are shown.
Reference numerals 11c and 211d correspond to the optical images 207a, 207b, 207c, and 207d on the image sensor 204 shown in FIG. 6, respectively. That is, the images 211a and 211d are green color filters, the image 211b is a blue color filter, and the image 211c is a red color filter. It should be noted that although an actual image is obtained by inverting the image horizontally from the imaging lens 202, the image is an erect image here for the sake of easy understanding. And here, a pair of images 211a by the green color filter,
The amount of parallax is calculated using 211d. This is because the accuracy is higher than that when the calculation is performed using different wavelengths, and the green color has the highest spatial resolution among RGB, and the calculation accuracy can be improved.
Further, in the image processing of this embodiment, uncompressed raw image data (quantized image signal of the image sensor 204) that has not been processed is used. By doing so, the accuracy of the image shift amount calculation can be improved and the processing time can be shortened as compared with the case where the image shift amount is calculated by performing back calculation from the compressed color image data.

FIG. 10 is a detailed flowchart of the flowchart of FIG. The detailed flow will be described using these flowcharts. 311 surrounded by a dotted line in the figure
Corresponds to 107 in FIG. Therefore, at 311, at least two different image processes, that is, the first image process 312 (corresponding to step S104 in FIG. 1) and the second image process 313 (corresponding to step S105 in FIG. 1) are performed. .

First, in step S301, four images having parallax taken by the image pickup unit 201 are input,
The image is divided into a plurality of regions in pixel units. FIG. 11 is a diagram for explaining this area division. As shown in the figure, the images 211a and 211d by the green color filter are divided into areas. Here, the smaller the number of pixels in one area, the smoother the boundary line with the background when the main subject is separated. However, if it is too small, the correlation calculation in the image shift amount calculation to be described later will not be established. In the present embodiment, it has a square shape of 10 × 10 [pixel].

Next, in step S302, the image shift amount in each area is calculated by using a known correlation calculation. For example, focusing on the area 212a of the image 211a in FIG. 11, one image shift amount is calculated using the image data of the area 212d of the image 211d corresponding to this area 212a. Here, the images 211a and 212b correspond to the optical images 207a and 207d of FIG. 6, and are formed by the lens units 202a and 202d of FIG. Therefore, the moving direction of the optical image depending on the distance is the direction indicated by the arrow 213, and the change in the image shift amount also changes in the arrow direction. Then, as described above, the intervals between the lens portions are set to be equal to each other, so that the arrow 213 in which the change in the image shift amount occurs is exactly inclined at 45 degrees. In order to detect the image shift amount in the direction of the arrow 213, the parallax amount in the vertical direction and the horizontal direction can be calculated and calculated from the combined vector, but in the present embodiment, in order to reduce the calculation load, The image shift amount in the direction of arrow 213 is directly calculated by the correlation calculation.

Next, in step S303, an image shift amount distribution is created based on the calculated image shift amount in the entire area. Here, the image shift amount is divided into two. Figure 8
Assuming that the main subject exists within 1000 [mm], the parallax amount at that time is about 0.3 [pixel] or more. Therefore, the image shift amount is 0.3 here.
It is determined whether it is greater than or less than [pixel].

FIG. 12 is a diagram visually showing the distribution of image shift amounts in the image 211a or 211d divided into regions. In this distribution chart, the area filled with black indicates an image shift amount of 0.3 [pixel] or more, and the white region indicates an image shift amount of less than 0.3 [pixel]. From this figure, it can be seen that only the main subject can be separated in the image 211a or 211d. It should be noted that the area filled with gray in the figure is an area where an image shift amount calculation error has occurred, and is likely to occur in a low contrast portion such as the background sky.

Next, in step S304, one representative image shift amount in the area filled with black in FIG. 12, that is, the main subject is calculated. Specifically, an average value is calculated from the image shift amounts of a plurality of regions filled with black, and this value is stored as the representative image shift amount of the main subject.

Next, in step S305, it is determined whether or not the image shift amount has been calculated in each area, that is, whether or not there is a calculation error. Then, in the case of an area that is a calculation error (gray area in FIG. 12), step S31
Go to 0. On the other hand, if the region is not a calculation error, the process proceeds to step S306.

Next, in step S306, it is determined whether or not each area is the main subject in the area where no calculation error has occurred. In the case of the area determined to be the main subject (black area in FIG. 12), the process proceeds to the shift correction image processing in step S307 and subsequent steps, and the area determined to be not the main subject (white area in FIG. 12), that is, the background. In the case of the region determined to be, the process proceeds to the blurring filter process of step S308.

In steps S307 and 309, shift correction image processing for returning the image shift to a predetermined amount is performed so as to reduce the image quality deterioration at the time of image combination in the main subject due to the image shift.

First, in step S307, an image misregistration return amount is calculated so that a factor of image quality deterioration such as color misregistration during image combination is almost eliminated. Since the representative image shift amount of the main subject has been calculated in step S304, the image shift return amount Xcmp is calculated by the following equation (6) when the representative image shift amount is Xavg. Xcmp = {Xavg − (− 0.5)} / 2 (6) Here, 0.5 in the above formula (6) is because the image shift amount at the time of pixel shift is used as a reference. In addition, this image misalignment return amount Xcmp is rounded off to be an integer in pixel units.
By doing so, it becomes possible to facilitate the subsequent processing.

Then, in step S309, step S
The image data is replaced based on the image shift return amount calculated in 308. FIG. 13 is a diagram for explaining this, in which the central point So of the image sensor 204 shown in FIG. 6 and the vicinity of the central point Sa of the imaging region 204a are enlarged. Now
Consider the one area 214. It is assumed that a value of 1.0 [pixel] is calculated as the image misalignment return amount by Expression (6). Since the main subject exists on the short distance side, the optical image formed on the imaging area 204a has a center point S
It is moving outward with respect to o. Therefore, the area 21
By replacing the image data in the area indicated by the dotted line in the drawing, which is one pixel inward from the center point 4 with respect to 4 as the image data of the area 214, the image shift can be restored. In addition,
Similarly, in the other imaging areas 204b, 204c, and 204d, the displacement of the image can be restored by replacing the image data with one pixel inside toward the center point So. Then, the misalignment return process is performed on the entire area determined to be the main subject.

In this way, if the image return amount is calculated as an integer in pixel units in step S307, step S307
In replacing the image data in 309, it is only necessary to shift the area. Alternatively, the image return amount may be calculated by a value including the decimal point, and the image data at the position where the actual pixel does not exist may be calculated from the surrounding image data by using the interpolation calculation, and the replacement may be performed. By doing so, the effect of pixel shifting can be maximized. However, in the present embodiment, the maximum value of the correction remaining amount from the proper image displacement amount that occurs when the image displacement return amount is an integer is 0.5 [pixel]. The configuration is as described above considering that there is almost no effect and that the processing speed is increased.

The above is the shift correction image processing in steps S307 and S309.

On the other hand, in step S308, the blurring filter process is applied to the area determined to be other than the main subject in the photographing screen, that is, the background portion. FIG. 14 shows a 9 × 9 filter matrix used in the blur filtering process, and blur is added by convolving this matrix with each pixel of the image. The matrix shown in (a) shows a large blur, and the matrix shown in (b) shows a small blur. Note that these matrixes are the point spread function (Point Spread) of the imaging lens.
d Function). Then, for example, the F number of the image pickup optical system is F5.
6 is assumed, (a) is equivalent to F2.8, (b)
Is set to be equivalent to F4.0, and both are considered so as to add a blur that cannot be realized by the imaging optical system.

Then, the image data of the pixel in the i-th row and the j-th column in the area determined to be other than the main subject is I (i, j), and the row direction of the matrix in FIG. 14 is k and the column direction is l. , And the value of the matrix at the k-th row and the 1-th column is Fil (k, l) using the matrix numbers from -4 to 4 as shown in the figure, the image data I (I, J) after blurring is added. Is expressed by the following equation (7).

[0065]

[Equation 1]

The background image can be effectively blurred by repeating such calculation for all the pixels in the area shown in white in FIG. In addition, image 21
No other image 211b, 211c, 2
Similar processing is performed in 11d.

Since two filter matrices are prepared here, they can be switched according to the photographer's preference. Also, in the above example, the background is 1
I did blur filtering with different types of filters,
It is also possible to subdivide the background in the image shift amount distribution chart of FIG. 12 according to the image shift amount and add blurring filter processing with a large number of filter matrixes according to the distance estimated from the image shift amount. In that case, if each parameter of the equation (5) is changed and the change amount of the image shift amount due to the distance is made large even at a position which is somewhat apart, the background can be easily subdivided according to the image shift amount. You can Furthermore, the filter matrix may be calculated one by one according to the image shift amount. However, in this embodiment, only one type of blur is applied to the background in consideration of speeding up the processing. Therefore, the effect can be more exerted on an image such as a portrait photograph in which the distance between the main subject and the background is large.

Finally, in step S310, the image 211
A composite image is created using a, 211b, 211c, and 211d. Here, an image corresponding to the Bayer array is created by shifting the pixels. FIG. 15 is a diagram showing a part of a color filter array in a general Bayer array, in which R
Indicates a red color filter, G indicates a green color filter, and B indicates a blue color filter. Further, it has a repeating structure of RGGB and a set of 4 pixels. In the present embodiment, since the Bayer array is based on pixel shifting, considering the image 211a as a reference, the images 211b and 211 are arranged between the pixels of the image 211a.
By embedding the image data of c and 211d, a Bayer array image can be formed. Therefore, the images 211a, 211
By rearranging the image data of b, 211c, and 211d, a Bayer array composite image can be created.

The above is the image processing method in this embodiment.

In the flowchart of FIG. 10,
In the area where the image shift amount cannot be calculated, no image processing is performed, and the process proceeds to the image combining processing in step S310. However, such a region that cannot be calculated has a low contrast in most cases, and there is no problem in image quality even if special processing is not performed on an image in such a region. For example, assume that some areas of the main subject cannot be calculated. With respect to the main subject, shift correction image processing is performed in order to prevent color shift during image combination, but since the contrast is low, color shift is hardly noticeable without this processing. Further, even when a partial area of the background cannot be calculated, this area has a low contrast, and thus it is not necessary to perform the blurring process. because,
This is because blurring image processing is performed on other backgrounds, and blurring is equivalent to reducing the contrast of an image. On the other hand, if such an uncalculatable area is separated into the main subject and the background by some means, if the separation is wrong, the opposite image processing will be performed, and the image will be unnatural and low in image quality. It becomes an image.

From the above viewpoints, in the present embodiment, the processing is not performed for the area where the image shift amount cannot be calculated. By doing so, the processing algorithm can be simplified and the speed can be increased. In addition, the number of pixels installed in mobile devices is small,
In a device including a simple image pickup optical system and including an image pickup unit having a relatively low resolution, a difference between a portion where image processing is not performed and a portion where image processing is performed due to calculation failure is hardly noticeable. Therefore, it can be said that the present embodiment is more suitable for a device including such an imaging unit. Note that the main subject and the background may be completely separated by using known processing such as grouping, and some image processing may be performed even in the uncalculatable area.

The above is the image processing method according to the first embodiment. In this way, it is effective to calculate the image shift amount based on the image having parallax, perform the shift correction image processing on the main subject to generate a high quality image, and perform the blurred image processing on the background. It is possible to add clear blurring and obtain an image with high image quality and natural blurring. Furthermore, even if there is a region in the image in which the image shift amount cannot be calculated, the image quality is not significantly deteriorated.

(Second Embodiment) The second embodiment is the first
The present invention relates to an image processing device that integrally includes an imaging unit and an image processing unit. In addition, the portions denoted by the same reference numerals as those in the first embodiment are the same as those in the first embodiment.

FIG. 16 is a perspective view of the image processing apparatus of this embodiment, which uses a digital camera as the image processing apparatus. In the figure, reference numeral 215 denotes a photographing button, and the photographing operation is executed by the photographer pressing this button. Reference numeral 216 is an optical viewfinder, 217 is a strobe, and 218 is a photographing section protected by parallel plate glass, and the image pickup section 201 described in the first embodiment is provided inside. Note that the configuration of the image pickup unit is the same as that of the first embodiment, and therefore description thereof will be omitted.
A shooting mode setting member 219 sets various shooting modes. On the other hand, 220 is an external storage medium insertion portion, which can be opened and closed by a lid 221. Then, by inserting the external storage medium 222 into the external storage medium insertion section 220, the photographed image can be stored.

FIG. 17 is a block diagram of an electric system in the image processing apparatus of FIG. The image processing apparatus includes an image pickup unit including an image pickup lens 202 and an image pickup element 204. The image pickup element 204 receives a clock signal from a TG (timing generation) unit 402 and sends an analog image signal to the A / D conversion unit 401. Further, the A / D conversion unit 401 converts the analog image signal into a digital image signal, and this digital image signal is sent to the digital signal processing unit 403. Further, the digital signal processing unit 403 is connected to the central control unit 404 and the IF (interface) unit 405, and here, AE
(Auto Exposure), AWB (Auto
Well-known image processing such as white balance), gamma correction, and image compression is performed.

The IF unit 405 can be connected to an external storage medium or the like, and can be connected to the external storage medium 222 of FIG. 16 to store photographed image data.

The central control unit 404 is a CP for performing various calculations.
A U (central processing unit) 406, a ROM 407 that stores various operation programs, and a RAM 408 that temporarily holds a digital image signal are provided. The central control unit 404 also includes a TG unit 402 and a digital signal processing unit 403.
Also performs various controls. Further, the central control unit 404 includes a photographing switch 409 and a photographing mode setting switch 4 corresponding to the photographing button 215 and the photographing mode setting member 219 of FIG.
10 are connected. Then, in the present embodiment, by connecting the photographing mode setting switch 410 to the contact 411, the image processing of the first embodiment (the flowchart of FIG. 10) by the filter matrix of FIG. 14A is executed, By connecting to the contact 412, FIG.
The image processing of the first embodiment (the flowchart of FIG. 10) is executed by the filter matrix of 4 (b). Here, the shooting mode in which such processing is executed will be hereinafter referred to as a digital portrait mode. When it is connected to another contact, a shooting mode such as auto shooting, macro, or night view is selected and executed.

The above is the configuration of the image processing apparatus in the second embodiment using a digital camera as an example. Next, the actual operation relating to the image processing of this embodiment will be described with reference to the flowchart of FIG.

FIG. 18 is a flow chart showing the operation of the digital camera relating to the image processing of this embodiment. This flowchart is stored in the ROM 407 of the central control unit 404 as a program.

First, in step S501, the image pickup button 21
Pressing 5 causes A in the digital signal processing unit 403.
Exposure calculations such as E and AWB are performed, and a clock signal for the TG unit 402 is generated based on the calculated values, whereby charge is accumulated in the image sensor 204 and an analog image signal is read. Then, the process proceeds to step S502.

Next, in step S502, the analog image signal is converted into a quantized digital image signal in the A / D converter 401.

Next, in step S503, it is determined whether or not the photographing mode setting switch 410 is set to the digital portrait mode. Then, when the digital portrait mode is selected, the corresponding filter matrix in FIG. 14A or 14B is selected, and the process proceeds to step S504. On the other hand, if not selected, step S504 is skipped and step S5 is executed.
Go to 05.

In the digital portrait image processing in step S504, the flowchart in FIG. 10 of the first embodiment is executed. That is, the main subject and the background of the image are separated, the shift correction image processing is performed on the main subject, and the blurred image processing is performed on the background.

Next, in the digital signal processing of step S505, known image processing such as color processing and gamma correction is performed in the digital signal processing unit 403, and an image to be finally stored is generated through image compression processing.

Finally, in the image data writing in step S506, the image for saving is stored in the external storage medium via the IF unit 405.

The above is the processing operation in the digital camera. In the above description, the digital portrait image processing in step S504 uses the digital image signal before the digital signal processing in the digital signal processing unit 403 in step S505, that is, the raw digital image signal immediately after A / D conversion. I did it. This is because the image shift amount can be detected with higher accuracy by using the raw digital image signal as described in the first embodiment. The order may be reversed. Furthermore, the raw digital image signal may be calculated by back-calculating the digital image signal that has been subjected to the digital signal processing in step S505, but in the present embodiment, the image compression processing is irreversible processing. It is in order.

The example in which the image processing method described in the first embodiment is applied to the digital camera has been described above.

In this embodiment, the digital portrait image processing is performed by the digital camera. However, in order to improve the processing speed, when the digital portrait mode is selected, this image processing is not performed inside the digital camera, only normal digital image processing is performed, and the digital portrait mode is added as additional information of the image. Is stored and the image is
Image processing may be performed on an application of a device having a high-speed CPU such as C. However, in the present embodiment, the liquid crystal (not shown) provided on the rear surface of the digital camera of FIG. 16 internally performs processing for confirming the captured image.

The present invention is not limited to digital cameras, but can be applied to other image processing devices such as digital video cameras and scanners.

Further, in each of the above-described embodiments, the present invention can be applied regardless of whether a plurality of images are combined or only one image is used as an image region to which blurring is added. It is a thing.

(Other Embodiments) The purpose of each embodiment is to supply a storage medium (or recording medium) recording a program code of software for realizing the functions of the above-described embodiments to a system or apparatus, Computer of the system or device (or CPU or MPU)
Needless to say, can also be achieved by reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiment are realized, but also an operating system (OS) running on the computer is executed based on the instruction of the program code. It goes without saying that a case where some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Further, after the program code read from the storage medium is written in the memory provided in the function expansion card inserted into the computer or the function expansion unit connected to the computer, based on the instruction of the program code, , The CPU provided in the function expansion card or the function expansion unit performs some or all of the actual processing,
It goes without saying that the processing includes the case where the functions of the above-described embodiments are realized.

When the present invention is applied to the above storage medium, the storage medium stores the program code corresponding to the above-described flowcharts (shown in FIGS. 1, 10, and 18).

[0094]

As described above, according to the present invention,
From an image having a plurality of parallax images captured by an imaging unit having an imaging lens having a small aperture, it is possible to obtain an image having a blur as if the image was captured by an imaging lens having a large aperture.

Further, by applying pixel shift to an image having a plurality of parallaxes, a higher definition image can be generated.

Further, in the area where the parallax cannot be calculated, the special image processing is not performed, so that the image quality is not deteriorated.

[Brief description of drawings]

FIG. 1 is a diagram showing a concept of an image processing method according to an embodiment of the present invention.

FIG. 2 is a side view of an imaging unit that captures a plurality of images having parallax.

FIG. 3 is a diagram of the imaging lens viewed from the incident surface side.

FIG. 4 is a diagram of the imaging lens viewed from the exit surface side.

FIG. 5 is a diagram of the image pickup element viewed from the incident surface side.

FIG. 6 is a diagram of the image pickup element viewed from the incident surface side, and is a diagram for explaining pixel shifting.

FIG. 7 is a cross-sectional view of an optical system in the image pickup unit, which is a diagram for explaining a change in image shift amount.

FIG. 8 is a graph showing a relationship between an image shift amount and a distance in the embodiment of the present invention.

[Fig. 9] Fig. 9 is a diagram illustrating four images having parallax obtained by an imaging unit.

FIG. 10 is a flowchart of an image processing method showing FIG. 1 in more detail.

[Fig. 11] Fig. 11 is a diagram for describing area division in an image having four parallaxes obtained from an imaging unit.

FIG. 12 is a diagram visually showing the distribution of image shift amounts in an image divided into regions.

FIG. 13 is a partially enlarged view in which the vicinity of the center of the image sensor is enlarged, and is a diagram for explaining replacement of image data.

FIG. 14 is a diagram showing two types of filter matrices used in blurring filter processing.

FIG. 15 is a diagram showing a part of a color filter array in a Bayer array.

FIG. 16 is a perspective view of an image processing apparatus (digital camera) according to a second embodiment.

FIG. 17 is a block diagram of an electric system in the image processing apparatus (digital camera).

FIG. 18 is a flowchart showing the operation of the image processing apparatus (digital camera).

FIG. 19 is a diagram showing a concept of image processing in a conventional example.

[Explanation of symbols]

201 Imaging unit 202 Imaging lens 204 image sensor 205 aperture 206 color filter 207 Optical image 215 Shoot button 219 Shooting mode selection member 222 External storage medium

Claims (24)

[Claims] 1. An image shift amount calculation means for calculating an image shift amount in a plurality of regions in a plurality of images having an input parallax, and an image shift amount calculated by the image shift amount calculation means. An image processing device, comprising: an image processing unit that performs at least two different image processes for the images of the plurality of areas. 2. The two image processes include a first image process for correcting an image so that the image shift amount becomes a predetermined amount, and a second image process for adding a blur different from a blur condition of the image.
The image processing apparatus according to claim 1, wherein the image processing is the image processing according to claim 1. 3. An image shift amount in a plurality of regions in a plurality of images having an input parallax is calculated, and at least two different images are calculated for the images in the plurality of regions according to the calculated image shift amount. An image processing method characterized by performing image processing by switching. 4. The two image processes are a first image process for correcting an image so that the image shift amount is a predetermined amount, and a second image process for adding a blur that is different from a blur condition of the image.
The image processing method according to claim 3, wherein the image processing is the image processing according to claim 3. 5. A control program for causing a computer to execute the image processing method according to claim 3 or 4. 6. A recording medium on which the control program according to claim 5 is recorded. 7. An image shift amount calculating means for calculating an image shift amount in a plurality of regions in a plurality of images having an input parallax, and an image shift amount calculated by the image shift amount calculating means according to the image shift amount. Blurred image processing means for performing image processing for adding blur to images in the plurality of areas, and control means for prohibiting image processing for adding blur to images in areas where the image shift amount cannot be calculated. An image processing apparatus comprising: 8. An image shift amount in a plurality of regions in a plurality of images having an input parallax is calculated, and blurring is added to the images in the plurality of regions according to the calculated image shift amount. An image processing method for performing blurred image processing, characterized in that image processing of adding blur is prohibited for an image of an area in which the image shift amount cannot be calculated. 9. A control program for causing a computer to execute the image processing method according to claim 8. 10. A recording medium on which the control program according to claim 9 is recorded. 11. A correspondence between the first image, an input means for inputting a second image having a parallax with respect to the first image, and the first and second images input by the input means. A first area and a second area different from the first area, different image processing is performed, and image processing means for combining the first image and the second image is provided. A characteristic image processing device. 12. The image processing means includes the first and second image processing means.
12. The image processing apparatus according to claim 11, wherein the image shift due to the parallax in the first region of the image is corrected to perform the synthesizing process of the first image and the second image. 13. The image processing means includes the first and second image processing means.
The image processing apparatus according to claim 11 or 12, wherein a predetermined blur is added to the second area when the image of the image is combined. 14. The image processing means adds the predetermined blur when the image shift amount due to the parallax of the first and second images in the second region is equal to or lower than a predetermined detection level. The image processing device according to claim 13, wherein the image processing device is not performed. 15. The image processing means includes the first and second image processing means.
15. The image processing apparatus according to claim 11, wherein the first region and the second region are determined based on an image shift amount of the image due to the parallax. 16. The image processing apparatus according to claim 11, wherein the image processing apparatus is an image pickup apparatus. 17. The image processing apparatus according to claim 11, wherein the image processing apparatus is a camera. 18. A displacement amount between a first image, an input unit for inputting a second image having a parallax with respect to the first image, and an image shift amount due to the parallax between the first and second images. Based on the above, at least a predetermined blur is added to the first image, and when the image shift amount due to the parallax between the first and second images is equal to or less than a predetermined detection level, the predetermined blur is added. An image processing device characterized by the absence thereof. 19. The image processing apparatus according to claim 18, wherein the image processing apparatus is an image pickup apparatus. 20. The image processing apparatus according to claim 18, wherein the image processing apparatus is a camera. 21. A first image and a second image having parallax with respect to the first image are input, and corresponding first regions of the input first and second images are provided. An image processing method, wherein different image processing is applied to a second area different from the first area, and the first image and the second image are combined. 22. A first image and a second image having a parallax with respect to the first image are input, and at least based on a shift amount of the images due to the parallax between the first and second images. A predetermined blur is added to the first image, and the first,
An image processing method, wherein the predetermined blur is not added when the amount of displacement of the second image due to the parallax is equal to or lower than a predetermined detection level. 23. A control program for causing a computer to execute the image processing method according to claim 21 or 22. 24. A recording medium on which the control program according to claim 23 is recorded.