KR100935392B1 - Aperture device and imaging device with same, automatic focus restoration device and method based on movement of color channel - Google Patents

Abstract

Disclosed are an aperture device capable of estimating distance information of an object, an imaging device having the same, and an autofocus restoration device and method based on movement of a color channel. The aperture device according to the present invention includes a plate having a plurality of openings and a color filter having different colors coupled to each opening. The automatic focus restoring apparatus according to the present invention calculates a focus value of each pixel constituting the input original image, determines a restoring area based on the focus value, and adjusts the focus in the restoring area by moving the color channel. After generating the images, the fusion region is determined from each of the original image and the intermediate image to generate a reconstructed image in which the focus is restored in the entire region by fusion of the fusion regions. According to the present invention, an automatic focus restoration based on the movement of a color channel can be used to restore a degraded image more quickly while preventing a phenomenon such as blurring or ringing without degrading the quality of the restored image, and making the image red. The distance information of the subject may be estimated from the respective images when the green, blue and blue color channels are separated.

Color filter, aperture, color channel, shift

Description

Aperture, image pick-up device having same, automatic focus restoring device and method based on movement of color channel {Aperture, image pick-up device, and apparatus and method for digital autofocusing based on color channel shift}

According to the present invention, an aperture device capable of estimating distance information of an object, an imaging device having the same, and a focus of an image including a plurality of subjects located at different distances from a lens may be restored based on movement of a color channel. An automatic focus restoring apparatus and method are provided.

Recently, with the development of video technology fields such as camcorders, digital cameras, and video surveillance systems, the demand for autofocus technology is rapidly increasing. That is, as the resolution and precision of the imaging apparatuses increase, the focal point of the image is blurred by the fine movement of the subject or the imaging apparatus during the shooting. In particular, when there are a plurality of subjects located at different distances from the lens, deterioration of focus occurs except for a subject positioned on a focal length, and thus a blur occurs in an image. Therefore, the need for an automatic focusing technology that can restore this is increasing.

Conventional techniques for autofocusing usually consist of two modules: analysis and control. The analysis module estimates the focus of the image on the image plane, and the control module performs a function of moving the lens assembly to the optical focus position based on the focus information estimated by the analysis module to focus. Systems employing this autofocus technology include infrared autofocusing (IRAF), through-the-lens autofocusig (TTLAF), semi-digital autofocusing (SDAF) and fully digital autofocusing ( There is a fully digital autofocusing (FDAF) system. These systems use reconstruction and fusion methods based on multiple original images. However, when restoring an image in which a plurality of subjects having different depths of field exist, a phenomenon such as reblurring or ringing may be caused, and image restoration may cause serious degradation of image quality. Can be. In addition, there is a problem that takes a long time to obtain a completely reconstructed image due to the repeated reconstruction process.

1 is a diagram illustrating a convergence pattern of photographing light according to a position of a subject in a general lens when a plurality of subjects located at different distances from a lens exist. Referring to FIG. 1, the center of the diaphragm 120 is positioned on the optical axis 130 of the lens 110, and the photographing light of the subject converges in the shape of a dot or a circle on the photographing element 140 accordingly. Form a pattern. In this case, the convergence pattern of the image can be expressed only in a bi-axial plane. Therefore, even when the plurality of subjects 150, 160, 170 are located at different distances from the lens, a pattern is formed in which the photographing light converges at the same point on the photographing element. It is impossible to estimate the depth.

FIG. 2 is a view showing an image obtained by using an aperture having one opening through which light passes, and FIG. 3 is a view showing an image restored to focus using a regularization restoration method, which is one of the conventional autofocus techniques. Drawing. Referring to FIG. 2, when a general aperture is used, the phenomenon is blurred evenly in the deteriorated area of the image, and it is not known whether the subject of the deteriorated area is located near or far from the lens. Referring to FIG. 3, the focus of the deteriorated area is restored but the deterioration phenomenon occurs in the area in which the original focus is achieved. As described above, the quality of the image is greatly reduced.

An object of the present invention is to provide an aperture for estimating distance information of a subject in an image in which a plurality of subjects located at different distances from a lens exist.

Another technical problem to be solved by the present invention is to estimate the distance information of the subject from the image, and to solve the problem of deterioration of the image quality due to the blurring or ringing phenomenon and the image restoration shown in the conventional autofocus technique. An imaging device is provided.

Another technical problem to be solved by the present invention is to provide an apparatus and method for restoring a focus of an image which can solve the problem of deterioration of image quality due to the reblur or ringing phenomenon and the image restoration in the conventional autofocus technique. have.

Another technical problem to be solved by the present invention is to execute a focus restoration method of an image on a computer that can solve the problem of deterioration of image quality due to the blurring or ringing phenomenon and the image restoration in the conventional autofocus technique. The present invention provides a computer-readable recording medium for recording a program.

In order to solve the above technical problem, the aperture device according to the present invention comprises a plate with a plurality of openings through which light passes; And color filters having different colors provided in the respective openings.

In accordance with another aspect of the present invention, an imaging device includes: an aperture having a plurality of openings provided with color filters having different colors; And an imaging device in which photographing light of a subject passing through the respective color filters is formed.

In order to solve the above further technical problem, the automatic focus restoring apparatus according to the present invention includes photographing light of a subject passing through each of the color filters of an aperture having a plurality of openings provided with color filters having different colors. Receives the original image generated by the image pickup device, the sum of the difference in the pixel intensity with respect to the surrounding pixels for each pixel constituting the original image to obtain the protrusion value, and the predetermined size of each pixel A focus value calculator configured to calculate a focus value for each pixel based on a sum of protrusion values corresponding to pixels located in the window; Based on the calculated focus value, a reconstructed region having a predetermined size including pixels smaller than a reference value preset in the original image is determined, and each of the color channels calculated from position information of the color filter of the aperture is determined. A channel alignment unit configured to generate an intermediate image in focus in the reconstruction area by performing movement and alignment of the respective color channels based on a motion vector corresponding to; And matching the remaining images to the reference image by using an image in which the matched focus is located at the center of the image among the original image and the intermediate image as the reference image, and forming a fusion region centered on the matched pixel. And an image fusion unit configured to determine each of the original image and the intermediate image, and to fuse the fusion regions to generate a reconstructed image of which the focus is restored in the entire region.

In order to solve the above technical problem, the automatic focus restoring method according to the present invention includes photographing light of a subject passing through each of the color filters of an aperture having a plurality of openings provided with color filters having different colors. Receives the original image generated by the image pickup device, the sum of the difference in the pixel intensity with respect to the surrounding pixels for each pixel constituting the original image to obtain the protrusion value, and the predetermined size of each pixel A focus value calculating step of calculating a focus value for each pixel by a sum of protrusion degree values corresponding to pixels located in the window; Based on the calculated focus value, a reconstructed region having a predetermined size including pixels smaller than a reference value preset in the original image is determined, and each of the color channels calculated from position information of the color filter of the aperture is determined. A channel alignment step of generating an intermediate image in focus in the reconstruction area by performing the movement and alignment of the respective color channels based on the motion vector corresponding to; And matching the remaining images to the reference image by using an image in which the matched focus is located at the center of the image among the original image and the intermediate image as the reference image, and forming a fusion region centered on the matched pixel. And an image fusion step of determining each of the original image and the intermediate image, and fusing the fusion regions to generate a reconstructed image of which focus is restored in all regions.

According to the present invention, when using an aperture having a plurality of openings provided with color filters having different colors, the focal lengths of the respective color filters are different. Therefore, the generated image may be separated into red, green, and blue color channels. When the distance information of the subject can be estimated from each image. In addition, the automatic focus restoration based on the movement of the color channel is used to restore the image using only one original image, so that the deteriorated image can be restored more quickly, but the quality of the reconstructed image is not deteriorated, but the blurring in the already focused area is performed. You can prevent the sound from happening or ringing.

Hereinafter, with reference to the accompanying drawings will be described in detail a preferred embodiment of the aperture device, image pickup device, automatic focus restoration device and method according to the present invention. First, in designating reference numerals to components of each drawing, the same components have the same number even though they are displayed on different drawings.

4 is a view showing a schematic structure of a preferred embodiment of the aperture device according to the present invention.

Referring to FIG. 4, the diaphragm 400 according to the present invention includes a plate 410 in which a plurality of openings 420, 430, and 440 are formed, and a color filter 450 installed in each of the openings 420, 430, and 440. , 460, 470.

The first opening 420 is formed on the center line C of the plate 410, and the second opening 430 and the third opening 440 are formed to be symmetrical to the center line C. Preferably, the openings are formed such that the distances from the center of the plate 410 to the center of each opening are all the same.

The color filters 450, 460, and 470 are installed at respective openings, and the green color filter 460 at the first opening 420, the red color filter 450 at the second opening 430, and the third opening ( The blue color filter 470 is installed in the 440. Since the positions of the respective color filters 450, 460, and 470 are different from each other in the aperture 400, the convergence pattern of the photographing light is displayed at different positions even in an image generated by the photographing light passing through the color filters 450, 460, and 470. Is formed.

5 is a view showing an image generated by the aperture 400 according to the present invention. Compared with the image shown in FIG. 2, the image generated by the aperture 400 according to the present invention has a different position where the photographing light of the subject passing through each color filter 450, 460, 470 forms a convergence pattern. Therefore, the shape of the subject is separated from the defocused area. Therefore, the distance from the lens to the place where the subject is located can be estimated based on the separated pattern.

6 is a block diagram showing the configuration of an image pickup apparatus according to the present invention, and FIG. 7 is a view showing a schematic structure of a preferred embodiment of the image pickup apparatus 600 according to the present invention.

6 and 7, the imaging device 600 according to the present invention includes an aperture 400, an imaging device 610, and a focus restoration unit 620.

The diaphragm 400 is installed in the lens unit 710 in front of the imaging device 600, and the imaging device 610 is installed in the main body 720 of the imaging device. The photographing light of the subject passing through the color filters 450, 460, and 470 of the aperture 400 is imaged on the imaging device 610 to generate an image, and the convergence pattern of the photographing light is red, green, and blue color extracted from the image. It appears different for the channel.

8 illustrates a convergence pattern of color channels corresponding to each color filter installed in the aperture. Referring to FIG. 8, the photographed light of the subject 840 located at a distance from the lens 830 passes through the color filters of the red 450, the green 460, and the blue 470 of the aperture 400. Convergence patterns are formed in the color channels of red 850, green 860, and blue 870. At this time, the convergence pattern in each color channel is displayed at the positions where the top, bottom, left, and right are changed at the positions of the corresponding color filters 450, 460, and 470. That is, the photographing light passing through the red color filter 450 forms a convergence pattern at the position indicated by reference numeral 855 of the red color channel 850, and the photographing light passing through the green color filter 460 is the green color channel. At the position indicated by reference numeral 865 of 860, the photographing light passing through the blue color filter 470 forms a convergence pattern at the position indicated by reference numeral 875 of the blue color channel 870. Therefore, the distance to the subject can be estimated based on the arrangement of the color filters and the convergence pattern shown in the color channel.

The focus restorer 620 is connected to the image pickup device 610 to be installed inside the main body 720 of the image pickup device 600, and receives an image from the image pickup device 610 to restore the focus of the deteriorated area. To this end, the focus restoring unit 620 includes a focus value calculating unit 630, a channel alignment unit 640, and an image fusion unit 650.

The focus value calculator 630 calculates the protrusion value by summing the difference of pixel intensities with neighboring pixels for each pixel constituting the original image input from the image pickup device 610, and determines a protrusion value based on each pixel. The focus value of each pixel is calculated by the sum of the protrusion values corresponding to the pixels in the window of size.

The channel alignment unit 640 determines the reconstructed areas having a predetermined size having a lower focus value than the surrounding area in the original image based on the calculated focus value, and thus positions the color filters 450, 460, and 470 of the aperture 400. Based on the motion vectors of the color channels 850, 860, and 870 calculated from the information, intermediate images whose focus is matched in each reconstructed region are generated by the movement of the color channels 850, 860, and 870.

The image fusion unit 650 combines the remaining images into the reference image by using the image in which the matched focus is located at the center of the image among the original image and the intermediate image as the reference image, and the fusion region centered on the pixel whose focus is matched. After deciding on each of the original image and the intermediate image, the fusion regions are fused to generate a reconstructed image in which the focus is restored in the entire region.

A plurality of openings 420, 430, and 440 which implement the focus restoring unit 620 as a separate device and have color filters 450, 460, and 470 having different colors in front of the focus value calculating unit 630. Is provided with an input interface (not shown) for receiving an image generated by the aperture 400, and an output interface (not shown) for outputting an image whose focus is restored at the rear end of the image fusion unit 650 is provided. If provided, it may be implemented as a separate exclusive autofocus restoring apparatus.

9 is a diagram illustrating an image in which the focus is restored by the imaging apparatus 600 and the auto focus restoration apparatus according to the present invention.

Referring to FIG. 9, when the imaging apparatus 300 and the autofocus restoration apparatus according to the present invention are used, the focus is completely restored in the region where the shape of the subject is separated, and the quality of the image is maintained as a whole and the original focus is achieved. The restore process does not affect anything.

10 is a flowchart illustrating a process of performing an embodiment of the automatic focus restoration method according to the present invention.

Referring to FIG. 10, the focus value calculator 630 protrudes each pixel constituting the original image when the original image generated by the aperture having the plurality of openings in which the color filters having different colors are installed is input. The degree value is calculated (S1000). The protrusion value increases when the image is well focused and can be calculated using the following absolute Laplacian operator.

Where, L _k is a function, to obtain the value x is the horizontal component of the Laplacian function L _k, and y denotes the vertical direction component of the function L _k.

Quadratic derivatives in the x and y directions often use opposite signs to cancel each other, so use the absolute Laplacian operator to prevent this. The protrusion value is defined as the sum of the pixel intensity between the pixel for which the protrusion value is to be obtained and the pixels positioned in the horizontal and vertical directions with respect to the pixel. This protrusion value is obtained by the following equation.

Here, ML _k (i, j) is a modified Laplacian value representing the protrusion value of a pixel having a coordinate of (i, j) in the image, and I (i, j) has a coordinate of (i, j) The pixel intensity of the pixel is shown.

The focus value calculator 630 calculates the protrusion values for all the pixels constituting the image by using Equation 2, and then calculates the focus values of the respective pixels by using the following equation (S1010). The focal value of the pixel is calculated by summing protrusion values of pixels whose protrusion values are greater than or equal to a preset threshold value among pixels in a window of a predetermined size with respect to each pixel.

Here, f (i, j) is a focus value of a pixel having coordinates of (i, j) in the image, N is a size of a window centered on a pixel for which the focus value is to be obtained, and ML _k (p, q) is The protrusion value of the pixel having coordinates of (p, q), and T ₁ represent a threshold value of a preset protrusion value.

Threshold T ₁ is determined experimentally and gives mostly acceptable results when selected in the range of 40-60. Meanwhile, the size of the window for calculating the focus value is preferably set to 3 * 3 (pixel) size in order to obtain high accuracy.

The channel alignment unit 640 determines a reconstructed region having a predetermined size having a lower focus value than the surrounding region in the original image for focus reconstruction based on the calculated focus value (S1020). This process divides the original image into regions of a predetermined size, for example, 1/16 size of the original image, calculates an average focus value of the pixels included in each region, By comparison, the difference is made by determining a region having an average focus value that is equal to or greater than a preset value as a restoration region. The number of restoration areas depends on the number of subjects present in the image and the distance from the lens to the subject.

Next, the channel alignment unit 640 calculates a motion vector of the red 850, green 860, and blue 870 color channels from the position information of the color filters 450, 460, and 470 of the aperture 400. The color channels 850, 860, and 870 are moved by the motion vectors to generate an intermediate image in which focus is matched in each reconstructed region (S1030). In the case of the image shown in FIG. 8, since the position of the green color filter 460 is between the remaining color filters 450 and 470, the remaining color channels 860 and 870 are aligned with the green color channel 860.

11 shows the motion vectors of the red and blue color channels 850 and 870 relative to the green color channel 860. 11, and r _i, and c _j is the vertical and the horizontal direction movement amount of a red color channel 850 is calculated from the position of the green color filter 460 of the red color filter 450, respectively, r _k and c _l is the amount of movement in the vertical and horizontal directions of the blue color channel 870, respectively, calculated from the position of the blue color filter 470 with respect to the green color filter 470.

The channel alignment unit 640 generates the intermediate image by moving the red and blue color channels 850 and 870 by the following equation based on the motion vector calculated by the method described with reference to FIG. 11.

Here, I _RGB is a generated intermediate image, S is a shift operator, I _R , I _B and I _G are color channels of red 850, blue 870, and green 860, respectively, and a motion vector (r _1). , c ₁ ) or (r ₂ , c ₂ ) represents the amount of movement in the vertical and horizontal directions of the red 850 or blue 860 color channel.

12A and 12B are diagrams illustrating an intermediate image generated by alignment of an original image and a color channel. Referring to FIG. 12A, the center portion of the image is focused, and the region indicated by reference numeral 1210 is not in focus, and thus positions of photographed light converge on the color channel are different, and thus the shape of the subject is separated. At this time, if the area indicated by reference number 1210 is determined as the reconstructed area and the color channels 850, 860, and 870 are moved and aligned, the focus of the reconstructed area indicated by reference number 1220 is restored as shown in FIG. 12B. As a result, a phenomenon in which the shape of the subject is separated and the focus is blurred in the middle portion where the focus is originally focused is generated, but in the region indicated by reference numeral 1220, an intermediate image in which the focus is matched is generated.

Next, the image fusion unit 650 matches the remaining images to the reference image by using the image in which the matched focus is located at the center of the image among the original image and the intermediate image as the reference image (S1040). Since the images generated by the movement of the color channels do not all exist at the same position, overlapping or missing images occur in the process of fusing the images. In order to prevent such a case, a process of matching the remaining images based on any one image is required, and this is performed by inversely moving the remaining images with respect to the reference image as shown in the following equation. A new image is not generated by the matching process, but the position of the pixels between the images is matched by the movement of the image itself.

Here, I _r is a reference image, S ^-1 is a reverse movement operation, and I _k is an image obtained by matching itself to the reference image.

The image fusion unit 650 matches an image and then determines a fusion region having a predetermined size centered on the pixels in focus among the original images and the intermediate images, and fuses the fusion regions to restore the focus in all regions. Generated reconstructed image (S1050). The fusion region may be determined to be the same as or similar to the restoration region, and the process of fusing the respective fusion regions is performed by the following equation.

Here, I _ck is a reconstructed image, F _k is a fusion region, and I _ak and I _bk are original images or intermediate images for fusion. Equation 6 shows a process of generating a reconstructed image I _ck by fusing the images I _ak and I _bk . According to equation (6), a reconstructed image is generated by combining the region except for the region corresponding to the fused region of the fusion region of the I _ak I _ak and _bk I. It is preferable to apply smoothing or low pass filtering to each of the fusion regions in order to remove unnatural boundaries between the fusion regions that may be caused by the image fusion.

Finally, the image fusion unit 650 replaces the missing pixels in the fusion process by the following equation (S1060). In this process, the image fusion unit 650 detects the position of the missing pixel in the reconstructed image, selects an image having a focal point that is closest to the missing pixel among the original image and the intermediate image, Pixels located at the same point are matched to positions of missing pixels in the reconstructed image.

Where mindis {x, y} is the function representing the shortest distance between x and y, I (x, y) is the missing pixel of the reconstructed image with coordinates of (x, y), and f _p (x, y) Is a pixel of the image in which the focal point corresponding to the coordinate closest to (x, y) is located among the original image and the intermediate image.

The invention can also be embodied as computer readable code on a computer readable recording medium. The computer-readable recording medium includes all kinds of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, and may also be implemented in the form of a carrier wave (for example, transmission over the Internet). Include. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion.

Although the preferred embodiments of the present invention have been shown and described above, the present invention is not limited to the specific preferred embodiments described above, and the present invention belongs to the present invention without departing from the gist of the present invention as claimed in the claims. Various modifications can be made by those skilled in the art, and such changes are within the scope of the claims.

1 is a view showing a convergence pattern of photographing light caused by a general aperture according to the prior art;

2 is a view showing an image generated by a conventional aperture according to the prior art;

3 is a view illustrating an image in which focus is restored by an autofocus technique according to the prior art;

4 is a view showing the configuration of a preferred embodiment of the aperture device according to the present invention,

5 is a view showing an image generated by the aperture device according to the present invention,

6 is a block diagram for explaining the operation of the preferred embodiment of the imaging apparatus according to the present invention;

7 is a diagram showing the configuration of a preferred embodiment of an image pickup apparatus according to the present invention;

8 is a view showing a convergence pattern of photographing light formed by the imaging device according to the present invention;

9 is a view showing an image in which the focus is restored by the image capturing apparatus and the automatic focus restoring apparatus according to the present invention;

10 is a flowchart illustrating a process of performing a preferred embodiment of the automatic focus restoration method according to the present invention;

FIG. 11 is a diagram illustrating an example of calculating a motion vector of color channels 850, 860, and 870 from position information of color filters 450, 460, and 470 of an aperture 400, and

12A and 12B illustrate an intermediate image generated by movement of an original image and color channels 850, 860, and 870, respectively.

Claims (18)

A plate having a plurality of openings through which light passes; And And a color filter having a different color from each other provided in each of the openings. The method of claim 1, One opening is formed on the center line of the plate, the remaining opening is formed so as to be symmetrical to the center line. delete delete delete delete delete delete delete Each of the constituents of the original image is obtained by receiving an original image generated by imaging light of a subject passing through each of the color filters of the aperture having a plurality of openings provided with color filters having different colors from each other. For each pixel, the difference between pixel intensities and neighboring pixels is summed to obtain the protrusion value, and each pixel is the sum of the protrusion values corresponding to pixels located in a window of a predetermined size centering on each pixel. A focus value calculator for calculating a focus value with respect to the focus value; Based on the calculated focus value, a reconstructed region including pixels smaller than a reference value preset in the original image is determined, and corresponding to each of the color channels calculated from position information of the color filter of the aperture. A channel alignment unit configured to generate an intermediate image in focus in the reconstruction area by performing movement and alignment of each color channel based on a motion vector; And Among the original image and the intermediate image, the matched focal point is an image positioned at the center of the image as a reference image, and the remaining images are matched with the reference image, and the fusion regions centered on the matched pixels are respectively formed. And an image fusion unit configured to determine the original image and the intermediate image of the image, and to fuse the fusion regions to generate a reconstructed image of which the focus is restored in the entire region. The method of claim 10, And the focus value calculator calculates a focus value by summing protrusion values of pixels having a protrusion value greater than or equal to a preset threshold value among pixels in the window centered on the pixel. The method of claim 10, The channel alignment unit divides the original image into a plurality of divided regions, calculates an average focus value of pixels included in each divided region, and then, a difference from the divided region having the highest average focus value is equal to or greater than a preset value. And a partition area having an average focus value as the recovery area. The method according to any one of claims 10 to 12, And the image fusion unit replaces the missing pixel in the fusion process with a pixel at the same position as the missing pixel of the intermediate image having a focal point coinciding with the vicinity of the missing pixel among the intermediate images. Focus Restoration Device. Each of the constituents of the original image is obtained by receiving an original image generated by imaging light of a subject passing through each of the color filters of the aperture having a plurality of openings provided with color filters having different colors from each other. For each pixel, the difference between pixel intensities and neighboring pixels is summed to obtain the protrusion value, and each pixel is the sum of the protrusion values corresponding to pixels located in a window of a predetermined size centering on each pixel. A focus value calculating step of calculating a focus value for the; Based on the calculated focus value, a reconstructed region including pixels smaller than a reference value preset in the original image is determined, and corresponding to each of the color channels calculated from position information of the color filter of the aperture. A channel alignment step of generating an intermediate image in focus in the reconstruction area by performing movement and alignment of each color channel based on a motion vector; And Among the original image and the intermediate image, the matched focal point is an image positioned at the center of the image as a reference image, and the remaining images are matched with the reference image, and the fusion regions centered on the matched pixels are respectively formed. And an image fusion step of determining the original image and the intermediate image of the image, and fusing the fusion regions to generate a reconstructed image of which the focus is restored in the entire region. The method of claim 14, In the focus value calculating step, the focus value is calculated by summing the protrusion values of pixels whose protrusion values are greater than or equal to a preset threshold value among pixels in the window centered on the pixels. The method of claim 14, The channel sorting step, Dividing the original image into a plurality of divided regions and calculating an average focus value of pixels included in each divided region; And And determining, as the reconstructed region, a divided region having an average focus value whose difference from the divided region having the highest average focus value is greater than or equal to a preset value. The method according to any one of claims 14 to 16, In the image fusion step, the pixel missing in the fusion process is replaced with a pixel at the same position as the missing pixel of the intermediate image having a focal point in the vicinity of the missing pixel among the intermediate images. Automatic focus restoration method. A computer-readable medium having recorded thereon a program for executing the automatic focus restoration method according to any one of claims 14 to 16 on a computer.

Images