JP2017138240A - Image processing device, image processing method, and program - Google Patents

Abstract

It is possible to obtain an F-number that increases the accuracy of distance information derived from a plurality of viewpoint images obtained by a plenoptic camera.
An error estimator for estimating an error in distance to a subject derived from a plurality of viewpoint images obtained by an imaging device from an arbitrary F value input, and an F value input to the error estimator are updated. However, an F value that minimizes the error in distance to the subject estimated by the error estimation unit is searched from among F values that can be set in the imaging apparatus, and the searched F value is an F value at the time of imaging of the imaging apparatus. F value determining means for determining as follows.
[Selection] Figure 7

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program for determining an F value at the time of imaging by an imaging apparatus.

  An imaging device (plenoptic camera) that collects light passing through different pupil regions on different pixels on an imaging device is known. In addition, by performing stereo matching using an image obtained by imaging with a plenoptic camera, that is, a plurality of images (multi-viewpoint images) generated from light passing through the different pupil regions, A technique for estimating the distance is known (Patent Document 1).

  Non-Patent Document 1 shows that the longer the baseline length, the higher the accuracy of distance information derived by stereo matching. In the plenoptic camera, the smaller the F value, the longer the baseline length. Therefore, according to the contents shown in Non-Patent Document 1, when estimating the distance to the subject from a plurality of viewpoint images obtained by the plenoptic camera, an F value as small as possible is set in the plenoptic camera. It is desirable to shoot.

JP 2009-229125 A Masatoshi Okutomi and Takeo Kanade, "A Multiple-Baseline Stereo", IEEE Trans. On Pattern Analysis and Machine Intelligence, 1993

  However, if the F value is decreased to increase the baseline length, the blur on the captured image increases. Therefore, the accuracy of stereo matching is lowered, and as a result, the accuracy of distance information derived by stereo matching is also lowered. That is, in the plenoptic camera, two parameters, the base line length and the blur size, are determined by the F value, and there is a trade-off regarding the accuracy of the distance information between the two parameters. Therefore, in the present invention, it is possible to obtain an F value that increases the accuracy of distance information derived from a plurality of viewpoint images obtained by a plenoptic camera.

  An image processing apparatus according to the present invention is an image processing apparatus that determines an F value at the time of imaging by a plenoptic imaging apparatus, and from an arbitrary F value that is input, from a plurality of viewpoint images obtained by the imaging apparatus. An error estimator that estimates an error in the distance to the subject to be derived, and an F value that minimizes the error in the distance to the subject estimated by the error estimator while updating the F value input to the error estimator. F value determination means for searching from among F values that can be set in the imaging apparatus and determining the searched F value as an F value at the time of imaging of the imaging apparatus.

  According to the present invention, it is possible to obtain an F value that increases the accuracy of distance information derived from a multi-viewpoint image obtained by a plenoptic camera.

It is a block diagram which shows an example of a structure of the image processing apparatus which concerns on a 1st Example. It is a figure which shows the structure of an imaging device. It is a figure for demonstrating the function of an imaging device. It is a figure for demonstrating the derivation | leading-out method of the distance information in an imaging device. It is the figure which showed the relationship between the error of pixel deviation | shift amount, and the error of distance information. It is the figure which showed the relationship between an aperture diameter and the error of distance information. FIG. 3 is a block diagram illustrating a software configuration of the image processing apparatus according to the first embodiment. It is a flowchart which shows the search process of the optimal F value in the image processing apparatus which concerns on a 1st Example. It is a figure which shows the relationship between F value and the error of distance information in case the error of pixel deviation | shift amount is constant irrespective of the diameter of a blur. It is a figure which shows the relationship between F value and the error of distance information in case the error of pixel deviation | shift amount is proportional to the diameter of a blur. It is a figure which shows the relationship between F value and the error of distance information in case the error of pixel shift amount increases rapidly as the diameter of blur increases. It is a figure which shows the relationship between F value and the error of distance information in case the error of pixel shift | displacement increases rapidly when the diameter of a blur exceeds a predetermined threshold value. It is a figure which shows the other structure of an imaging device. It is an example of the table which shows the relationship between the magnitude | size of a blur, and the difference | error of a pixel shift amount.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The configuration shown below is merely an example, and the present invention is not limited to the illustrated configuration.

<Example 1>
FIG. 1 is a block diagram illustrating an example of the configuration of the image processing apparatus according to the first embodiment.

  The image processing apparatus 100 includes a CPU 101, a RAM 102, a ROM 103, an HDD I / F 104, an HDD 105, an input unit 106, and a system bus 107. An imaging device 108 and an external memory 109 are connected to the image processing device 100. For example, the image processing apparatus 100 is mounted on a digital camera together with the imaging apparatus 108. The external memory 109 is installed in, for example, a card slot installed in a digital camera.

  The CPU 101 executes a program stored in the ROM 103 using the RAM 102 as a work memory, and comprehensively controls each component via the system bus 107. Thereby, various processes described later are executed.

  The HDD I / F 104 is an interface such as serial ATA (SATA), for example, and connects the HDD 105 as a secondary storage device. The CPU 101 reads data from the HDD 105 via the HDD I / F 104. Further, the CPU 101 writes data to the HDD 105 via the HDD I / F 104.

  The HDD 105 is a hard disk drive. The secondary storage device may be a storage device such as an optical disk drive in addition to the HDD.

  The input unit 106 is a serial bus interface such as USB or IEEE1394. The CPU 101 acquires data from the imaging device 108, the external memory 1009 (for example, a hard disk, a memory card, a CF card, an SD card, a USB memory) and the like via the input unit 106.

  The imaging device 108 is a plenoptic camera. Details of the imaging device 108 will be described later.

  Note that the image processing apparatus 100 may include other components.

[Details of Imaging Device 108]
FIG. 2 is a diagram illustrating an example of an internal configuration of the imaging apparatus (plenoptic camera) 108. The imaging device 108 includes a main lens 201, an aperture 202, a shutter 203, a microlens array 204, an optical low-pass filter 205, an iR cut filter 206, a color filter 207, an imaging element 208, and an A / D converter (simply referred to as A / D). 209. In the following description, the main lens 201 is assumed to be a single lens, but in actuality, it is composed of a plurality of zoom lenses and focus lenses.

  By adjusting the diaphragm 202, the base line length, blur size, depth of field, and incident light amount of the imaging device 108 can be adjusted. Note that the “baseline length” in the imaging device 108 has a different meaning from the “baseline length” that represents the distance between the stereo cameras. This point will be described later.

  The microlens array 204 is different from the condensing microlens disposed immediately before the image sensor 208. In general, a condensing microlens is provided with one optical system (for example, a convex lens) for each pixel of an image sensor, but the microlens array 204 has one optical system (for example, one for every two pixels). Two optical systems). In this embodiment, even if the size of each lens of the microlens array 204 is in units of micrometers or millimeters, it is called a microlens array regardless of the size of each lens.

  FIG. 3 is a diagram for explaining the function of the imaging apparatus 108. The imaging device (plenoptic camera) 108 has a function of condensing the light beam passing through the pupil region 301 and the light beam passing through the pupil region 302 of the main lens 201 on different pixels. In the example shown in FIG. 3, the light beam passing through the pupil region 301 is condensed on the pixels 303 to 305 (R pixel), and the light beam passing through the pupil region 302 is collected on the pixels 306 to 308 (L pixel). Here, FIG. 3 shows only the light beams incident on the pixels 303 to 308, but the same applies to other pixels on the image sensor 208. When only R pixels are collected to generate an image, an image (right viewpoint image) with the center (or centroid) 309 of the pupil region 301 as a viewpoint can be generated. Further, when an image is generated by collecting only L pixels, an image (left viewpoint image) with the center (or centroid) 310 of the pupil region 302 as a viewpoint can be generated. The distance between the two viewpoints, that is, the distance L between the center 309 of the pupil region 301 and the center 310 of the pupil region 302 is referred to as a baseline length. In the following, a plenoptic camera that divides the area of the main lens 201 into two partial pupil areas will be described as an example. However, the present invention is not limited to this, and the area of the main lens 201 is increased to more partial pupil areas. The present invention can also be applied to a plenoptic camera that divides.

[Distance information estimation method]
FIG. 4 is a diagram for explaining a method for deriving distance information in the imaging apparatus 108. A subject point 401 represents a certain point on the subject. The subject surface 402 is a surface represented by a set of points that are at the same distance as the subject point 401 in the direction perpendicular to the main lens 201 from the main lens 201. The condensing surface 403 is a surface represented by a set of points on which light emitted from each point on the object surface 402 is condensed in the camera. The focus surface 404 is a surface at a distance where the imaging apparatus 108 has focused. The microlens surface 405 is a surface on which the microlens array 204 is disposed.

  In the derivation of the distance information, first, a pixel shift amount B shown in FIG. 4 is derived by stereo matching with a plurality of viewpoint images (right viewpoint image and left viewpoint image) obtained by the imaging device 108. Any method can be used for stereo matching. For example, the right viewpoint image and the left viewpoint image are divided into small blocks, and the small block of the left viewpoint image having the highest correlation with each small block of the right viewpoint image is searched. By doing so, the pixel shift amount B can be calculated. Thereafter, the distance s is calculated from the pixel shift amount B. Hereinafter, the relationship between the pixel shift amount B and the distance s is shown. First, equation (2) can be derived from equation (1), which is the lens formula.

  Here, i is a distance from the main lens 201 to the condensing surface 403 (hereinafter referred to as an image distance). f is the focal length of the main lens. The image distance i can be expressed by Expression (3) based on the similarity relationship shown in FIG.

  Here, D is the aperture diameter. m is a distance from the main lens 201 to the microlens surface 405 (hereinafter referred to as a microlens distance). The microlens distance m can be expressed by Expression (5) using Expression (4) which is a lens formula.

  Moreover, Formula (6) is formed from the definition of F value.

  If the F value, the focal length f of the main lens, and the focus distance p are known in advance from the equations (1) to (6), the distance s can be calculated. Specifically, the image distance i is calculated by substituting the microlens distance m and the aperture diameter D calculated by the equations (5) and (6) into the equation (3), and the calculated image distance i is expressed by the equation (2). By substituting into, the distance s can be calculated. In this way, the distance s is derived using the distance s as distance information. Note that the units of s, f, D, i, m, p, and B in the equations (1) to (6) are all the same (for example, [mm]). The unit of the pixel shift amount that can be calculated by stereo matching is [pixel]. Therefore, the unit is aligned by calculating the product of the pixel shift amount that can be calculated by stereo matching and the pixel pitch μ [mm / pixel] of the image sensor 208 as the pixel shift amount B.

[Relationship between pixel shift amount, F value, blur, and distance information error]
If the pixel shift amount B calculated by stereo matching includes an error, an error also occurs in the distance information. FIG. 5 is a diagram showing the relationship between the error of the pixel shift amount B and the error of distance information according to the equations (1) to (6). Specifically, a distance calculated by adding an error to the pixel shift amount B calculated as an object distance 1250.0 [mm] and an F value 8.0 (distance s ′ described later), and an object distance 1250. It is the figure which plotted the difference with 0 [mm]. As shown in FIG. 5, the error in the distance information increases as the error in the pixel shift amount B increases. One of the factors that include an error in the pixel shift amount B is a decrease in stereo matching accuracy due to blurring on the image. This decrease in accuracy can be suppressed, for example, by reducing the blur on the image when the error of the pixel shift amount is proportional to the size of the blur. In order to reduce the blur on the image, the F value may be increased.

  FIG. 6 is a diagram showing the relationship between the aperture diameter D and the distance information error according to the equations (1) to (6). As shown in FIG. 6, even if the error of the pixel shift amount B is the same, the smaller the aperture diameter D (the shorter the base line length), the greater the error in distance information. In order to increase the aperture diameter D in order to reduce the error in the distance information, the F value may be decreased.

  As described above, depending on the relationship between the error of the pixel shift amount and the size of the blur, the error in the distance information can be reduced by reducing the blur on the image and increasing the aperture diameter D. However, to reduce the blur on the image, the F value must be increased. On the other hand, to increase the aperture diameter D, the F value must be decreased. That is, depending on the relationship between the error of the pixel shift amount and the size of the blur, there is a trade-off regarding the accuracy of the distance information between the size of the blur and the aperture diameter D. Therefore, the image processing apparatus according to the present embodiment calculates the F value so that the accuracy of the distance information is as high as possible in consideration of such a trade-off.

[Processing Flow of Example 1]
Processing of the image processing apparatus 100 will be described. FIG. 7 is a block diagram illustrating a software configuration of the image processing apparatus 100 according to the first embodiment. As illustrated in FIG. 7, the image processing apparatus 100 includes an initial F value input unit 701, a baseline length calculation unit 702, a blur amount calculation unit 703, a pixel shift amount error estimation unit 704, and a distance error estimation unit 705. F value determination unit 706. The image processing apparatus 100 calculates an F value (hereinafter referred to as an optimal F value) suitable for estimation of distance information using these components.

  FIG. 8 is a flowchart illustrating the optimal F value calculation process in the image processing apparatus 100 according to the first embodiment. In this embodiment, the ROM 103 and the HDD 105 store a program for performing the processing shown in FIG. 8, and each component shown in FIG. 7 functions when the CPU 101 loads and executes the program.

  Here, the image processing apparatus 100 mounted on the digital camera is taken as an example. In this embodiment, when pre-photographing is executed by an imaging control device (not shown) included in the digital camera, the image processing device 100 starts the processing shown in FIG.

  In step S801, the initial F value input unit 701 inputs an initial F value. Although any value may be used as the initial F value, in this embodiment, an F value that has been frequently selected as the optimum F value in the past distance estimation is input as the initial F value.

  In step S802, the baseline length calculation unit 702 calculates the baseline length. The baseline length L is calculated by the following formula.

  Here, D is the aperture diameter, and is calculated using Equation (6). n is the number of viewpoints of the plenoptic camera, and n = 2 in this embodiment.

  In step S803, the blur amount calculation unit 703 calculates the size of the blur. In this embodiment, the blur amount calculation unit 703 calculates the blur diameter as the blur size. The blur diameter r is calculated by the following equation.

  Here, D is the aperture diameter, and is calculated using Equation (6). i is an image distance, and is calculated using Equation (3). m is the microlens distance, and is calculated using Equation (5). n is the number of viewpoints of the plenoptic camera, and n = 2 in this embodiment.

  In step S804, the pixel shift amount error estimation unit 704 estimates a pixel shift amount error. Details of the processing of the pixel shift amount error estimation unit 704 will be described later.

  In step S805, the distance error estimation unit 705 estimates an error in distance information. Details of the processing of the distance error estimation unit 705 will be described later.

  In step S806, the F value determination unit 706 updates the F value so that the error in the distance information estimated in step S805 becomes smaller. An arbitrary method can be used to update the F value, for example, the steepest descent method can be used.

  In step S807, the F value determination unit 706 determines whether or not to update the F value. An arbitrary determination criterion may be used for the end determination. For example, the update may be terminated when the F value update amount becomes smaller than a predetermined threshold value.

  When it is determined that the update is finished (Yes in step S807), the image processing apparatus 100 finishes the process. In this way, the optimum F value is calculated.

  If it is not determined that the update has ended (No in step S807), the image processing apparatus 100 returns to step S802 and repeats the processes in steps S802 to S806 until it is determined that the update has ended. If it is not determined that the update is completed even if the processes in steps S802 to S806 are repeated for all the F values that can be set in the imaging apparatus 108, the F value with the smallest error in the distance information estimated in step S805 is used. Is calculated as the optimum F value.

  The F value determination unit 706 determines the calculated optimum F value as the F value at the time of image capturing by the image capturing apparatus 108.

[Details of Processing of Pixel Deviation Amount Error Estimating Unit 704]
The pixel shift amount error estimation unit 704 estimates an error of the pixel shift amount in the current F value. The error of the pixel shift amount differs depending on the stereo matching method used for calculating the pixel shift amount. Therefore, in the present embodiment, the relationship between the blur diameter and the pixel shift error is modeled after being classified into four based on the stereo matching method. Error estimation information (corresponding to equations (9) to (12) shown below) generated by the modeling is stored in a storage unit such as the ROM 103. The pixel shift amount error estimation unit 704 estimates a pixel shift amount error based on the error estimation information.

  The first classification is a case where the error of the pixel shift amount is constant regardless of the blur diameter. For example, when the pixel value noise and the design error of the imaging device are so small that they can be ignored, it can be considered that they are included in this classification. This is because the stereo matching accuracy degradation is considered to occur because the texture information on the surface of the subject is lost due to the blur and the influence of the error factor becomes relatively large. That is, when the error factor can be ignored, it is considered that the accuracy of stereo matching does not decrease. In this classification, the error e of the pixel shift amount is calculated by the following equation.

  Here, α is an arbitrary constant. Note that the pixel shift amount error e originally takes a different value for each pixel, but in this embodiment, a certain value is set as a representative value.

  The second classification is a case where the error of the pixel shift amount is proportional to the blur diameter. General stereo matching (for example, a scheme using Sum of Absolute Difference for calculating correlation between blocks) can be considered to be included in this classification. In this classification, the error e of the pixel shift amount is calculated by the following equation.

  Here, k is an arbitrary proportional constant, and r is a blur diameter.

  The third classification is a case where the error of the pixel shift amount increases more rapidly than the second classification as the blur diameter increases. For example, a stereo matching method that emphasizes accuracy at the expense of robustness against error factors, such as using Sum of Squared Difference for the correlation calculation between blocks, can be considered to be included in this classification. In this classification, the error e of the pixel shift amount is calculated by the following equation.

  Here, β is an arbitrary power root.

  The fourth classification is a case where the error of the pixel shift amount increases rapidly when the diameter of the blur exceeds a predetermined threshold value. For example, a stereo matching method in which the block size for calculating the pixel shift amount is constant regardless of the blur diameter can be considered to be included in this classification. In this classification, the error e of the pixel shift amount is calculated by the following equation.

  Here, z is an arbitrary threshold value. In this embodiment, the pixel shift amount error estimation unit 704 determines which one of the equations (9) to (12) is used based on the stereo matching method set in the digital camera. For example, when the digital camera supports a plurality of stereo matching methods, error estimation information corresponding to each stereo matching method is stored in advance in a storage unit such as the ROM 103. The pixel shift amount error estimation unit 704 determines the stereo matching method selected by the user, and determines which error estimation information to use based on the determination result. In addition, when it is known in advance that the pixel deviation amount error estimation unit 704 has a small influence on the pixel deviation amount due to pixel value noise, the design error of the imaging apparatus, or the like, the pixel deviation amount estimation unit 704 uses the stereo matching method selected by the user. Without using equation (9) as the error estimation information, it is determined.

  In this embodiment, the relationship between the blur diameter and the pixel shift amount error is modeled. However, the relationship between the F value and the pixel shift amount error may be modeled. That is, error estimation information that can estimate the magnitude of the pixel shift amount from the F value may be stored in advance in a storage unit such as the ROM 103. According to such a form, it is not necessary to calculate the amount of blur, and the configuration and processing of the image processing apparatus 100 can be simplified. Further, in this embodiment, the relationship between the blur diameter and the pixel shift amount error is classified into four, but may be classified into five or more. In addition, the pixel shift amount error estimation unit 704 is not limited to the above method, and the pixel shift amount error may be estimated by an arbitrary method.

[Details of Processing of Distance Error Estimation Unit 705]
The distance error estimation unit 705 estimates the distance information error d from the baseline length L, the pixel shift amount error e, and the subject distance s. Here, for the baseline length L, the value calculated by the baseline length calculation unit 702 is used. For the pixel shift amount error e, the value estimated by the pixel shift amount error estimation unit 704 is used. For the subject distance s, distance information calculated by a method such as Time of Flight may be used in addition to the distance information estimated from the image acquired by the above-described pre-shooting. Note that a method of selecting a target subject (hereinafter referred to as an important subject) for calculating the subject distance s is arbitrary. For example, a subject near the center of the shooting angle of view may be selected as an important subject. In the present embodiment, the subject distance s is not necessarily the distance information to be finally calculated, but is used for estimating an error in the distance information, and therefore does not necessarily have to be highly accurate.

  The pixel shift amount B of the subject at the distance s can be calculated by the following equation.

  Here, D is the aperture diameter. i is an image distance, and is calculated by the equation (1). m is a microlens distance, and is calculated by the equation (5). Note that, when using the distance information estimated from the image acquired by the pre-photographing, the pixel shift amount B is calculated by stereo matching in the calculation process of the subject distance s. Is not calculated.

  On the other hand, when the error e estimated by the pixel shift amount error estimation unit 704 is included in the pixel shift amount, the pixel shift amount B ′ is expressed by the following equation.

  The distance s ′ estimated using the pixel shift amount B ′ can be calculated by the following equation.

  Therefore, the error d of the distance information can be estimated by the following equation.

  The procedure for estimating the distance information error d will be summarized below.

  First, the distance error estimation unit 705 calculates a baseline length and a blur diameter from the F value. Next, the distance error estimation unit 705 calculates an error of the pixel shift amount from the blur diameter using the equations (9) to (12). Finally, the distance error estimation unit 705 estimates the error d of the distance information using the equations (13) to (16).

  In the above description, one important subject is selected and the error of the distance information in the important subject is calculated. However, the present invention is not limited to this. A plurality of important subjects may be selected, and an average value of distance information errors in the plurality of important subjects may be calculated as the distance information error d in Equation (16).

[Effect of Example 1]
For each of the four categories of the relationship between the blur diameter error and the pixel deviation amount error, the distance information error calculated by the optimum F value and the average value of the distance information error calculated by the F value of 1.0 to 32.0. By comparing, the effect of the present embodiment is shown. In the following, the error of distance information is calculated with f = 50.0, p = 1000.0, and s = 1250.0.

  FIG. 9 is a diagram illustrating the relationship between the F value and the distance information error in the first classification (when the error of the pixel shift amount is constant regardless of the blur diameter). The error of the distance information at the optimum F value shown in FIG. 9 = 1.0 (the portion surrounded by a broken-line circle in FIG. 9) is calculated as follows. First, the subject distance s = 1250.0 is calculated by a method such as Time of Flight. Next, the distance s ′ is calculated from the equation (15). B ′ (= B + e) required when calculating the distance s ′ from the equation (15) is calculated using B calculated from the equation (13). Here, the error e is 0.03. L, m, and i required when calculating B from Expression (13) are calculated from Expression (7), Expression (5), and Expression (1), respectively. In this way, the distance s ′ is calculated as 1251.9. From equation (17), the error d in the distance information is calculated as 1.9 [mm]. Thus, the error of distance information = 1.9 [mm] at the optimum F value = 1.0 is smaller than the average value 31.6 [mm] of the error of distance information at all F values, and the optimum F value It can be seen that the distance information calculated by is highly accurate.

  FIG. 10 is a diagram illustrating a relationship between the F value and the error of distance information in the second classification (when the error of the pixel shift amount is proportional to the blur diameter). As shown in FIG. 10, in this classification, the error of the distance information is constant regardless of the F value. From this, it is understood that when performing stereo matching included in this classification, photographing may be performed with an arbitrary F value.

  FIG. 11 shows the relationship between the F value and the distance information error in the third classification (when the error in pixel shift increases more rapidly than in the second classification as the diameter of the blur increases). FIG. As shown in FIG. 11, the error of the distance information at the optimum F value = 32.0 (the portion surrounded by the broken-line circle in FIG. 11) is 79.7 [mm], and the error of the distance information in all the F values Compared with the average value of 88.3 [mm], the accuracy is high.

  FIG. 12 shows the relationship between the F value and the distance information error in the fourth classification (when the blur diameter exceeds a predetermined threshold value, the error of the pixel shift amount increases rapidly). FIG. Here, the threshold value was set to 3.0 [pixel]. As shown in FIG. 12, the error of the distance information at the optimum F value = 8.0 (the portion surrounded by the broken-line circle in FIG. 12) is 15.0 [mm], and the error of the distance information in all the F values Compared to the average value of 69.1 [mm], the accuracy is high.

  In the above description, the plenoptic camera in which the microlens array 204 is disposed on the focal plane of the main lens 201 has been described as an example. However, the configuration of the imaging device 108 is not limited thereto. Therefore, the present invention can be applied to an arbitrary imaging device in which light passing through each region of the main lens 201 is received by separate pixels by the microlens array 204. For example, in the present invention, as shown in FIG. 13, the microlens array 204 is arranged so that the focal plane 1301 of the main lens 201 and the image sensor 208 are conjugate to the microlens array 204. It can also be applied to optic cameras. The image picked up by the plenoptic camera shown in FIG. 13 can be converted into a left and right viewpoint image like the plenoptic camera shown in FIG. 3, and the same processing as described above is applied to this left and right viewpoint image. By applying, distance information can be estimated. Note that the method of converting the imaging data of the plenoptic camera shown in FIG. 13 into the left and right viewpoint images is not the main point of the present invention, and thus the description thereof is omitted.

  In the above, the distance information error is estimated by using the distance estimation obtained by the pre-photographing method or the Time of Flight method, and the F value (optimum F value) suitable for the distance information estimation from the distance information error. ) Is calculated, but the method of calculating the optimum F value is not limited to this. As shown in FIGS. 9 to 12, if the relationship between the blur diameter and the pixel shift amount error can be modeled, an F value suitable for estimating the distance information can be obtained. Therefore, the optimum F value may be calculated directly from the model of the relationship between the blur diameter and the pixel shift amount error without estimating the distance information or estimating the error of the distance information.

  Further, a “distance estimation mode” may be provided in the shooting mode, and the image processing apparatus 100 may perform the above processing only when the shooting mode is the distance estimation mode.

  As described above, in the present embodiment, the relationship between the blur size and the pixel shift amount error is modeled. Then, based on the relationship, an F value that minimizes the error of the pixel shift amount is calculated. Thereby, it is possible to calculate the F value so that the accuracy of the distance information is as high as possible.

[Example 2]
In the first embodiment, the relationship between the blur size and the pixel shift error is modeled, and the optimum F value is searched based on the relationship. In the second embodiment, the image processing apparatus stores in advance in a storage unit such as the ROM 103 as a table the relationship between parameters such as blur size, pixel shift error, F value, and distance information error. Then, the image processing apparatus calculates the optimum F value by calculating the error of the distance information with reference to the table. The table is created based on, for example, the relationship between parameters obtained in past distance estimation. The ROM 103 or the like may store a table corresponding to each stereo matching method.

  FIG. 14 is an example of a table showing the relationship between the blur diameter and the pixel shift amount error. The pixel shift amount error estimation unit 704 in the present embodiment calculates a pixel shift amount error with reference to the table. The other processes in the present embodiment are the same as those in the first embodiment, and a description thereof will be omitted.

  As described above, in this embodiment, the pixel shift amount error estimation unit 704 refers to the table stored in the ROM 103 or the like, and calculates the pixel shift amount error from the size of the blur. By storing the relationship between the blur size and the pixel shift amount as a table, it is not necessary to search for the optimum F value as shown in FIG. 8, and the processing time required to derive the optimum F value is as follows. Can be shortened. Also, by maintaining the relationship between blur size and pixel shift error as a table, even when it is difficult to model the blur size and pixel shift error, the accuracy of distance information is increased. The F value can be calculated.

  In the above description, a table indicating the relationship between the blur size and the pixel shift error is used. However, the table used for calculating the error of the distance information is not limited to this, and an arbitrary table can be used. For example, the optimal F value may be calculated using a table indicating the relationship between the F value and the error in the pixel shift amount, and a table indicating the relationship between the F value and the error in the distance information. According to such a configuration, it is possible to estimate the error of the pixel shift amount and the error of the distance information directly from the F value, so that it is possible to further reduce the processing time required to derive the optimum F value.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

704 Pixel shift amount error estimation unit 705 Distance error estimation unit 706 F value determination unit

Claims (12)

An image processing apparatus for determining an F value at the time of imaging by a plenoptic imaging apparatus,
Error estimation means for estimating an error of a distance from a subject derived from a plurality of viewpoint images obtained by the imaging device from an arbitrary F value input;
While updating the F value input to the error estimation unit, search for an F value that minimizes an error in distance to the subject estimated by the error estimation unit from among F values that can be set in the imaging apparatus. And an F value determining means for determining the searched F value as an F value at the time of image capturing by the image capturing apparatus. The error estimation means includes
From an arbitrary F value that is input, an error of the pixel shift amount of the multi-viewpoint image obtained by the imaging device is estimated,
The image processing apparatus according to claim 1, wherein an error in a distance from a subject derived from the multi-viewpoint image is estimated from an error in the estimated pixel shift amount. A blur amount calculating means for calculating a blur size in a plurality of viewpoint images obtained by the imaging device from an arbitrary F value input;
Storage means for storing error estimation information capable of estimating an error of a pixel shift amount from the size of blur in the plurality of viewpoint images,
The error estimation means includes
The image processing apparatus according to claim 2, wherein an error of a pixel shift amount is estimated from a blur size calculated by the blur amount calculation unit using the error estimation information stored in the storage unit. The error estimation information is
The image processing apparatus according to claim 3, wherein the image processing apparatus is a formula representing a relationship between a blur size and an error of a pixel shift amount modeled based on a stereo matching method used for deriving a pixel shift amount. Storage means for storing error estimation information capable of estimating the error of the pixel shift amount from the F value;
The error estimation means includes
The image processing apparatus according to claim 2, wherein an error of a pixel shift amount is estimated from an arbitrary input F value using the error estimation information stored in the storage unit. The error estimation information is
The image processing apparatus according to claim 5, wherein the image processing apparatus is an expression indicating a relationship between an F value and an error of a pixel shift amount modeled based on a stereo matching method used for deriving the pixel shift amount. The storage means stores a plurality of error estimation information,
The error estimation means includes
The error of the pixel shift amount is estimated based on error estimation information corresponding to a stereo matching method used for derivation of the pixel shift amount among the plurality of error estimation information. An image processing apparatus according to 1. The error estimation means includes
Using a pixel shift amount derived by stereo-matching the multi-viewpoint images, deriving a first subject distance indicating a distance from the subject;
Using a value obtained by adding an error of the estimated pixel shift amount to a pixel shift amount derived by stereo matching of the multiple viewpoint images, a second subject distance indicating a distance from the subject is derived,
The image processing apparatus according to claim 2, wherein a difference between the first subject distance and the second subject distance is estimated as an error in the distance to the subject. . An image processing apparatus for determining an F value at the time of imaging by a plenoptic imaging apparatus,
Storage means for storing a table indicating correspondence between each F value that can be set in the imaging apparatus and an error in distance from a subject derived from a plurality of viewpoint images, which is obtained by the imaging apparatus in which each F value is set When,
With reference to the table stored in the storage means, an F value that minimizes an error in distance to the subject is determined, and the F value obtained by the determination is determined as an F value at the time of image capturing by the image capturing apparatus. An image processing apparatus comprising: an F value determination unit that performs the processing. The storage means holds a plurality of the tables,
The F value determining means includes
Of the plurality of tables stored in the storage means, refer to a table corresponding to a stereo matching method used when deriving the distance to the subject, and minimize an error in the distance to the subject. The image processing apparatus according to claim 9, wherein a value is determined. An image processing method for determining an F value at the time of imaging by a plenoptic imaging device,
An error estimation step of estimating an error of a distance from a subject derived from a plurality of viewpoint images obtained by the imaging device from an arbitrary F value;
While updating the F value, the error estimation step is repeated, and among the F values that can be set in the imaging apparatus, an F value that minimizes the error in the distance to the subject estimated in the error estimation step is searched for And an F value determination step of determining the searched F value as an F value at the time of imaging by the imaging apparatus.   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1-10.