JP2011242134A - Image processor, image processing method, program, and electronic device - Google Patents

Abstract

The position and orientation of a camera can be estimated more easily with a simple configuration.
A composite image generation unit 103 projects a position of a pixel constituting a reference image serving as a reference for estimation onto a position of a pixel on a captured image corresponding to a pixel constituting the reference image. Then, a composite image composed of each pixel existing at a position on the captured image projected by the projection parameter is generated, and the evaluation unit 104 generates an evaluation function representing a correlation between the composite image and the reference image, and updates the parameters The unit 106 updates the projection parameter based on the evaluation function, and estimates at least one of the position or orientation of the imaging unit based on the updated projection parameter. The present invention can be applied to, for example, a computer that estimates the position and orientation of a camera and performs processing based on the estimation result.
[Selection] Figure 3

Description

  The present invention relates to an image processing device, an image processing method, a program, and an electronic device, and in particular, an image suitable for use when, for example, estimating the position and orientation of a camera based on a captured image obtained by imaging with a camera. The present invention relates to a processing device, an image processing method, a program, and an electronic device.

  For example, there are the following first and second estimation methods as an estimation method for performing calibration for estimating the position and orientation of the camera based on a captured image obtained by imaging of the camera.

  In the first estimation method, the camera captures a wallpaper or the like having a specific pattern (for example, a checkered pattern). Then, the difference square sum between corresponding pixels of the reference image prepared in advance and the captured image is calculated, and the calculated difference square sum is used as an evaluation function for estimating the position and orientation of the camera. Thus, the position and orientation of the camera are estimated (see, for example, Patent Document 1).

  The reference image represents an image obtained when a wallpaper having a specific pattern is captured by a camera whose position and orientation are known.

  Further, in the second estimation method, for example, from an 8-bit image configured by pixels having an 8-bit pixel value, bits existing at eight different positions among the respective bits configuring an 8-bit bit string are respectively determined. Eight 1-bit images are generated as pixel values.

  Then, eight 1-bit images are sequentially displayed on an LCD (Liquid Crystal Display), and the eight 1-bit images displayed on the LCD are captured based on the captured images obtained by the camera. The position and orientation of the camera are estimated (see, for example, Non-Patent Document 1).

JP 2002-27507 A

Yasutoshi Nakamura, Hitoshi Watanabe: High-precision stereo camera calibration based on image differences, Image Sensing Symposium SSII09, IS1-10, 2009.

  However, in the first estimation method described above, when calculating the evaluation function, in order to reduce the adverse effect due to the level difference of the pixel value (luminance value) that occurs between the captured image and the reference image, In addition to capturing images such as wallpaper, it is necessary to normalize (divide) the pixel values of each pixel constituting each of the reference image and the captured image and binarize them with a predetermined value.

  In the second estimation method described above, the number of pixels of the 1-bit image displayed on the LCD is limited according to the total number of pixels that can be displayed on the LCD. Further, in the second estimation method, in order to capture eight 1-bit images displayed on the LCD, it has been necessary to perform imaging at least eight times by the camera.

  Further, in the second estimation method, since an LCD is used for calibration, for example, the configuration is large compared with a case where calibration is performed without using an LCD.

  The present invention has been made in view of such a situation, and makes it possible to more easily estimate the position and orientation of a camera with a simple configuration.

  An image processing apparatus according to a first aspect of the present invention is an image processing apparatus that estimates at least one of the position or orientation of the imaging unit based on a captured image obtained by imaging of the imaging unit, On the captured image projected by the projection parameter based on the projection parameter that projects the position of the pixel constituting the reference image to the position of the pixel on the captured image corresponding to the pixel constituting the reference image. Based on the evaluation function, a composite image generation unit that generates a composite image composed of each pixel existing at a position, an evaluation generation unit that generates an evaluation function that represents a correlation between the composite image and the reference image, Updating means for updating the projection parameter; and estimation means for estimating at least one of the position or orientation of the imaging unit based on the updated projection parameter. An image processing apparatus.

  The evaluation generation unit generates the evaluation function represented by a quadratic polynomial, and the update unit updates the projection parameter calculated by the steepest descent method using the evaluation function as a new projection parameter. be able to.

  The evaluation generation means generates the evaluation function having a candidate projection parameter representing the updated projection parameter candidate as a variable, and the update means newly sets the candidate projection parameter when the evaluation function is minimized. It can be updated as a new projection parameter.

  An initial parameter generating means for generating the projection parameter based on the position of each pixel constituting the reference image and the corresponding position on the captured image can be further provided, and the composite image generating means In response to the projection parameter being generated by the parameter generation unit, the composite image is generated based on the projection parameter, and the updated unit is updated in response to the update of the projection parameter by the update unit. The composite image can be generated based on the projection parameters.

  The reference image is obtained by imaging with another imaging unit having a known position and orientation different from the imaging unit, and the estimation unit includes the position of the imaging unit with respect to the other imaging unit and It is possible to estimate at least one of the position or the posture of the imaging unit based on the projection parameter representing the difference in posture.

  An image processing method according to a first aspect of the present invention is an image processing method of an image processing apparatus that estimates at least one of the position or orientation of the imaging unit based on a captured image obtained by imaging of the imaging unit, The image processing apparatus includes a composite image generation unit, an evaluation generation unit, an update unit, and an estimation unit, and the image synthesis unit determines a position of a pixel constituting a reference image serving as a reference for estimation. Based on a projection parameter projected to the position of the pixel on the captured image corresponding to the pixel constituting the image, a composite image composed of each pixel present at the position on the captured image projected by the projection parameter Generating the evaluation function representing the correlation between the composite image and the reference image, and updating the projection parameter based on the evaluation function. The estimating means, based on said projection parameter after update, an image processing method comprising the step of estimating at least one of the position or orientation of the imaging unit.

  The program according to the first aspect of the present invention is based on a captured image obtained by imaging by an imaging unit, and uses a computer of an image processing apparatus that estimates at least one of the position or orientation of the imaging unit as a criterion for estimation. Based on the projection parameter that projects the position of the pixel that constitutes the image to the position of the pixel on the captured image that corresponds to the pixel that constitutes the reference image, the position on the captured image that is projected by the projection parameter Based on the evaluation function, a composite image generation unit that generates a composite image composed of each existing pixel, an evaluation generation unit that generates an evaluation function representing a correlation between the composite image and the reference image, and the projection based on the evaluation function Update means for updating parameters, and estimation means for estimating at least one of the position or orientation of the imaging unit based on the updated projection parameters Is a program for making the function Te.

  According to the first aspect of the present invention, based on a projection parameter for projecting the position of a pixel constituting a reference image serving as a reference for estimation to the position of a pixel on a captured image corresponding to the pixel constituting the reference image. A composite image composed of each pixel existing at a position on the captured image projected by the projection parameter is generated, and an evaluation function representing a correlation between the composite image and the reference image is generated. Based on the evaluation function, the projection is performed. The parameter is updated, and at least one of the position and orientation of the imaging unit is estimated based on the updated projection parameter.

  An electronic device according to a second aspect of the present invention is an electronic device that estimates at least one of the position or orientation of the imaging unit based on a captured image obtained by imaging of the imaging unit, and serves as a reference for estimation. Based on the projection parameter that projects the position of the pixel that constitutes the image to the position of the pixel on the captured image that corresponds to the pixel that constitutes the reference image, the position on the captured image that is projected by the projection parameter Based on the evaluation function, a composite image generation unit that generates a composite image composed of each existing pixel, an evaluation generation unit that generates an evaluation function representing a correlation between the composite image and the reference image, and the projection based on the evaluation function Update means for updating parameters, estimation means for estimating at least one of the position or orientation of the imaging unit based on the updated projection parameters, and the estimation means Based on more estimated estimation result, an electronic device and an execution means for executing a predetermined process.

  According to the second aspect of the present invention, based on a projection parameter that projects the position of a pixel constituting a reference image serving as a reference for estimation to the position of a pixel on a captured image corresponding to the pixel constituting the reference image. A composite image composed of each pixel existing at a position on the captured image projected by the projection parameter is generated, and an evaluation function representing a correlation between the composite image and the reference image is generated. Based on the evaluation function, the projection is performed. The parameter is updated, and at least one of the position and orientation of the imaging unit is estimated based on the updated projection parameter, and a predetermined process is executed based on the estimation result.

  According to the present invention, it is possible to estimate the position and orientation of a camera more easily with a simple configuration.

It is a 1st figure for demonstrating the outline | summary of this invention. It is a 2nd figure for demonstrating the outline | summary of this invention. It is a block diagram which shows the structural example of the image processing apparatus which is this Embodiment. It is a flowchart for demonstrating the parameter update process which an image processing apparatus performs. It is a flowchart for demonstrating the 1st initial stage matrix calculation process which a synthetic | combination image generation part performs. It is a flowchart for demonstrating the 2nd initial stage matrix calculation process which a synthesized image generation part performs. It is a flowchart for demonstrating the composite image generation process which a composite image generation part performs. Is a diagram illustrating an example of the estimated result by projection matrix P ₀ obtained by the first and second initial matrix calculation process. It is a 1st figure which shows an example of the estimation result which changes according to noise. It is a 1st figure which shows an example of the estimation result which changes according to a pitch angle. It is a 1st figure which shows an example of the estimation result which changes according to a yaw angle. It is a 1st figure which shows an example of the estimation result which changes according to a roll angle. It is a 1st figure which shows an example of the estimation result which changes according to the partial area | region in a captured image. It is the 1st figure for explaining field matching processing. It is the 2nd figure for explaining field matching processing. It is a 2nd figure which shows an example of the estimation result which changes according to noise. It is a 2nd figure which shows an example of the estimation result which changes according to a pitch angle. It is a 2nd figure which shows an example of the estimation result which changes according to a yaw angle. It is a 2nd figure which shows an example of the estimation result which changes according to a roll angle. It is a 2nd figure which shows an example of the estimation result which changes according to the partial area | region in a captured image. It is a block diagram which shows the structural example of a computer.

Hereinafter, modes for carrying out the invention (hereinafter referred to as the present embodiment) will be described. The description will be given in the following order.
Embodiment (an example of updating a projection matrix based on the cross-correlation between a composite image and a reference image)
Modified example

[Outline of the present invention]
FIG. 1 shows an overview of the present invention.

  FIG. 1A shows a three-dimensional object whose three-dimensional position (x, y, z) is known. In FIG. 1A, the three-dimensional object is arranged on the XYZ coordinate axes defined by the X axis, the Y axis, and the Z axis.

  On the upper side of FIG. 1B, a reference image 41 obtained when a three-dimensional object is imaged by the camera 21 is shown. It is assumed that the three-dimensional position (x, y, z) and posture of the camera 21 are known.

Here, the posture of the camera 21 is represented by, for example, the rotation angle (θ _r , θ _p , θ _y ) of the camera 21. The rotation angles (θ _r , θ _p , θ _y ) are the imaging direction of the camera 21 when the center of the camera 21 is the origin and the roll axis, pitch axis, and yaw axis that are orthogonal to each other are defined at the origin. On the other hand, the roll angle θ _r representing the angle formed with the roll axis, the pitch angle θ _p representing the angle formed with the pitch axis, and the yaw angle θ _y representing the angle formed with the yaw axis are represented. The same applies to the camera 22.

  1B shows a captured image 42 obtained when a camera 22 captures a three-dimensional object. It is assumed that the three-dimensional position (x, y, z) and posture of the camera 22 are unknown.

  In the present invention, the three-dimensional position (x, y, z) and posture of the camera 22 are estimated based on the reference image 41 obtained by the camera 21 and the captured image 42 obtained by the camera 22.

That is, for example, in the present invention, based on the reference image 41 obtained by the imaging of the camera 21 and the captured image 42 obtained by the imaging of the camera 22, the positional deviation from the reference image 41 occurring in the captured image 42 is detected. As a correction parameter for correction, for example, a projection matrix P ₀ is calculated.

In the present invention, the projection matrix P ₁ is calculated by applying the steepest descent method such as the Levenberg-Marquardt method to the evaluation function generated based on the calculated projection matrix P ₀ . Furthermore, in the present invention, the projection matrix P ₂ is calculated by applying the steepest descent method to the evaluation function generated based on the calculated projection matrix P ₁ .

In this way, in the present invention, so as to calculate a new projection matrix P _{i + 1} based on the projection matrix P _i, and updates the projection matrix P _i to the new projection matrix P _{i + 1.} Then, the three-dimensional position (x, y, z) and posture of the camera 22 are estimated based on the projection matrix P _i finally obtained. Note that the projection matrix P _i represents the i + 1-th (i represents a natural number) projection matrix.

Next, FIG. 2 shows an example of how the projection matrix P _i is updated using the steepest descent method.

In the present invention, as shown in FIG. 2, there is a three-dimensional object identical to the partial region 42 a (a three-dimensional object (subject) existing on the reference image 41) in the captured image 42 obtained by imaging by the camera 22. A projection matrix P ₀ for correcting the positional deviation occurring in (region) to the position of the reference image 41 is calculated. In the present invention, based on the calculated projection matrix P ₀ , the positional deviation occurring in the partial area 42 a in the captured image 42 is corrected to the position of the reference image 41 to generate the composite image 61.

Further, in the present invention, an evaluation function representing the degree of correlation between the composite image 61 generated based on the projection matrix P ₀ and the reference image 41 is calculated.

Then, the present invention applies the steepest descent method to the calculated evaluation function to calculate the projection matrix P ₁ when the evaluation function is minimized. As a result, the projection matrix P ₀ is updated to the projection matrix P ₁ with higher accuracy for correcting the positional deviation occurring in the captured image 42 to the position of the reference image 41.

Further, the present invention newly generates a composite image 61 based on the projection matrix P ₁ , and calculates an evaluation function representing the degree of correlation between the generated composite image 61 and the reference image 41. Then, the steepest descent method is applied to the calculated evaluation function to calculate a projection matrix P ₂ when the evaluation function is minimum, and the projection matrix P ₁ is updated to a new projection matrix P ₂ . Thereafter, similarly, the projection matrix P _i is updated using the steepest descent method.

In the present invention, for example, if the projection matrix P _i obtained by the update converges, that is, if the projection matrix P _i-1 and almost the same projection matrix P _i is obtained by the update, based on the projection matrix P _i The three-dimensional position (x, y, z) and posture of the camera 22 are estimated.

Note that the projection matrix P _i is used to correct a positional deviation from the reference image 41 occurring in the partial region 42a in the captured image 42 as described above. Therefore, the projection matrix P _i represents a positional deviation that occurs between the partial area 42 a in the captured image 42 and the reference image 41.

In addition, the positional shift that occurs between the partial area 42 a in the captured image 42 and the reference image 41 is a three-dimensional position (x, y, z) of the camera 22 and a rotation angle (θ _r , θ _p , θ _y ).

That is, the projection matrix P _i changes in accordance with the three-dimensional position (x, y, z) and rotation angle (θ _r , θ _p , θ _y ) of the camera 22. This represents the difference between the three-dimensional position (x, y, z) and the rotation angle (θ _r , θ _p , θ _y ).

For this reason, in the present invention, 3 of the camera 22 is based on the projection matrix P _i that changes according to the three-dimensional position (x, y, z) and the rotation angle (θ _r , θ _p , θ _y ) of the camera 22. The posture represented by the dimension position (x, y, z) and the rotation angle (θ _r , θ _p , θ _y ) can be estimated.

[Configuration Example of Image Processing Device 81]
FIG. 3 shows a configuration example of the image processing apparatus 81 according to the present embodiment.

  The image processing apparatus 81 includes a three-dimensional information holding unit 101, a reference image holding unit 102, a composite image generation unit 103, an evaluation unit 104, an optimization calculation unit 105, and a parameter update unit 106.

The three-dimensional information holding unit 101 holds a three-dimensional position (x, y, z) ^T of a three-dimensional object existing on the reference image 41 in advance. For convenience of explanation, the three-dimensional position (x, y, z) is represented by a matrix. That is, T represents transposition, and therefore the three-dimensional position (x, y, z) ^T represents a 3 × 1 matrix.

Specifically, for example, the three-dimensional information holding unit 101 associates the three-dimensional position (x, y, z) ^T of the three-dimensional object displayed on each pixel constituting the reference image 41 with each pixel. Is held.

  Note that the two-dimensional position (x, y) of the pixels constituting the reference image 41 is the captured image with the position of the pixel located at the lower left among the pixels constituting the reference image 41 as the origin (0, 0). The XY coordinates are defined by defining the X axis in the horizontal direction 42 and the Y axis in the vertical direction.

  That is, in the present embodiment, (x, y, z) out of the XY coordinates representing the two-dimensional position (x, y) of the pixels constituting the reference image 41 and the three-dimensional position (x, y, z) of the three-dimensional object. It is assumed that the XY coordinates representing y) coincide with each other.

Three-dimensional information holding unit 101, two-dimensional position of each pixel constituting the reference image 41 (x _j, y _j) as the feature point m _j, for each feature point m _j, features representing the characteristics of the feature point m _j The amount is held in advance. Note that the feature amount for each feature point m _j is generated in advance based on the reference image 41.

In this embodiment, for the sake of simplicity, the two-dimensional position (x _j , y _j ) of each pixel constituting the reference image 41 is set as the feature point m _j , but the feature point m _j The position adopted as is not limited to this.

In other words, for example, the feature point m _j includes pixels that are separated by predetermined pixels (for example, 5 pixels or 25 pixels) in the horizontal direction and the vertical direction in each pixel constituting the reference image 41 2. Only the dimension position (x _j , y _j ) may be adopted as the feature point m _j .

  The reference image holding unit 102 holds a reference image 41 obtained by imaging a three-dimensional object with the camera 21 in advance.

A captured image 42 is supplied from the camera 22 to the composite image generation unit 103. The composite image generation unit 103 generates a composite image 61 from the captured image 42 by correcting the positional deviation of the captured image 42 from the camera 22 from the reference image 41 based on the projection matrix P _i .

That is, for example, the composite image generation unit 103 obtains the feature points m _j on the reference image 41 held in the 3D information holding unit 101 and the feature quantities for each feature point m _j from the 3D information holding unit 101. read out.

Then, based on the feature amount for each feature point m _j that has been read, the composite image generation unit 103 determines from the captured image 42 from the camera 22 the feature point n _j (imaging image) on the captured image 42 that corresponds to the feature point m _j. The position (u _j , v _j )) of the pixels constituting the image 42 is extracted.

  Note that the position (u, v) of the pixel constituting the captured image 42 is the origin (0,0) of the pixel located at the lower left among the pixels constituting the captured image 42. It is represented by UV coordinates that define the U axis in the horizontal direction and the V axis in the vertical direction.

The composite image generation unit 103 _selects the three-dimensional position (x _j , y _j) corresponding to the feature point m _j between the read feature point m _j on the reference image 41 and the corresponding feature point n _j on the captured image 42. , z _j ) ^T and the two-dimensional position (x _j , y _j , z _j ) ^T using the two-dimensional position (u _j , v _j ) ^T as the feature point n _j by the least square method or the like. Projection matrix P ₀ for conversion (projection) to position (u _j , v _j ) ^T is calculated.

Based on the calculated projection matrix P ₀ , the composite image generation unit 103 converts the three-dimensional position (x _j , y _j , z _j ) ^T corresponding to the read feature point n _j to the two-dimensional position on the captured image 42. (U _j , v _j ) Convert to ^T.

The composite image generation unit 103 extracts a pixel existing at the two-dimensional position (u _j , v _j ) ^T obtained by the conversion from each pixel constituting the captured image 42 from the camera 22, and uses the extracted pixel. The composed composite image 61 is generated. Then, the composite image generation unit 103 supplies the generated composite image 61 and the projection matrix P ₀ to the evaluation unit 104.

In addition, the composite image generation unit 103 reads a three-dimensional position (x, y, z) ^T that is stored in advance in the three-dimensional information storage unit 101 and is associated with each pixel constituting the reference image 41. Further, the composite image generation unit 103 converts the read three-dimensional position (x, y, z) ^T into a two-dimensional image on the captured image 42 based on the projection matrix P _i (≠ P ₀ ) from the parameter update unit 106. Position (u, v) Convert to ^T.

The composite image generation unit 103 extracts a pixel existing at the two-dimensional position (u _j , v _j ) ^T obtained by the conversion from each pixel constituting the captured image 42 from the camera 22, and uses the extracted pixel. The composed composite image 61 is generated. Then, the composite image generation unit 103 supplies the generated composite image 61 and the projection matrix P _i from the parameter update unit 106 to the evaluation unit 104.

  The evaluation unit 104 reads the reference image 41 held in the reference image holding unit 102 from the reference image holding unit 102.

Evaluation unit 104 reads the reference image 41, and based on the composite image 61 and the projection matrix P _i from the composite image generation unit 103, as shown in the following equation (1), the reference image 41 and the synthetic image 61 An evaluation function f (x _n ) representing the correlation is calculated.

・・・（１）

... (1)

Here, in Expression (1), the pixel values a _i and a _j represent the pixel values of each pixel constituting the reference image 41, and the pixel values b _i and b _k represent the pixels of the composite image 61. Represents a pixel value. In addition, i, j, and k each take a value from 1 to the total number of pixels constituting the reference image 41 (or the composite image 61). The reference image 41 and the composite image 61 are configured with the same number of pixels.

Further, in Equation (1), the candidate matrix x _n represents a candidate for a new projection matrix P _{i + 1} , and x _n = P _i + d _n . D _n is a matrix represented by the same row and column as the projection matrix Pi, and represents a different matrix.

Incidentally, the pixel values a _i and a _k of the pixels constituting the reference image 41 are constants, and the pixel values b _i and b _k of the pixels constituting the synthesized image 61 change according to the candidate matrix x _n. It is a variable. Therefore, the evaluation function f (x _n ) is a function having the pixel values b _i and b _k as variables.

The evaluation unit 104 supplies the calculated evaluation function f (x _n ) to the optimization calculation unit 105.

Optimizing operation unit 105, the evaluation function f (x _n) from the evaluation unit 104, by applying the steepest descent method, the evaluation function f (x _n) candidate matrix when is minimized x _n (= P _i + d _n ) is calculated as a new projection matrix P _{i + 1} and supplied to the parameter update unit 106.

The parameter update unit 106 supplies the projection matrix P _{i + 1} from the optimization calculation unit 105 to the composite image generation unit 103. In this case, the composite image generation unit 103 generates a composite image 61 based on the projection matrix P _{i + 1} from the parameter update unit 106.

For example, when the parameter update unit 106 determines that the projection matrix P _{i + 1} from the optimization calculation unit 105 has converged, the parameter update unit 106 determines whether the camera 22 is based on the projection matrix P _{i + 1} from the optimization calculation unit 105. The three-dimensional position (x, y, z) and the rotation angle (θ _r , θ _p , θ _y ) representing the posture are estimated and output to the subsequent stage.

By the way, according to the steepest descent method performed in the optimization calculation unit 105, when the evaluation function f (x _n ) is represented by a quadratic polynomial, it is easier than when it is not represented by a quadratic polynomial. The solution can be calculated.

Therefore, it is desirable that the evaluation function f (x _n ) is represented by a quadratic polynomial. Therefore, consider expressing the evaluation function f (x _n ) shown in the equation (1) by a quadratic polynomial.

  First, the right side of Expression (1) is squared to obtain an inverse, and the evaluation function is changed to that shown in Expression (2) below.

・・・（２）

... (2)

In the formula ^(2), Σa _j 2 are constants, not a value that changes by the candidate matrix x _n, the evaluation function f which varies depending on the candidate matrix x _n (x _n), excludes? A _j ² Equation (3) is derived.

・・・（３）

... (3)

  In equation (3), Σ (summation) for variable k is moved to the outside and converted to equation (4).

・・・（４）

... (4)

Here, the evaluation function f (x _n ) shown in Expression (4) is a quadratic polynomial for the variables b _i and b _k .

Therefore, when the steepest descent method is applied to the evaluation function f (x _n ) shown in Equation (4), the evaluation function f (x _n ) is compared with the case where the evaluation function f (x _n ) is not represented by a quadratic polynomial. Thus, since the projection matrix P _{i + 1} as a solution of the steepest descent method can be easily calculated, the projection matrix P _{i + 1} can be calculated more quickly.

In equation (4), f (B _k ) represents (b _k / Σa _i b _i ).

Hereinafter, the evaluation unit 104 calculates the evaluation function f (x _n ) shown in the equation (4) instead of the evaluation function f (x _n ) shown in the equation (1), and sends it to the optimization calculation unit 105. Then, the optimization calculation unit 105 calculates a new projection matrix P _{i + 1} by the steepest descent method based on the evaluation function f (x _n ) shown in Expression (4) from the evaluation unit 104. The description will be made assuming that the image is supplied to the composite image generation unit 103.

[Description of Operation of Image Processing Device 81]
Next, parameter update processing performed by the image processing apparatus 81 will be described with reference to the flowchart of FIG.

  This parameter update process is started when, for example, the captured image 42 is supplied from the camera 22 to the composite image generation unit 103 of the image processing apparatus 81.

In step S < _b > 1, the composite image generation unit 103 is based on the feature points m _j on the reference image 41 stored in the three-dimensional image holding unit 101 and the feature points n _j on the captured image 42 from the camera 22. An initial matrix calculation process for calculating the projection matrix P _i (= P ₀ ) is performed. The initial matrix calculation process will be described later with reference to FIGS.

In step S <b> 2, the composite image generation unit 103 stores the three-dimensional position (x, y, z) ^{T corresponding} to each pixel constituting the reference image 41, which is stored in advance in the three-dimensional information storage unit 101. and based on the projection matrix P _i calculated in the processing of step S1, the composite image generation processing for generating a composite image 61 from the captured image 42. Details of the composite image generation process will be described later with reference to FIG.

  The composite image generation unit 103 supplies the composite image 61 generated by the composite image generation process to the evaluation unit 104.

In step S <b> 3, the evaluation unit 104 reads the reference image 41 held in advance in the reference image holding unit 102 from the reference image holding unit 102. Then, the evaluation unit 104 evaluates the evaluation function f (x _n ) as shown in Expression (4) based on the read reference image 41 and the composite image 61 and the projection matrix P _i from the composite image generation unit 103. Is supplied to the optimization calculation unit 105.

In step S4, the optimization calculation unit 105 applies the steepest descent method based on the evaluation function f (x _n ) from the evaluation unit 105, and the candidate matrix x when the evaluation function f (x _n ) is minimized. _n is calculated as a new projection matrix P _{i + 1} and supplied to the parameter update unit 106.

In step S5, the parameter update unit 106 supplies the projection matrix P _{i + 1} from the optimization calculation unit 105 to the composite image generation unit 103, and returns the process to step S2.

In step S <b> 2, the composite image generation unit 103 stores the three-dimensional position (x, y, z) associated with each pixel constituting the reference image 41 that is stored in advance in the three-dimensional information storage unit 101. ^{Based on T} and the new projection matrix P _{i + 1} updated in the immediately preceding step S5, a composite image generation process for generating the composite image 61 from the captured image 42 is performed, and thereafter the same process is performed. It is.

In step S5, for example, when the parameter updating unit 106 determines that the projection matrix P _{i + 1} from the optimization calculation unit 105 has converged, the parameter update unit 106 is based on the projection matrix P _{i + 1} from the optimization calculation unit 105. Thus, the three-dimensional position (x, y, z) and orientation of the camera 22 are estimated and output to the subsequent stage, and the parameter update process is terminated.

[Details of initial matrix calculation process]
Next, the first initial matrix calculation process as the initial matrix calculation process performed by the composite image generation unit 103 in step S1 of the parameter update process will be described with reference to the flowchart of FIG.

In step S21, the composite image generation unit 103, the 3-dimensional information holding unit 101 reads the feature quantity of the feature point m _j and the feature point m _j on the reference image 41. The variable j represents a natural number from 1 to J.

In step S _< b> 22, the composite image generation unit 103 captures an image supplied from the camera 22 as a feature point having the same feature quantity as the feature quantity of the read feature point m _j (= (x _j , y _j ) ^T ). 42, feature points n _j (= (u _j , v _j ) ^T ) are extracted.

Thus, the composite image generation unit 103, a feature point m _j on the reference image 41, corresponding to the feature point m _j, J-number of combinations (m _j of the feature points n _j on the captured image 42, n _j ) _Is acquired as a combination corresponding point (m _j , n _j ).

In step S23, the composite image generation unit 103, J-number of combinations corresponding points (m _j, n _j) from extracted different Q (<J) pieces of combination corresponding points (m _j, n _j), respectively, extracted A set U _k composed of the Q combination corresponding points (m _j , n _j ) is generated.

That is, the composite image generation unit 103 extracts Q combination corresponding points (mj, nj) from _J combination corresponding points (m _j , n _j ) by _J C _Q combinations. Each set of Q combination corresponding points (mj, nj) is generated as _J C _Q sets U _k . Note that k is a natural number from 1 to _J C _Q.

In step S24, the composite image generation unit 103 calculates a projection matrix P _k obtained based on Q combination corresponding points (mj, nj) constituting the set U _k for each generated set U _k .

Specifically, for example, the composite image generation unit 103 has a three-dimensional position (x _j , y) corresponding to the feature point m _j in Q combination corresponding points (m _j , n _j ) constituting the set U _{k. j} , z _j ) ^T and the two-dimensional position (u _j , v _j ) ^T as the feature point n _j , and the error Err1 = Σ _{j = 1} ^J {(u _j , v _j ) Calculate a projection matrix P _k that minimizes ^T −P _k (x _j , y _j , z _j ) ^T }.

In step S25, the composite image generation unit 103 calculates the calculated error Err1 = Σ _{j = 1} ^J {(u _j , v _j ) ^T −P _k (x _j , y _j among the plurality of calculated projection matrices P _k. , z _j ) Let the projection matrix P _k when ^T } is minimum be the projection matrix P _min .

In step S _< b> 26, the composite image generation unit 103 determines the three-dimensional position (x _j , y _j , z _j ) ^T corresponding to the feature point m _j and the feature _among the J combination corresponding points (m _j , n _j ). Using the two-dimensional position (u _j , v _j ) ^T as the point n _j , the error E _n = Σ _{j = 1} ^J (u _j , v _j ) ^T -P _n (x _j , y _j , z _j ) ^T = Σ _{j = 1} ^J (u _j , v _j ) ^T − (P _min + d _n ) (x _j , y _j , z _j ) ^T is calculated.

Then, the composite image generation unit 103, based on the calculated value E _n, calculates a function _{g (E n) = αE n} 2 / (1 + αE n 2). Α is a constant, for example, a value that is 0 or more and 1 or less.

In step S27, the composite image generation unit 103, the steepest descent method to calculate the E _n when minimizing the function g (E _n), corresponding to the calculated _{E n P n = (P min} + d n ) Is a projection matrix P ₀ . Then, after the process of step S27 is completed, the process is returned to step S1 of FIG. 4, the process is advanced to step S2, and the subsequent processes are performed.

Note that the composite image generation unit 103 calculates the projection matrix P ₀ by the first initial matrix calculation process described above, but the projection matrix P by the second initial matrix calculation process shown in the flowchart of FIG. You may make it calculate ₀ .

[Details of second initial matrix calculation process]
Next, the second initial matrix calculation process performed as the initial matrix calculation process performed by the composite image generation unit 103 in step S1 of the parameter update process will be described with reference to the flowchart of FIG.

  In steps S41 to S45, processing similar to that in steps S21 to S25 in FIG. 5 is performed.

In step S46, the composite image generation unit 103, for each of the J combination corresponding points (m _j , n _j ), a three-dimensional position (x _j , y _j , z _j ) ^T corresponding to the feature point m _j , and Based on the two-dimensional position (u _j , v _j ) ^T as the feature point n _j , the error Err2 = (u _j , v _j ) ^T −P _min (x _j , y _j , z _j ) ^T is calculated. Then, the composite image generation unit 103 selects a combination corresponding point (m _j , n _j ) from which the corresponding error Err2 is less than a predetermined threshold among the _J combination corresponding points (m _j , n _j ). .

In step S47, the composite image generation unit 103 selects the three-dimensional position (x _j , y _j , z _j ) ^T corresponding to the feature point m _j and the feature point at the selected combination corresponding point (m _j , n _j ). Using the two-dimensional position (u _j , v _j ) ^T as n _j , (u _j , v _j ) ^T -P ₀ (x _j , y _j , z _j ) ^T is minimized by the least square method or the like The projection matrix P ₀ is calculated, the process is returned to step S1 in FIG. 4, the process proceeds to step S2, and the subsequent processes are performed.

[Details of composite image generation processing]
Next, the composite image generation process performed by the composite image generation unit 103 in step S2 of the parameter update process will be described with reference to the flowchart of FIG.

In step S <b> 61, the composite image generation unit 103 reads the three-dimensional position (x, y, z) ^T associated with each pixel constituting the reference image 41 from the three-dimensional information holding unit 101.

Then, in step S62, the composite image generation unit 103, the projection matrix P ₀ is generated in step S1, when advanced to step S2, based on the generated projection matrix P _0, the read reference image A three-dimensional position (x, y, z) ^T on 41 is converted into a two-dimensional position (u, v) ^T on the captured image 42.

In step S62, the composite image generation unit 103 is supplied with a new projection matrix P _{i + 1} from the parameter update unit 106 and proceeds from step S5 to step S2. The three-dimensional position (x, y, z) ^T read from the three-dimensional information holding unit 101 is converted into a two-dimensional position (u, v) ^T on the captured image 42 based on the correct projection matrix P _{i + 1.} .

In step S63, the composite image generation unit 103 exists at the two-dimensional position (u, v) ^T on the captured image 42 obtained by the process of step S62 among the pixels constituting the captured image 42 from the camera 22. Extract the pixels to be used.

  Then, the composite image generation unit 103 generates an image composed of the extracted pixels as the composite image 61, returns the processing to step S2 in FIG. 4, and supplies the generated composite image 61 to the evaluation unit 104. Then, the subsequent processing is performed.

As described above, according to the parameter update process, in step S3, the captured image 42 (the synthesized image 61) and the reference image 41 are changed according to the difference in performance between the camera 21 and the camera 22 and the difference in the imaging conditions. An evaluation function P (x _n ) representing the correlation between the composite image 61 and the reference image 41 is calculated as an evaluation function that is not easily affected by the level difference that occurs in the pixel value.

Therefore, for example, in the parameter updating process, the difference square sum Σ (a _i −b _i ) ² for the pixel values of the corresponding pixels of the composite image 61 and the reference image 41 is used as the evaluation function P (x _n ). As in the case, it is not necessary to image a wallpaper or the like on which a specific pattern (pattern) is displayed as a subject, or normalize the composite image 61 and the reference image 41 to convert them into a binary image.

  That is, according to the parameter update process, the subject to be imaged by the cameras 21 and 22 is not limited to a wallpaper or the like on which a specific pattern is displayed, and any object may be imaged as a subject. The image 61 and the reference image 41 do not need to be binarized images, and can be used as images represented by RGB (Red, Green, Blue) values in addition to grayscale.

In the parameter updating process, as shown in the equation (4), the evaluation function P (x _n ) represented by the quadratic polynomial is used. Therefore, the evaluation function P (x _n ) is not a quadratic polynomial. It is possible to calculate a new projection matrix P _{i + 1} as a solution of the steepest descent method more quickly than in the case where there is no such method.

Further, for example, in the parameter updating process, the evaluation function P (x _n ) represented by the quadratic polynomial is used as shown in the equation (4), so that the evaluation function P ( x _n ), even if the pixel values a _i , a _j , b _i and b _k used in the evaluation function P (x _n ) are reduced in order to increase the calculation speed of the evaluation function P (x _n ) Therefore, it is possible to calculate the projection matrix P _{i + 1} with high accuracy.

Note that even if the pixel values a _i , a _j , b _i, and b _k used in the evaluation function P (x _n ) are reduced, a relatively accurate projection matrix P _{i + 1} can be calculated. This has been confirmed by experiments conducted by the applicant. The experiment conducted by the applicant and the items confirmed by the experiment will be described with reference to FIGS.

Further, for example, even when noise is generated in the captured image 61 in the parameter update process, when using the evaluation function (x _n ) as shown in Expression (4), for example, the sum of squares Σ (a _i − b _i) as compared to when using a ² evaluation function P (x _n), it can be calculated with higher accuracy projection matrix P _{i + 1.}

Note that the fact that a relatively accurate projection matrix P _{i + 1} can be calculated even if noise occurs in the captured image 42 will be described with reference to FIG. 16 described later.

[Estimation result by projection matrix P ₀ obtained by first and second initial matrix calculation processes]
Next, FIG. 8 shows that 121 combinations corresponding points (m _j , n _j ) are prepared for 15 different patterns, and the first and second initial matrix calculation processes are performed based on the prepared 15 patterns. It shows the estimation results estimated from projection matrix P ₀ obtained when performing.

8A and 8B, black bar graphs are obtained when the first initial matrix calculation process is performed based on 121 combination corresponding points (m _j , n _j ) of 15 different patterns. An average of errors in the estimation result based on the projection matrix P _{0 obtained} is shown.

8A and 8B, the bar graphs shown in white are obtained when the second initial matrix calculation process is performed based on 121 combination corresponding points (m _j , n _j ) in 15 different patterns. The average of the errors in the estimation results based on the projection matrix P ₀ obtained is shown.

  Specifically, in the center of FIG. 8A, the bar graph shown in white is an error as the average error of the three-dimensional position when the noise for the captured image 42 is 0 (when no noise is generated for the captured image 42). 0.00011633 is represented, and the bar graph shown in black represents the error 0.000145667.

  Note that the average error of the three-dimensional position refers to the three-dimensional position (x, y, z) of the camera 22 estimated as an estimation result obtained for 15 different patterns, and the actual three-dimensional position (x ′, y ′). , z ′) and | x−x ′ | + | y−y ′ | + | z−z ′ |

  On the right side of FIG. 8A, as an average error of the rotation angle when the noise with respect to the captured image 42 is 0, a bar graph shown in white represents an error 0.000123333, and a bar graph shown in black represents an error 0.000142667. .

Note that the average error of the rotation angle is a rotation angle (θ _r , θ _p , θ _y ) representing the posture of the camera 22 estimated as estimation results obtained for 15 different patterns, and an actual rotation angle (θ _r ', θ _p ', θ _y ') and an average of | θ _r -θ _r ' | + | θ _p -θ _p '| + | θ _y -θ _y ' |.

On the left side of FIG. 8A, when the noise with respect to the captured image 42 is 0, the three-dimensional position and rotation angle estimated from the newly generated projection matrix P ₁ based on the projection matrix P ₀ by the parameter update process. An average error addition value obtained by adding the respective average errors is shown. On the left side of FIG. 8A, as the average error addition value based on the projection matrix P ₁ , the average error addition value 0 is represented for any bar graph shown in white and black.

  Further, in the center of FIG. 8B, the average error of the three-dimensional position in the case where the noise with respect to the captured image 42 is 3.0 (originally, the pixel value 3 is added to the pixel value of each pixel of the captured image 42 without noise). As shown, the bar graph shown in white represents the error 1.58628, and the bar graph shown in black represents the error 1.067658667.

  On the right side of FIG. 8B, as an average error of the rotation angle when the noise with respect to the captured image 42 is 3.0, a bar graph shown in white represents an error 6.623128333, and a bar graph shown in black represents an error 1.421813333. .

On the left side of FIG. 8B, when the noise with respect to the captured image 42 is 3.0, the three-dimensional position and rotation angle estimated from the newly generated projection matrix P ₁ based on the projection matrix P ₀ by the parameter update process. An average error addition value obtained by adding the respective average errors is shown. In Figure 8B the left, as the average error adding value based on projection matrix P _1, bar graph indicated by white represents the average error sum value 0.032973, bar graph shown in black represents the average error sum value 0.333367333.

As shown in the center and right side of FIG. 8A and the center and right side of FIG. 8B, when no noise is generated in the captured image 42 (noise is 0), it is generated by the first and second initial matrix calculation processes. There is no significant difference in the error of the estimation results for the projected projection matrix P ₀ .

However, as shown in the center and right side of FIG. 8A and the center and right side of FIG. 8B, when noise occurs in the captured image 42 (noise is 3.0), it is generated by the first initial matrix calculation process. and towards the estimation results for the projection matrix P _0, compared with the estimation results for the projection matrix P ₀ generated by the second initial matrix calculation process, the error of the estimation result becomes very small.

Next, referring to FIGS. 9 to 13, the three-dimensional position (x, y, z) and rotation of the camera 22 estimated based on the projection matrix P ₀ calculated using the first initial matrix calculation process The three-dimensional position (x, y, z) estimated based on the angle (θ _r , θ _p , θ _y ), and the projection matrix P ₀ calculated using the second initial matrix calculation process, and the rotation angle (θ _{r, θ p, θ y)} will be described.

[State of estimation result that changes according to noise generated on captured image 42]
FIG. 9 shows a state in which the estimated three-dimensional position (x, y, z) and the rotation angle (θ _r , θ _p , θ _y ) change according to noise generated in each pixel constituting the captured image 42. An example is shown.

  9A and 9B represent pixel values added as noise generated in each pixel.

  The vertical axis in FIG. 9A represents the error | x−x ′ | + | between the estimated three-dimensional position (x, y, z) and the actual three-dimensional position (x ′, y ′, z ′). y-y '| + | z-z' | This also applies to the vertical axis in FIGS. 10A, 11A, 12A, and 13A.

Furthermore, the vertical axis in FIG. 9B represents the error | θ _r − between the estimated rotation angle (θ _r , θ _p , θ _y ) and the actual rotation angle (θ _r ′, θ _p ′, θ _y ′). θ _r '| + | θ _p -θ _p ' | + | θ _y -θ _y '| This also applies to the vertical axis in FIGS. 10B, 11B, 12B, and 13B.

In FIG. 9, graphs 121 to 123 represent graphs of estimation results estimated based on the projection matrix P ₀ calculated using the second initial matrix calculation process.

Further, in FIG. 9, graphs 141 to 143 represent graphs of estimation results estimated based on the projection matrix P ₀ calculated using the first initial matrix calculation process.

Note that, in the combination of the graphs 121 and 141, the combination of the graphs 122 and 142, and the combination of the graphs 123 and 143, a different feature point m _j is adopted as the feature point m _j on the reference image 41 for each different combination. Shows the combination of graphs obtained.

  The same applies to the graphs 121 to 123 and the graphs 141 to 143 in FIGS. 10 to 13 described later.

[State of estimation result that changes according to pitch angle θ _p of camera 22]
FIG. 10 shows an example of how the estimated three-dimensional position (x, y, z) and rotation angle (θ _r , θ _p , θ _y ) change according to the change in the pitch angle θ _p of the camera 22. Show. 10A and 10B represents the pitch angle θ _p of the camera 22, and the roll angle θ _r and yaw angle θ _y of the camera 22 are set similarly to the camera 21.

[State estimation result changes according to the yaw angle theta _y Camera 22]
FIG. 11 shows an example of how the estimated three-dimensional position (x, y, z) and rotation angle (θ _r , θ _p , θ _y ) change according to the change in the yaw angle θ _y of the camera 22. Show. 11A and 11B represents the yaw angle θ _y of the camera 22, and the roll angle θ _r and the pitch angle θ _p of the camera 22 are set similarly to the camera 21.

[State of estimation result that changes according to roll angle θ _r of camera 22]
FIG. 12 shows an example of how the estimated three-dimensional position (x, y, z) and rotation angle (θ _r , θ _p , θ _y ) change according to the change in the roll angle θ _r of the camera 22. Show. 12A and 12B represents the roll angle θ _r of the camera 22, and the pitch angle θ _p and the yaw angle θ _y of the camera 22 are set similarly to the camera 21.

[State of the estimation result that changes according to the area of the partial region 42a in the captured image 42]
FIG. 13 shows an example of how the estimated three-dimensional position (x, y, z) and rotation angle (θ _r , θ _p , θ _y ) change according to the area of the partial region 42 a in the captured image 42. Is shown. 13A and 13B represents the ratio of the area of the partial region 42a when the area of the entire region on the captured image 42 is 100.

The three-dimensional position (x ′, y ′, z ′) and rotation angle (θ _r ′, θ _p ′, θ _y ′) estimated from the projection matrix P ₀ calculated using the first initial matrix calculation process In any of the cases shown in FIGS. 9 to 13, the three-dimensional position (x ′, y ′, z ′) estimated from the projection matrix P ₀ calculated using the second initial matrix calculation process and It can be seen that the estimation is more accurate than the rotation angles (θ _r ′, θ _p ′, θ _y ′).

Next, referring to FIG. 14 and FIG. 15, a combination corresponding point (m _j , n _j ) between the reference image 41 and the composite image 61 is extracted by region matching processing, and the extracted combination corresponding point (m _j , n _j , A case where the projection matrix P ₁ is generated based on n _j ) will be described.

  The region matching processing described with reference to FIGS. 14 and 15 is used for comparison with the present invention in FIGS. 16 to 20 described later.

FIG. 14 shows how the combination corresponding points (m _j , n _j ) are extracted by the region matching process.

Incidentally, the composite image 61, as in the case of the parameter update process in the present invention, the calculated projection matrix P _0, is described as being generated based on the calculated projection matrix P _0.

In the area matching process, the feature point m _j on the reference image 41 is sequentially set as the feature point of interest, and a rectangular region 41a composed of a plurality of pixels existing around the feature point of interest including pixels existing at the feature point of interest. Set. In the area matching process, an area most similar to the rectangular area 41a is detected among all areas on the composite image 61, and the position where the pixel existing at the center of the detected area is associated with the feature point m _j . It is extracted as a feature point n _j on the composite image 61 to be performed.

Then, the region matching processing is performed based on the combination corresponding point (m _j , n _j ) representing the combination of the feature point m _j on the reference image 41 and the corresponding feature point n _j on the composite image 61. A projection matrix P ₁ is calculated by multiplication or the like.

Thereby, in the area matching process, as shown in FIG. 15, the three-dimensional position (x, y, z) corresponding to the feature point m _j on the reference image 41 is converted into the two-dimensional position on the corresponding captured image 42. A projection matrix P ₁ for conversion to (u, v) is calculated.

Next, referring to FIG. 16 to FIG. 20, the three-dimensional position (x, y, z) and rotation angle (θ _r , θ) of the camera 22 estimated based on the projection matrix P ₁ calculated by the region matching process. _p , θ _y ) and the three-dimensional position (x, y, z) and rotation angle (θ _r , θ _p , θ _y ) estimated based on the projection matrix P ₁ calculated by the parameter update process will be described.

[State of estimation result that changes according to noise generated on captured image 42]
FIG. 16 shows a state in which the estimated three-dimensional position (x, y, z) and rotation angle (θ _r , θ _p , θ _y ) change according to noise generated in each pixel constituting the captured image 42. An example is shown.

  16A and 16B represent pixel values added as noise generated in each pixel.

  Also, the vertical axis in FIG. 16A represents the error | x−x ′ | + | between the estimated three-dimensional position (x, y, z) and the actual three-dimensional position (x ′, y ′, z ′). y-y '| + | z-z' | This also applies to the vertical axis in FIGS. 17A, 18A, 19A, and 20A.

Further, the vertical axis in FIG. 16B represents the error | θ _r − between the estimated rotation angle (θ _r , θ _p , θ _y ) and the actual rotation angle (θ _r ′, θ _p ′, θ _y ′). θ _r '| + | θ _p -θ _p ' | + | θ _y -θ _y '| This also applies to the vertical axis in FIGS. 17B, 18B, 19B, and 20B.

In FIG. 16, graphs 161 to 163 represent graphs of estimation results estimated based on the projection matrix P ₁ calculated by the parameter update process.

Further, in FIG. 16, graphs 181 to 183 represent graphs of estimation results estimated based on the projection matrix P ₁ calculated by the region matching process.

Note that the graphs 161 and 181 indicate the two-dimensional positions (x, y) of the pixels that are separated by one pixel in the vertical direction and the horizontal direction among the pixels constituting the reference image 41, and the characteristics of the reference image 41. The graph about the case where it is set as the point m _j is shown.

The graphs 162 and 182 indicate the two-dimensional positions (x, y) of the pixels that are separated by 5 pixels in the vertical direction and the horizontal direction among the pixels constituting the reference image 41, and the characteristics of the reference image 41. The graph about the case where it is set as the point m _j is shown.

Furthermore, the graphs 163 and 183 show the two-dimensional positions (x, y) of the pixels that are separated by 25 pixels in the vertical direction and the horizontal direction among the pixels constituting the reference image 41, and the characteristics of the reference image 41. The graph about the case where it is set as the point m _j is shown.

  The same applies to the graphs 161 to 163 and the graphs 181 to 183 in FIGS.

[State of estimation result that changes according to pitch angle θ _p of camera 22]
FIG. 17 shows an example of how the estimated three-dimensional position (x, y, z) and rotation angle (θ _r , θ _p , θ _y ) change according to the change in the pitch angle θ _p of the camera 22. Show. 17A and 17B represents the pitch angle θ _p of the camera 22, and the roll angle θ _r and yaw angle θ _y of the camera 22 are set similarly to the camera 21.

[State estimation result changes according to the yaw angle theta _y Camera 22]
FIG. 18 shows an example of how the estimated three-dimensional position (x, y, z) and rotation angle (θ _r , θ _p , θ _y ) change according to the change in the yaw angle θ _y of the camera 22. Show. 18A and 18B represents the yaw angle θ _y of the camera 22, and the roll angle θ _r and the pitch angle θ _p of the camera 22 are set in the same manner as the camera 21.

[State of estimation result that changes according to roll angle θ _r of camera 22]
FIG. 19 shows an example of how the estimated three-dimensional position (x, y, z) and rotation angle (θ _r , θ _p , θ _y ) change according to the change in the roll angle θ _r of the camera 22. Show. 19A and 19B represents the roll angle θ _r of the camera 22, and the pitch angle θ _p and yaw angle θ _y of the camera 22 are set in the same manner as the camera 21.

[State of the estimation result that changes according to the area of the partial region 42a in the captured image 42]
FIG. 20 shows an example of how the estimated three-dimensional position (x, y, z) and rotation angle (θ _r , θ _p , θ _y ) change according to the area of the partial region 42 a in the captured image 42. Is shown. 20A and 20B represents the ratio of the area of the partial region 42a when the area of the entire region on the captured image 42 is 100. In FIG.

The three-dimensional position (x ′, y ′, z ′) and rotation angle (θ _r ′, θ _p ′, θ _y ′) estimated from the projection matrix P ₁ calculated by the parameter update process are the same as those in FIG. In any case shown in FIG. 20, the three-dimensional position (x ′, y ′, z ′) estimated from the projection matrix P ₁ calculated by the region matching process and the rotation angle (θ _r ′, θ _p ′, θ _It can be seen that the estimation is more accurate than _y ′).

In addition, at the three-dimensional position (x ′, y ′, z ′) and the rotation angle (θ _r ′, θ _p ′, θ _y ′) estimated from the projection matrix P ₁ calculated by the parameter update process, the reference image Even if 41 feature points exist every 25 pixels, it can be estimated with higher accuracy than in the case of region matching processing.

Therefore, according to the parameter update process of the present invention, even when the feature points of the reference image 41 used for the evaluation function f (x _n ) are sparse when generating the projection matrix P1 in the parameter update process, It is possible to estimate the three-dimensional position (x ′, y ′, z ′) and the rotation angle (θ _r ′, θ _p ′, θ _y ′) of the camera 22 with relatively high accuracy.
<2. Modification>
In the present embodiment, the three-dimensional information holding unit 101 holds the three-dimensional position (x, y, z) ^T of each pixel constituting the reference image 41 in advance. Instead of x, y, z) ^T , a two-dimensional position (x, y) ^T may be held.

In this case, if the two-dimensional position (x, y) ^T is expressed as the three-dimensional position (x, y, c) ^T , the parameter update process is performed in the same manner as described in the present embodiment. be able to. Note that c at the three-dimensional position (x, y, c) ^T represents a constant, for example, c = 0.

  Further, the image processing apparatus 81 in the present embodiment is configured not to include the camera 22, but may be configured to include the camera 22, for example.

Furthermore, in the present embodiment, the three-dimensional position (x, y, z) and posture of the camera 22 are estimated based on the projection matrix P _i. Any one of x, y, z) and posture may be estimated.

  As the image processing device 81, for example, a mobile terminal with a camera that performs predetermined processing based on the position and orientation of the camera 22, and an electronic device such as an HMD (head mount display) are adopted. Can do. That is, for example, as the image processing device 81, an AR (Augmented Reality) technique is used, and an image captured by the camera 22 is displayed on a monitor or the like (not shown) according to the position and orientation of the camera 22. An electronic device that performs processing for displaying an object or the like can be employed.

  This electronic apparatus acquires related information related to an object existing on a captured image obtained by imaging by the camera 22 from a server connected to the network.

  For example, each feature point pattern representing a pattern of feature points for an object existing on a captured image (for example, when a square paper as an object exists, four feature points respectively representing four corners of the paper are represented by 1 When the server is configured to hold the related information of the corresponding object in association with each other), the electronic device extracts the feature points from the captured image obtained by the imaging of the camera 22 The related information associated with the feature point pattern for the extracted feature points is acquired from the server.

  Further, when the server is configured so as to associate and hold the corresponding related information for each combination of the position and direction of the electronic device (for example, the imaging direction of the camera 22 incorporated in the electronic device), the electronic device The position of the electronic device is measured using a GPS (global positioning system) or the like, and the direction of the electronic device is measured using an acceleration sensor, a gyro sensor, or the like.

  And an electronic device will acquire the relevant information matched with the combination of the measured position and direction from a server.

  In addition, when the server is configured to hold corresponding related information in association with each position of the electronic device, the electronic device acquires the related information associated with the measured position, and the direction of the electronic device When the server is configured to hold corresponding related information in association with each other, the electronic apparatus acquires related information associated with the measured direction.

  Based on the acquired related information, the electronic device generates a composite image to be displayed by being combined (superimposed) on the captured image. Then, the electronic device estimates the position and orientation of the camera 22 as described above in the present embodiment, and combines the synthesized image with the captured image and displays it in consideration of the estimation result obtained as a result. Let

  The electronic device acquires the related information from the server connected to the network. However, for example, when the electronic device has a storage unit that holds the related information in advance, the electronic device is replaced with the server. Thus, related information can be acquired from the storage unit.

  In this case, the electronic device can acquire the related information from the storage unit without connecting to the network. Therefore, even in an environment where the electronic device cannot be connected to the network, the composite image corresponding to the related information is synthesized on the captured image. It is possible to display.

  Next, the series of processes described above can be executed by dedicated hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software can execute various functions by installing a so-called embedded computer or various programs. For example, it is installed from a recording medium in a general-purpose personal computer.

[Computer configuration example]
FIG. 21 shows a configuration example of a personal computer that executes the above-described series of processing by a program.

  A CPU (Central Processing Unit) 201 executes various processes according to a program stored in a ROM (Read Only Memory) 202 or a storage unit 208. A RAM (Random Access Memory) 203 appropriately stores programs executed by the CPU 201 and data. The CPU 201, ROM 202, and RAM 203 are connected to each other via a bus 204.

  An input / output interface 205 is also connected to the CPU 201 via the bus 204. Connected to the input / output interface 205 are an input unit 206 made up of a keyboard, mouse, microphone, and the like, and an output unit 207 made up of a display, a speaker, and the like. The CPU 201 executes various processes in response to commands input from the input unit 206. Then, the CPU 201 outputs the processing result to the output unit 207.

  A storage unit 208 connected to the input / output interface 205 includes, for example, a hard disk, and stores programs executed by the CPU 201 and various data. The communication unit 209 communicates with an external device via a network such as the Internet or a local area network.

  Further, a program may be acquired via the communication unit 209 and stored in the storage unit 208.

  The drive 210 connected to the input / output interface 205 drives a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the programs and data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 208 as necessary.

  As shown in FIG. 21, a recording medium that is installed in a computer and records a program that can be executed by the computer includes a magnetic disk (including a flexible disk), an optical disk (CD-ROM (compact disc-read only). memory), DVD (including digital versatile disc)), magneto-optical disc (including MD (mini-disc)), or removable media 211, which is a package media consisting of semiconductor memory, or the program is temporary or permanent ROM 202 recorded in the memory, a hard disk constituting the storage unit 208, and the like. Recording of a program on a recording medium is performed using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 209 that is an interface such as a router or a modem as necessary. Is called.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. It also includes processes that are executed individually.

  Furthermore, the present embodiment is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.

  81 image processing apparatus, 101 three-dimensional information holding unit, 102 reference image holding unit, 103 composite image generation unit, 104 evaluation unit, 105 optimization calculation unit, 106 parameter update unit

Claims (8)

In an image processing apparatus that estimates at least one of the position or orientation of the imaging unit based on a captured image obtained by imaging of the imaging unit,
Based on a projection parameter that projects the position of a pixel constituting a reference image serving as a reference for estimation onto the position of a pixel on the captured image corresponding to the pixel constituting the reference image, the projection parameter projects the position A composite image generating means for generating a composite image composed of each pixel present at a position on the captured image;
Evaluation generation means for generating an evaluation function representing a correlation between the composite image and the reference image;
Updating means for updating the projection parameter based on the evaluation function;
An image processing apparatus comprising: an estimation unit configured to estimate at least one of a position and a posture of the imaging unit based on the updated projection parameter. The evaluation generation unit generates the evaluation function represented by a quadratic polynomial,
The image processing apparatus according to claim 1, wherein the updating unit updates a projection parameter calculated by a steepest descent method using the evaluation function as a new projection parameter. The evaluation generation means generates the evaluation function having a candidate projection parameter representing a candidate of the updated projection parameter as a variable,
The image processing apparatus according to claim 2, wherein the updating unit updates the candidate projection parameter when the evaluation function is minimized as a new projection parameter. An initial parameter generating means for generating the projection parameter based on the position of each pixel constituting the reference image and the corresponding position on the captured image;
The composite image generation means includes
In response to the projection parameter being generated by the initial parameter generation means, the composite image is generated based on the projection parameter,
The image processing apparatus according to claim 1, wherein the composite image is generated based on the updated projection parameter in response to the update of the projection parameter by the update unit. The reference image is obtained by imaging with another imaging unit having a known position and orientation different from the imaging unit,
The image processing according to claim 1, wherein the estimation unit estimates at least one of a position or a posture of the imaging unit based on the projection parameter representing a difference in position and posture of the imaging unit with respect to the other imaging unit. apparatus. In the image processing method of the image processing apparatus that estimates at least one of the position or orientation of the imaging unit based on a captured image obtained by imaging of the imaging unit,
The image processing apparatus includes:
A composite image generating means;
Evaluation generation means;
Update means;
Including estimation means and
The image synthesizing unit projects the position of a pixel constituting a reference image serving as a reference for estimation based on a projection parameter that projects the position of a pixel on the captured image corresponding to the pixel constituting the reference image. Generating a composite image composed of each pixel present at a position on the captured image projected by the parameter;
The evaluation generation means generates an evaluation function representing a correlation between the composite image and the reference image;
The updating means updates the projection parameter based on the evaluation function,
An image processing method, comprising: a step of estimating at least one of a position or a posture of the imaging unit based on the projection parameter after update. A computer of an image processing apparatus that estimates at least one of the position or orientation of the imaging unit based on a captured image obtained by imaging of the imaging unit,
Based on a projection parameter that projects the position of a pixel constituting a reference image serving as a reference for estimation onto the position of a pixel on the captured image corresponding to the pixel constituting the reference image, the projection parameter projects the position A composite image generating means for generating a composite image composed of each pixel present at a position on the captured image;
Evaluation generation means for generating an evaluation function representing a correlation between the composite image and the reference image;
Updating means for updating the projection parameter based on the evaluation function;
A program for functioning as an estimation means for estimating at least one of the position or orientation of the imaging unit based on the updated projection parameter. In an electronic device that estimates at least one of the position or orientation of the imaging unit based on a captured image obtained by imaging of the imaging unit,
Based on a projection parameter that projects the position of a pixel constituting a reference image serving as a reference for estimation onto the position of a pixel on the captured image corresponding to the pixel constituting the reference image, the projection parameter projects the position A composite image generating means for generating a composite image composed of each pixel present at a position on the captured image;
Evaluation generation means for generating an evaluation function representing a correlation between the composite image and the reference image;
Updating means for updating the projection parameter based on the evaluation function;
An estimation means for estimating at least one of the position or orientation of the imaging unit based on the updated projection parameters;
An electronic device comprising: execution means for executing predetermined processing based on the estimation result estimated by the estimation means.

Images