JP2003065714A - Guide device and method for camera calibration, and camera calibration device - Google Patents

Abstract

(57) [Summary] [Problem] To provide a guideline of a calibration method in order to reduce a user's burden of a calibration operation without using a dedicated large-scale facility. SOLUTION: A camera to be calibrated is fixed to a tripod, and three planes are imaged on a guide sheet. On the guide sheet, the installation positions of the target planes for imaging three planes and the installation positions of the tripod at the time of each imaging are indicated. The installation location specified on the guide sheet has a constraint condition recommended by the present inventors. By imaging three planes along the guide sheet, the user can perform calibration easily and with sufficient accuracy.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a guide device and a guide method for camera calibration for guiding a user when performing a camera calibration operation for calculating a parameter representing a characteristic of a camera, and BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a camera calibration device, and in particular, a guide device for camera calibration that guides a user when performing a camera calibration operation for a camera of a type that images an object and outputs electronic image data. And a guide method, and a camera calibration device.

More specifically, the present invention relates to a method of performing a camera calibration operation of a type in which a parameter is calculated by using a captured image of an arbitrary plane without spatial positional constraint in the camera calibration of the stereo method. The present invention relates to a guide device and a guide method for camera calibration for guiding a user to a user, and a camera calibration device, and in particular, obtains sufficient accuracy without requiring expensive equipment or precision machined equipment. In addition, the present invention relates to a guide device and a guide method for camera calibration that reduce the burden of user's calibration operation, and a camera calibration device.

[0003]

2. Description of the Related Art A stereo method can be mentioned as one of distance measuring methods for an object. The stereo method uses an image captured from a plurality of viewpoints (projection centers) having a predetermined positional relationship, and determines the distance between each point in the scene, that is, the captured image, and the projection center by the so-called "triangulation" principle. It is a method of measuring.

Here, for convenience, there are two viewpoints, that is, two viewpoints.
The principle of the stereo method using a single camera will be described. One camera is used as a reference camera, picks up an image of an object from a position facing the front, and outputs a reference image. Further, the other camera is used as a detection camera, picks up an object from an oblique direction, and outputs a detection image. FIG. 7 schematically shows the arrangement of the reference camera and the detection camera with respect to the imaging plane, and FIG. 8 shows the reference image and the detection when a substantially square pattern is picked up by each of the reference camera and the detection camera. The image is shown schematically.

[0005] As shown in FIG. 7, in the captured image of the reference camera, projection screen S _b of the straight line and the reference camera connecting the projection center C _b of M and the reference camera points on a plane to be imaged
A point M is observed at the intersection point m with. A straight line connecting the point M and the projection center C _b of the reference camera is the line of sight of the reference camera.
Further, in the image captured by the detection camera, the point M is observed at the intersection point m ′ between the straight line connecting the point M and the projection center C _{d of the} detection camera and the projection screen S _d of the detection camera. The straight line connecting the point M and the projection center C _d of the detection camera is the line of sight of the detection camera.

The line of sight of the reference camera is observed as a straight line on the projection screen of the detection camera, and this straight line is called an "epipolar line".

Further, in the examples shown in FIGS. 7 and 8, the image picked up by the reference camera facing the substantially square pattern is square, whereas the image picked up by the detection camera which obliquely looks at this pattern is It appears as a trapezoid as a result of the reduction of the side with a long distance from the viewpoint. Even if an object of the same size, this is projected as a large image as it approaches the projection center C of the camera, and conversely, the projection center C
It relies on the basic property of central projection that it is projected smaller as it moves away from it.

As described above, when the object to be imaged is a plane.
In this case, the image captured by the detection camera is the image captured by the reference camera.
This is a projective transformed image. That is, the image captured by the reference camera
Point m (x in the image_b, Y_b) And the corresponding detection turtle
Point m '(x _d, Y_dBetween the following)
The formula holds. However, H in the equation is a 3 × 3 projective transformation matrix.
Is.

[0009]

[Equation 1]

The projective transformation matrix H is a matrix that implicitly includes the internal parameters and external parameters of the camera, and the equation of the plane, and since the scale factor has a degree of freedom,
It has 8 degrees of freedom. In addition, Kenichi Kanaya's "Image Understanding"
(Morikita Shuppan, 1990) describes that it is possible to find corresponding points between a standard image and a reference image by projective transformation.

The line of sight of the reference camera appears as a straight line on the projection screen of the camera of the detection camera, called the epipolar line (see above and FIG. 7). The point M existing on the line of sight of the reference camera appears on the same observation point m on the projection screen of the reference camera regardless of the depth of the point M, that is, the size of the distance from the reference camera. On the other hand, the observation point m ′ of the point M on the projection screen of the detection camera appears on the epipolar line at a position according to the distance between the reference camera and the observation point M.

FIG. 9 shows the epipolar line and the detection curve.
Illustration of the observation point m'on the Mera projection screen
I understand. As shown in the figure, the position of the point M is M₁，
M₂, M ₃The observation point in the reference image changes as
m '₁, M '₂, M '₃Shift to. In other words, epi
The position on the polar line corresponds to the depth of the observation point M
It is a translation.

By utilizing the above geometrical optical properties, the observation point m'corresponding to the observation point m of the reference camera is searched on the epipolar line to identify the distance from the reference camera to the point P. You can This is the "stereo method"
Is the basic principle of.

A perspective image is generated based on a front image obtained by actually capturing an imaged object, or the distance of an object is measured from a plurality of images by a plurality of cameras based on a stereo method. It is assumed that the optical system has the characteristics that are in perfect agreement with the theory. For this reason, it is necessary to perform a predetermined correction on the image obtained by actual shooting. For example, a camera lens generally has a distortion parameter, and the observation point is imaged at a position displaced from the theoretical point. Therefore, if a camera-specific parameter is calculated and the image data is not corrected according to this parameter in the projective transformation, an accurate projected image cannot be obtained from the front image, and an accurate depth measurement is performed by the stereo method. Can't do.

The parameters of the camera are classified into lens distortion parameters, internal parameters representing camera characteristics, and external parameters representing the three-dimensional position of the camera.
The task of calculating camera parameters is generally
This is called "camera calibration". Although many methods for camera calibration have been proposed in the past, the current situation is that none have been established yet. In general, it is also necessary to use a dedicated device etc. for restraining the data used for calibration,
This is a very complicated process.

Recently, some camera calibration methods using an arbitrary plane without position constraint have been proposed, and the burden on the user performing the calibration operation is being reduced.

For example, a paper by Zhengyou Zhang: "A Flex
ible New Technique for Camera Calibration ", Micros
oft Research Technical Report, 1999 (http://www.re
search.microsoft.com/~zhang/) describes how to calculate the internal parameters of the camera. this
According to Zhang's method, the planes with known patterns are imaged from different directions (that is, the line-of-sight directions are not parallel), and the internal parameters of the camera are calculated using the projective transformation matrix in each plane. It is possible to calculate the camera parameters necessary for distance measurement.

However, even if the object to be imaged for camera calibration is an arbitrary plane as in the method of Zhang, there is actually some constraint condition for the angle at the time of image pickup. The current situation.

Even with the Zhang method, when the three planes are close to parallel, for example, sufficient constraint cannot be obtained for the parameters, and as a result, the parameters cannot be calculated. In order to calculate the parameters stably and with sufficient accuracy, it is considered desirable that the three planes have a certain angle.

Moreover, although it is possible to calibrate the camera by the method of Zhang etc. even under the condition that there is no dedicated experimental equipment, at present, anyone can easily and sufficiently perform the calibration. It is not possible to perform calibration with high accuracy. The reason for this is that, as described above, since it is an ambiguous category of imaging on an arbitrary plane, there is too much freedom on the part of the user side, and work is confusing. As a result, it is difficult to perform calibration with sufficient accuracy without some imaging training.

[0021]

SUMMARY OF THE INVENTION It is an object of the present invention to suitably guide a user when performing a camera calibration operation for a camera of a type that images an object and outputs electronic image data. An object of the present invention is to provide an excellent guide device and guide method for camera calibration that can be performed, and a camera calibration device.

A further object of the present invention is to perform a camera calibration operation of a type in which a parameter is calculated by using a captured image of an arbitrary plane without spatial position constraint in the camera calibration of the stereo method. An object of the present invention is to provide an excellent guide device and guide method that can appropriately guide a user, and a camera calibration device.

A further object of the present invention is camera calibration performed using three image data obtained by imaging a plane on which a known pattern is drawn from an arbitrary direction without position constraint, as in the Zhang method. An object of the present invention is to provide an excellent guide device and guide method, and a camera calibration device that can obtain sufficient accuracy and reduce the burden of the user's calibration operation.

[0024]

The present invention has been made in consideration of the above problems, and the first side surface thereof uses an imaged image of an arbitrary plane without spatial positional constraint. A guide device or a guide method for guiding a user when performing a camera calibration operation of a type for calculating a parameter, wherein a camera and a target plane for obtaining an imaging plane non-parallel to a target plane including a known pattern The guide device or the guide method for camera calibration, characterized in that the guide device or the guide method is provided with instructing means or steps for instructing three or more positional relationships.

The instructing means or step indicates one positional relationship between the object plane and a camera capable of picking up an image of the object plane almost directly from the front side, and sufficiently in front of the object plane, that is, from its normal line. The two positional relationships between the camera capable of capturing an image of the target plane and the target plane from the inclined line-of-sight direction are designated. However, regarding the other two positional relationships, it is possible to obtain an imaging plane that is not too inclined with respect to the target plane so that the amount of information per pixel does not decrease. More specifically,
The line-of-sight directions of the cameras in these positional relationships are inclined about 45 degrees with respect to the normal line of the target plane.

As described above, according to the camera calibration method of Zhang, the calibration is performed using three images of a plane on which a known pattern is drawn, taken from an arbitrary direction without position constraint. You can
However, since it is an ambiguous category of imaging an arbitrary plane, there is too much freedom on the part of the user, and the work is confusing. As a result, it is difficult to perform calibration with sufficient accuracy without some imaging training.

On the other hand, according to the guide device and the guide method for camera calibration according to the first aspect of the present invention, the user is guided by the guide to set the positional relationship between the target plane and the camera. It is possible to easily acquire the image data necessary for performing the camera calibration with sufficient accuracy by sequentially changing and repeating the imaging.

Further, since the Zhang method does not require a strict positional relationship between the plane and the calibration, even if the user does not strictly follow the guide by the guide device or the guide method according to the first aspect of the present invention, it is possible to some extent. If it is installed according to the guide, it is possible to perform calibration with sufficient accuracy.

The instructing means or step may instruct a single installation location for the target plane and a plurality of installation locations for the camera.

Alternatively, the instructing means or step is
A plurality of installation locations for the target plane and a single installation location for the camera may be designated.

The guiding device and the guiding method for camera calibration according to the first aspect of the present invention uses the images taken in the respective positional relationships designated by the pointing means or steps to distort the camera. It is applied to the camera calibration operation that estimates the parameters, calculates each projective transformation matrix that projects each captured image onto a predetermined virtual plane, and calculates the internal parameters of the camera based on the projective transformation matrix of each captured image. can do.

A second aspect of the present invention is a camera calibration device for performing a camera calibration operation of a type in which a parameter is calculated by using a captured image of an arbitrary plane without spatial constraint. And
A calibration jig in which three planes each having a known pattern of C (cyan), M (magenta), and Y (yellow) are arranged on a transparent object at a predetermined angle, and the calibration jig. A camera comprising: a light source that illuminates from behind the jig; and a processing unit that performs arithmetic processing for camera calibration based on an image obtained by capturing the transmitted light of the calibration jig with the camera. It is a calibration device.

Based on the complementary color relationship between RGB and CMY, the processing means can simultaneously extract images for calibration on three planes by extracting colors with sufficiently large G + B, R + B, and R + G from the captured image. it can.

Therefore, according to the camera calibration device of the second aspect of the present invention, the camera position,
Since it is possible to set a certain degree of constraint on the plane position and the image for calibration of three planes can be acquired by one image pickup, the work load on the user is significantly reduced.

Further objects, features and advantages of the present invention are as follows.
It will be clarified by a more detailed description based on embodiments of the present invention described below and the accompanying drawings.

[0036]

BEST MODE FOR CARRYING OUT THE INVENTION The present invention provides a guideline for a calibration method in order to reduce the burden of the user's calibration operation without using a dedicated large-scale facility.

However, the present invention, like the method of Zhang,
It is applied when the calibration of the stereo camera is performed by using three plane images obtained by capturing the plane on which the known pattern is drawn from three different directions.

Before describing the guide device for camera calibration according to the present invention, the camera calibration itself for calculating various parameters of the camera will be described. Below, the points on the captured image are

[0039]

[Equation 2]

And the points in space are

[0041]

[Equation 3]

The description of each point in the linear coordinate system is as follows.

[0043]

[Equation 4]

In such a case, the relationship between the point M in space and the point m on the captured image is given by the following equation.

[0045]

[Equation 5]

Here, s is a scale factor, and the matrix [R, t] is called an extrinsic parameter and represents the position of the camera in space. In addition, R and t respectively represent the rotation and translation row of the image. Further, the matrix A is called an internal parameter of the camera and is given by the formula shown below.

[0047]

[Equation 6]

Here, (u0, v0) represents the center of the image, and α (= − f · ku) and β (= − f · kv / sin).
θ) represents the scale factors of the u-axis and v-axis, respectively, and γ
(= F · ku · cotθ) represents the twist of the two axes.
The matrix P = A · [R, t] is a 3 × 4 projection matrix for projecting points in space onto the image plane.

In the above description, conversion under ideal conditions has been shown without considering the effect of lens distortion. However, in reality, the influence of distortion cannot be ignored, so lens distortion is also added as a camera parameter. Distorted coordinate point md = [ud, vd] ^T and point m =
The relationship of [u, v] ^T can be expressed by the following equation.

[0050]

[Equation 7]

However, rd ² = (ud-cu) ² + (vd-
cv) ² sv ² , where (cu, cv) is the strain center and s
v represents an aspect ratio.

The principle of the distance measuring method in the stereo method is as already described with reference to FIG. That is, the projection matrices to the reference camera and the detection camera are P and P ′, and the points on the reference image and the detection image are m and m ′. However, it is assumed that the influence of the distortion is removed from the points on each image by using the equation (3).

When measuring the distance to the point M, it is necessary to detect the point m'on the detected image corresponding to the point m on the reference image. Since the point m ′ is located on a straight line called an epipolar line as shown in FIG. 9 (described above), in order to detect the point corresponding to the point m on the reference image, the search on this straight line is performed. It should be done.

The epipolar line is the center of the camera and the point m.
It is a set of points that are projected onto the detection camera on the connecting straight line.
(See above). For example, if the measurement range in the Z direction in space is Z₀−
Z_nAnd the epipolar line corresponds to each distance
Point M₀, M_nPoint m ₀', M_nAnd a straight line connecting
It In the actual search, the distance Z_iImage of the point located at
Project on top, measure the similarity with points on the reference image, and
Determine the correspondence of points between.

Then, it is projected to the point m on the reference camera.
Distance Z_iPoint M_iThe point m on the detected image _i'To the procedure to project
explain about. For a point on the line of sight passing through the point m, use the formula (1)
It can be described using the following equation.

[0056]

[Equation 8]

Here, P ⁺ is a pseudo inverse matrix of P and is defined by the following equation.

[0058]

[Equation 9]

Further, P⊥ is a vector that satisfies the following equation, and is always projected to the origin so that it actually indicates the optical center. However, ω is an arbitrary four-dimensional vector.

[0060]

[Equation 10]

Equation (4) represents all points passing through the optical center and the point m on the reference image, but the scale factor can be determined by setting the distance to Z _i, and the point in space can be determined. M _i can be determined. If this point is projected by the projection matrix P ′, the point m _i ′ on the detected image can be calculated.

From the above, in order to obtain the distance between points in space, the camera parameters A, R, and
t, A '. R ′ and t ′ are individually calculated, or the projection matrices P and P ′ are directly calculated, and the distortion parameters κ1, cu1, cv1, sv1 κ2 of each camera are further calculated.
It is sufficient to calculate cu2, cv2 and sv2.

In the present invention, the distortion parameter of the camera is calculated from three images obtained by photographing a predetermined target plane from different line-of-sight directions.

First, the target plane on which a predetermined pattern is drawn is imaged from three different viewpoints. There is also no particular restriction on the pattern of the target plane to be imaged, but in the following, it is assumed that the target plane having a black and white binary lattice pattern formed on the surface is used.

At the time of imaging, the spatial position (or viewpoint) of the plane need not be relatively parallel. However,
It is preferable that one of the three picked-up images is an image picked up from a substantially front surface of the target plane. Regarding other viewpoints, it is not necessary that they are parallel to each other, but as the line-of-sight direction approaches 90 degrees with respect to the normal direction of the target plane, the captured image will be captured from the direction close to the side of the target plane. However, this is not preferable because the amount of information of the target plane that the captured image has per pixel is reduced.

Then, using each captured image of the camera, the distortion parameter of the camera is estimated, and at the same time, the projective transformation matrix to the composite image is calculated.

The composite image referred to here is an image in which the dimensions of a virtual plane arranged in the XYZ coordinate system in parallel with the XY plane and at the position Z = 0 are dropped.
The coordinate system is XY coordinates. As for the distortion parameter estimation method used here, for example, the camera calibration method disclosed in Japanese Patent Application Laid-Open No. 2000-350239 (Japanese Patent Application No. 11-161217) already assigned to the present applicant is extended. Will be realized in.

A point on the composite image is m _o = [X _o , Y _o ] ^T , a point on the picked-up image is m _d = [u _d , v _d ] ^T , and a point on the image from which distortion has been removed is m =. Assuming that [u, v] ^T , the relationship between the points can be represented by the projective transformation matrix H _o and the distortion parameter (κ, cu, cv, sv) as shown in FIG.
That is, the point m _d on the captured image is converted into m using the equation (3), and further converted by the following equation, so that the point m _d is associated with the point m _o on the composite image.

[0069]

[Equation 11]

Based on the relationship shown in the above equation, as shown in FIG. 11, the picked-up images are combined into the same composite image (Im
By performing age registration), parameter estimation of the stereo camera, that is, camera calibration can be performed.

In this case, the parameters to be estimated include the distortion parameter (4 parameters) and the projective transformation matrix H _oi (8 parameters) from each captured image I _i to the composite image I _o , which is a total of 28 parameters. . As an evaluation value,
Each parameter is estimated by minimizing the sum of squares of the combined image I _o and the captured image I _i as shown in the following equation. However, i
= 1, 2, and 3 represent the number of images. By this processing, for example, even if the position where the pattern is imaged is local as shown in FIG. 11, it is possible to relatively stably estimate the distortion parameter.

[0072]

[Equation 12]

For the minimization of the evaluation value, the Levenberg-Marquardt method (hereinafter simply referred to as "LM method"), which is a general method for solving nonlinear problems, can be applied. For example, SA Teukolsky, WT Vetterling, B.
"NUMERICAL RECIPES in C" by P. Flannery (WH Pres
s) describes the LM method.

Here, the procedure of the distortion parameter estimation processing will be summarized.

1: A composite image I _o is created in the computer according to the imaged pattern. 2: For each of the camera images I ₁ , I ₂ , and I ₃ , the evaluation value of the equation (8) is evaluated by the LM method using the relationship between the captured image and the synthetic image represented by the equations (3) and (7). , And distortion parameters κ, cu, cv, sv, and projective transformation matrices H _o1 , H _o2 , H _o3 from each image to a composite image are calculated.

When the relation between the virtual plane used for estimating the distortion parameter and the captured image is rewritten by the above equation (1), the following equation is obtained.

[0077]

[Equation 13]

However, R = [r1, r2, r3]. The following equation can be derived from the above equation (9) and the above equation (7).

[0079]

[Equation 14]

Further, by setting H = [h1 h2 h3], the following equation can be obtained from the above equation (10).

[0081]

[Equation 15]

However, λ is a scale factor. r1 and r
2 is orthonormal, that is, <r1, r2> = 0 and | r1
Using the fact that | = | r2 | = 1, the above equation (1
The following relational expression can be derived from 2).

[0083]

[Equation 16]

However, it is assumed that B = A ^-T A ^-1 . Formula (12)
In, B is a symmetric matrix and is represented by the following equation.

[0085]

[Equation 17]

Here, the column vector b = [B11, B12,
B22, B13, B23, B33] T, and a row of H
Tor h_i= [H_i1, H_i2, H_i3]^TThen, we get
You can However, v_ij= [H_i1h_j1, H_i1h_j2+
h_i2h_j1, H_i2h_j2, H_i3h _j1+ H_i1h_j3, H_i3h_j2+
h_i2h_j3, H_i3h_j3]^TAnd

[0087]

[Equation 18]

By substituting the equation (13) into the equation (12), the following equation regarding the column vector b can be obtained.

[0089]

[Formula 19]

Since the above equation (14) is obtained from one imaging plane (projection transformation matrix), the following equation is finally obtained by creating equation (14) for all the captured images, By solving this, the column vector b can be obtained.

[0091]

[Equation 20]

However, V is as follows.

[0093]

[Equation 21]

If the vector b can be calculated by the above equation (15), each element of the internal parameter can be calculated by the following equation.

[0095]

[Equation 22]

Here, the procedure of the internal parameter estimation processing of the reference camera will be summarized.

1: Inverse matrix H of projective transformation matrix to composite image
Calculate _i = H _oi ^-1 . 2: The relational expression of Expression (14) is calculated from each inverse matrix H _i . 3: The equation (15) obtained using all the matrices H _i is solved, and each element value of the internal parameter is calculated from the equation (17).

Next, the position and orientation of the imaged plane in the space are estimated using the internal parameters of the camera calculated above.

At the time of calculating the distortion parameter, the lattice points of the picked-up image can be extracted by using the inverse matrix of the projective transformation matrix H _o from each image simultaneously calculated to the combined image. That is, by projecting an arbitrary grid point in the composite image (the composite image is created and its position is known) by the matrix H _o ⁻¹ , the grid point on the image from which the distortion is removed is projected. It is possible to calculate m = [u, v] ^T.

As shown in FIG. 12, an image from which distortion has been removed
Grid point m above and point M on the virtual plane _o= [X, Y, O]^T
The relationship is that the virtual plane rotates and moves in space,
By the projection matrix P onto the camera 1, the imaging plane of the reference camera is set.
It will be projected. Rotation in the space of the virtual plane and
If the movement is a matrix [Rw tw], then point m and point M_oSeki
The clerk is expressed by the following equation.

[0101]

[Equation 23]

However, in the above equation (19), the matrix D is as follows.

[0103]

[Equation 24]

By modifying the equation (18), the following equation is derived.

[0105]

[Equation 25]

Since the point M is the spatial coordinate of the point on the imaging plane, the plane position can be determined by estimating the matrix [Rwtw], and the spatial coordinate of the grid point on the imaging plane can be determined. It can be calculated.

The matrix [Rw tw] has a total of 6 degrees of freedom of rotation angles (θ1, θ2, θ3) about each axis and translation (tx, ty, tz) as shown in the following equation (20). is there.

[0108]

[Equation 26]

The projection matrix P can be calculated using the internal parameter A calculated previously and the external parameter whose value is determined according to the measurement range. For example, when the camera position of the reference camera 1 is C _b = [xc, yc, zc] ^T , the projection matrix P can be calculated using the following equation (21).

[0110]

[Equation 27]

The six parameters t shown in the above equation (20)
The estimation of r = [θ1, θ2, θ3, tx, ty, tz] is
It can be performed by the LM method (described above) using the following expression as an evaluation value.

[0112]

[Equation 28]

The above expression (22) represents the sum of the squares of the differences between the grid points m _j on the image picked up by the reference camera 1 and the points projected by the above expression (18). By applying the estimation according to the above equation (22) to each imaging plane, the rotation and movement parameters Rw1, Rw2, Rw3, tw1, tw in the space of the virtual plane.
2. tw3 can be obtained.

Rotation and movement parameters Rw1, Rw2, Rw3, tw1, t in the space of the virtual plane calculated above.
Using w2 and tw3, the points on the detection camera m ′ _i = [u ′
The relationship between _i , v ′ _i ] ^T and the point on the imaging plane M _i = [X _i , Y _i , Z _i ] ^{T obtained} by converting the point on the virtual plane by D _i satisfies the above-mentioned expression (1). . Based on the equation, the following equation can be further obtained.

[0115]

[Equation 29]

However, v _{1i and} v _2i satisfy the following expression (24), and the vector p ′ is the vector of each element of the projection matrix P ′ onto the detection camera 2 as shown in the following expression (25). It is assumed to be represented by.

[0117]

[Equation 30]

[0118]

[Equation 31]

By solving the above equation (23), the projection matrix P'to the detection camera can be obtained.

Finally, the parameters A, Rw1, Rw2, Rw3, tw1, calculated in the above processing steps are
For each of tw2, tw3, P ', the reference camera 1,
By using all the images captured by the detection camera 2,
Perform optimization processing.

The evaluation formula is given by the following formula (26) and can be solved by the LM method (described above). However,
i = 1, 2, 3 are image numbers, and j = 1, 2, ..., N
Represents the score on the image. Also, each point in equation (26) is
Each of the projected points is represented by equation (27).

[0122]

[Equation 32]

However, the matrix [R, t] is an external parameter indicating the camera position of the reference camera 1, and is determined by the measurement range as described above.

As described in the section “Prior Art”, Zhan
In the camera calibration method of g, the calibration can be performed using three images obtained by capturing a plane on which a known pattern is drawn from an arbitrary direction without position constraint. However, even if the present inventors say that there is no position constraint with respect to the imaging directions of the three images (the viewing direction of the camera), it is actually sufficient to consider the following items with sufficient accuracy. I think that the parameters can be estimated.

(1) One image is an image of a target plane of a known pattern taken almost directly in front. (2) The other two images are non-parallel to this. It is preferable to have the line-of-sight direction directly in front of the target plane, that is, sufficiently inclined from the normal line. (3) However, if the line-of-sight directions of the other two images are too inclined with respect to the target plane, the amount of information of the known pattern per pixel decreases, which is not preferable.

As described above, the ambiguous guideline of imaging an arbitrary plane has too much freedom on the user side,
Without some degree of imaging training, it is difficult to perform camera calibration with sufficient accuracy.

Therefore, the inventors of the present invention propose the following guidelines for the calibration method in order to reduce the burden of the user's calibration operation.

A. Method using tripod + guide sheet Fix the camera to be calibrated on the tripod,
Imaging of three planes is performed on the guide sheet. On this guide sheet, the installation positions of the target planes for capturing images of the three planes and the installation positions of the tripod during each image capturing are designated. The target plane indicated on the guide sheet and the respective installation locations of the tripod are provided with the constraint conditions recommended by the present inventors. In such a case, the user can easily perform calibration with sufficient accuracy by capturing images of the three planes along the guide sheet.

FIG. 1 shows the structure of the guide sheet and
It shows how to obtain three imaging planes using a guide sheet.

As shown in the figure, on the guide sheet, there are drawn a position on which a tripod is set, in which the numbers of imaging orders are drawn, and an image on which a target plane having a known pattern is set. From the first place where the tripod is installed, the target plane is imaged almost directly in front.

From the second and third places where the tripod is installed, the object plane can be imaged in front of the object plane, that is, from the line-of-sight direction sufficiently inclined from the normal line. However, it is possible to obtain an imaging plane that is not too inclined with respect to the target plane so that the amount of information per pixel does not decrease. From the place where the second and third tripods are installed, for example, 45 degrees each with respect to the normal line of the target plane.
It is tilted about a degree.

The user may be guided by the guide formed on such a guide sheet to set the target plane, and change the position of the tripod according to the order of image pickup to pick up the image of the target.

In the Zhang method or the like, a strict positional relationship between the plane and the calibration is unnecessary. Therefore,
Even if the user does not strictly install the tripod and the target plane on the guide sheet, if the installation follows the guide specified on the guide sheet to some extent, it is possible to perform calibration with sufficient accuracy. It will be possible.

Of course, each installation place formed on the guide sheet may be changed depending on the distance measurement range, the type of camera, the focal length, and the like.

In the guide sheet shown in FIG. 1, the installation place of the target plane is fixed, and the installation place of the tripod, that is, the camera position is changed every time the image is picked up. On the other hand, even if the position of the camera is fixed and the position of the plane of the imaging target is moved, it is of course possible to obtain the same operational effect as in the case shown in FIG. FIG. 2 shows the configuration of the guide sheet in which the camera position is fixed and the installation location is instructed so as to move the position of the plane of the imaging target.

By performing the camera calibration according to the guide sheet as shown in FIGS. 1 and 2, the positional relationship between the camera and the target plane is limited to some extent by the camera tripod and the guide sheet. In calibration that requires a strict positional relationship between the camera position and the calibration target, it is necessary to pay a lot of attention to how to mount the camera on a tripod and how to place the camera and a flat surface. In the camera calibration method premised on the invention, since the positional relationship between the camera and the imaging plane is basically arbitrary, it is not necessary to pay such attention, and by providing a certain degree of restriction, a sufficiently accurate calibration can be performed. Is possible.

B. Method using color filters Prepare three planes in which a lattice-shaped pattern is created by a C (cyan), M (magenta), and Y (yellow) color filters on a transparent flat transparent object such as an acrylic plate.
Let each plane be plane 1, plane 2, and plane 3, respectively. For example, as shown in FIG. 3, these three planes are arranged so as to have a predetermined angle, and a calibration jig is created.

Then, as shown in FIG. 4, the plane of the calibration jig is illuminated from behind by a light source. When this is taken by a camera, a point P on the image plane
The color at is represented by the following formula.

[0139]

[Expression 33]

However, α, β and γ are coefficients. The RGB values and the CMY values have a complementary color relationship as shown in FIG. Therefore, if a point on the image plane where the value of G + B is sufficiently large is extracted, the calibration pattern of plane 1 as shown on the left side of FIG. 6 can be extracted. Similarly, if a point on the image plane where the value of R + B is sufficiently large is extracted, a calibration pattern of plane 2 as shown in the center of FIG. 6 can be extracted, and an image plane where the value of R + G is sufficiently large is extracted. By extracting the upper points, it is possible to extract the calibration pattern of the plane 2 as shown on the right side of FIG. Therefore, by capturing the transmitted light of the calibration jig only once with the camera you want to calibrate, and extracting sufficiently large colors from the captured image, it is possible to simultaneously extract independent three-plane patterns. That is why.

By using such a calibration jig, it is possible to provide some restraint on the camera position and the plane position. Further, since the images for calibration of the three planes can be acquired at the same time by one image pickup, the burden on the user is significantly reduced.

The pattern formed on the color filter may be any pattern as long as it is a known pattern.
Further, the positional relationship among the plane 1, the plane 2, and the plane 3 may be arbitrary.

[Supplement] The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the scope of the present invention. That is, the present invention has been disclosed in the form of exemplification, and the contents of this specification should not be construed in a limited manner. In order to determine the gist of the present invention, the section of the claims described at the beginning should be taken into consideration.

[0144]

As described above in detail, according to the present invention,
A camera that can suitably guide the user when performing a camera calibration operation for a camera that captures an object and outputs electronic image data.
An excellent guide device and guide method for calibration, and a camera calibration device can be provided.

Further, according to the present invention, in the camera calibration of the stereo method, when performing the camera calibration operation of the type in which the parameters are calculated by using the captured image of an arbitrary plane without spatial positional constraint. It is possible to provide an excellent guide device and guide method that can appropriately guide the user, and a camera calibration device.

Further, according to the present invention, in the camera calibration that is performed by using the three image data of the plane in which the known pattern is drawn from the arbitrary direction without the position constraint like the Zhang method. It is possible to provide an excellent guide device and guide method, and a camera calibration device that can obtain sufficient accuracy and reduce the burden of the user's calibration operation.

According to the guide device and the guide method for camera calibration according to the present invention, any general user can easily, practically and sufficiently accurately perform the calibration that requires a complicated process. It is possible to calibrate. Further, the user can perform the calibration easily and with sufficiently high accuracy without requiring expensive equipment.

Further, according to the guide device and the guide method for camera calibration according to the present invention, the three planes (Zhang's method etc.) necessary for the calibration are provided.
Can be acquired by one-time shooting, and the burden on the user for calibration can be reduced.
Further, since there is no strict restriction on acquisition of the imaging plane, it is possible to acquire an image required for calibration without paying special attention.

[Brief description of drawings]

FIG. 1 is a diagram showing a configuration of a guide sheet and a state in which three image pickup planes are obtained using the guide sheet.

FIG. 2 is a diagram showing a configuration of a guide sheet in which a camera position is fixed and a position of a plane of an imaging target is moved.

FIG. 3 is a calibration jig configured by arranging three planes having a grid pattern formed by C (cyan), M (magenta), and Y (yellow) color filters so as to have a predetermined angle. It is the figure which showed.

FIG. 4 is a diagram showing how the calibration jig shown in FIG. 3 is illuminated by a light source installed at the rear.

FIG. 5 is a diagram showing a complementary color relationship between RGB values and CMY values.

FIG. 6 is a diagram showing each plane pattern extracted by one shooting by extracting a color in which G + B, R + B, and R + G are sufficiently large.

FIG. 7 is a diagram schematically showing an arrangement of a reference camera and a detection camera with respect to an imaging target.

FIG. 8 is a diagram showing an image obtained by capturing a substantially square pattern by each of a reference camera and a detection camera.

FIG. 9 is a diagram illustrating a state of an epipolar line and an observation point m ′ in a reference image.

[10] point _{m o = [X o, Y} o] on the composite image ^T, the point _{m d = [u d, v} d] of the captured image ^T, and a point on the image to remove distortions m = It is a figure showing the relation of [u, v] ^T.

FIG. 11 is a diagram depicting a state in which captured images are subjected to the same combined image registration (Image Registration).

FIG. 12 is a diagram depicting a relationship between a lattice point m on an image from which distortion is removed and a point M _o = [X, Y, O] ^T on a virtual plane.

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Kaguya             1-14-10 Higashi Gotanda, Shinagawa-ku, Tokyo Stock             Ceremony company Sony Kihara Laboratory F term (reference) 2F065 AA04 AA20 AA31 BB02 BB05                       BB27 EE05 EE08 FF05 FF63                       JJ03 JJ05 JJ26 LL22 QQ18                       QQ31 UU09                 5C061 AB06 AB08                 5L096 AA02 AA06 BA08 CA04 CA17                       DA05 FA15 GA38 GA40

Claims (15)

[Claims] 1. A guide device for guiding a user when performing a camera calibration operation of a type in which a parameter is calculated using a captured image of an arbitrary plane that is not spatially constrained, and includes a known pattern. A guide device for camera calibration, comprising: an instruction means for instructing three or more positional relationships between a camera and an object plane for obtaining an imaging plane that is non-parallel to the object plane. 2. The instructing means indicates one positional relationship between a camera capable of picking up an image of a target plane almost directly from the front and a target plane, and from a line-of-sight direction sufficiently inclined from a normal line of the target plane. The guide device for camera calibration according to claim 1, wherein the guide device indicates two positional relationships between a camera capable of capturing an image of the target plane and the target plane. 3. The pointing means is capable of picking up an image of the target plane from a line-of-sight direction that is sufficiently inclined from the normal line of the target plane, but the camera and the target plane do not reduce the amount of information per pixel. The guide device for camera calibration according to claim 2, wherein the guide device indicates one positional relationship. 4. The guide device for camera calibration according to claim 1, wherein the instructing means indicates a single installation location on the target plane and a plurality of installation locations on the camera. . 5. The guide device for camera calibration according to claim 1, wherein the instructing means indicates a plurality of installation locations for the target plane and a single installation location for the camera. . 6. A distortion parameter of the camera is estimated using images captured in each positional relationship designated by the pointing means, and each projective transformation matrix for projecting each captured image onto a predetermined virtual plane is calculated. The guide for camera calibration according to claim 1, wherein the guide is applicable to a camera calibration operation for calculating internal parameters of a camera based on a projective transformation matrix of each captured image. apparatus. 7. A guide method for guiding a user when performing a camera calibration operation of a type in which a parameter is calculated using a captured image of an arbitrary plane without spatial positional constraint, and includes a known pattern. A guide method for camera calibration, comprising an instructing step of instructing three or more positional relationships between a camera and an object plane for obtaining an imaging plane that is non-parallel to the object plane. 8. In the instructing step, one of a camera capable of picking up an image of a target plane from almost directly in front and a target plane
The two positional relations, and also the two positional relations between the camera and the target plane that can image the target plane from the line-of-sight direction that is sufficiently inclined from the normal line of the target plane.
The guiding method for camera calibration according to claim 7, wherein. 9. In the instructing step, the target plane can be imaged from a line-of-sight direction that is sufficiently inclined from the normal line of the target plane, but the camera and the target plane do not reduce the amount of information per pixel. 9. The guiding method for camera calibration according to claim 8, wherein one of two positional relationships is designated. 10. The camera according to claim 7, wherein in the instructing step, a single installation location for the target plane and a plurality of installation locations for the camera are instructed.
Guide method for calibration. 11. The camera according to claim 7, wherein in the instructing step, a plurality of installation locations for the target plane and a single installation location for the camera are instructed.
Guide device for calibration. 12. A distortion parameter of a camera is estimated using images captured in each positional relationship designated by the instructing step, and each projective transformation matrix for projecting each captured image onto a predetermined virtual plane is calculated. The guide for camera calibration according to claim 7, wherein the guide is applicable to a camera calibration operation for calculating an internal parameter of a camera based on a projective transformation matrix of each captured image. Method. 13. A camera of a type in which a parameter is calculated by using a captured image of an arbitrary plane without spatial positional constraint.
A camera calibration device for performing a calibration operation, in which a known pattern on a transparent object is formed by three color filters of C (cyan), M (magenta), and Y (yellow) on a three-plane surface at a predetermined angle. A calibration jig arranged with a light source that illuminates the calibration jig from behind, and calculation processing for camera calibration based on an image obtained by capturing the transmitted light of the calibration jig with a camera. A camera calibration device, comprising: 14. The processing means simultaneously acquires images for calibration of three planes by extracting from the captured image a color in which G + B, R + B, and R + G are sufficiently large based on the complementary color relationship between RGB and CMY. The camera calibration device according to claim 13, wherein: 15. A distortion parameter of a camera is estimated using an image for calibration, each projective transformation matrix that projects each captured image onto a predetermined virtual plane is calculated, and based on the projective transformation matrix of each captured image. The camera according to claim 13, which is applicable to a camera calibration operation for calculating an internal parameter of the camera.
Calibration device.