JP2006098065A - Calibration device and method, and three-dimensional modelling device and system capable of using the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a calibration device and method allowing freedom for a photographing position, and a three-dimensional modeling device and system capable of using the same. <P>SOLUTION: A positional relation between a rotary table 10 and a three-dimensional coordinate measuring articulated arm system 40 with a camera is specified by a transformation matrix M<SB>1</SB>from a rotary table coordinate system to an arm reference coordinate system, and a positional relation between a camera installation shaft 20d and a position of the camera 30 is specified by a transformation matrix M<SB>2</SB>from a camera installation coordinate system to a camera coordinate system. The positional relations are calibrated in advance. A transformation matrix M<SB>a</SB>from the arm reference coordinate system for specifying a positional relation between an arm reference joint 18a and a camera installation joint 18c to the camera installation coordinate system is obtained from an arm output data of the three-dimensional coordinate measuring articulated arm system 40. A rotary table-camera transformation matrix M<SB>0</SB>in an optional arm position is calculated pursuant to the calculation expression M<SB>0</SB>=M<SB>2</SB>×M<SB>a</SB>×M<SB>1</SB>. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a three-dimensional modeling technique, and more particularly to a three-dimensional modeling apparatus and a three-dimensional modeling system that generate a three-dimensional shape from an image obtained by photographing an object, and a calibration apparatus and method that calibrate the three-dimensional modeling apparatus.

  Computer graphics is used in various fields, and there are business applications that require high-precision 3D graphics images such as 3D CAD. Also in end-user applications such as computer games, there is a high demand for quality of 3D graphics images from users, and there is a need to express more complicated 3D shapes on the screen. In order to draw a complex and fine 3D graphics image, it is necessary to model an actual 3D object and input the 3D model data. However, the amount of data is enormous and the efficiency of the initial input is high. Is desired. Therefore, a three-dimensional scanner that captures a target object and automatically acquires three-dimensional shape data is used.

For example, in Patent Document 1, a target object placed on a turntable provided with a position detection mark is shot with a camera together with the turntable, and shooting position information is obtained from position detection marks in each image of the plurality of shot images. Also disclosed is a three-dimensional modeling apparatus that obtains a silhouette image from the photographed image and generates a three-dimensional model.
JP 2001-108421 A

  However, the position of the camera is fixed in the three-dimensional modeling apparatus of Patent Document 1, and it is not suitable for acquiring a three-dimensional shape by photographing a target object having a complicated shape from various angles. In order to increase the degree of freedom of the camera position, it is conceivable to attach the camera to the articulated arm for 3D coordinate measurement, but when the position of the camera is moved by the articulated arm for 3D coordinate measurement, the camera is Accurate position information is required. In the three-dimensional modeling, since the three-dimensional shape is reproduced from the captured image, the reproduction accuracy of the three-dimensional shape is deteriorated due to a slight error. In a 3D modeling system using an articulated arm for 3D coordinate measurement with a camera, it is indispensable to calibrate the system and accurately and efficiently calculate the position information of the camera.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a three-dimensional modeling technique capable of giving a degree of freedom to a photographing position. Another object is to provide a calibration technique for a three-dimensional modeling apparatus in which a shooting position is given a degree of freedom.

  In order to solve the above problems, a calibration apparatus according to an aspect of the present invention is an apparatus that calibrates a three-dimensional modeling apparatus including a multi-joint arm for camera-equipped three-dimensional coordinate measurement and a platform on which an object is placed. The measurement point provided on the calibration panel installed on the table is measured by the articulated arm for three-dimensional coordinate measurement with the camera, based on the coordinate value of the measurement point, And an arm installation position calibration unit that calibrates the positional relationship of the articulated arm for 3D coordinate measurement with the camera, and a reference mark provided on the calibration panel installed on the table is photographed by the camera. Based on the acquired coordinate value of the reference mark, the photographing position calculation unit for calculating the position information of the camera with respect to the table at the time of photographing, and the calibration panel An arm position calculation unit for calculating position information of a three-dimensional coordinate measurement articulated arm when the reference mark is photographed by the camera; and a position of the table and the camera-equipped three-dimensional coordinate measurement articulated arm. The camera-installed joint of the articulated arm for 3D coordinate measurement with the camera using the calibration result of the relationship, the positional information of the camera with respect to the table at the time of shooting, and the positional information of the articulated arm for 3D coordinate measurement And a camera installation position calibration unit that calibrates the positional relationship of the cameras.

  According to this aspect, it is possible to efficiently perform the calibration of the three-dimensional modeling apparatus including the articulated arm for three-dimensional coordinate measurement with a camera having a high degree of freedom of the camera position and the platform for placing the object.

  Using the calibration result of the positional relationship between the table and the articulated arm for three-dimensional coordinate measurement with the camera and the calibration result of the positional relationship between the camera-installed joint and the camera, the camera with respect to the table at an arbitrary arm position It may further include a camera position calculation unit for calculating the position information. This makes it possible to obtain an accurate camera position when shooting an object at an arbitrary arm position.

  The platform may be a turntable on which an object can be placed and rotated. By using the turntable, the degree of freedom of the camera position is further increased. Based on the coordinate value of the measurement point obtained by measuring the measurement point on the calibration panel with the articulated arm for three-dimensional coordinate measurement with the camera while varying the rotation angle of the turntable, the calibration It may further include a calibration panel correction unit that corrects a deviation of the panel for the rotary table. The “deviation of the calibration panel with respect to the turntable” includes a positional deviation of the calibration panel from the rotation center of the turntable, an inclination of the turntable with respect to the rotation plane, and the like.

  A slit light extraction unit for extracting slit light projected on an object placed on the table from an image photographed by the camera, and a plane equation created by the slit light irradiated with the coordinate position of the projected slit light. A slit light position calibration unit that calibrates the projection position of the slit light with respect to the camera by fitting may be further included. In the case where the slit light projection method is used for obtaining the three-dimensional shape data of the object, the slit light position can be calibrated to obtain the three-dimensional shape data with high accuracy.

  The plurality of reference marks provided on the calibration panel may be randomly arranged. Due to the randomness of the arrangement of the reference marks, it is possible to avoid a pattern composed of an arbitrary number of reference marks from taking the same shape when photographed from different angles, and to eliminate indefiniteness. The plurality of reference marks provided on the calibration panel may be randomly classified into a plurality of types. Even if a plurality of reference marks provided on the calibration panel are arranged with some regularity, for example, symmetrically with respect to the center of the table, the reference marks are randomly classified into a plurality of types. Due to the difference, it is possible to eliminate indefiniteness due to the shooting direction. In addition, the shooting position calculation unit determines the correspondence between the reference mark extracted from the image shot by the camera and the reference mark of the calibration panel based on the type of the reference mark, and processes a meaningless combination as a processing target. Can be removed.

  The plurality of reference marks provided on the calibration panel may be classified into a plurality of types, and the measurement points may be provided in association with different types of the reference marks. For example, the reference mark may be classified by color or shape, and the measurement point may be provided in the reference mark. Thereby, the measurement point can be specified by the type such as the color and shape of the reference mark, and the order of measuring the plurality of measurement points can be specified by the type of the reference mark.

  Another aspect of the present invention is a three-dimensional modeling apparatus. The three-dimensional modeling apparatus includes a multi-joint arm for measuring a three-dimensional coordinate on which a camera is installed, a table for placing an object photographed by the camera, and a calibration panel installed on the table. The calibration panel includes a plurality of measurement points measured by the articulated arm for 3D coordinate measurement in order to calibrate a positional relationship between the table and the articulated arm for 3D coordinate measurement, In order to calculate the position information of the camera, a plurality of reference marks photographed by the camera are included.

  According to this aspect, the degree of freedom in setting the position of the camera is increased, and fine three-dimensional modeling is possible. Further, using the calibration panel, it is possible to efficiently calibrate the positional relationship between the camera, the multi-joint arm for measuring the three-dimensional coordinates, and the table on which the object is placed.

  Another aspect of the present invention is a three-dimensional modeling system. This three-dimensional modeling system includes a three-dimensional modeling device including a multi-joint arm for camera-equipped three-dimensional coordinate measurement and a platform for placing an object, and a calibration device for calibrating the three-dimensional modeling device. The calibration device is based on the coordinate value of the measurement point obtained by measuring the measurement point provided on the calibration panel installed on the table by the articulated arm for three-dimensional coordinate measurement with the camera. , An arm installation position calibration unit that calibrates the positional relationship between the table and the articulated arm for three-dimensional coordinate measurement with the camera, and a reference mark provided on the calibration panel installed on the table by the camera. Based on the coordinate value of the reference mark obtained by photographing, a photographing position calculating unit for calculating positional information of the camera with respect to the table at the time of photographing, and the reference mark provided on the calibration panel as the camera An arm position calculation unit for calculating position information of a multi-joint arm for three-dimensional coordinate measurement at the time of photographing, and the table and the multi-joint for three-dimensional coordinate measurement with camera The three-dimensional coordinate measuring articulated arm with the camera using the calibration result of the positional relationship of the camera, the positional information of the camera with respect to the table at the time of photographing, and the positional information of the articulated arm for measuring the three-dimensional coordinate And a camera installation position calibration unit that calibrates the positional relationship of the cameras.

  Yet another embodiment of the present invention is a calibration method. This method is a method for calibrating a three-dimensional modeling apparatus including an articulated arm for three-dimensional coordinate measurement with a camera and a table on which an object is placed, and is provided on a calibration panel installed on the table. Based on the coordinate value of the measurement point obtained by measuring the measured point by the articulated arm for 3D coordinate measurement with camera, the positional relationship between the table and the articulated arm for 3D coordinate measurement with camera Calibrating, and based on the coordinate value of the reference mark obtained by photographing the reference mark provided on the calibration panel installed on the table by the camera, Calculating the position information of the camera; and a multi-function for measuring three-dimensional coordinates when the reference mark provided on the calibration panel is photographed by the camera. Calculating the position information of the arm, the calibration result of the positional relationship between the table and the articulated arm for three-dimensional coordinate measurement with the camera, the position information of the camera with respect to the table at the time of shooting, and the three-dimensional coordinate measurement And using the position information of the articulated arm to calibrate the positional relationship between the camera-installed joint of the camera-equipped articulated arm for three-dimensional coordinate measurement and the camera.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, and the like are also effective as an aspect of the present invention.

  According to the present invention, it is possible to acquire a three-dimensional shape efficiently and with high accuracy by photographing a target object while freely changing the photographing position. Further, the calibration of the three-dimensional modeling apparatus having a degree of freedom in the photographing position can be performed efficiently and accurately.

  FIG. 1 is a configuration diagram of a three-dimensional modeling system 300 according to the embodiment. The 3D modeling system 300 includes a 3D coordinate processing articulated arm system 40 with a camera, a turntable 10, and a 3D data processing apparatus 100 connected to the 3D coordinate measuring articulated arm system 40 with a camera.

  The multi-joint arm system 40 with a camera for three-dimensional coordinate measurement is configured by installing the camera 30 on multi-joint arms 20a to 20d having five rotation axes. The first arm 20 a installed on the support base 12 can rotate in the first rotation direction 21. The second arm 20b is connected to the first arm 20a via the first joint 18a, and is rotatable in the second rotation direction 22 around the first joint 18a. The first joint 18a is also referred to as an arm reference joint. The third arm 20c is connected to the second arm 20b via the second joint 18b, and is in a third rotation direction 23 around the second joint 18b and a fourth rotation direction 24 around the arm axis. It can be rotated.

  The third joint 18c at the tip of the third arm 20c is provided with a camera installation shaft 20d, and the camera installation shaft 20d is rotatable in a fifth rotation direction 25 around the axis. The third joint 18c is also referred to as a camera installation joint. The camera installation shaft 20d is provided with a camera 30, a line laser 36, and a measurement probe 32.

  The turntable 10 mounts a target object for three-dimensional modeling, and can rotate in a rotation direction indicated by reference numeral 11 around a rotation axis.

  The three-dimensional data processing apparatus 100 is photographed by the camera 30 using arm output data such as rotation angle data of each joint, normal vector of each axis, and position vector data of the multi-joint arm system 40 for measuring three-dimensional coordinates with a camera. The received image data and measurement data obtained by the measurement probe 32 are received from the multi-joint arm system 40 for 3D coordinate measurement with a camera, and the system is calibrated and 3D model data is generated.

  FIG. 2 is a detailed configuration diagram of the camera installation axis 20d. The camera installation shaft 20 d can rotate around the rotation shaft 26. An attachment portion 37 is provided on the camera installation shaft 20 d, and the camera 30 and the line laser 36 are fixed to the attachment portion 37. The camera 30 and the line laser 36 are connected to each control unit via a connection cable (not shown). Further, the camera installation axis 20d is provided with a measurement probe 32, and the position information of the measurement target point can be acquired by applying the tip of the measurement probe 32 to the surface of the target object.

  FIG. 3 is a diagram for explaining a calibration panel provided on the turntable 10. The top plate of the turntable 10 may also serve as a calibration panel, and the calibration panel may be placed on the turntable 10 during calibration. The calibration panel includes a reference pattern including a plurality of measurement points 15 for measurement by the measurement probe 32 of the articulated arm system 40 for three-dimensional coordinate measurement with a camera and a plurality of reference marks 16 for imaging by the camera 30. Is provided. The measurement point 15 is a hole into which the tip of the measurement probe 32 is inserted, and the plurality of reference marks 16 are circle marks randomly colored with several colors such as blue, red, yellow, and green.

  Since the plurality of reference marks 16 on the calibration panel are randomly colored with several kinds of colors, when the camera position is calibrated based on the reference marks 16 extracted from the photographed image by voting processing described later, Since the reference mark 16 extracted from the photographed image can easily be associated with the reference mark 16 on the calibration panel and the number of combinations is reduced, the calculation time can be shortened.

  In addition, since the plurality of reference marks 16 on the calibration panel are randomly colored with several kinds of colors, even when images are taken from different angles, a pattern including any number of the plurality of reference marks 16 is a combination of colors. Therefore, it is possible to avoid the same pattern and to eliminate indefiniteness.

  In the calibration panel of FIG. 3, for ease of design, a plurality of reference marks 16 are regularly arranged and symmetrically arranged with respect to the center of the calibration panel. However, this arrangement is randomized. Also good. In that case, even if the reference mark 16 is the same color, the above-described indefiniteness can be excluded from the randomness of the arrangement of the reference mark 16. This is because the arrangement of the reference marks 16 of the same color is random.

  The measurement point 15 is provided in the reference mark 16. This is because the measurement point 15 to be measured by the measurement probe 32 is different by changing the color of the reference mark 16 provided with the measurement point 15. This is because the color can be specified by color. For example, the measurement order of the plurality of measurement points 15 can be designated to the user by the color of the reference mark 16 such as red, blue, and yellow. If the measurement point 15 is measured in the color order designated by the user, it is possible to uniquely identify the measurement point 15 at which the measurement point 15 measured by the user is located on the calibration panel. .

  FIG. 4 is a diagram for explaining how the 3D modeling system 300 according to the present embodiment acquires 3D model data. The target object 14 is placed on the turntable 10, the multi-joint arm of the multi-joint arm system 40 for 3D coordinate measurement with camera is manually operated, and the camera 30 is moved to a position where the target object 14 can be photographed. The line laser 36 irradiates the target object 14 on the turntable 10 with slit light, and the camera 30 images the laser line 38 projected onto the target object 14. By rotating the turntable 10 or moving the position of the camera 30, the laser line 38 projected onto the target object 14 from various angles is photographed. The three-dimensional data processing apparatus 100 acquires the three-dimensional shape data of the target object 14 by the slit light projection method (also referred to as a light cutting method) using the captured image of the projected laser line 38, and 3 of the target object 14. Perform dimensional modeling.

  FIG. 5 is a diagram for explaining a coordinate system set on the turntable 10. An orthogonal coordinate system x0-y0-z0 is defined in which the surface of the turntable 10 is the xz plane and the rotation axis of the turntable 10 is the y axis. This is called a turntable coordinate system.

  FIG. 6 is a diagram illustrating a coordinate system set for the arm reference joint 18a of the multi-joint arm system 40 for 3D coordinate measurement with a camera. An orthogonal coordinate system x1-y1-z1 is defined at the arm reference joint 18a connecting the first arm 20a and the second arm 20b with the rotation axis of the first arm 20a as the z-axis. This is called an arm reference coordinate system.

  FIG. 7 is a diagram illustrating a coordinate system set for the camera installation joint 18 c and a coordinate system set for the camera 30. The camera installation axis 20d is an axis whose positional relationship with the camera 30 is invariable among the joint axes of the arm. An orthogonal coordinate system x2-y2-z2 is defined with the camera installation joint 18c as the origin and the camera installation axis 20d as the y axis. This is called a camera installation axis coordinate system. Further, an orthogonal coordinate system x3-y3-z3 is defined in which the focal point of the camera 30 is the origin, the focal plane 34 is the xy plane, and the optical axis is the z-axis. This is called a camera coordinate system.

As shown in FIGS. 5 to 7, the three-dimensional modeling system 300 has four coordinate systems of a rotary base coordinate system, an arm reference coordinate system, a camera installation axis coordinate system, and a camera coordinate system, and these different coordinate systems. Coordinate conversion is required between In general, the coordinate conversion is performed when the coordinate value in the conversion source coordinate system is X = (x, y, z) ^t and the coordinate value in the conversion destination coordinate system is X ′ = (x ′, y ′, z ′) ^t. It is expressed by the coordinate conversion formula.

X ′ = M _r · X + t
Here, the matrix _Mr is a 3-by-3 matrix that gives the rotation component of the transformation source coordinate system, and the vector t is a vector that gives the translation component of the transformation source coordinate system.

The first row of the matrix _{M r} that gives a rotational component, the second row, third row, respectively _{M r1 = (m 11, m} 12, m 13), M r2 = (m 21, m 22, m 23), M _{If r3} = (m ₃₁ , m ₃₂ , m ₃₃ ) and the vector t giving the translation component is t = (m ₁₄ , m ₂₄ , m ₃₄ ), the above coordinate conversion formula is
x ′ = m ₁₁ x + m ₁₂ y + m ₁₃ z + m ₁₄
y ′ = m ₂₁ x + m ₂₂ y + m ₂₃ z + m ₂₄
z ′ = m ₃₁ x + m ₃₂ y + m ₃₃ z + m ₃₄
Can be written.

Each row of the 3 × 4 coordinate transformation matrix M is expressed as M ₁ = (m ₁₁ , m ₁₂ , m ₁₃ , m ₁₄ ), M ₂ = (m ₂₁ , m ₂₂ , m ₂₃ , m ₂₄ ), M ₃ = ( m ₃₁ , m ₃₂ , m ₃₃ , m ₃₄ ) and defining a vector X ⁺ = (x, y, z, 1) ^t obtained by adding 1 as the fourth component to the coordinate value X in the transformation source coordinate system The coordinate conversion formula is
X ′ = M · X ⁺
Can be written. Thus, the positional relationship between different coordinate systems can be uniquely determined by the 3 × 4 conversion matrix M.

  Although the calculation of the product of the coordinate transformation matrices of 3 rows and 4 columns and the calculation of the inverse matrix of the coordinate transformation matrix of 3 rows and 4 columns cannot be originally performed, the following description is given for convenience in description. is doing. This means calculating as follows.

For the calculation of the product of a 3 × 4 matrix, M ₀ = M ₁ · M ₂ is
M ₀ · X ⁺ = M ₁ · M ₂ · X ⁺
M _r0 · X + t ₀ = M _r1 · (M _r2 · X + t ₂ ) + t ₁
In this way, the calculation is divided into a rotation component of 3 rows and 3 columns and a translation component of 1 column,
M _r0 = M _r1 · M _r2
t ₀ = M _r1 · t ₂ + t ₁
And

The same applies to the calculation of the inverse matrix.
X ′ = M · X ⁺
X ′ = M _r · X + t
X = M _r ⁻¹ · (X′−t)
= M _r ⁻¹ · X′−M _r ⁻¹ · t
As described above, the rotation component and the translation component are divided into the rotation component M _r ⁻¹ and the translation component t ⁻¹ of the inverse matrix M ⁻¹ of the M, respectively. M _r ⁻¹ = M _r ⁻¹
t ⁻¹ = −M _r ⁻¹ · t
And calculate.

Each component m _ij of the matrix M _r of 3 rows and 3 columns that gives the rotation component can also be written as follows using Euler angles α, β, and γ, so the number of parameters of the rotation component is substantially Three.

m ₁₁ = cos α cos β cos γ-sin α sin γ
m ₁₂ = −cos α cos β sin γ-sin α cos γ
m ₁₃ = cos α sin β
m ₂₁ = sin α cos β cos γ + cos α sin γ
m ₂₂ = −sin α cos β sin γ + cos α cos γ
m ₂₃ = sin αsin β
m ₃₁ = −sin βcos γ
m ₃₂ = sin αsin γ
m ₃₃ = cos β

  If a vector t giving a translation component is set to t = (tx, ty, tz), the Euler angles α, β, γ and the six parameters of the translation components tx, ty, tz are coordinates defining the coordinate transformation matrix M. The number of unknown parameters when obtaining the coordinate transformation matrix M is six.

FIG. 8 is a diagram for explaining the calculation principle of the calibration of the articulated arm system 40 with a camera for three-dimensional coordinate measurement and the turntable 10. In order to acquire 3D data by the articulated arm system 40 for 3D coordinate measurement with a camera, accurate position information of the camera 30 with respect to the turntable 10 is necessary. In order to grasp the position of the camera 30 with respect to the turntable 10 when the articulated arm system 40 for 3D coordinate measurement with a camera is operated to move the camera 30 to an appropriate position, from the turntable coordinate system A transformation matrix M ₀ (referred to as a turntable-camera coordinate transformation matrix) to the camera coordinate system is required.

Turntable - for obtaining the camera coordinate transformation matrices M ₀ exactly, the calibration of the positional relationship of the turntable 10 and the camera three-dimensional coordinate measuring articulated arm system 40, and the position of the camera installation shaft 20d and the camera 30 Relationship calibration is required.

The positional relationship between the turntable 10 and the articulated arm system 40 for 3D coordinate measurement with a camera is a conversion matrix M ₁ from the turntable coordinate system to the arm reference coordinate system (this is called a turntable-arm reference joint coordinate conversion matrix). The positional relationship between the camera installation axis 20d and the camera 30 is defined by a conversion matrix M ₂ from the camera installation coordinate system to the camera coordinate system (this is called a camera installation joint-camera coordinate conversion matrix). When the calibration results of these positional relationships are given, a transformation matrix M _a from the arm reference coordinate system that defines the positional relationship between the arm reference joint 18a and the camera installation joint 18c to the camera installation axis coordinate system (this is referred to as an arm reference joint). (Referred to as “camera-installed joint coordinate transformation matrix”) is obtained from the arm output data of the articulated arm system 40 for three-dimensional coordinate measurement with a camera, and therefore the turntable-camera coordinate transformation matrix M ₀ can be calculated by it can.

M ₀ = M ₂ · M _a · M ₁
Here, each of the coordinate transformation matrices M ₀ , M ₁ , M ₂ , and Ma is _a 3 × 4 matrix.

Thus, turntable - the arm reference joint coordinate transformation matrix M _1, the camera installation joints - if in calibrating the camera coordinate transformation matrix M ₂ in advance, using the arm output data, rotation in any arm position turntable defining a positional relationship between the base coordinate system and the camera coordinate system - can be calculated camera coordinate transformation matrix M _0.

When the coordinate values of the rotary table coordinate system X0 = (x0, y0, z0 ) to the coordinate value X3 = (x3, y3, z3 ) in the camera coordinate system, turntable - following the camera coordinate transformation matrices _{M 0} Can be requested.

X3 = M ₀ · X0 ⁺
Here, X0 ⁺ = (x0, y0, z0, 1).

FIGS. 9A to 9C are diagrams for explaining the calibration procedure of the articulated arm system 40 with a camera for three-dimensional coordinate measurement and the turntable 10. As shown in FIG. 9A, the position of the rotary base 10 and the articulated arm system 40 for 3D coordinate measurement with a camera is determined, and the measurement points on the calibration panel on the rotary base 10 are measured by the measurement probe 32. turntable - Request arm reference joint coordinate transformation matrix M _1.

Next, as shown in FIG. 9B, while the positions of the rotary base 10 and the articulated arm system 40 for 3D coordinate measurement with a camera are kept in the same state as in FIG. The reference pattern of the upper calibration panel is photographed to obtain a turntable-camera coordinate transformation matrix M ₀ ′ at the time of calibration panel photography. At this time, it obtains an arm output data at the time of calibration panels shooting arm reference joint - previously obtained camera installed joint coordinate transformation matrix M _a. Later turntable obtained in FIG. 9 (a) - to use the arm reference joint coordinate transformation matrix M _1, at the time of calibration panels photographing, the rotary table 10 and the camera three-dimensional coordinate measuring articulated arm system 40 positional relationship turntable in FIG 9 (a) - must be the same when calculated arms reference joint coordinate transformation matrix M _1, the position of the camera 30 is arbitrary.

All the data necessary for calibration are obtained by the two procedures of FIG. 9A and FIG. 9B. Finally, as shown in FIG. 9C, the turntable-arm reference joint coordinate transformation matrix M ₁ acquired in FIG. 9A and the turntable at the time of photographing the calibration panel acquired in FIG. 9B. - camera coordinate transformation matrix M _{0 'arm} reference joint - using a camera installed joint coordinate transformation matrix M _a, by the following equation, camera installation joints - calculates the camera coordinate transformation matrix M _2.

M ₂ = M ₀ ′ · M ₁ ⁻¹ · M _a ⁻¹
Here, the arm reference joint - camera installation joint coordinate transformation matrix M _a are those obtained from the arm the output data when the photographed calibration panel in Figure 9 (b).

  FIG. 10 is a configuration diagram of the three-dimensional data processing apparatus 100. These configurations can be realized in hardware by any computer's CPU, memory, or other LSI, and in software, by a program with a 3D modeling function and an image processing function loaded in the memory. However, here, functional blocks realized by their cooperation are depicted. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof.

  The captured image input unit 160 acquires a calibration panel image captured by the camera 30 of the articulated arm system for 3D coordinate measurement with camera 40 and an image of the target object irradiated with the slit light, and the calibration panel The image is given to the photographing position calculation unit 120, and the image of the target object irradiated with the slit light is given to the slit light extraction unit 200.

  The arm data input unit 170 acquires arm output data of the articulated arm system 40 for 3D coordinate measurement with a camera and measurement data by the measurement probe 32, an arm installation position calibration unit 110, an arm position calculation unit 180, and calibration. This is given to the panel correction unit 190.

Arm position calculation unit 180, based on the arm output data, arm reference joints in the current arm position - calculates the camera installation joint coordinate transformation matrix M _a, camera installation position calibration unit 130 and the camera position calculated as the arm position information Part 140.

The arm installation position calibration unit 110 uses the measurement results of the plurality of measurement points 15 of the calibration panel on the turntable 10 by the measurement probe 32 of the articulated arm system 40 for 3D coordinate measurement with a camera. 10 and the positional relationship between the camera-equipped articulated arm system 40 for three-dimensional coordinate measurement are calibrated. The arm installation position calibration unit 110 calculates a coordinate conversion parameter based on the known coordinate value of the measurement point 15 on the calibration panel in the rotary base coordinate system and the coordinate value in the arm reference coordinate system measured by the measurement probe 32. by obtaining, turntable - calculating the arm reference joint coordinate transformation matrix M _1.

Arm installation position calibration unit 110, a position calibration result of the relationship between the turntable 10 and the camera three-dimensional coordinate measuring articulated arm system 40, the turntable calculated - camera installation arm reference joint coordinate transformation matrix M ₁ The data is given to the position calibration unit 130, the camera position calculation unit 140, and the calibration panel correction unit 190.

  The photographing position calculation unit 120 uses an image obtained by photographing a reference pattern composed of a plurality of reference marks 16 on the calibration panel on the turntable 10 by the camera 30 of the articulated arm system 40 for 3D coordinate measurement with a camera. The positional relationship between the turntable 10 and the camera 30 during calibration panel shooting is obtained. The photographing position calculation unit 120 extracts all the reference marks 16 from the photographed image of the calibration panel by image processing.

The photographing position calculation unit 120 obtains a coordinate conversion parameter based on the known coordinate value of the reference mark 16 of the calibration panel in the rotary base coordinate system and the coordinate value of the reference mark 16 extracted from the photographed image in the photographed image. Thus, a turntable-camera coordinate conversion matrix M ₀ ′ at the time of calibration panel shooting is calculated and provided to the camera installation position calibration unit 130 as shooting position information.

The camera installation position calibration unit 130 calibrates the positional relationship between the camera installation joint 18 c and the camera 30 of the articulated arm system 40 for 3D coordinate measurement with a camera based on the imaging position information given by the imaging position calculation unit 120. Do. The camera installation position calibration unit 130 includes an arm reference joint-camera installation joint coordinate transformation matrix M _a at the time of photographing the calibration panel given from the arm position calculation unit 180, and a turntable-arm given from the arm installation position calibration unit 110. Using the reference joint coordinate transformation matrix M ₁ and the rotating table-camera coordinate transformation matrix M ₀ ′ at the time of photographing the calibration panel given from the photographing position calculation unit 120, the camera-installed joint-camera coordinate transformation matrix is expressed by the following equation. to calculate the M _2.

M ₂ = M ₀ ′ · M ₁ ⁻¹ · M _a ⁻¹

The camera installation position calibration unit 130 gives the calculated camera installation joint-camera coordinate conversion matrix M ₂ to the camera position calculation unit 140 as a calibration result of the positional relationship between the camera installation joint 18 c and the camera 30.

  The camera position calculation unit 140 rotates after the arm installation position calibration unit 110 and the camera installation position calibration unit 130 complete the calibration of the system including the articulated arm system 40 with a camera and the turntable 10. Based on the calibration result, camera position information at an arbitrary arm position required when acquiring the three-dimensional shape data of the target object 14 placed on the table 10 is calculated.

Camera position calculator 140 is given as the calibration results from the arm installation position calibration unit 110 turntable - the arm reference joint coordinate transformation matrix M _1, camera installation given as calibration result from the camera installation position calibration unit 130 joint - a camera coordinate transformation matrix M _2, arm reference joint given as the arm position information from the arm position calculation unit 180 - using a camera installed joint coordinate transformation matrix M _a, by the following equation, in the current arm position turntable - it calculates the camera coordinate transformation matrix M _0.

M ₀ = M ₂ · M _a · M ₁

The camera position calculation unit 140 gives the calculated turntable-camera coordinate conversion matrix M ₀ to the three-dimensional data generation unit 150 as camera position information at the current arm position.

  The slit light extraction unit 200 extracts slit light from a photographed image of the target object irradiated with the slit light by the line laser 36 of the articulated arm system 40 for three-dimensional coordinate measurement with a camera, and obtains information on the extracted slit light. This is given to the three-dimensional data generation unit 150.

The three-dimensional data generation unit 150 uses the turntable-camera coordinate conversion matrix M ₀ given as the camera position information from the camera position calculation unit 140 and the slit light information extracted from the photographed image by the slit light extraction unit 200. Utilizing the slit light projection method, the three-dimensional shape data of the target object is generated and recorded.

When the calibration panel is deviated from the rotation center of the turntable 10 or tilted with respect to the rotation plane of the turntable 10, the calibration panel correction unit 190 corrects the displacement of the calibration panel with respect to the turntable 10. Perform the process. The calibration panel correction unit 190 measures the measurement points 15 on the calibration panel while changing the rotation angle θ of the turntable 10 with the measurement probe 32, and the turntable calculated by the arm installation position calibration unit 110. - using an arm reference joint coordinate transformation matrix M _1, calculates a coordinate transformation correction matrix M _m for correcting the deviation of the calibration panel against turntable 10.

The calibration panel correction unit 190 supplies the calculated coordinate conversion correction matrix M _m to the arm installation position calibration unit 110, the camera installation position calibration unit 130, and the camera position calculation unit 140. The arm installation position calibration unit 110 corrects the coordinate value in the rotary base coordinate system of the measurement point 15 on the calibration panel using the coordinate conversion correction matrix M _m, and then performs the above-described rotary base-arm reference joint coordinate conversion. The matrix M ₁ is calculated. Camera installation position calibration unit 130, the above-described camera installation joints - to correct the camera coordinate transformation matrix M ₂ the equation by the coordinate transformation correction matrix M _m. The camera position calculation unit 140 corrects the above-described calculation formula of the turntable-camera coordinate conversion matrix M ₀ with the coordinate conversion correction matrix M _m .

  The slit light position calibration unit 210 calibrates the positional relationship between the camera 30 and the line laser 36. The slit light extraction unit 200 extracts slit light from the line laser 36 from an image obtained by photographing a calibration flat plate while changing the position with the camera 30. The slit light position calibration unit 210 calibrates the slit light position by obtaining the three-dimensional position of the slit light extracted from the captured image and applying it to the plane equation created by the slit light from the line laser 36. The slit light position calibration unit 210 provides the slit light position calibration result to the slit light extraction unit 200, and the slit light extraction unit 200 corrects the position information of the extracted slit light based on the calibration result.

  A system calibration procedure by the three-dimensional modeling system 300 having the above configuration will be described.

  FIG. 11 is a flowchart for explaining a rough procedure of system calibration. The positional relationship between the turntable 10 and the articulated arm system 40 for 3D coordinate measurement with a camera is calibrated (S100). Next, the positional relationship between the camera-installed joint 18c of the articulated arm system 40 for 3D coordinate measurement with camera 40 and the camera 30 is calibrated (S200). Based on the calibration results of step S100 and step S200, the camera position is calculated at an arbitrary arm position (S300).

  FIG. 12 is a flowchart for explaining the procedure of the calibration process S100 of the positional relationship between the turntable 10 of FIG. 11 and the articulated arm system 40 for 3D coordinate measurement with a camera.

  A plurality of measurement points 15 on the calibration panel installed on the turntable 10 are measured by the measurement probe 32 of the articulated arm system 40 for 3D coordinate measurement with camera (S110).

  When the measurement point 15 on the calibration panel is measured by the measurement probe 32, the coordinate value of the measurement point 15 in the arm reference coordinate system is obtained from the arm output data. Further, since the position of the measurement point 15 on the calibration panel is determined by the design specifications, the coordinate value of the measurement point 15 in the rotary base coordinate system is known. Assuming that n measurement points 15 are measured, the coordinate values of the n measurement points 15 in the rotary table coordinate system are t0, t1,..., tn, and the coordinate values of the same measurement point 15 in the arm reference coordinate system are a0, a1, ..., an.

Arm installation position calibration unit 110, in order to calibrate the positional relationship of the turntable 10 and the camera three-dimensional coordinate measuring articulated arm system 40, the turntable - calculating the arm reference joint coordinate transformation matrix M ₁ (S120) . Turntable - coordinate transformation parameters of the arm reference joint coordinate transformation matrix M ₁ is in principle can be calculated by the coordinate values of the three measurement points 15 on the calibration panel.

Three certain measurement points 15 are selected, and three vectors v _t1 , v _t2 , v _t3 are obtained by the following equation using the coordinate values t0, t1, t2 of the three measurement points 15 in the rotating base coordinate system, Furthermore, v _a1 , v _a2 and v _a3 are obtained using the coordinate values a0, a1 and a2 of the same three measurement points 15 in the arm reference coordinate system.
v _t1 = t1-t0, v _t2 = t2-t0, v _t3 = v _t1 × v _t2
v _a1 = a1-a0, v _a2 = a2-a0, v _a3 = v _a1 × v _a2
Here, the symbol x represents the outer product of the vectors.

A matrix T of 3 rows and 3 columns having three vectors v _t1 , v _t2 , and v _t3 in the rotary table coordinate system as columns, and 3 columns having three vectors v _a1 , v _a2 , and v _a3 in the arm reference coordinate system as columns. A matrix A of 3 rows and columns is defined as follows.
T = (v _t1 v _t2 v _t3 )
A = (v _a1 v _a2 v _a3 )

Turntable to be obtained - rotating component M _r of the arm reference joint coordinate transformation matrix M ₁ is given by the following equation.
M _r = A · T ⁻¹

Further, the turntable - translational component t of the arm reference joint coordinate transformation matrix M ₁ is given by the following equation.
t = a0−M _r · t0

Next, the arm installation position calibration unit 110, turntable calculated in step S120 - in order to increase the accuracy of the arm reference joint coordinate transformation matrix M _1, using three or more measurement points 15 on the calibration panel, turntable - optimized by the least squares method coordinate transformation parameters of the arm reference joint coordinate transformation matrix M ₁ (S130).

Step S120 in the turntable was determined - by the arm reference joint coordinate transformation matrix M _1, to convert the known coordinates of the measuring point 15 in the turntable coordinate system into coordinate values in the arm reference coordinate system (called computational coordinate values) The same measurement point 15 is compared with the coordinate value measured by the measurement probe 32 (referred to as the actually measured coordinate value), and the sum of the squares of the difference between the calculated coordinate value and the actually measured coordinate value is used to calculate these values. Find the error of the coordinate value. For all the measurement points 15 on the calibration panel, the error between the calculated coordinate value and the actually measured coordinate value is obtained, and the sum of the errors is obtained for all the measurement points 15. So that the sum of the error is minimized, turntable - arm based matrix impart a rotational component of the joint coordinate transformation matrix M ₁ M Euler angles _r alpha, beta, gamma, and each component tx of vector t giving the translation component, The results obtained by obtaining ty and tz in step S120 are set as initial values and optimized by the least square method.

Turntable is calculated by the above - arm reference joint coordinate transformation matrix M ₁ is the rotation angle of the turntable 10 is the case of the 0. In order to generalize when the rotation angle of the turntable 10 is θ, t0 ′, t1 ′ obtained by rotating the coordinate values t0, t1,..., Tn in the turntable coordinate system by the rotation matrix R as follows. ,..., Tn ′.
ti ′ = R · ti (i = 0, 1,..., n)

Here, the first row R ₁ , the second row R ₂ , and the third row R ₃ of the rotation matrix R are
R ₁ = (cos θ, 0, −sin θ)
R ₂ = (0,1,0)
R ₃ = (sin θ, 0, cos θ)
It is.
In the following calculation formula, the rotation matrix R is expressed as a conversion matrix of 3 rows and 4 columns by adding 0 as the fourth column of each row.

  FIG. 13 is a flowchart for explaining the procedure of the calibration process S200 of the positional relationship between the camera installation joint 18c and the camera 30 in FIG.

  The camera 30 of the articulated arm system 40 for 3D coordinate measurement with camera is moved to an appropriate position, and a reference pattern composed of a plurality of reference marks 16 on the calibration panel is photographed (S210). At this time, the relative positional relationship between the turntable 10 and the articulated arm system 40 for 3D coordinate measurement with a camera needs to be the same as that at the time of data acquisition in step S100. This is because the calibration result of the positional relationship between the turntable and the arm in step S100 is used in later calculations.

The imaging position calculation unit 120 calculates a turntable-camera coordinate transformation matrix M ₀ ′ at the time of calibration panel imaging using the captured image (S220). This process will be described in detail with reference to FIG.

Arm position calculating unit 180 as position information of the arm at the time of calibration panels taken from the arm output data, arm reference joint - calculates the camera installation joint coordinate transformation matrix M _a (S230).

Arm reference joint - camera installation joint coordinate transformation matrix M _a is calculated from the arm output data. As the arm output data, information on the position vector e and direction vector v of the camera installation axis 20d and the position vector p of the measurement probe 32 is used. Since these data are obtained from the articulated arm system 40 for three-dimensional coordinate measurement with a camera, they are values in the arm reference coordinate system.

Arm reference joint - camera installation joint coordinate transformation matrix M _a is calculated by the following procedure. First, the coordinate axis vectors vx, vy, vz of the camera installation axis coordinate system viewed from the arm reference coordinate system are calculated by the following formula.
vy = v / | v |
vp = pe
vn = v × vp
vx = vn / | vn |
vz = vx × vy

These coordinate axes vectors vx, vy, the use of vz, arm reference joint - matrix M _r giving the rotational component of the camera installation joint coordinate transformation matrix M _a is
M _r = (vx vy vz)
Given in. Arm reference joint - vector t which gives the translation component of the camera installation joint coordinate transformation matrix M _a is equal to the position vector e of the camera installation shaft 20d,
t = e
It is.

The camera installation position calibration unit 130 includes a turntable-arm reference joint coordinate transformation matrix M ₁ obtained in step S100, a turntable-camera coordinate transformation matrix M ₀ ′ at the time of calibration panel photography obtained in step S220, and arm reference joint obtained in step S230 - using the camera installation joint coordinate transformation matrix M _a, camera installation joints - the camera coordinate transformation matrix M ₂ is calculated by the following equation (S240).
M ₂ = M ₀ ′ · M ₁ ⁻¹ · M _a ⁻¹

Camera installation joint of the - formula for the camera coordinate transformation matrix M _2, the rotation angle of the turntable is if 0. If the rotation angle is generalized to the case of theta, using a rotation matrix R described above, the camera installation articulated by the formula - may be determined camera coordinate transformation matrix M _2.
M ₂ = M ₀ ′ · R ⁻¹ · M ₁ ⁻¹ · M _a ⁻¹

FIG. 14 is a flowchart for explaining the procedure of calculation processing S220 of the turntable-camera coordinate conversion matrix M ₀ ′ at the time of photographing the calibration panel in FIG. Here, it is assumed that internal parameters such as the focal length, distortion center, and distortion coefficient of the lens of the camera 30 are obtained in advance.

  The photographing position calculation unit 120 extracts a plurality of reference marks 16 from the photographed image of the calibration panel by image processing (S222).

The imaging position calculation unit 120 votes based on the known coordinate value of the reference mark 16 on the calibration panel in the rotary base coordinate system and the actually measured coordinate value of the reference mark 16 extracted from the captured image. A coordinate conversion parameter of the turntable-camera coordinate conversion matrix M ₀ ′ at the time of shooting is calculated by processing (S224).

A procedure for calculating coordinate conversion parameters of the turntable-camera coordinate conversion matrix M ₀ ′ by voting will be described.

  (1) A six-dimensional parameter space (also referred to as a Hough space) corresponding to the coordinate transformation parameters (α, β, γ, tx, ty, tz) is set, and 0 votes are set in all regions.

  (2) List all pairs of arbitrary three reference marks 16 among all the reference marks 16 on the calibration panel. All combinations of the three reference marks 16 listed from the calibration panel are referred to as a reference set.

  (3) List all combinations of three reference marks 16 among the reference marks 16 extracted from the captured image of the calibration panel. All combinations of the three reference marks 16 listed from the photographed image are called an extraction set.

  (4) The following processing is performed for all combinations between a set of arbitrary three reference marks 16 included in the extracted set and a set of arbitrary three reference marks 16 included in the reference set.

  (4-1) When the combination of three reference marks 16 selected from the extracted set and the set of three reference marks 16 selected from the reference set are different in color combination of the reference marks 16, the following ( The processes of 4-2) and (4-3) are not performed.

  (4-2) The coordinate values in the captured image of the three reference marks 16 selected from the extracted set and the coordinate values in the rotary table coordinate system of the three reference marks 16 selected from the reference set are converted into the camera coordinate system. Further, coordinate conversion parameters (α, β, γ, tx, ty, tz) are calculated based on the coordinate values obtained by projecting them onto the captured image plane.

The calculation for projecting a point in the camera coordinate system onto the captured image plane is (x, y, z) as the point in the camera coordinate system and (x ′, y ′) as the point projected onto the captured image plane.
x ′ = x · f / z + cx
y ′ = y · f / z + cy
Can be written. Here, f is the focal length, and (cx, cy) is the image center. However, the unit of the focal length is a pixel.

  (4-3) In the parameter space, vote for an area corresponding to the value of the calculated coordinate transformation parameters (α, β, γ, tx, ty, tz). This increases the number of votes in the corresponding area of the parameter space by one.

  (5) When all the votes are completed, the peak of the number of votes is detected from the parameter space, and the values of the coordinate conversion parameters (α, β, γ, tx, ty, tz) corresponding to the peak region are correct. Output as.

  By the voting process described above, a coordinate conversion parameter calculated from the correct correspondence between the set of reference marks 16 selected from the extracted set and the set of reference marks 16 selected from the reference set is obtained as a correct answer. Become. Since the reference mark 16 is color-coded by a plurality of types of colors in the calibration panel, there is no difference between the set of the reference mark 16 selected from the extracted set and the set of the reference mark 16 selected from the reference set. A meaningful combination can be discriminated and skipped, unnecessary calculation is eliminated, and the processing time can be increased.

  Since the plurality of reference marks 16 provided on the calibration panel are randomly colored, when a set of three reference marks 16 is selected in the voting process described above, when shooting them from different angles, It is possible to avoid the same pattern as the set of the other three reference marks 16 as much as possible, to eliminate indefiniteness due to the shooting direction, and to provide uniqueness. Further, if the arrangement of the plurality of reference marks 16 is also random, the effect can be further enhanced, and the accuracy of the voting process is improved.

Furthermore, the imaging position calculation unit 120 uses the coordinate conversion parameters of the rotary table-camera coordinate conversion matrix M ₀ ′ obtained in step S224 to know the reference mark 16 in the rotary table coordinate system based on these coordinate conversion parameters and camera internal parameters. Is converted into a camera coordinate system, the coordinate value projected onto the captured image plane is compared with the coordinate value of the reference mark 16 extracted from the captured image, and the difference of each component of these coordinate values is compared. Find the error by sum of squares. This error is obtained for all the reference marks 16, and the coordinate transformation parameter is optimized by the least square method with the coordinate transformation parameter obtained in step S224 as an initial value so that the sum is minimized (S226).

  FIG. 15 is a flowchart illustrating camera position calculation processing based on the calibration result of the system of FIG.

The target object 14 is placed on the turntable 10, and the camera 30 is moved to an appropriate position by the articulated arm system 40 for 3D coordinate measurement with a camera. Arm position calculation unit 180, based on the arm output data of camera 3-dimensional coordinate measuring articulated arm system 40, the arm reference joints in the current arm position - calculates the camera installation joint coordinate transformation matrix M _a (S310) . Arm reference joints - the calculation of camera installation joint coordinate transformation matrix M _a is the calculation method described in step S230 of FIG. 13 is used.

The camera position calculation unit 140 includes a turntable-arm reference joint coordinate transformation matrix M ₁ obtained by the calibration process of step S100, a camera installation joint-camera coordinate transformation matrix M ₂ obtained by the calibration process of step S200, and the arm reference joints in the current arm position obtained in step S310 - using the camera installation joint coordinate transformation matrix M _a, by the following equation, turntable - calculates the camera coordinate transformation matrix M ₀ (S320).
M ₀ = M ₂ · M _a · M ₁

Above the turntable - formula of the camera coordinate transformation matrices M ₀ is the rotation angle of the turntable 10 is 0, the case where the rotation angle of the turntable 10 is generally a θ is rotated by the following formula table - can be calculated camera coordinate transformation matrix M _0.
M ₀ = M ₂ · M _a · M ₁ · R
Here, R is the rotation matrix of the rotation angle θ described above.

  The calibration procedure of the system composed of the articulated arm system for camera 3D coordinate measurement 40 and the turntable 10 has been described above, but it is necessary to calibrate the turntable 10 and the calibration panel. There is a positional relationship and a positional relationship of slit light by the camera 30 and the line laser 36. A supplementary explanation will be given below regarding the calibration of these positional relationships.

  First, calibration of the positional relationship between the turntable 10 and the calibration panel will be described.

  The calibration panel also serves as the top plate of the turntable 10, and the origin of the calibration panel coordinate system is designed to be the center of rotation of the turntable 10. However, in practice, the origin of the calibration panel may deviate from the rotation center of the turntable 10. In addition, the calibration panel on the turntable 10 may be tilted.

  Therefore, parameters for correcting the deviation of the calibration panel with respect to the turntable 10 such as the deviation from the rotation center of the calibration panel and the inclination with respect to the rotation plane are calculated. For the calculation of the correction parameter, data obtained by measuring the measurement point 15 installed on the calibration panel with the measurement probe 32 is used. However, it is necessary to acquire data by changing the rotation angle of the turntable 10.

Calibration panel displacement correction is performed according to the following procedure. First obtains the measurement data of the measuring point 15 at each rotation angle theta, the turntable by the procedure of step S100 - calculating the arm reference joint coordinate transformation matrix M _1.

A parameter for correcting the displacement of the calibration panel with respect to the turntable 10 is represented by a conversion matrix M _m (referred to as a coordinate conversion correction matrix), and a coordinate value in the turntable coordinate system of the measurement point 15 of the calibration panel is represented by p. When the coordinate value of the measurement point 15 in the arm reference coordinate system is represented by a, the following conversion equation is established.
a = M ₁・ R ・ M _m・ p

By the above conversion formula, the difference between each component of the coordinate value obtained by converting the known coordinate value of the measurement point 15 in the rotary table coordinate system into the arm reference coordinate system and the coordinate value of the measurement point 15 measured by the measurement probe 32. Find the error by the sum of squares. This error is obtained for all the measurement points 15 on the calibration panel, and the coordinate transformation parameters of the transformation matrix M _m for correcting the deviation from the center of rotation are optimized by the least square method so that the total sum of errors is minimized. However, the turntable - arm reference joint coordinate transformation matrix M ₁ is the mean value of M ₁ obtained by step S100 to change the rotation angle θ of the turntable 10, the coordinate transformation correction matrix M _m, for the first measurement point 15 The obtained rotation components α, β, γ and translation components tx, ty, tz are set as initial values. However, the coordinate transformation correction matrix M _m, rotational component alpha, beta, one of gamma, translation components tx, ty, one of tz is keep fixed.

In this way the use of the coordinate transformation correction matrix M _m which is calculated is the calculation of the step S100, the coordinate values t0, t1 in rotary table coordinate system of the n measurement points 15, ..., the tn by the following formula The corrected t0 ′, t1 ′,..., Tn ′ may be replaced.
ti ′ = R · M _m · ti (i = 0, 1,..., n)

Also, the camera installation joints step S200 - in determining the camera coordinate transformation matrix M _2, using the coordinate transformation correction matrix M _m, it may be calculated by correcting the following equation.
M ₂ = M ₀ ′ · M _m ⁻¹ · R ⁻¹ · M ₁ ⁻¹ · M _a ⁻¹

Further, in step S320, the turntable - when obtaining the camera coordinate transformation matrices M _0, using the coordinate transformation correction matrix M _m, may be calculated by correcting as the following equation.
M ₀ = M ₂ · M _a · M ₁ · R · M _m

  Next, the calibration of the slit light position will be described.

  FIG. 16 is a diagram for explaining a method of calibrating the slit light position. A measurement unit including a camera 30 and a line laser 36 is attached to the camera installation shaft 20d. In order to perform the slit light projection method using the line laser 36, calibration of the positional relationship between the slit light by the camera 30 and the line laser 36 is required. The positional relationship between the camera 30 and the slit light can be defined by a plane equation of the slit light in the camera coordinate system.

In the camera coordinate system, the plane equation of the slit light is
a.x + b.y + c.z + d = 0
Can be written. The projected slit light position is calibrated by obtaining the coefficients a, b, c, and d of the plane equation.

  In order to acquire calibration data, slit light is projected onto the flat plate 42 provided with a calibration circle pattern, and the image is captured. The photographing is performed twice or more by changing the position of the flat plate 42.

  The circular pattern on the flat plate 42 is disposed only in the peripheral portion of the flat plate 42, and the central portion is blank. When the slit light extraction unit 200 extracts slit light from the captured image of the flat plate 42, processing is limited to only the central blank area, so that the accuracy of extraction of the slit light is ensured. Further, the slit light extraction unit 200 can obtain the relative positional relationship between the flat plate 42 and the camera 30 by extracting a circular pattern in the peripheral region from the captured image. Therefore, by capturing the plane plate 42, it is possible to simultaneously acquire the positional relationship of the plane plate 42 with respect to the camera 30 and measure the two-dimensional coordinate position of the slit light on the captured image.

  The slit light position calibration by the slit light position calibration unit 210 is performed according to the following procedure.

  (1) The slit light position calibration unit 210 extracts a circular pattern from the captured image of the flat plate 42, and calculates the positional relationship between the flat plate 42 and the camera. This positional relationship is expressed by a conversion matrix from the plane plate coordinate system to the camera coordinate system. Thereby, the position of the flat plate 42 is obtained.

  (2) The slit light extraction unit 200 extracts a blank area at the center of the flat plate without the circular pattern for calibration from the captured image of the flat plate 42 using the conversion matrix obtained by the process (1).

  (3) The slit light extraction unit 200 extracts slit light from the line laser 36 from the blank area extracted in the process (2).

  (4) The slit light position calibration unit 210 coordinates in the camera coordinate system of the point where the straight line connected with the focal point and the plane plate intersect for the pixel determined as the slit light projection position in the slit light extraction process of (3). Calculate the value.

(5) slit-position calibration unit 210, by changing the position of the plane plate 42, a plurality of points determined that the slit light projection position _{p i = (px i, py} i, pz i) (i = 0, 1,..., N), and coefficients a, b, c, and d that minimize the following expression f are calculated by the method of least squares.
_{f = Σ i (a · px} i + b · py i + c · pz i + d) 2

  FIG. 16 shows a state in which the plane plate 42 is moved to positions indicated by reference numerals 42a and 42b, laser light is irradiated from the line laser 36, and the laser lines 44a and 44b on the plane plates 42a and 42b are photographed by the camera 30. ing. Pixels p1 and p2 determined as slit light projection positions are extracted from the captured images of the flat plates 42a and 42b, and the three-dimensional coordinate values in the camera coordinate system of the pixels p1 and p2 are calculated. The three-dimensional coordinate values of the pixels p1 and p2 are applied to the equation of the plane 46 formed by the slit light from the line laser 36, and the coefficients a, b, c, and d of the plane equation are obtained, and the slit light position is calibrated.

  In the calibration procedure of the slit light position described above, the slit light extraction process is limited to the blank area in the center of the flat plate that does not overlap with the calibration circle pattern. Can be reduced. Further, by photographing a flat plate provided with a circular pattern, it is possible to acquire the position information of the flat plate and extract the slit light by using the same photographed image, thereby reducing the trouble of photographing. The two-dimensional position of the projection laser is extracted from the captured image, the three-dimensional position of the projection laser is obtained from the position information of the plane plate obtained in advance, and the laser plane can be calculated by connecting a plurality of projection laser lines.

  As described above, according to the 3D modeling system 300 of the present embodiment, since the imaging system is a combination of the rotary base 10 and the articulated arm system 40 for 3D coordinate measurement with a camera, the position of the camera 30 is determined. The degree of freedom is high and fine modeling of the target object is possible. Further, when shooting with the camera 30 moved, the camera position information at the time of shooting necessary for the calculation of the three-dimensional modeling is accurately obtained from the rotation angle of the turntable 10 and the arm position information based on the result of system calibration. It is possible to guarantee the accuracy of the three-dimensional shape data acquired from the captured image.

  FIG. 17 is a diagram for explaining an extension example of the three-dimensional modeling system 300. In addition to the rotary table 10 and the articulated arm system 40 for 3D coordinate measurement with a camera described above, an articulated arm system 50 for 3D coordinate measurement without a camera and a fixed camera 60 are added to the system. It is assumed that the calibration of the combination of the turntable 10 and the articulated arm system 40 with a camera for three-dimensional coordinate measurement has already been completed by the calibration procedure described so far.

  Once the fixed camera 60 is installed, the camera position is fixed and does not move. Therefore, in order to add the fixed camera 60 to the system, it is sufficient to calibrate the positional relationship between the turntable 10 and the fixed camera 60 by photographing the reference pattern of the calibration panel on the turntable 10. Specifically, the shooting position calculation unit 120 performs a reference pattern including a plurality of reference marks 16 on the calibration panel on the turntable 10 by the fixed camera 60 according to a procedure similar to the procedure of step S220 described in FIG. The positional relationship between the turntable 10 and the fixed camera 60 is obtained using an image obtained by capturing the image.

  Thus, by performing the calibration of the positional relationship between the fixed camera 60 and the rotary table 10 already installed, the fixed camera 60 is incorporated into the three-dimensional modeling system 300, and the system can be expanded. The fixed camera 60 is used, for example, to capture a target object on the turntable 10 from a distance and acquire rough three-dimensional shape data of the target object. On the other hand, the articulated arm system 40 with a camera for three-dimensional coordinate measurement is used for taking close-up of a target object and acquiring more detailed three-dimensional shape data of the target object.

  The articulated arm system 50 for three-dimensional coordinate measurement without a camera has a measuring probe attached to the tip of the articulated arm, but is not provided with a camera. The articulated arm system 50 for camera-less three-dimensional coordinate measurement acquires the three-dimensional coordinates of the measurement point on the target object by applying a measurement probe to the surface of the target object. In order to add the articulated arm system 50 for camera-less three-dimensional coordinate measurement to the system, the measurement point of the calibration panel on the turntable 10 is measured by the measurement probe, and the turntable 10 and the camera-free three-dimensional system are measured. It is sufficient to calibrate the positional relationship of the multi-joint arm system 50 for coordinate measurement. Specifically, the arm installation position calibration unit 110 follows the procedure similar to the procedure of step S100 described in FIG. 12 on the turntable 10 by the measurement probe of the articulated arm system 50 for camera-less 3D coordinate measurement. The positional relationship between the turntable 10 and the articulated arm system 50 for 3D coordinate measurement without a camera is calibrated using the measurement results of the plurality of measurement points 15 of the calibration panel.

  In this way, by performing the calibration of the positional relationship between the articulated arm system 50 for camera-less 3D coordinate measurement and the turntable 10 already installed, the articulated arm system 50 for camera-less 3D coordinate measurement 3 The system can be expanded by being incorporated in the dimensional modeling system 300. The articulated arm system 50 for three-dimensional coordinate measurement without a camera has a simple configuration because it does not have a camera or line laser mounted on the tip of the arm, but has only a measurement probe. There is an advantage of high.

  The articulated arm system 50 for three-dimensional coordinate measurement without a camera can be used as follows, for example. When the target object is transparent or translucent, or is a specular reflection object, it is difficult to obtain the three-dimensional shape data of the target object by shooting with a camera, but the articulated arm for three-dimensional coordinate measurement without a camera If the measurement probe of the system 50 is used, three-dimensional shape data can be acquired by measuring the three-dimensional coordinates of the surface of such a target object.

  If the articulated arm system 50 for 3D coordinate measurement without a camera is used, it is assumed that the size of the sphere at the tip of the measurement probe is known. The three-dimensional space in which the sphere at the tip of the measurement probe can pass by operating the arm of the articulated arm system 50 for three-dimensional coordinate measurement without a camera and moving the measurement probe around the target object. Assuming that the target object does not exist in this area, it is possible to narrow the candidate area in the three-dimensional space occupied by the target object by cutting the area from the candidate.

  As described above, if the system including the articulated arm system 40 for 3D coordinate measurement with a camera and the turntable 10 is first calibrated, then the articulated arm system 50 for 3D coordinate measurement without a camera and the fixed system are fixed. A camera 60 can be added to easily expand the system.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

  In the above description, the reference mark 16 on the calibration panel is color-coded by a plurality of types of colors, but the reference mark 16 may be classified according to the shape of the reference mark 16. By providing the measurement point 15 in the reference mark 16 having a different shape, the measurement point 15 can be specified by the shape of the reference mark 16, and the measurement order of the plurality of measurement points 15 can be designated by the shape. Further, the measurement point 15 may not be provided inside the reference mark 16 but may be provided outside the reference mark 16 as long as the correspondence with the reference mark 16 is known. For example, the measurement point 15 may be associated with the reference mark 16 by providing the measurement point 15 in the vicinity of the reference mark 16. Further, the measurement point 15 may be provided regardless of the position of the reference mark 16, and the measurement point 15 itself may be color-coded. For example, a color can be painted around the hole of the measurement point 15 or a hole can be colored.

It is a block diagram of the three-dimensional modeling system which concerns on embodiment. It is a detailed block diagram of the camera installation axis | shaft of the articulated arm system for three-dimensional coordinate measurement with a camera of FIG. It is a figure explaining the calibration panel provided on the turntable of FIG. It is a figure explaining a mode that 3D model data is acquired by the 3D modeling system which concerns on embodiment. It is a figure explaining the coordinate system set to the turntable of FIG. It is a figure explaining the coordinate system set to the arm reference joint of the articulated arm system for three-dimensional coordinate measurement with a camera of FIG. It is a figure explaining the coordinate system set to the camera installation joint of the articulated arm system for three-dimensional coordinate measurement with a camera of FIG. 1, and the coordinate system set to a camera. It is a figure explaining the calculation principle of the calibration of the articulated arm system for three-dimensional coordinate measurement with a camera of FIG. 1, and a turntable. It is a figure explaining the procedure of the calibration of the articulated arm system for 3D coordinate measurement with a camera of FIG. 1, and a turntable. It is a block diagram of the three-dimensional data processing apparatus of FIG. It is a flowchart explaining the rough procedure of the calibration of the three-dimensional modeling system which concerns on this Embodiment. It is a flowchart explaining the procedure of the calibration process of the positional relationship of the turntable of FIG. 11, and the articulated arm system for three-dimensional coordinate measurement with a camera. 12 is a flowchart for explaining a calibration processing procedure for the positional relationship between the camera-installed joint and the camera in FIG. It is a flowchart explaining the procedure of the calculation process of the turntable-camera coordinate transformation matrix at the time of calibration panel imaging | photography of FIG. It is a flowchart explaining the calculation process of the camera position based on the calibration result of the system of FIG. It is a figure explaining the method of calibration of a slit light position. It is a figure explaining the example of an extension of the three-dimensional modeling system concerning an embodiment.

Explanation of symbols

  10 rotary table, 12 support table, 15 measurement point, 16 reference mark, 18a arm reference joint, 18c camera installation joint, 20d camera installation axis, 30 camera, 32 measurement probe, 36 line laser, 40 3D coordinate measurement with camera Articulated arm system, 50 Articulated arm system for 3D coordinate measurement without camera, 60 Fixed camera, 100 3D data processing device, 110 Arm installation position calibration unit, 120 Shooting position calculation unit, 130 Camera installation position calibration unit, 140 camera position calculation unit, 150 three-dimensional data generation unit, 160 captured image input unit, 170 arm data input unit, 180 arm position calculation unit, 190 calibration panel correction unit, 200 slit light extraction unit, 210 slit light position calibration unit, 300 3 Based modeling system.

Claims (15)

An apparatus for calibrating a three-dimensional modeling apparatus including a multi-joint arm for measuring three-dimensional coordinates with a camera and a table for placing an object,
Based on the coordinate values of the measurement points obtained by measuring the measurement points provided on the calibration panel installed on the table by the articulated arm for three-dimensional coordinate measurement with the camera, the table and the camera An arm installation position calibration unit for calibrating the positional relationship of the articulated arm for 3D coordinate measurement,
Based on the coordinate value of the reference mark obtained by photographing the reference mark provided on the calibration panel installed on the table by the camera, position information of the camera with respect to the table at the time of photographing is calculated. A shooting position calculator to
An arm position calculation unit that calculates position information of a multi-joint arm for three-dimensional coordinate measurement when the reference mark provided on the calibration panel is photographed by the camera;
Using the calibration result of the positional relationship between the table and the articulated arm for 3D coordinate measurement with the camera, the position information of the camera with respect to the table at the time of shooting, and the position information of the articulated arm for 3D coordinate measurement A calibration apparatus comprising: a camera installation joint of the articulated arm for three-dimensional coordinate measurement with a camera and a camera installation position calibration unit that calibrates the positional relationship of the camera.   Using the calibration result of the positional relationship between the table and the articulated arm for three-dimensional coordinate measurement with the camera and the calibration result of the positional relationship between the camera-installed joint and the camera, the camera with respect to the table at an arbitrary arm position The calibration apparatus according to claim 1, further comprising a camera position calculation unit that calculates position information of the camera. The stand is a turntable that can be rotated by placing an object on it,
Based on the coordinate value of the measurement point obtained by measuring the measurement point on the calibration panel with the articulated arm for three-dimensional coordinate measurement with the camera while varying the rotation angle of the turntable, the calibration The calibration apparatus according to claim 1, further comprising a calibration panel correction unit that corrects a deviation of the panel for the rotary table. A slit light extraction unit that extracts slit light projected on an object placed on the table from an image photographed by the camera;
2. A slit light position calibration unit that calibrates the projection position of the slit light with respect to the camera by applying the coordinate position of the projected slit light to a plane equation created by the irradiated slit light. 4. The calibration device according to any one of items 1 to 3.   The calibration apparatus according to claim 1, wherein the plurality of reference marks provided on the calibration panel are randomly arranged.   5. The calibration apparatus according to claim 1, wherein the plurality of reference marks provided on the calibration panel are randomly classified into a plurality of types. 6.   The plurality of reference marks provided on the calibration panel are classified into a plurality of types, and the measurement points are provided in association with different types of the reference marks. 5. The calibration device according to any one of 4 to 4. An articulated arm for 3D coordinate measurement with a camera installed;
A table for placing an object photographed by the camera;
A calibration panel installed on the table,
The calibration panel comprises:
A plurality of measurement points measured by the articulated arm for 3D coordinate measurement in order to calibrate the positional relationship between the table and the articulated arm for 3D coordinate measurement;
A three-dimensional modeling apparatus comprising a plurality of reference marks photographed by the camera in order to calculate position information of the camera with respect to the table.   9. The three-dimensional modeling apparatus according to claim 8, wherein the plurality of reference marks provided on the calibration panel are randomly color-coded by a plurality of types of colors.   The plurality of reference marks provided on the calibration panel are color-coded by a plurality of types of colors, and the measurement points are provided in the reference marks of different colors. The three-dimensional modeling apparatus described. A three-dimensional modeling apparatus including an articulated arm for three-dimensional coordinate measurement with a camera and a platform for placing an object;
A calibration device for calibrating the three-dimensional modeling device,
The calibration device
Based on the coordinate values of the measurement points obtained by measuring the measurement points provided on the calibration panel installed on the table by the articulated arm for three-dimensional coordinate measurement with the camera, the table and the camera An arm installation position calibration unit for calibrating the positional relationship of the articulated arm for 3D coordinate measurement,
Based on the coordinate value of the reference mark obtained by photographing the reference mark provided on the calibration panel installed on the table by the camera, position information of the camera with respect to the table at the time of photographing is calculated. A shooting position calculator to
An arm position calculation unit that calculates position information of a multi-joint arm for three-dimensional coordinate measurement when the reference mark provided on the calibration panel is photographed by the camera;
Using the calibration result of the positional relationship between the table and the articulated arm for 3D coordinate measurement with the camera, the position information of the camera with respect to the table at the time of shooting, and the position information of the articulated arm for 3D coordinate measurement A three-dimensional modeling system comprising: a camera installation joint of the articulated arm for 3D coordinate measurement with a camera and a camera installation position calibration unit that calibrates the positional relationship of the camera. A fixed camera with a fixed shooting position for shooting an object to be placed on the table;
The imaging position calculation unit of the calibration device fixes the reference mark provided on the calibration panel installed on the table in order to expand the system by adding the fixed camera to the system. The three-dimensional modeling system according to claim 11, wherein a positional relationship of the fixed camera with respect to the table is calculated based on a coordinate value of the reference mark acquired by photographing with a camera. Further comprising a multi-joint arm for camera-less three-dimensional coordinate measurement for measuring an object placed on the table,
The arm installation position calibration unit of the calibration device is provided on a calibration panel installed on the table in order to expand the system by adding the articulated arm for three-dimensional coordinate measurement to the system. Calibration of the positional relationship between the table and the articulated arm for 3D coordinate measurement without camera based on the coordinate value of the measurement point obtained by measuring the measurement point by the articulated arm for 3D coordinate measurement without camera The three-dimensional modeling system according to claim 11, wherein: A method for calibrating a three-dimensional modeling apparatus including a multi-joint arm for measuring three-dimensional coordinates with a camera and a table for placing an object,
Based on the coordinate values of the measurement points obtained by measuring the measurement points provided on the calibration panel installed on the table by the articulated arm for three-dimensional coordinate measurement with the camera, the table and the camera Calibrating the positional relationship of the articulated arm for 3D coordinate measurement with
Based on the coordinate value of the reference mark obtained by photographing the reference mark provided on the calibration panel installed on the table by the camera, position information of the camera with respect to the table at the time of photographing is calculated. And steps to
Calculating position information of a multi-joint arm for three-dimensional coordinate measurement when the reference mark provided on the calibration panel is photographed by the camera;
Using the calibration result of the positional relationship between the table and the articulated arm for 3D coordinate measurement with the camera, the position information of the camera with respect to the table at the time of shooting, and the position information of the articulated arm for 3D coordinate measurement And a step of calibrating the positional relationship between the camera-installed joint of the articulated arm for three-dimensional coordinate measurement with a camera and the camera. A program for calibrating a three-dimensional modeling device including a multi-joint arm for measuring three-dimensional coordinates with a camera and a table for placing an object,
Based on the coordinate values of the measurement points obtained by measuring the measurement points provided on the calibration panel installed on the table by the articulated arm for three-dimensional coordinate measurement with the camera, the table and the camera Calibrating the positional relationship of the articulated arm for 3D coordinate measurement with
Based on the coordinate value of the reference mark obtained by photographing the reference mark provided on the calibration panel installed on the table by the camera, position information of the camera with respect to the table at the time of photographing is calculated. And steps to
Calculating position information of a multi-joint arm for three-dimensional coordinate measurement when the reference mark provided on the calibration panel is photographed by the camera;
Using the calibration result of the positional relationship between the table and the articulated arm for 3D coordinate measurement with the camera, the position information of the camera with respect to the table at the time of shooting, and the position information of the articulated arm for 3D coordinate measurement A program for causing a computer to execute a step of calibrating a positional relationship between a camera-installed joint of the articulated arm for three-dimensional coordinate measurement with a camera and the camera.

Images