JP2001245323A - Three-dimensional input method and device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional input method and device that can reduce an arithmetic quantity to decrease the processing time in the case of obtaining three-dimensional data from images. SOLUTION: The three-dimensional input device has a camera 11 that picks up an image of an object Q to obtain a two-dimensional image, a shape information processing section 14 that decides a position of an extracted characteristic point on an image space, an attitude control signal 16 that measures a relative attitude between the camera 11 and the object, and an object shape calculation section 15 that restores the three-dimensional shape of the object in terms of the least square method on the basis of the position of characteristic points and the attitudes of the camera 11 as to images obtained through image pickup at different attitudes.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for obtaining three-dimensional data of an object based on a multi-view image obtained by imaging the object from multiple viewpoints or an image obtained by imaging the object in different irradiation directions. Method and apparatus.

[0002]

2. Description of the Related Art Conventionally, there has been known a method of obtaining three-dimensional data (three-dimensional shape data) of an object by a factor decomposition method based on a multi-viewpoint image (Kanede Takeo et al., "Factor Decomposition Method"). Restoration of Object Shape and Camera Motion "IEICE Transactions D-II, J76-D-II, No. 8, 1
993.8).

That is, according to this conventional method, first, F (F is an integer of 3 or more) images are captured by a camera that moves relative to an object. From the image of the first frame (first image), remarkable feature points such as corners
Points are extracted, and the movement is sequentially tracked between frames.

The position of the feature point p on the image of the f-th frame is (xfp, yfp), and a measurement matrix W of size (2F × P) is defined as follows.

[0005]

(Equation 1)

[0006] The upper half of the matrix W is the x coordinate of each feature point in the image, and the lower half is the y coordinate. Each column corresponds to the tracking result of one feature point, and each row corresponds to the x, y coordinates of all the feature points in one image.

Here, the following equation (1) is established between the matrix W, the matrix M indicating the camera posture, and the matrix S indicating the three-dimensional position of the object. W = M · S (1) Here, the matrix M indicating the camera posture and the matrix S indicating the three-dimensional position of the object are expressed as follows.

[0008]

(Equation 2)

Note that rxi, ryi, and rzi are orthogonal basis vectors representing the camera posture in the object coordinate system, and (X
i, Yi, Zi) are the three-dimensional coordinates of the object in the object coordinate system.

In order to factorize the matrix W, a singular value decomposition technique in linear algebra is used. That is, the matrix W is expressed as follows. W = UΣV Here, U is a 2F × P orthogonal matrix, Σ is a P × P diagonal matrix, and V is a P × P orthogonal matrix. Σ is shown as follows.

[0011]

(Equation 3)

In this Σ, the 0s on the diagonal line
The diagonal elements σ are not arranged in descending order as σ1 ≧ σ2 ≧... ≧ σp ≧ 0. If the rank of the matrix W is 3, it becomes 0 after σ4. Generally, since the matrix W contains noise, the rank is not necessarily 3 but σ4
Since the subsequent singular values are extremely small, decomposition is performed with σ4 and subsequent values set to 0 to obtain the best approximate solution in the least squares sense.

Conventionally, a plurality of images of the same object have been obtained by changing the position of the light source with respect to the object, that is, the irradiation direction (illumination direction), thereby obtaining a plurality of images of the same object. A method for obtaining the three-dimensional data, that is, a so-called photometric stereo method is known (Japanese Patent Laid-Open No. 6-341818).

In this conventional method, F (F is an integer of 3 or more) images or more are obtained by fixing the camera and the object and changing the irradiation direction of the light source to three or more directions. The following equation (2) holds between the image I of the obtained image and the normal S of the surface of the object and the direction L of the light source.

I = S · L (2) Here, the matrices I, S, and L indicating the image, the normal to the surface of the object, and the direction of the light source are expressed as follows.

[0016]

(Equation 4)

As described above, the size of each image is P (≧ 3)
The image matrix I is defined by the value of each pixel. (SiX, SiY, SiZ) is a direction vector of illumination in the object coordinate system, and (l Xi, l Yi, l
Zi) is the normal direction of a point on the object to which a certain pixel focuses.

[0018]

However, in the former method described above, only a matrix W composed of multi-viewpoint images is used, and the matrix W is subjected to mathematical processing to obtain a matrix M indicating the camera posture. Then, a matrix S indicating the three-dimensional position of the object is calculated.

Therefore, the amount of calculation for the mathematical processing of the matrix W is large, and the processing requires time. Further, since only the matrix W is used, if there is an error in the matrix W, the effect is large.

In the latter method, the matrix I composed of a plurality of images of the object with different shadings is used.
The matrix S is subjected to mathematical processing to calculate the respective matrices S and L indicating the normal to the surface of the object and the direction of the light source.

Therefore, the amount of calculation for the mathematical processing of the matrix I is large, and the processing requires time. The present invention has been made in view of the above-described problem, and has as its object to reduce the amount of calculation and shorten the processing time.

[0022]

According to a first aspect of the present invention, there is provided a method for obtaining a two-dimensional image by changing the relative posture between an image pickup means and an object to obtain a two-dimensional image of the object. Extracting a feature point from the image, determining the position of the extracted feature point in the image space, and restoring the three-dimensional shape of the object from the position and the attitude of the feature point for a plurality of images. And

According to a second aspect of the present invention, there is provided an image pickup device for obtaining a two-dimensional image by re-image of an object, extracting a feature point from the image, and determining a position of the extracted feature point in an image space. Means for determining, means for measuring the relative attitude of the imaging means and the object, and for a plurality of images obtained by imaging in a plurality of different attitudes, the position of the feature point and the attitude Means for restoring the three-dimensional shape of the object to the least squares.

The apparatus according to a third aspect of the present invention has an attitude changing means for changing a relative attitude between the imaging means and the object. The apparatus according to the fourth aspect of the present invention has a means for controlling the attitude changing means by a program.

According to a fifth aspect of the present invention, there is provided an apparatus for projecting pattern light for obtaining the characteristic point onto the object. The method according to the invention of claim 6 is a method for imaging a plurality of 2D objects by imaging an object by changing the irradiation direction of the light source.
A step of obtaining a three-dimensional shape of the object from a plurality of obtained images and an irradiation direction of a light source at the time of imaging.

According to a seventh aspect of the present invention, there is provided an image pickup means for obtaining a two-dimensional image by re-image of an object, and a light source for illuminating the object, the irradiation direction of the object being variable. Means for measuring the irradiation direction of the light source on the object, and a plurality of images obtained by imaging in a plurality of different irradiation directions and the irradiation direction of the light source at the time of imaging,
Means for restoring the dimensional shape to a least square.

[0027]

[First Embodiment] FIG. 1 is a block diagram showing a schematic configuration of a three-dimensional input device 1 according to a first embodiment of the present invention.

In FIG. 1, a three-dimensional input device 1 includes a camera 11, a camera moving device 12, a camera posture measuring unit 13,
It comprises a shape information processing unit 14, an object shape calculation unit 15, an attitude control unit 16, and pattern projection units 17a and 17b.

The object Q is fixed at a predetermined position in the object coordinate system. The camera 11 captures an image of the object Q and outputs image data DA. The camera 11 can be moved by a camera moving device 12.

The camera moving device 12 is controlled by the attitude control unit 16 or by a user's input operation.
The position and orientation of the camera 11 are changed. In the present specification, a position or a posture or both of them may be referred to as a “posture”.

The attitude control unit 16 holds a program created in advance for changing the attitude of the camera 11 and performing imaging by the camera 11 in various attitudes. Such a program controls, for example, the camera 11 to take an image of the object Q from upper, lower, left, and right directions.

The attitude control unit 16 also has operation buttons or keys for the user to perform an instruction or input operation. As the camera moving device 12, for example, a stage capable of precisely controlling the amount of movement and the amount of rotation is used. By moving the camera 11, the relative attitude between the camera 11 and the object Q is changed.

The camera posture measuring unit 13 measures the posture of the camera 11 and outputs posture data DT. Since the object Q is fixed, the relative attitude between the camera 11 and the object Q is measured by measuring the attitude of the camera 11. The camera posture measuring unit 13 may measure the posture based on a control signal output from the camera moving device 12 or the posture control unit 16. Further, a six-axis measuring arm attached to the camera 11 or the stage, a magnetic sensor, an acceleration sensor, a gyro, or a combination thereof may be used.

The pattern projection units 17a and 17b project the pattern light for obtaining the characteristic points from the image of the object Q, particularly when the object Q has few corners or patterns like a ball. It is for. As pattern light,
For example, a polka dot pattern or a lattice pattern is used. The pattern projecting unit 17 is installed near the target object Q and projects so that a constant pattern always appears on the surface of the target object Q. The number of pattern projection units 17 may be one or more.

The shape information processing section 14 extracts feature points from the input image (image data DA) and determines the positions of the extracted feature points in the image space. Object shape calculator 15
Is based on a plurality of images, and based on the positions of the feature points and the camera 11
From the posture (posture data DT), the three-dimensional shape of the object Q is reconstructed in a least-squares manner by calculation.

The shape information processing unit 14 and the object shape calculation unit 15 are provided with, for example, an RO storing a processing program.
M and a CPU that executes the program. It is also possible to use a personal computer, a workstation, or the like. The programs and data can be provided or stored by various recording media such as a hard disk, a magneto-optical disk, and a floppy disk.

An image of the object Q is taken by the camera 11 at a timing when the camera 11 is positioned in a predetermined posture by the camera moving device 12 or at an appropriate timing during the movement. Imaging is performed on the object Q from at least three directions. A plurality of obtained image data D
Based on A and the posture data DT, arithmetic processing is performed in the shape information processing unit 14 and the object shape calculation unit 15,
Three-dimensional data DC is generated.

FIG. 2 is a block diagram of the shape information processing section 14 of the three-dimensional input device 1. 2, the shape information processing unit 14 includes a feature point extracting unit 21, a feature point associating unit 22,
And a feature point tracking unit 23.

The feature point extracting section 21 extracts a feature point for each frame (each image) of the input image data DA. That is, corners, patterns, and the like of the object Q are extracted as feature points. When pattern light is projected by the pattern projection unit 17, a polka dot pattern or a lattice point is set as a feature point.

More specifically, a large number of points of interest are set in an image, and a small area composed of a plurality of pixels around each point of interest has a somewhat strong vertical density edge component in the small area. A feature point is such that the horizontal density edge component and the horizontal density edge component are appropriately mixed.

The feature point associating unit 22 associates feature points extracted in each frame between frames. That is, when a certain frame is used as a reference image and the next frame is used as a reference image, it is determined where the feature points of the reference image correspond in the reference image.

More specifically, a search window area having a predetermined size is set in the reference image and the reference image,
A point having a high correlation value is two-dimensionally searched using a correlation function expressed by the following equation (3). This process is performed for all frames in order from the first frame.

[0043]

(Equation 5)

Here, Il (x, y) and Ir (x, y)
Represents the image density at the point (x, y) of the reference image and the reference image, respectively. Also, al, ar and vl
, Vr respectively represent the average or variance within the window area. C on the left side is 2d of the reference image and the reference image.
It represents the similarity of the pattern in the window area of × 2d.

The feature point tracking unit 23 enumerates the positions of one certain feature point in the image through all the frames using the correspondence between the frames, and stores the locus data DB of the feature points.
Is constructed (generated) and output. This process is performed for all feature points.

FIG. 3 is a block diagram of the object shape calculator 15 of the three-dimensional input device 1. In FIG. 3, the object shape calculation unit 15 includes a measurement matrix configuration unit 31, a camera attitude matrix configuration unit 32, and a three-dimensional position calculation unit 33.

The measurement matrix forming unit 31 forms the matrix W described in the related art section based on the locus data DB of three or more feature points. When the feature point of the object Q becomes invisible from the camera 11 due to occlusion or the like or a new feature point that has not been seen appears, a sub-matrix of the matrix W is created, and the object shape is calculated separately.

The camera posture matrix forming section 32 forms a matrix M indicating the camera posture described in the section of the prior art. That is, based on the posture data DT, an orthogonal basis vector representing the camera posture in the object coordinate system is calculated.

Based on the matrix W from the measurement matrix construction unit 31 and the matrix M from the camera posture matrix construction unit 32, the three-dimensional position calculation unit 33 calculates a matrix S indicating the three-dimensional position of the object Q. Calculate. In the calculation, an inverse matrix M ⁻¹ of the matrix M is obtained, and the obtained result is applied to both sides of the above equation (1) from the left. Thereby, the matrix S is obtained. That is, S = M ^-1 · W. Thereby, the optimum value of the element of the matrix S is obtained by using the least squares method. The calculation method itself using the least squares method uses a known method.

FIG. 4 is a flowchart showing the flow of processing by the three-dimensional input device 1. In FIG. 4, first, the object Q is imaged by the camera 11 to obtain a plurality of image data DA (# 11). At this time, posture data DT indicating the relative positional relationship between the camera 11 and the object Q is obtained (# 12). For each two-dimensional image DA, a feature point is extracted and its position is obtained (# 13). Based on the position of the feature point and the posture data DT, three-dimensional data DC is obtained by using the least squares method (# 14).

As described above, according to the three-dimensional input device 1 of the present embodiment, a plurality of image data DA
And posture data DT indicating the relative positional relationship between the camera 11 and the object Q, and based on these data, a matrix operation using the posture data of all pixels and all positions is performed. 3D data DC by approximate solution
Is obtained, the three-dimensional data DC can be obtained in a short time with a small amount of calculation for ^obtaining the inverse matrix M ⁻¹ of the matrix M.

In the conventional factorization method or multi-view stereo method, the amount of calculation is large and it takes time to calculate three-dimensional data. However, in the present embodiment, the processing is performed in a short time because the calculation of the inverse matrix is sufficient. Speedup is achieved.

Further, since the image of the object Q from many viewpoints and many feature points and the posture data DT are used, the error of the matrix W and the measurement error of the camera posture are reduced, and the least squares In this sense, the best approximate solution can be obtained, and highly accurate three-dimensional data DC can be obtained. [Second Embodiment] FIG. 5 shows a third embodiment of the present invention.
FIG. 6 is a block diagram showing a schematic configuration of a dimension input device 1B.
FIG. 4 is a diagram for describing an example of a method of moving a target object Q.

In the second embodiment, instead of moving the camera 11 as in the first embodiment, the camera 11 is fixed and the object Q is moved. That is, in FIG. 5, the three-dimensional input device 1B includes a camera 11, a shape information processing unit 14, an object shape calculation unit 15, a posture control unit 16, an object moving device 12B, an object posture measurement unit 13B, and the like.

The object Q is fixed at a predetermined position on the table TB. The table TB can be moved in a three-dimensional direction along the x, y, and z axes by the object moving device 12B, and can be rotated around each axis. The object coordinate system moves with the movement of the table TB.

The object posture measuring unit 13B includes a table TB
Is measured, that is, the posture of the object Q, and the posture data DT
b is output. Since the camera 11 is fixed, by measuring the attitude of the object Q, the relative attitude between the camera 11 and the object Q is measured.

The shape information processing unit 14 and the object shape calculation unit 15 are the same as those described in the first embodiment. When using the pattern projection unit 17 as used in the first embodiment, the operation unit 17 is moved integrally with the table TB in order to keep the pattern on the target object Q constant.

In the three-dimensional input device 1B of the second embodiment,
While moving or rotating the object Q by the object moving device 12B, an image is taken by the fixed camera 11. The object posture measurement unit 13B measures the amount of movement of the object Q in synchronization with the frame period of the camera 11, and calculates the posture of the camera 11 in the object coordinate system from the amount of movement.

The shape information processing section 14 outputs locus data DB of characteristic points from an image taken by the camera 11. The object shape calculation unit 15 calculates the three-dimensional data DC of the target object Q from the trajectory data DB and the posture data DTb of the feature points in a least-square manner and outputs the result.

When the object Q is moved, the translation and the rotation are usually combined. This is because the three-dimensional shape of the target object Q may not be accurately restored only by the translation or the rotation.

A columnar object Q as shown in FIG.
In the case of 1 or a cubic object Q, occlusion hardly occurs because the shape is simple. In such a case,
Since only one movement of the object Q is sufficient, the object Q1 may be simply moved according to a predetermined program standardized for such a simple object Q. [Third Embodiment] FIG. 7 shows a third embodiment according to the present invention.
It is a block diagram showing a schematic configuration of a dimension input device 1C.

In the third embodiment, instead of moving the camera 11 or the object Q as in the first and second embodiments, they are fixed, and the lighting device for illuminating the object Q is moved. I do.

That is, in FIG. 7, the three-dimensional input device 1C includes a camera 11, an object shape restoring unit 10, a light source 18,
It comprises a light source moving device 19 and an illumination direction measuring unit 20.

When the light source 18 irradiates the object Q, the object Q is illuminated. The light source 18 is the light source moving device 1
9, the irradiation direction and the irradiation position on the object Q can be changed. The direction of irradiation by the light source 18 is changed to three or more directions, and three or more images are acquired by imaging the target object Q with the camera 11 at that time. Image data DA obtained from camera 11
Is input to the object shape restoration unit 10.

The illumination direction measuring unit 20 measures the direction vector of the light source 18 in the object coordinate system in synchronization with the frame period of the camera 11. The measured direction vector is posture data DTc indicating the posture of the light source 18 with respect to the object Q.
Is input to the object shape restoration unit 10.

The illumination direction measuring unit 20 includes the light source moving device 19
The attitude of the light source 18 may be measured based on a control signal output from the controller, for example, a signal from a precisely controlled stage for moving the light source 18. Alternatively, the measurement may be performed by a magnetic sensor, an acceleration sensor, a gyro, a combination thereof, or the like attached to the light source 18.

The light source moving device 19 may move the light source 18 according to a predetermined program. FIG. 8 shows an example in which illumination is always performed from a fixed direction regardless of the object Q.

In FIG. 8, a light source 18
By changing the posture of the camera, illumination is performed from various directions indicated by arrows, and images are obtained by capturing images with the camera 11 in each state. However, when lighting is performed, a condition is that the unit vectors in the light source direction are not on the same plane.

Then, the object shape restoring section 10 performs necessary image processing on the image data DA,
Calculate the dimension data DC. In calculating the three-dimensional data DC, the row components iN1, iN2,.
(1 ≦ N ≦ P), the matrix I
Is configured.

Then, an inverse matrix L ^-1 of the matrix L indicating the direction of the light source is obtained, and this is applied to both sides of the above equation (2) from the left. Thereby, the matrix S is obtained. That is, S = I · L
It becomes ^-1 . As a result, the optimal value of the element of the matrix S is set to at least 2
It is determined using multiplication.

When the three-dimensional coordinates of a certain point on the object Q are known, the three-dimensional shape can be restored by performing integration processing based on the data. 3 of the third embodiment
According to the dimension input device 1C, a plurality of image data DA for the object Q and attitude data DTc indicating the illumination direction are input, a matrix operation is performed based on these data, and a least square approximation solution is used. Since the three-dimensional data DC is obtained, the three-dimensional data DC can be obtained in a short time with a small amount of calculation for ^obtaining the inverse matrix L ⁻¹ of the matrix L.

Further, by using images illuminated from a number of directions, errors due to image noise and measurement errors in the illumination direction are reduced, and the best approximate solution can be obtained in a least square sense.

In the embodiment described above, the light source 18 is moved by the light source moving device 19, but the object Q and the camera 11 are moved without moving the light source 18 or at the same time (moving the object Q). The relative position between the camera and the camera 11 may be fixed). In addition, the object Q, the camera 11, the light source 18, and the like may be moved manually instead of being moved by a mechanical device. In this case, those postures may be measured by an appropriate sensor. In addition, the configuration, shape, processing content, processing order, and the like of the three-dimensional input devices 1, 1B, and 1C can be appropriately changed in accordance with the gist of the present invention.

[0074]

According to the present invention, in obtaining three-dimensional data from a plurality of images, the amount of calculation can be reduced and the processing time can be shortened.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a three-dimensional input device according to a first embodiment of the present invention.

FIG. 2 is a block diagram of a shape information processing unit.

FIG. 3 is a block diagram of an object shape calculation unit.

FIG. 4 is a flowchart showing a flow of processing by the three-dimensional input device.

FIG. 5 is a block diagram illustrating a configuration of a three-dimensional input device according to a second embodiment of the present invention.

FIG. 6 is a diagram for describing an example of a method of moving an object.

FIG. 7 is a block diagram illustrating a configuration of a three-dimensional input device according to a third embodiment of the present invention.

FIG. 8 is a diagram showing an example in which an illumination direction on an object Q is changed by a program.

[Explanation of symbols]

 1, 1B, 1C 3D input device 10 Object shape restoration unit 11 Camera (imaging means) 12 Camera moving device (posture changing unit) 12B Object moving device (posture changing unit) 13 Camera posture measuring unit 13B Object posture measuring unit 14 Shape information processing unit 15 Object shape calculation unit 16 Attitude control unit 17 Pattern projection unit 18 Light source 19 Light source moving device 20 Illumination direction measurement unit Q Object

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 2F065 AA53 BB05 DD06 FF01 FF04 FF05 FF07 JJ03 JJ26 MM06 QQ18 QQ23 QQ31 QQ36 RR02 SS13 UU05 UU07 5B050 BA09 DA07 EA05 EA18 5B057 CA08 CA12 CA16 CB16 DC13 5C061 AA29 AB02 AB08

Claims (7)

[Claims] A step of obtaining a two-dimensional image by changing the relative posture between the image pickup means and the object; obtaining a two-dimensional image; extracting feature points from the image; and an image space of the extracted feature points. A three-dimensional input method, comprising: determining an upper position; and restoring a three-dimensional shape of the target object from positions and orientations of the feature points for a plurality of images. 2. An image pickup means for obtaining a two-dimensional image by re-image of an object, a means for extracting a feature point from the image, and determining a position of the extracted feature point in an image space; Means for measuring a relative posture between the object and the object, and a position of the feature point and the posture for a plurality of images obtained by photographing in a plurality of different postures, the three-dimensional Means for restoring the shape to the least squares, and a three-dimensional input device. 3. The three-dimensional input device according to claim 2, further comprising a posture changing means for changing a relative posture between said imaging means and said object. 4. A three-dimensional input device according to claim 3, further comprising means for controlling said attitude changing means by a program. 5. The apparatus according to claim 2, further comprising means for projecting a pattern light for obtaining said characteristic point onto said object.
The three-dimensional input device according to any one of the above. 6. A step of obtaining a plurality of two-dimensional images by imaging an object by changing an irradiation direction of a light source, and obtaining the plurality of two-dimensional images from the obtained plurality of images and an irradiation direction of a light source at the time of imaging. Restoring a three-dimensional shape; and a three-dimensional input method. 7. An image pickup means for obtaining a two-dimensional image by re-image of an object, a light source for illuminating the object, the irradiation direction of the object being variable, and the object of the light source Means for measuring the irradiation direction to the object, a plurality of images obtained by imaging in a plurality of different irradiation directions, and the irradiation direction of the light source at the time of imaging, and the least-square reconstruction of the three-dimensional shape of the object. A three-dimensional input device, comprising: