JP2018195070A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

Provided is an information processing apparatus capable of estimating a position and orientation with high accuracy even if a measurement object cannot be three-dimensionally measured with high accuracy and high density. An information processing apparatus includes a captured image input unit that inputs captured images including a measurement object, and feature elements and feature elements obtained from images obtained by observing the measurement object from a plurality of viewpoints. The template input unit 112 that inputs information of the template 20 having information on the mutual depth relative relationship is collated with the feature element described in the template and the feature element detected from the shot image, and is included in the shot image. A candidate search unit 113 for searching for a matching candidate of the measurement object, a depth identification unit 114 for identifying a depth relative relationship between detected feature elements, and a depth relative relationship obtained by the depth identification unit and features described in the template A candidate determination unit 115 that determines whether a matching candidate is correct or not based on a relative depth relationship between elements. [Selection] Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program that acquire three-dimensional position and orientation information of a measurement target from an image.

  2. Description of the Related Art In recent years, there has been proposed an apparatus that uses a robot arm to capture and operate a target such as a randomly placed component and photograph a target with an imaging device to estimate a three-dimensional position and orientation. In such an apparatus, a plurality of images obtained by observing a measurement target from a plurality of viewpoints are acquired in advance, and a two-dimensional feature amount such as a contour detected in the image is extracted for each viewpoint to generate a lookup table called a template. Keep it. At the time of execution of the device, the feature quantity obtained from the input captured image is compared with the feature quantity described in the template, and the position and orientation of the measurement target is determined using the viewpoint information associated with the most similar template. Get an approximate value.

  In Patent Document 1, in addition to a two-dimensional image obtained by photographing with a camera, information on a three-dimensional contour and a surface normal estimated by three-dimensional measurement of a measurement target is used, and there is no error. A method for estimating the correct posture is described.

JP 2014-178967 A

  However, the technique described in Patent Document 1 has a problem that the position and orientation estimation accuracy is low if the measurement object cannot be three-dimensionally measured with high accuracy and high density. An object of the present invention is to provide an information processing apparatus capable of estimating a position and orientation with high accuracy even if a measurement object cannot be three-dimensionally measured with high accuracy and high density.

  An information processing apparatus according to the present invention includes a photographed image input unit that inputs a photographed image including an object, a feature element obtained from images obtained by observing the object from a plurality of viewpoints, and information on a depth relative relationship between the feature elements. A template input unit that inputs information on a template having a feature, and a feature element described in the template and a feature element detected from the photographed image are collated, and a matching candidate for an object included in the photographed image is obtained. Candidate search means for searching, depth identification means for identifying a depth relative relationship between at least two detected feature elements, information on the depth relative relationship obtained by the depth identification means, and features described in the template Candidate determining means for determining whether the matching candidate is correct or not based on information on the relative depth relationship between elements. And wherein the Rukoto.

  According to the present invention, it is possible to accurately estimate the position and orientation of a measurement target even if three-dimensional measurement cannot be performed with high density and high accuracy.

It is a figure which shows the structural example of the information processing apparatus in 1st Embodiment. It is a figure explaining the relationship between the imaging part in 1st Embodiment, and a measurement target object. It is a flowchart explaining the flow of the process in 1st Embodiment. It is a figure which shows the example in which a template production | generation part produces | generates a template image. It is a figure which shows the example of the data structure of a template. It is a figure which shows the example of the contrast acquired in 1st Embodiment. It is a figure which shows the structural example of the information processing apparatus in 2nd Embodiment. It is a figure explaining the relationship between the imaging part in 2nd Embodiment, a 2nd imaging part, and a measurement target object. It is a figure explaining the parallax comparison by the depth identification part in 2nd Embodiment. It is a figure which shows the structural example of the information processing apparatus in 3rd Embodiment. It is a figure explaining the relationship between the imaging part in 3rd Embodiment, a three-dimensional measuring part, and a measurement target object. It is a figure explaining the acquisition method of the feature element three-dimensional coordinate in 3rd Embodiment. It is a figure explaining the relationship between the imaging part in 4th Embodiment, a three-dimensional measuring part, and a measurement target object. It is a figure which shows the structural example of the information processing apparatus in 5th Embodiment. It is a flowchart explaining the flow of the process in 5th Embodiment. It is a figure which shows the computer function which can implement | achieve the information processing apparatus in this embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
A first embodiment of the present invention will be described. In the present embodiment, a description will be given of an apparatus that can be applied to the case of imaging a measurement object using a single imaging apparatus, for estimation of the three-dimensional position and orientation of the measurement object that exists on substantially the same plane. In the present embodiment, an appropriate one is selected from a plurality of obtained three-dimensional position and orientation candidates based on the contrast of two images taken by changing the focal length of the imaging apparatus. This makes it possible to accurately estimate the position and orientation of the measurement object without providing a three-dimensional measurement apparatus.

  FIG. 1 is a diagram illustrating a configuration example of the entire apparatus according to the first embodiment. In FIG. 1, a rectangular frame indicates a functional module that performs each process of the present embodiment, and an arrow indicates a data flow. The apparatus includes an information processing apparatus 100 that estimates a three-dimensional position and orientation, and an imaging unit 101 and a template generation unit 102 that are connected thereto. The configuration shown in FIG. 1 is an example, and the present invention is not limited to this.

  The imaging unit 101 acquires a captured image obtained by capturing a scene including a measurement target for estimating a three-dimensional position and orientation, and outputs the captured image to the captured image input unit 111 of the information processing apparatus 100. The imaging unit 101 has an automatic focus control mechanism (not shown) inside, and can arbitrarily change the focal length of the lens by external control. In the present embodiment, the imaging unit 101 acquires two captured images 10 and 11 at different focal lengths. The template generation unit 102 generates a template 20 that describes a feature element obtained from images obtained by observing the measurement target 201 from a plurality of viewpoints and a depth relative relationship between the feature elements. The template generation unit 102 outputs the generated template 20 to the template input unit 112 of the information processing apparatus 100.

  FIG. 2 is a diagram illustrating the relationship between the imaging unit 101 and the measurement object 201 in the first embodiment. A plurality of measurement objects 201 are randomly arranged on the plane 202, and the imaging unit 101 acquires the captured images 10 and 11 obtained by looking down on the whole from above. The individual measurement objects 201 take various positions and orientations, but rarely overlap each other, and are generally in contact with the plane 202. Further, the imaging unit 101 is arranged so that the optical axis of the imaging unit 101 and the plane 202 are substantially orthogonal.

  The information processing apparatus 100 according to the first embodiment includes a captured image input unit 111, a template input unit 112, a candidate search unit 113, a depth identification unit 114, and a candidate determination unit 115. The captured image input unit 111 captures the captured images 10 and 11 captured by the imaging unit 101 into the information processing apparatus 100 and supplies the captured images 10 and 11 to the candidate search unit 113 and the candidate determination unit 115. The template input unit 112 takes the template 20 generated by the template generation unit 102 into the information processing apparatus 100 and supplies the template 20 to the candidate search unit 113 and the candidate determination unit 115.

  The candidate search unit 113 collates the input captured image 10 with the template 20 by template matching, and searches for a matching candidate for the position and orientation of the measurement target. The candidate search unit 113 generates a matching candidate list 40 based on the search result, and outputs the matching candidate list 40 to the depth identification unit 40 and the candidate determination unit 115.

  The depth identifying unit 114 identifies the relative depth relationship between the feature elements detected from the captured images 10 and 11. The depth identifying unit 114 identifies the depth context of the feature elements included in the template 20 with reference to the two captured images 10 and 11, and outputs the identified depth context information to the candidate determining unit 115. . The candidate determination unit 115 determines the correctness of the matching candidate based on the depth relative relationship obtained by the depth identification unit 114 and the depth relative relationship between the feature elements described in the template 20. The candidate determination unit 115 determines the correctness of the position and orientation included in the matching candidate list 40 based on the depth context information of the feature element, and only the position and orientation information determined to be correct is the final position and orientation information 50. Output as.

  The operation in the first embodiment will be described. FIG. 3 is a flowchart illustrating a processing flow of the apparatus according to the first embodiment. FIG. 3A shows the flow of the entire process, and FIG. 3B shows the flow of the preparation process in step S301 shown in FIG. 3A.

  In step S <b> 301, pre-preparation processing is performed, and the template generation unit 102 generates the template 20 related to the measurement target 201. Since the template generation unit 102 is not included in the information processing apparatus 100, the process of step S301 is performed as advance preparation prior to the process in the present embodiment. Note that after the template 20 is generated, the process of step S301 can be omitted even when the process of this embodiment is repeated.

  With reference to FIG.3 (b), the detail of the preparation process in step S301 is demonstrated. In step S <b> 311, the template generation unit 102 reads a three-dimensional model of the measurement object 201. Here, the three-dimensional model is CAD data generated at the time of designing the measurement object 201, and the CAD data relates to the three-dimensional coordinates of the vertices of the measurement object 201 and the indices of the vertices constituting the sides and surfaces. It is assumed that information is described.

  In step S <b> 311, the template generation unit 102 sets a plurality of viewpoints for drawing the three-dimensional model of the measurement object 201. This viewpoint is such that images with various appearances covering the postures that the measurement target 201 can actually take are generated under the situation where the position and orientation of the measurement target 201 are estimated as shown in FIG. Set to. In the present embodiment, a regular polyhedron is assumed in which the measurement object 201 is arranged at the center of the circumscribed sphere, and the viewpoint for drawing is set at the position of the vertex of the regular polyhedron. In step S312, the image with the radius of the circumscribed sphere changed and the image with the rotation angle of the viewpoint changed in the image plane are also drawn, so these are set as different viewpoints. The template generation unit 102 sets hundreds to thousands of viewpoints. The viewpoints to be actually set are the size of the measurement object 201, possible postures, and resolution of postures required for estimation. What is necessary is just to determine suitably according to factors, such as.

  The processing from step S312 to step S315 is executed for each viewpoint set in step S311 and is repeatedly executed a number of times equal to the number of viewpoints. In step S312, the template generation unit 102 draws a three-dimensional model of the measurement target 201 from the viewpoint set in step S311 to generate a template image 30 as shown in FIG. At this time, the depth value from the viewpoint position to the three-dimensional coordinates constituting each pixel is held in a depth buffer (not shown). Next, in step S 313, the template generation unit 102 samples a predetermined number of pixels on the contour 411 from the template image 30, and extracts a feature element 412 that is a part of data constituting the template 20.

  FIG. 4A is a diagram illustrating an example in which the template generation unit 102 generates the template image 30 in the present embodiment. The template image 30 is generated by, for example, drawing a three-dimensional model of the measurement object 201 from the viewpoint position (401 to 405, etc.) set in step S311 using computer graphics. FIG. 4B shows an example of the generated template image 30 and the contour 411 and the sampled feature element 412 related to the measurement object 201 obtained as a result of the contour extraction process described later.

  FIG. 5 is a diagram illustrating an example of a data structure of the template 20 generated by the template generation unit 102. The template 20 is composed of a collection of a plurality of template elements. Each template element has the position and orientation of the viewpoint that generated the template image 30 and a plurality of feature elements 412. Each feature element 412 has a feature position, a feature parameter, and feature depth identification information.

  In the present embodiment, image coordinates of the contour 411 of the measurement target 201 extracted from the template image 30 are stored as feature positions, and values indicating the direction of the gradient of the contour 411 are stored as feature parameters. After applying the contour extraction process to each template image 30, the template generation unit 102 samples a predetermined number of pixels constituting the contour 411, and calculates the coordinates and contour gradient of the pixels. To the template 20.

  In step S314, the template generation unit 102 calculates feature depth identification information for the feature element 412 extracted in step S313. The template generation unit 102 calculates the feature depth identification information as follows. When sampling the feature element 412 of the measurement object 201, the template generation unit 102 obtains a depth value by referring to the depth buffer that holds the depth value in step S312. When a predetermined number of samplings are completed for one template image 30, the same number of depth values as the number of feature elements are obtained. The template generation unit 102 assigns “front” or “back” labels to these depth values. In the present embodiment, threshold values for distinguishing the front and back are determined by discriminant analysis for a plurality of depth values. In the discriminant analysis, a threshold value for minimizing the intra-group variance and maximizing the inter-group variance is determined for each group of values having the labels “front” and “back”. The template generation unit 102 assigns a “front” label to those having a depth value smaller than the threshold value, and assigns a “back” label to those having a depth value larger than the threshold value. Write into the template 20 as depth identification information (ID).

  In step S315, if the processes in steps S312 to S315 have been performed for all the viewpoints set in step S311, the process proceeds to step S316. If not, the process returns to step S312. In step S316, the template generation unit 102 holds the calculated template 20 in a storage unit (not shown).

  Returning to FIG. 3A, in step S <b> 302, the imaging unit 101 captures an image of a scene including the measurement object 201. In the first embodiment, the imaging unit 101 acquires two captured images 10 and 11 having different focal lengths and outputs them to the captured image input unit 111. The captured image 10 is an image captured in a state where the position of the plane 202 is in focus, and the captured image 11 is focused on the near side (the upper end of the measurement target 201) with respect to the plane 202. It is an image taken in a state. The captured image input unit 111 supplies the input captured images 10 and 11 to the candidate search unit 113 and the candidate determination unit 115. In step S303, the template input unit 112 reads the template 20 recorded in step S316 from the storage unit and inputs it to the information processing apparatus 100.

  In step S304, candidate search unit 113 searches for a position and orientation candidate. The candidate search unit 113 first performs contour extraction processing on the captured image 10. Next, the candidate search unit 113 shifts the region set on the photographed image 10 by so-called template matching for each template element included in the template 20 between the contour 411 and the contour detected from the photographed image 10. Calculate similarity. Information on the contour 411 is described in the feature element 412 as a feature position and a feature parameter. The similarity is set so as to increase when the position and direction of the contour 411 defined in the feature element 412 match the position and direction of the contour detected from the captured image 10. The sum of the similarities of all the feature elements 412 is calculated for each area of the template element and the captured image 10.

  The candidate search unit 113 holds all template elements whose sum of similarities exceeds a predetermined threshold as matching candidates in the matching candidate list 40. The matching candidate list 40 stores the identification number of template elements, the degree of similarity, and the position of the template on the captured image 10. Since the template 20 is generated by drawing the measurement object 201 from a viewpoint in any direction, the position and orientation described in the template element can be regarded as the three-dimensional attitude of the measurement object 201. Further, the three-dimensional position of the measurement object 201 can be calculated from the position of the template element detected on the captured image 10 and the position and orientation described in the template element. For example, the 3D of the measurement target 201 is calculated from the center coordinates representing the template element detected on the captured image 10 and the distance between the measurement target 201 and the viewpoint calculated from the position and orientation described in the template element. The position can be calculated.

  In step S <b> 305, the depth identifying unit 114 refers to the two captured images 10 and 11 and identifies the depth relationship between the feature elements included in the template 20, that is, the relative relationship between the depths of the feature elements. The depth identifying unit 114 first extracts one matching candidate from the matching candidate list 40. Then, the depth identifying unit 114 refers to the feature depth identification information from the feature elements defined in the template element, and obtains the feature element to which the “front” label is assigned and the feature element to which the “back” label is assigned. A feature pair is constructed by extracting one by one. In addition, when the feature element extracted on the picked-up image 10 is not observed, the feature element is not used. The depth identification unit 114 repeats this process to set a plurality of feature pairs.

  Next, the depth identifying unit 114 refers to the captured images 10 and 11 and compares the contrast levels of the feature elements 412. In the present embodiment, this is realized by comparing the luminance of the pixels in the captured images 10 and 11 corresponding to the feature element 412. Specifically, when the brightness in the captured image 10 is higher than the brightness in the captured image 11, it is determined that the contrast is high and a value “1” is assigned. In other cases, the contrast is determined to be low and the value “0” is determined. ". In the present embodiment, the focus position of each captured image is set to the position of the plane 202 in the captured image 10, and is set to the upper end of the measurement object 201 in the captured image 11. Accordingly, when the contrast of the captured image 10 is higher (that is, when the value indicating the comparison result is “1”), it is determined that the feature element 412 is relatively located on the far side, and otherwise, the front side Judged to be located on the side.

  FIG. 6 is a diagram illustrating an example of contrast acquired by the depth identifying unit 114. The positions of certain feature elements 412 defined in the template elements are indicated by black circles. FIG. 6A illustrates an example of contrast in the captured image 10, and FIG. 6B illustrates an example of contrast in the captured image 11. In FIG. 6, the graph shown below shows an example of the luminance change between the cross sections A and B orthogonal to the contour line 411.

  As shown in FIG. 6A, the captured image 10 is in a state where the contour line 411 of the measurement target 201 is in focus, the contrast is high, and the luminance at the position of the feature element 412 (black circle). The peak is also relatively large. On the other hand, as shown in FIG. 6B, the captured image 11 is out of focus from the outline 411 of the measurement object 201, has low contrast, and the luminance peak at the position of the feature element 412 is also relative. It is a small value. Therefore, comparing the luminance values at the positions of the characteristic elements 412 of the captured images 10 and 11 can be regarded as equivalent to comparing the contrast.

  Next, in step S <b> 306, the candidate determination unit 115 determines whether the matching candidates included in the matching candidate list 40 are correct, and outputs information determined to be correct as final position and orientation information 50. For each feature element 412 set as a feature pair, the candidate determination unit 115 refers to the contrast comparison result acquired in step S305 and the feature depth identification information, and determines whether the matching candidate posture is correct or incorrect. Specifically, the comparison result is 0 for the feature element 412 to which the “front” label of the feature depth identification information is assigned, and the comparison result for the feature element 412 to which the “back” label of the feature depth identification information is assigned. If is 1, the matching candidate is determined to be correct.

  In this embodiment, in order to improve the stability of the position / orientation determination, the above-described determination is performed on a plurality of feature pairs, and when the ratio determined to be correct is higher than a predetermined threshold, the matching candidate Is determined to be correct. This makes it possible to reduce the influence of characteristic elements and noise that are not observed because they are hidden. The candidate determination unit 115 outputs position and orientation information corresponding to the matching candidate determined to be correct as final position and orientation information 50. When there are a plurality of matching candidates determined to be correct, the candidate determination unit 115 selects the matching candidate having the highest ratio determined to be correct, and sets the corresponding position and orientation information as the final position and orientation information 50. To do.

  As described above, according to the first embodiment, a three-dimensional position can be obtained by photographing a measurement target with a single imaging device without providing a three-dimensional measurement device for the measurement target that exists on substantially the same plane. The posture can be estimated with high accuracy.

In addition to the example described above, the following first to eighth modifications may be applied to the first embodiment. Each modification may be applied alone or in combination.
[Modification 1] The generation of the template 20 by the template generation unit 102 is not limited to the example performed prior to the processing in the present embodiment, but may be performed in the step S303. Alternatively, the template input unit 112 may have the function of the template generation unit 102 and generate the template 20.

[Modification 2] When the template 20 is generated, the captured image 10 may be acquired by moving the imaging unit 101 to a predetermined position and orientation and capturing the measurement object 201. For example, the position and orientation of the captured image 10 may be measured by using the position and orientation sensor of the imaging unit 101. Alternatively, the position and orientation of the imaging unit 101 may be estimated from a series of captured images using SfM (Structure from Motion) or SLAM (Simultaneous Localization And Mapping) technology.

[Modification 3] In the present embodiment, the feature depth identification information has two types of “front” and “back”, but may be set to an arbitrary number. For example, in addition to “front” and “back”, a “middle” label may be set, or the depth value may be held as a continuous value.

[Modification 4] A coded aperture is provided at the aperture position of the imaging unit 201, and the candidate determination unit 115 applies blind deconvolution to the captured image 10 to estimate an approximate depth value. May be. In this case, since the imaging unit 201 only needs to acquire one captured image, the imaging unit 201 does not need to include a focus control mechanism. For the depth estimation, for example, a method described in A. Levin et al, Image and Depth from a Conventional Camera with a Coded Aperture, SIGGRAPH 2007 may be used.

[Modification 5] The template generation unit 102 sets the feature elements not only on the outline of the measurement target 201, but may set the feature elements 412 on the surface of the measurement target 201, for example.

[Modification 6] When the candidate search unit 113 generates the matching candidate list 40, not only the position and orientation candidates but also the type of the measurement object 201 may be included. When there are a plurality of types of measurement objects 201, some or all types of templates 20 may be generated. In this case, the type of the measurement target 201 may be recorded in the template element.

[Modification 7] The template generation unit 102 sets the viewpoints for all the ways of viewing the measurement object 201, but is not limited to this. When the position and orientation that the measurement object 201 can take is limited, the template image 30 may be generated only from the viewpoint corresponding to the position and orientation. Thereby, since the total number of templates is reduced, the processing speed can be improved.

[Modification 8] The candidate determination unit 115 outputs only the position / orientation candidates determined to be correct as the final position / orientation information, but a set of all position / orientation candidates and the evaluation values used for the determination, regardless of whether they are correct or incorrect. May be output. Further, when only one position / orientation candidate is included in the matching candidate list 40, the candidate determination unit 115 may output a result of the correctness determination of the candidate. The candidate determination unit 115 may calculate an evaluation value for each of the position and orientation and output the calculated evaluation value.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In the present embodiment, estimation of a three-dimensional position and orientation of a measurement object that exists on substantially the same plane will be described with respect to an apparatus that can be applied when the measurement object is imaged with two imaging devices. In the present embodiment, an appropriate one is selected from a plurality of obtained three-dimensional position / posture candidates based on the magnitude relationship of parallax in two images. This makes it possible to accurately estimate the position and orientation of the measurement object without providing a three-dimensional measurement apparatus.

  FIG. 7 is a diagram illustrating a configuration example of the entire apparatus according to the second embodiment. In FIG. 7, a rectangular frame indicates a functional module that performs each process of the present embodiment, and an arrow indicates a data flow. The apparatus includes an information processing apparatus 100 that estimates a three-dimensional position and orientation, and an imaging unit 101, a template generation unit 102, and a second imaging unit 103 that are connected to the information processing apparatus 100. The configuration illustrated in FIG. 7 is an example, and the present invention is not limited to this. In the following description, parts different from the first embodiment will be described, and description of parts similar to those of the first embodiment will be omitted.

  In the present embodiment, the imaging unit 101 does not need to have an automatic focus control mechanism therein, and acquires a single captured image 10. The second imaging unit 103 acquires a captured image obtained by capturing a scene including the measurement object 201 for estimating the three-dimensional position and orientation, and outputs the captured image to the second captured image input unit 116 of the information processing apparatus 100. The second imaging unit 103 acquires one captured image 11. Here, the imaging unit 101 and the second imaging unit 103 are calibrated in advance. Many methods have been proposed for calibration between multiple cameras, but any method may be used. In the present embodiment, internal parameters (focal length, principal point position, distortion coefficient) of the imaging unit 101 and the second imaging unit 103 and external parameters (relative tertiary between the imaging unit 101 and the second imaging unit 103). The original position / posture) is obtained in advance and is assumed to be known.

  FIG. 8 is a diagram illustrating the relationship between the imaging unit 101, the second imaging unit 103, and the measurement object 201 in the second embodiment. A plurality of measurement objects 201 are randomly arranged on the plane 202, and the imaging unit 101 and the second imaging unit 103 respectively acquire the captured images 10 and 11 obtained by bird's-eye view from above. The individual measurement objects 201 take various positions and orientations, but rarely overlap each other, and are generally in contact with the plane 202. Note that the imaging unit 101 and the second imaging unit 103 do not have to be arranged so that the optical axis and the plane 202 are substantially orthogonal.

  The information processing apparatus 100 according to the second embodiment includes a captured image input unit 111, a template input unit 112, a candidate search unit 113, a depth identification unit 114, a candidate determination unit 115, a second captured image input unit 116, and a second candidate. A search unit 117 is included. The second captured image input unit 116 captures the captured image 11 captured by the second imaging unit 103 into the information processing apparatus 100 and supplies the captured image 11 to the second candidate search unit 117.

  The second candidate search unit 117 collates the input captured image 11 with the template 20 corresponding to the position and orientation of the matching candidate included in the matching candidate list 40 by template matching. The depth identifying unit 114 calculates the parallax of the detected coordinates of the feature element 412 with reference to the two photographed images 10 and 11 and identifies the depth relationship of the feature elements included in the template 20.

  The operation in the second embodiment will be described. The processing flow of the apparatus in the second embodiment is the same as that of the first embodiment. However, there are steps whose processing contents are different from those of the first embodiment, which will be described with reference to FIG.

  In step S302, the imaging unit 101 captures the captured image 10 of the scene including the measurement target 201, and the second imaging unit 103 similarly captures the captured image 11 of the scene including the measurement target 201. The imaging unit 101 outputs the captured image 10 to the captured image input unit 111, and the second imaging unit 103 outputs the captured image 11 to the second captured image input unit 116. The captured image input unit 111 supplies the input captured image 10 to the candidate search unit 113 and the candidate determination unit 115, and the second captured image input unit 116 supplies the input captured image 11 to the second candidate search unit 117. Supply.

  Since the imaging unit 101 and the second imaging unit 103 in the present embodiment are calibrated, the second captured image input unit 116 uses the internal parameters and the external parameters at the time of calibration for the input captured image 11. Perform parallelization. Hereinafter, in the present embodiment, the captured image 10 and the captured image 11 will be described assuming that the epipolar line is parallel to the x axis (lateral direction).

  In step S304, candidate search unit 113 and second candidate search unit 117 search for position and orientation candidates. The process performed by the candidate search unit 113 is the same as that in the first embodiment. Hereinafter, the content of the process in which the second candidate search unit 117 searches for a position / posture candidate will be described. Similar to the candidate search unit 113, the second candidate search unit 117 first performs contour extraction processing on the captured image 11. For each template element included in the template 20, the similarity between the contour 411 defined in the feature element 412 and the contour detected from the captured image 11 is determined by template matching while shifting the region in the captured image 11. calculate. Information on the contour 411 is described in the feature element 412 as a feature position and a feature parameter.

  In the present embodiment, since the matching candidate has already been acquired by the candidate search unit 113, the process of the second candidate search unit 117 is efficiently performed using this. Specifically, the second candidate search unit 117 extracts matching candidates from the matching candidate list 40, refers to the template element identification number and the position of the region on the captured image 10, and determines the template element on the captured image 11. Estimate the position. Since the position and orientation of the template element can be referred to from the identification number of the template element, the three-dimensional position and orientation of the measurement object 201 can be obtained.

  With respect to this three-dimensional position and orientation, using external parameters at the time of calibration, three-dimensional conversion from the position and orientation of the imaging unit 101 to the position and orientation of the second imaging unit 103 is obtained, and the internal parameters of the second imaging unit 103 are determined. Using this, the projected image 11 is projected. Thereby, the position of the template element on the captured image 11 is obtained. The second candidate search unit 117 performs template matching on the captured image 11 in the vicinity thereof in the same manner as the candidate search unit 113 with the position of the template element on the captured image 11 obtained as an initial value. In this way, the position of the template on the final captured image 11 is finally calculated. The second candidate search unit 117 records the coordinates of the template element position on the captured image 11 in the matching candidate list 40.

  In step S <b> 305, the depth identifying unit 114 identifies the depth context of the feature elements included in the template 20. The depth identifying unit 114 first extracts one matching candidate from the matching candidate list 40. The depth identifying unit 114 refers to the feature depth identification information from the feature element 412 defined in the template element, and the feature element to which the “front” label is assigned and the feature element to which the “back” label is assigned. Are extracted one by one to form a feature pair. In addition, when the feature element extracted on the picked-up image 10 or the picked-up image 11 is not observed, the feature element is not used. The depth identification unit 114 repeats this process to set a plurality of feature pairs.

Next, the depth identifying unit 114 compares the parallax on the captured image 10 and the captured image 11 in which the feature element 412 is detected for each feature pair. In the present embodiment, since the epipolar lines of the captured image 10 and the captured image 11 are parallel to the x axis, the depth identifying unit 114 compares the difference of the x coordinates of the captured image 10 and the captured image 11 in the feature element 412. . FIG. 9 is a diagram for explaining a comparison of parallax by the depth identifying unit 114 in the second embodiment. Determining a difference x-coordinate for the detection coordinates in the two feature elements 412 captured image 10 and the captured image 11 (412a and 412b in the drawing) is detected making up the feature pairs, the parallax d _a parallax d _b calculate.

  In step S <b> 306, the candidate determination unit 115 determines the correctness of the matching candidates included in the matching candidate list 40 and outputs the final position / orientation information 50. The candidate determination unit 115 determines the correctness of the matching candidate based on the triangulation of the two captured images 10 and 11, and the position / orientation information corresponding to the matching candidate determined to be correct is the final position / orientation information. 50 is output.

Specifically, the candidate determination unit 115 compares the relative magnitude relationship between the disparity d _a parallax d _b calculated in feature depth identification information and step S305 of the feature element. In the example shown in FIG. 9, it is assumed that the feature depth identification information of the feature element 412a is “front” and the feature depth identification information of the feature element 412b is “back”. Due to the relative geometric relationship between the feature element 412a and the feature element 412b, the parallax of the feature element 412a located on the near side is larger than the parallax of the feature element 412b located on the far side. Therefore, the candidate determination unit 115 determines that the matching candidate is correct if d _a > d _b , and determines that the matching candidate is incorrect if d _a <d _b . In this embodiment, in order to improve the stability of the position / orientation determination, the above-described determination is performed on a plurality of feature pairs, and when the ratio determined to be correct is higher than a predetermined threshold, the matching candidate It is determined that the position and orientation corresponding to is correct. This makes it possible to reduce the influence of characteristic elements and noise that are not observed because they are hidden.

  As described above, according to the second embodiment, the measurement object is photographed with the two imaging devices with respect to the measurement object that is substantially on the same plane, so that the measurement object can be provided without providing a three-dimensional measurement device. It is possible to accurately estimate the three-dimensional position and orientation. Since the comparison is based only on the difference (parallax amount) of the coordinates at which the feature elements are detected in the captured image, the three-dimensional coordinates are not calculated clearly and can be executed only by integer arithmetic without performing floating point arithmetic. Therefore, the processing time is shortened and the processing speed is improved. Furthermore, based on the position and orientation candidates searched on one of the captured images, the position on the other captured image is estimated based on the geometric relationship at the time of calibration, and the search is limited to the vicinity thereof. Therefore, the processing time can be shortened and the processing speed can be improved as compared with the correspondence search required in normal stereo measurement.

In addition, not only the example mentioned above but regarding the second embodiment, the following modified examples 9 and 10 may be applied, the modified examples may be applied alone, or in combination. You may make it apply.
[Modification 9] The second imaging unit 103 may use a lower performance such as resolution and optical characteristics than the imaging unit 101. In this case, the apparatus according to the present embodiment is realized at low cost. It becomes possible.

[Modification 10] In this embodiment, the imaging unit 101 and the second imaging unit 103 need not be calibrated accurately. Since the candidate determination unit 115 determines only the context of the feature pair, the internal parameters and the external parameters of the image capturing unit 101 and the second image capturing unit 103 are accurate enough to distinguish the context of the feature pair. I just need it.

(Third embodiment)
Next, a third embodiment of the present invention will be described. In the present embodiment, the estimation of the three-dimensional position and orientation of the measurement object will be described with respect to an apparatus applicable when the three-dimensional coordinates measured by the three-dimensional measurement apparatus are sparse. In the present embodiment, the three-dimensional coordinates at the feature element positions are estimated from the sparse three-dimensional coordinates measured by the three-dimensional measuring apparatus. Based on the estimated context of the three-dimensional coordinates, an appropriate one is selected from a plurality of obtained three-dimensional position and orientation candidates. This makes it possible to accurately estimate the position and orientation of the measurement target even in a situation where the three-dimensional coordinates measured by the three-dimensional measurement apparatus are sparse.

  FIG. 10 is a diagram illustrating a configuration example of the entire apparatus according to the third embodiment. In FIG. 10, a rectangular frame indicates a functional module that performs each process of the present embodiment, and an arrow indicates a data flow. The apparatus includes an information processing apparatus 100 that estimates a three-dimensional position and orientation, and an imaging unit 101, a template generation unit 102, and a three-dimensional measurement unit 104 that are connected thereto. The configuration shown in FIG. 10 is an example, and the present invention is not limited to this. In the following description, parts different from those of the first embodiment will be described, and description of parts similar to those of the first embodiment will be omitted.

  In the present embodiment, the imaging unit 101 does not need to have an automatic focus control mechanism therein, and acquires a single captured image 10. The three-dimensional measurement unit 104 measures the three-dimensional coordinates on the surface of the measurement object 201 and outputs the measured three-dimensional coordinates 1101 to the three-dimensional coordinate input unit 118. In this embodiment, as an example, a multi-slit projection three-dimensional measurement apparatus is used as the three-dimensional measurement unit 104, and the resolution of the captured image 10 is dense in the direction parallel to the projection slit but sparse in the vertical direction. Assume that the three-dimensional coordinates 1101 to be obtained are acquired.

  FIG. 11 is a diagram illustrating the relationship among the imaging unit 101, the three-dimensional measurement unit 104, and the measurement object 201 in the third embodiment. There are a plurality of measurement objects 201, and the imaging unit 101 acquires the captured image 10 obtained by bird's-eye view from above. Further, the three-dimensional measuring unit 104 projects a plurality of slit lights onto the measurement object 201 and acquires a three-dimensional position on the projection slit line 1102 as a set of three-dimensional coordinates 1101. The individual measurement objects 201 take various positions and orientations at random, and do not necessarily exist on the same plane, and may overlap each other. Note that the imaging unit 101 does not have to be arranged so that its optical axis and the plane on which the measurement object 201 is placed are substantially orthogonal.

  Here, the imaging unit 101 and the three-dimensional measurement unit 104 are calibrated in advance. Many methods have been proposed for calibration between the imaging unit 101 such as a camera and the three-dimensional measuring unit 104 such as a three-dimensional measuring device, but any method may be used. In the present embodiment, internal parameters (focal length, principal point position, distortion coefficient) of the imaging unit 101 and a relative three-dimensional position and orientation between the imaging unit 101 and the three-dimensional measurement unit 104 are obtained in advance and are known. Consider it.

  The information processing apparatus 100 according to the third embodiment includes a captured image input unit 111, a template input unit 112, a candidate search unit 113, a depth identification unit 114, a candidate determination unit 115, and a three-dimensional coordinate input unit 118. The captured image input unit 111 captures the captured image 10 captured by the imaging unit 101 into the information processing apparatus 100 and supplies the captured image 10 to the candidate search unit 113, the candidate determination unit 115, and the depth identification unit 114. The three-dimensional coordinate input unit 118 takes the three-dimensional coordinates 1101 measured by the three-dimensional measurement unit 104 into the information processing apparatus 100 and supplies the information to the depth identification unit 114.

  The depth identifying unit 114 estimates a plurality of feature element three-dimensional coordinates 1201 at the feature element positions of the measurement target 201 using the captured image 10, the three-dimensional coordinates 1101, the template 20, and the matching candidate list 40. Then, the depth identifying unit 114 identifies the depth context of the feature elements included in the template 20 based on these positional relationships.

  The operation in the third embodiment will be described. The processing flow of the apparatus in the third embodiment is the same as that in the first embodiment. However, there are steps whose processing contents are different from those of the first embodiment, which will be described with reference to FIG.

  In step S <b> 302, the imaging unit 101 captures the captured image 10 of the scene including the measurement target 201, and the three-dimensional measurement unit 104 measures the three-dimensional coordinates 1101 on the measurement target 201. The imaging unit 101 outputs the captured image 10 to the captured image input unit 111, and the three-dimensional measurement unit 104 outputs the three-dimensional coordinates 1101 to the three-dimensional coordinate input unit 118. The captured image input unit 111 supplies the input captured image 10 to the candidate search unit 113, the depth identification unit 114, and the candidate determination unit 115. Further, the three-dimensional coordinate input unit 118 supplies the input three-dimensional coordinates 1101 to the depth identification unit 114.

  In step S305, the depth identifying unit 114 estimates the feature element three-dimensional coordinates 1201 in the captured image 10 with reference to the three-dimensional coordinates 1101 and the template 20. The depth identifying unit 114 first extracts one matching candidate from the matching candidate list 40. Then, the depth identifying unit 114 refers to the contour of the captured image 10 extracted by the candidate searching unit 113, the feature element 412 and the three-dimensional coordinate 1101 defined in the template element, and the feature element three-dimensional coordinate. 1201 is calculated.

  FIG. 12 is a diagram illustrating a method for acquiring the feature element three-dimensional coordinates 1201 in the present embodiment. The depth identifying unit 114 uses the three-dimensional coordinates 1101 on the measurement object 201 to acquire the feature element three-dimensional coordinates 1201 that are the three-dimensional coordinates at the position of the feature element 412 in the captured image 10. In FIG. 12, the feature element 412 and the feature element three-dimensional coordinate 1201 exist in the same place. The feature element three-dimensional coordinates 1201 at the position of the feature element 412 are unknown at this stage. Therefore, the depth identifying unit 114 estimates the feature element three-dimensional coordinates 1201 using the surrounding three-dimensional coordinates 1101.

  Specifically, the depth identifying unit 114 searches for the three-dimensional coordinates 1101 near the feature element 412 measured on the same contour for the contour 411 of the captured image 10 extracted by the candidate searching unit 113. In FIG. 12, two three-dimensional coordinates 1101a and 1101b in the vicinity where two black circles on the outline 411 are searched are shown. Next, the depth identifying unit 114 projects the three-dimensional coordinates 1101 a and 1101 b on the captured image 10 using the internal parameters of the imaging unit 101 and the external parameters between the imaging unit 101 and the three-dimensional measurement unit 104. Image coordinates are calculated. The depth identification unit 114 calculates the distance between the image coordinates of the feature element and the image coordinates on which the three-dimensional coordinates 1101a and 1101b are projected on the captured image 10, and uses the two distances calculated as the three-dimensional coordinates 1101a and 1101b. By dividing, the feature element three-dimensional coordinates 1201 are calculated.

  Next, as in the first embodiment, the depth identification unit 114 refers to the feature depth identification information from the feature elements 412 defined in the template element, and the feature elements assigned the “front” label and the “back” The feature elements to which the label is given are extracted one by one to form a feature pair. In addition, when the feature element extracted on the picked-up image 10 is not observed, the feature element is not used. The depth identification unit 114 repeats this process to set a plurality of feature pairs.

  In step S <b> 306, the candidate determination unit 115 determines the correctness of the matching candidates included in the matching candidate list 40 and outputs the final position / orientation information 50. The candidate determination unit 115 compares the feature element three-dimensional coordinates 1201 calculated in step S305 for each feature pair with the feature depth identification information. If the feature element three-dimensional coordinate 1201 corresponding to the feature element 412 to which the “front” label is assigned is closer to the viewpoint than the feature element three-dimensional coordinate 1201 to which the feature element 412 to which the “back” label is assigned is closer to the viewpoint. The candidate determination unit 115 determines that the matching candidate is correct. The candidate determination unit 115 outputs position and orientation information corresponding to the matching candidate determined to be correct as final position and orientation information 50.

Thus, according to the third embodiment, it is possible to accurately estimate the three-dimensional position and orientation of the measurement object even in a situation where the three-dimensional coordinates measured by the three-dimensional measurement apparatus are sparse. Become. In addition to the example described above, the following modification 11 may be applied to the third embodiment.
[Modification 11] The depth identification unit 114 estimates the feature element three-dimensional coordinates 1201 as the three-dimensional coordinates at the position where the feature element 412 is detected in the captured image 10, but the three-dimensional coordinates measured on the contour 411. 1101 may be used as it is. In this case, the candidate determination unit 115 may determine the front-rear relationship of the contour 411 instead of the feature element 412. Moreover, the template production | generation part 102 should just calculate the feature depth identification information with respect to the straight line (line segment) which comprises an outline instead of the sample point on the outline 411. FIG.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. In this embodiment, estimation of the three-dimensional position and orientation of the measurement target will be described with respect to an apparatus that can be applied when measuring only one point of the three-dimensional coordinates related to the measurement target. In the present embodiment, an appropriate one is selected from a plurality of obtained three-dimensional position / posture candidates based on the distance between one point of three-dimensional coordinates and a known plane. This makes it possible to accurately estimate the position and orientation of the measurement target even in a situation where the three-dimensional coordinates that can be measured are one point. Note that the overall configuration of the apparatus in the fourth embodiment is the same as the configuration in the third embodiment shown in FIG. Below, a different part from 3rd Embodiment is demonstrated and description is abbreviate | omitted about the part similar to 3rd Embodiment.

  The three-dimensional measuring unit 104 measures the three-dimensional coordinates of the surface of the measurement object 201 and outputs the measured three-dimensional coordinates 1101 to the three-dimensional coordinate input unit 118. In the present embodiment, as an example, a three-dimensional coordinate 1101 is acquired for one point on the surface of the measurement target 201 using a spot projection three-dimensional measurement device as the three-dimensional measurement unit 104. The depth identifying unit 114 identifies the depth context of the feature elements included in the template 20 based on the distance between one point of the three-dimensional coordinates 1101 and the plane 202.

  FIG. 13 is a diagram for explaining a relationship among the imaging unit 101, the three-dimensional measurement unit 104, and the measurement object 201 in the fourth embodiment. A plurality of measurement objects 201 are randomly arranged, and the imaging unit 101 acquires a captured image 10 obtained by bird's-eye view from above. In addition, the three-dimensional measurement unit 104 projects spot light onto the measurement object 201 and acquires a three-dimensional position on the projection spot as three-dimensional coordinates 1101. The individual measurement objects 201 take various positions and orientations, but rarely overlap each other, and generally exist on the plane 202. The distance from the three-dimensional measuring unit 104 to the plane 202 is measured in advance. This is realized by measuring the three-dimensional coordinate 1101 without placing the measurement object 201 and holding the value.

  The operation in the fourth embodiment will be described. The processing flow of the apparatus in the fourth embodiment is the same as that of the third embodiment. However, there are steps whose processing contents are different from those of the third embodiment, which will be described with reference to FIGS. 3 (a) and 3 (b).

  In step S <b> 313 in FIG. 3B, the template generation unit 102 extracts the feature element 412 from the template image 30. At this time, sampling is performed not only on the pixels on the contour 411 of the measurement object 201 but also on the pixels on the surface.

  In step S314, the template generation unit 102 calculates feature depth identification information for the feature element 412 extracted in step S313. The template generation unit 102 calculates the feature depth identification information as follows. When sampling the feature element 412 of the measurement object 201, the template generation unit 102 obtains a depth value by referring to the depth buffer that holds the depth value in step S312. When a predetermined number of samplings are completed for one template image 30, the same number of depth values as the number of feature elements are obtained. The template generation unit 102 converts these depth values into distances from the innermost surface. In FIG. 13, since the measurement object 201 is placed on the plane 202, the measurement object 201 observed from the viewpoint is in contact with the plane 202 at the innermost surface. The template generation unit 102 calculates the distance between the depth of the sampled feature element 412 and the innermost surface (that is, the surface in contact with the plane 202 when the measurement object 201 is placed on the plane 202). Write to the template 20 as depth identification information.

  Further, in step S302 in FIG. 3A, the imaging unit 101 captures the captured image 10 of the scene including the measurement target 201, and the three-dimensional measurement unit 104 determines the three-dimensional coordinates 1101 on the measurement target 201. Measure. The imaging unit 101 outputs the captured image 10 to the captured image input unit 111, and the three-dimensional measurement unit 104 outputs the three-dimensional coordinates 1101 to the three-dimensional coordinate input unit 118. The captured image input unit 111 supplies the input captured image 10 to the candidate search unit 113 and the depth identification unit 114. Further, the three-dimensional coordinate input unit 118 supplies the input three-dimensional coordinates 1101 to the depth identification unit 114.

  In step S305, the depth identifying unit 114 estimates the feature element three-dimensional coordinates 1201 in the captured image 10 with reference to the three-dimensional coordinates 1101 and the template 20. The depth identifying unit 114 first extracts one matching candidate from the matching candidate list 40. Next, the depth identifying unit 114 searches for the nearest feature element 412 of the image coordinates of the spot light detected on the captured image 10 based on the position and orientation defined in the template element. The depth identifying unit 114 calculates the distance between the three-dimensional coordinate 1101 and the plane 202 with reference to the input three-dimensional coordinate 1101 and the previously calculated distance between the three-dimensional measurement unit 104 and the plane 202.

  In step S <b> 306, the candidate determination unit 115 determines the correctness of the matching candidates included in the matching candidate list 40 and outputs the final position / orientation information 50. The candidate determination unit 115 compares the distance between the three-dimensional coordinates 1101-the plane 202 calculated in step S <b> 305 and the feature depth identification information described in the template 20. Candidate determination section 115 determines that the matching candidate is correct if the difference between these values is within a certain threshold value (that is, in a state in which it can be regarded as approximately equal within the threshold range). To do. The candidate determination unit 115 outputs position and orientation information corresponding to the matching candidate determined to be correct as final position and orientation information 50.

As described above, according to the fourth embodiment, it is possible to accurately estimate the three-dimensional position and orientation of the measurement object even in a situation where the number of three-dimensional coordinates that can be measured is one point. Note that the present invention is not limited to the example described above, and the following modification 12 may be applied to the fourth embodiment.
[Modification 12] In the present embodiment, the template generation unit 102 calculates the difference between the innermost surface of the measurement target 201 and the depth value of the feature element as the feature depth identification information. However, the present invention is not limited to this. A surface where the measurement object 201 is in contact with the plane 202 may be specified using physical simulation, and the difference between the surface and the depth value of the feature element may be calculated.

(Fifth embodiment)
Next, a fifth embodiment of the present invention will be described. In the present embodiment, the estimation of the three-dimensional position and orientation of the measurement target will be described with respect to an apparatus that can be applied to a case where a plurality of captured images are acquired continuously while a single imaging unit moves. In the present embodiment, a single imaging unit is attached to a drive unit such as a robot hand, and a plurality of images with different viewpoint positions are captured in the process of the drive unit continuously operating. An appropriate one is selected from a plurality of obtained three-dimensional position and orientation candidates based on the magnitude relationship of the parallax between the past captured image and the current captured image. This makes it possible to accurately estimate the position and orientation of the measurement object without providing a three-dimensional measurement apparatus.

  FIG. 14 is a diagram illustrating a configuration example of the entire apparatus according to the fifth embodiment. In FIG. 14, a rectangular frame indicates a functional module that performs each process of the present embodiment, and an arrow indicates a data flow. The apparatus includes an information processing apparatus 100 that estimates a three-dimensional position and orientation, and an imaging unit 101 and a template generation unit 102 that are connected thereto. Note that the configuration shown in FIG. 14 is an example. It is not limited to this. In the following description, parts different from those of the first embodiment will be described, and description of parts similar to those of the first embodiment will be omitted.

  The imaging unit 101 acquires a captured image 10 obtained by capturing a scene including a measurement object 201 for estimating a three-dimensional position and orientation. The imaging unit 101 is attached to a drive unit (not shown), and a plurality of captured images 10 having different viewpoint positions in time series are acquired by continuously operating the drive unit. The imaging unit 101 includes a position and orientation sensor (not shown), and acquires captured image position and orientation information 14 synchronized with the captured image 10 according to the movement of the drive unit. The imaging unit 101 outputs the acquired captured image 10 and captured image position / orientation information 14 to the captured image input unit 111 of the information processing apparatus 100. In the present embodiment, the imaging unit 101 does not have to have an automatic focus control mechanism therein.

  The information processing apparatus 100 according to the fifth embodiment includes a captured image input unit 111, a template input unit 112, a candidate search unit 113, a depth identification unit 114, a candidate determination unit 115, and a candidate holding unit 119. The captured image input unit 111 captures the captured image 10 and the captured image position / orientation information 14 acquired by the imaging unit 101 into the information processing apparatus 100 and supplies them to the candidate search unit 113 and the candidate determination unit 115.

  The candidate search unit 113 collates the input captured image 10 with the template 20 by template matching, generates a measurement target matching candidate list 40, and outputs it. The candidate search unit 113 always receives the latest captured image 10 and captured image position / orientation information 14 and generates a matching candidate list 40 at that time. The candidate search unit 113 also outputs the captured image 10, the captured image position / orientation information 14, and the matching candidate list 40 to the candidate holding unit 119.

  The candidate holding unit 119 holds the captured image 10, the captured image position / orientation information 14, and the matching candidate list 40 that are input or generated in the past by the candidate search unit 113. The candidate determination unit 115 receives the latest captured image 10, the captured image position / orientation information 14, and the matching candidate list 40 output from the candidate search unit 113. Further, the candidate determination unit 115 receives the past captured image 10, the captured image position / orientation information 14, and the matching candidate list 40 that are stored in the candidate storage unit 119. Based on the input information, the candidate determination unit 115 determines whether the matching candidates included in the latest matching candidate list 40 are correct or incorrect, and finally outputs only the position and orientation information corresponding to the matching candidates determined to be correct. Output as position and orientation information 50.

  The operation in the fifth embodiment will be described. FIG. 15 is a flowchart showing the flow of processing of the apparatus in the fifth embodiment. The flow of the preparation process in step S1501 is the same as the preparation process in the first embodiment shown in FIG.

  In the present embodiment, the process of step S1503 corresponding to the process of step S302 in the first embodiment is repeated many times. On the other hand, the process of step S1502 corresponding to the process of step S303 in the first embodiment may be processed only once when the apparatus is activated. Therefore, in the fifth embodiment, after the template 20 is input in step S1502, the captured image 10 and the captured image position / orientation information 14 are input in step S1503.

  In step S1503, the imaging unit 101 captures an image of a scene including the measurement target 201, and acquires position and orientation information of the imaging unit 101 in synchronization with the image. The imaging unit 101 outputs the acquired captured image 10 and captured image position / orientation information 14 to the captured image input unit 111. The captured image input unit 111 supplies the input captured image 10 and captured image position / orientation information 14 to the candidate search unit 113 and the candidate determination unit 115.

  In step S1504, the candidate search unit 113 searches for position and orientation candidates in the same manner as in the first embodiment. Next, in step S <b> 1505, the depth identifying unit 114 identifies the depth context of the feature elements included in the template 20. Note that if the candidate holding unit 119 does not hold the past captured image 10, the captured image position / orientation information 14, and the matching candidate list 40 (that is, at the time of the first execution), the processing of step S1505 and step S1506 is performed. Without proceeding to step S1508.

  In step S1504, the depth identifying unit 114 first extracts one position / orientation candidate from the matching candidate list 40 input from the candidate searching unit 113. Next, the depth identifying unit 114 inputs the current captured image 10 and the captured image position / orientation information 14 from the candidate searching unit 113. At the same time, the past captured image 10, the captured image position and orientation information 14, and the matching candidate list 40 are input from the candidate holding unit 119.

  The depth identification unit 114 refers to the feature depth identification information from the feature elements 412 defined in the template element, and includes one feature element to which the “front” label is assigned and one feature element to which the “back” label is assigned. A feature pair is constructed by extracting each one. In addition, when the feature element extracted on the present captured image 10 or the past captured image 10 is not observed, the feature element is not used. The depth identification unit 114 repeats this process to set a plurality of feature pairs.

  In addition, the depth identifying unit 114 refers to the current captured image position / orientation information 14 and the past captured image position / orientation information 14 to parallelize the past captured image 10. Thereby, the epipolar line of the present captured image 10 and the past captured image 10 is in a state parallel to the x axis. The depth identifying unit 114 compares the current captured image 10 in which the feature element 412 is detected for each feature pair with the parallax (that is, the difference between the detected x coordinates) in the past captured image 10. The comparison of the parallax between the two images may be performed by the same method as in the second embodiment.

  In step S <b> 1506, the candidate determination unit 115 determines whether the matching candidates included in the matching candidate list 40 are correct and outputs the final position / orientation information 50. The candidate determination unit 115 determines the correctness of the matching candidate based on triangulation of the two images (the current captured image 10 and the past captured image 10), and the position and orientation corresponding to the matching candidate determined to be correct. The information is output as final position and orientation information 50.

  In step S <b> 1507, the candidate holding unit 119 holds the latest matching candidate list 40, the captured image 10, and the captured image position / orientation information 14 input from the candidate search unit 113. These values are referred to when the next process of step S1506 is executed. In step S1508, based on an instruction from the user, the three-dimensional position / orientation estimation process of this embodiment is terminated, or the process returns to step S1503 to continue the process.

As described above, according to the fifth embodiment, the three-dimensional position of the measurement object can be obtained without providing a three-dimensional measurement device by continuously acquiring a plurality of captured images while the single imaging unit moves. The posture can be estimated with high accuracy. The following modified example 12 may be applied to the fifth embodiment, not limited to the example described above.
[Modification 12] In the present embodiment, the imaging unit 101 acquires the captured image position / orientation information 14 using the position / orientation sensor, but the present invention is not limited to this. The captured image input unit 111 may generate the captured image position / orientation information 14 based on the feature point coordinates detected on the captured image 10 using SfM or SLAM technology.

(Sixth embodiment)
Next, a sixth embodiment of the present invention will be described. In the present embodiment, an object gripping apparatus that can be applied when a driving unit such as a robot grips a measurement target object using the estimation result of the three-dimensional position and orientation of the measurement target object will be described. In the present embodiment, only portions different from the fifth embodiment will be described.

  In the present embodiment, a control unit (not shown) is provided outside the information processing apparatus 100 that estimates the three-dimensional position and orientation, and final target position and orientation information is output to the drive unit based on the output final position and orientation information 50. Is generated. This final target position / posture information is position / posture information that enables an end effector (not shown) of the drive unit to grip the measurement target 201. The final target position / orientation information is generated by applying predetermined three-dimensional transformation to the final position / orientation information 50. The predetermined three-dimensional transformation is defined from the relationship between the shape of the measurement target object 201 and the end effector, and the position and orientation of the drive unit most suitable for the end effector to hold the measurement target object 201 is determined as the measurement target object 201. It is expressed in the coordinate system. The predetermined three-dimensional conversion is calculated in advance before the start of processing in the present embodiment, and is input to the apparatus as a parameter at the start of processing.

  As described above, according to the sixth embodiment, the final target position / orientation information is generated by using the estimation result of the three-dimensional position / orientation of the measurement object output as the final position / orientation information 50, so that the robot or the like The drive unit can grip the measurement object.

  The “captured image input unit 111” may be any unit as long as it can input an image captured by the imaging unit 101 to the information processing apparatus 100. The input image may be any type of image such as an RGB color image, a gray scale image, a black and white image, or a depth image. The “feature element” is not limited to a point as long as it is an element detected in the image, and may be defined by an area such as a line, a circle, or a rectangle. In addition, the “feature element” can be obtained from images obtained by observing the measurement object from a plurality of viewpoints, and in each embodiment, the configuration in which the CAD model of the measurement object is rendered and generated has been described. It is not limited to this. For example, as described in Modification 2, an actual measurement object may be photographed from a plurality of directions, and a template may be generated in association with the position and orientation.

  Further, the “candidate search unit 113” is not limited to the configuration for searching for a candidate for the position and orientation of the measurement object from the input captured image. For example, as shown as Modification 6, it is possible to include a configuration in which the type of the measurement object is included in the position and orientation candidates and the type of the measurement object is searched in addition to the position and orientation. Further, the “candidate determination unit 115” determines the correctness of the matching candidate based on the detected depth relative relationship between the feature elements and the depth relative relationship between the feature elements described in the template. Various depth relative relationships can be used. Further, the “candidate determination unit 115” outputs the matching candidate determined to be correct as a result of the correctness determination of the matching candidate as final position and orientation information, but may output an intermediate result such as an evaluation value. For example, as shown in an example in Modification 8, a configuration for outputting all matching candidates, a configuration for outputting including evaluation values, a configuration for outputting evaluation values for positions and orientations, and the like can be applied.

(Other embodiments of the present invention)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  For example, the information processing apparatus shown in each embodiment described above has a computer function 1600 as shown in FIG. 16, and the CPU 1601 performs operations in each embodiment. As shown in FIG. 16, the computer function 1600 includes a CPU 1601, a ROM 1602, and a RAM 1603. Further, a controller (CONSC) 1605 of the operation unit (CONS) 1609 and a display controller (DISPC) 1606 of a display (DISP) 1610 as a display unit such as a CRT or LCD are provided. Further, a hard disk (HD) 1611, a controller (DCONT) 1607 of a storage device (STD) 1612 such as a flexible disk, and a network interface card (NIC) 1608 are provided. These functional units 1601, 1602, 1603, 1605, 1606, 1607, 1608 are configured to be communicably connected to each other via a system bus 1604.

  The CPU 1601 generally controls each component connected to the system bus 1604 by executing software stored in the ROM 1602 or the HD 1611 or software supplied from the STD 1612. That is, the CPU 1601 reads out from the ROM 1602, the HD 1611, or the STD 1612 and executes a processing program for performing the above-described operation, thereby performing control for realizing the operation in each of the above-described embodiments. The RAM 1603 functions as a main memory or work area for the CPU 1601. The CONSC 1605 controls instruction input from the CONS 1609. The DISPC 1606 controls the display of the DISP 1610. The DCONT 1607 controls access to the HD 1611 and the STD 1612 that store a boot program, various applications, user files, a network management program, and the processing program in each of the above-described embodiments. The NIC 1608 exchanges data with other devices on the network 1613 in both directions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed as being limited thereto. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

DESCRIPTION OF SYMBOLS 100: Information processing apparatus 101: Image pick-up part 102: Template production | generation part 103: 2nd image pick-up part 104: Three-dimensional measurement part 111: Captured image input part 112: Template input part 113: Candidate search part 114: Depth identification part 115: Candidate Determination unit 116: second captured image input unit 117: second candidate search unit 118: three-dimensional coordinate input unit 119: candidate holding unit 201: measurement object

Claims (11)

A photographed image input means for inputting a photographed image including the object;
A template input means for inputting information of a template having information on a feature element obtained from images obtained by observing the object from a plurality of viewpoints and a depth relative relationship between the feature elements;
Candidate search means for collating the feature elements described in the template with the feature elements detected from the photographed image and searching for matching candidates for the object included in the photographed image;
Depth identification means for identifying a depth relative relationship between at least two of the detected features;
Candidate determining means for determining whether the matching candidate is correct or not based on information on the relative depth relation obtained by the depth identifying means and information on the relative depth relation between the feature elements described in the template. A characteristic information processing apparatus.   When the candidate determination unit has the same information on the relative depth relationship in the feature element in which the matching candidate is detected from the captured image and the information on the relative depth relationship between the feature elements described in the template, The information processing apparatus according to claim 1, wherein the matching candidate is determined to be correct.   The depth identifying means identifies a depth relative relationship based on a comparison between a contrast relationship of a plurality of the captured images having different focal lengths and a positional relationship between feature elements acquired based on the template. The information processing apparatus according to claim 1 or 2.   The information processing apparatus according to claim 1, wherein the depth identifying unit identifies a depth relative relationship based on a comparison of detected coordinates of the feature elements in a plurality of the captured images. A second photographed image input means for inputting a second photographed image containing the object;
The feature element described in the template is matched with the feature element detected from the second photographed image, and the matching candidate of the object included in the second photographed image is searched from the matching candidates. The information processing apparatus according to claim 1, further comprising: a second candidate search unit that performs the processing. Having a three-dimensional measuring means for measuring the three-dimensional coordinates of the object;
The depth identifying means estimates a three-dimensional position of the detected feature element based on the three-dimensional coordinates, and identifies a depth relative relation based on the comparison of the three-dimensional positions. Or the information processing apparatus of 2.   The information processing apparatus according to claim 1, wherein the candidate determination unit determines whether the position and orientation of an object related to the matching candidate is correct or incorrect.   The information processing apparatus according to claim 1, wherein the candidate determination unit determines whether the type of an object related to the matching candidate is correct or incorrect. A photographed image input means for inputting a photographed image including the object;
A template input means for inputting information of a template having information on a feature element obtained from images obtained by observing the object from a plurality of viewpoints and a depth relative relationship between the feature elements;
Candidate search means for collating the feature elements described in the template with the feature elements detected from the photographed image and searching for matching candidates for the object included in the photographed image;
Depth identification means for identifying a depth relative relationship between at least two of the detected features;
Candidate determination means for determining whether the matching candidate is correct or not based on information on the relative depth relation obtained by the depth identification means and information on the relative depth relation between feature elements described in the template;
Drive means for gripping the object;
Control means for generating target position and orientation information of the driving means,
The candidate determination unit, when the information on the relative depth relationship in the feature element in which the matching candidate is detected from the captured image matches the information on the relative depth relationship between the feature elements described in the template, The matching candidate is determined to be correct,
The control means generates the target position and orientation information for gripping the target object based on the position and orientation information of the target object corresponding to the matching candidate determined to be correct. . A photographed image input step for inputting a photographed image including the object;
A template input step of inputting information of a template having information on a feature element obtained from images obtained by observing an object from a plurality of viewpoints and a depth relative relationship between the feature elements;
A candidate search step of performing a matching between the feature element described in the template and the feature element detected from the photographed image, and searching for a matching candidate of an object included in the photographed image;
A depth identification step for identifying a depth relative relationship between at least two of the detected features;
A candidate determining step of determining whether the matching candidate is correct or not based on the information on the relative depth relationship obtained in the depth identifying step and the information on the relative depth relationship between the feature elements described in the template. A characteristic information processing method.   The program for functioning a computer as each means of the information processing apparatus of any one of Claims 1-8.

Images