DE102020204829B4 - Modeling objects using robot-guided cameras - Google Patents

Abstract

Method for modeling an object (1) by means of a camera (3) guided by a robot (2), comprising the steps of:- aligning (S30) the robot-guided camera to a target point;- moving (S40) the robot-guided camera into various poses relative to the object while maintaining the alignment to the target point;- taking (S50) images of the object with the camera in the various poses;- storing (S50) pose data that depend on the respective pose and image data that depend on an image taken in this pose; and- determining (S60, S70) a model of the object on the basis of the stored pose and image data.

Description

The present invention relates to a method, system and computer program product for modeling an object by means of a camera guided by a robot.

The DE 10 2017 007 737 A1 relates to a method for capturing an image of a plant with a sensor device, in which the sensor device is guided around the plant with an arm of a robot.

The DE 10 2016 223 665 A1 relates to a method for determining the position of a manipulator, wherein the manipulator has a manipulator arm on which at least one camera is set up for optical recordings, wherein the method is characterized by the steps: moving the camera into the vicinity of at least one reference object by the manipulator, recording at least two images of the reference object by means of the camera from different perspectives, determining the position of the manipulator relative to the reference object based on the at least two recorded images, wherein the camera is part of a smartphone.

The DE 11 2016 002 057 T5 relates to a motorized mobile platform comprising an articulated robot arm and a triangulation scanner for performing three-dimensional measurements, wherein the robot arm and the triangulation scanner are detachably coupled with connectors.

The US 2015 / 0 000 410 A1 relates to a method and a device for scanning an object, wherein two virtual, orthogonal axes are positioned on a surface of the object, a scanning path of a moving probe is controlled depending on the two virtual, orthogonal axes, the scanning path can comprise several probe positions which are determined according to a desired coverage of the object, a single probe can be used or optionally a pair of probes or an arrangement of probes can be used, wherein optionally the probes are mounted on a multi-axis movable carrier.

The EP 2 831 539 B1 relates to a detection probe for size detection of an object by irradiating the object with light and detecting reflected light, comprising: a detection unit comprising an imaging sensor and a sensor lens arrangement for detecting the reflected light, a light projection unit comprising a light source for generating light and source optics for focusing the light and light plane generation optics for generating a light plane for irradiating the object, and an adjustment mechanism contained in the light projection unit for adjusting the position and/or orientation of the light plane relative to the detection unit so that it coincides with an object-side imaging plane of the sensor lens arrangement, and wherein the adjustment of the light plane is fixable for operational stability, wherein the sensor lens arrangement comprises a plurality of lenses, wherein all of the lenses in the sensor lens arrangement are arranged in a fixed, non-adjustable alignment with one another, the detection unit and the light projection unit are attached to a rigid base plate, the sensor lens arrangement is rigidly and non-adjustably attached to the rigid base plate, and this rigid base plate is designed for mounting the detection probe on a locator.

The EP 2 010 863 B1 relates to a scanning probe for recording data from a plurality of points on the surface of an object by irradiating the object with a strip of light and detecting the light reflected from the object surface, the scanning probe comprising: (a) a strip generating means for generating and emitting a strip of light; (b) a camera comprising an image sensor having a pixel array for detecting the light beam reflected from the object surface; (c) a means for adjusting the intensity of the light strip during the recording of a single image section in dependence on the intensities detected by the camera, wherein: the camera and/or the strip generating means are arranged such that during the recording of the image section different subgroups of the image sensor pixels detect reflected light from the strip at different times, the different subgroups corresponding to different positions on the detected light strip, the means for adjusting the light intensity is arranged such that the intensity of a light beam for a pixel subgroup is adjusted in dependence on the detected intensity of the previous pixel subgroup.

The object of the present invention is to improve the modeling of objects.

This object is achieved by a method having the features of claim 1. Claims 8, 9 protect a system or computer program product for carrying out a method described here. The subclaims relate to advantageous further developments.

• - Ausrichten der robotergeführten Kamera (mittels des Roboters) auf einen, insbesondere vorgegebenen und/oder objektfesten, Zielpunkt;
• - Bewegen der robotergeführten Kamera (mittels des Roboters) in verschiedene Posen relativ zu dem Objekt unter Beibehaltung der Ausrichtung (der Kamera) auf den Zielpunkt;
• - Aufnehmen von Bildern des Objekts mit der Kamera in diesen verschiedenen Posen;
• - Abspeichern
  • - von Posendaten, die von der jeweiligen Pose abhängen, in einer Ausführung die jeweilige Pose angeben bzw. definieren und/oder auf Basis von gemessenen Gelenk- bzw. Achsstellungen des Roboters ermittelt werden, und
  • - von Bilddaten, die von einem in dieser bzw. der jeweiligen Pose aufgenommenem Bild abhängen, in einer Ausführung auf Basis des aufgenommenen Bildes ermittelt, insbesondere berechnet, werden; und
• - Ermitteln, in einer Ausführung Berechnen, eines, in einer Ausführung zwei- oder dreidimensionalen, Modells des Objekts, in einer Ausführung Generieren eines neuen Modells oder Modifizieren eines vorhandenen Modells, auf Basis bzw. in Abhängigkeit von diesen abgespeicherten Posen und Bilddaten, wobei in einer Ausführung Bilddaten, die von einem in einer Pose aufgenommenem Bild abhängen, den Posendaten, die von dieser Pose abhängen, jeweils zugeordnet sind bzw. werden.

According to one embodiment of the present invention, a method for modeling an object using a camera guided by a robot comprises the steps:

• - Aligning the robot-guided camera (by means of the robot) to a target point, in particular a predetermined and/or object-fixed one;
• - Moving the robot-guided camera (by means of the robot) into different poses relative to the object while maintaining the orientation (of the camera) to the target point;
• - Taking pictures of the object with the camera in these different poses;
• - Save
  • - of pose data that depend on the respective pose, in one embodiment specify or define the respective pose and/or are determined on the basis of measured joint or axis positions of the robot, and
  • - from image data that depend on an image taken in this or the respective pose, are determined, in particular calculated, in an embodiment on the basis of the image taken; and
• - Determining, in one embodiment calculating, a model of the object, in one embodiment two- or three-dimensional, in one embodiment generating a new model or modifying an existing model on the basis of or depending on these stored poses and image data, wherein in one embodiment image data that depend on an image taken in a pose are or are respectively assigned to the pose data that depend on this pose.

In one embodiment, this allows a model of the object to be determined advantageously, in particular quickly and/or precisely. In one embodiment, a pose comprises a one-, two- or three-dimensional position and/or a one-, two- or three-dimensional orientation.

In one embodiment, the robot-guided camera is moved into one or more of the poses by applying forces to the robot, in one embodiment by manual application or application or manual guidance by an operator. In one embodiment, this allows a model of the object to be determined advantageously, in particular quickly and/or precisely.

Additionally or alternatively, in one embodiment, the robot-guided camera is automatically moved into one or more of the poses, in one embodiment without external forces exerted on the robot or without movement command inputs from an operator. In this way, in one embodiment, a model of the object can be advantageously determined, in particular quickly and/or precisely.

In one embodiment, when or to move the robot-guided camera into one or more of the poses (respectively), drives of the robot are controlled by means of or on the basis of the Operational Space Framework and/or on the basis of a predetermined minimum distance of the robot and/or the camera to the target point, in particular in such a way or with the proviso that the robot and/or the camera maintain a predetermined minimum distance to the target point, and/or on the basis of predetermined joint limits of the robot, in particular in such a way or with the proviso that the robot maintains predetermined joint limits. In one embodiment, this allows a model of the object to be determined advantageously, in particular quickly and/or precisely.

The Operational Space Framework uses an (operational) coordinate system to describe the position and orientation of an operational point in a reference system as a function of a task, whereby the space of this task is referred to as the operational space. Accordingly, only the coordinates required to describe the task need to be specified in the operational space.

Tasks can be described in particular in the Cartesian or workspace and/or in the joint coordinate space of the robot.

In particular, multiple simultaneous tasks can be implemented by multiple operational spaces. Additionally or alternatively, tasks in an execution can be described as operational forces. These operational forces can be used, for example, as forces to move an end effector into a target position and to drive or hold joints away from their joint limits. These operational forces can be mapped to forces, in particular (torsional) moments, in the joint coordinate space using a Jacobi matrix in order to control the robot, whereby in this case a regulation is also generally referred to as control.

mit dem Vektor der (virtuellen) operationellen Kräfte im Operational Space f_t, der symmetrischen Trägheits- bzw. Massenmatrix Λ_t(x_t), dem Vektor der Zentrifugal- und Corioliskräfte µ_t(x_t,ẋ_t) und dem Vektor der Gravitationskräfte p_t(x_t) (alle im Operational Space) beschrieben werden. Ergänzend wird zum Operational Space Framework auf Khatib, O., A unified approach for motion and force control of robot manipulators: The operational space formulation, IEEE Journal on Robotics and Automation, 1987, 3(1), S. 43-53 , Bezug genommen und der Inhalt dieses Artikels vollumfänglich in die vorliegende Offenbarung einbezogen.The dynamic behavior of a point (“operational point”) in the so-called operational space can be determined in particular by $$f_t={\Lambda }_t\left(x_t\right){\ddot{x}}_t+{\mathrm{\mu }}_t\left(x_t,{\dot{x}}_t\right)+p_t\left(x_t\right)$$

with the vector of (virtual) operational forces in the operational space f _t , the symmetric inertia or mass matrix Λ _t (x _t ), the vector of centrifugal and Coriolis forces µ _t (x _t ,ẋ _t ) and the vector of gravitational forces p _t (x _t ) (all in the operational space). In addition to the operational space framework, Khatib, O., A unified approach for motion and force control of robot manipulators: The operational space formulation, IEEE Journal on Robotics and Automation, 1987, 3(1), pp. 43-53 , and the contents of this article are fully incorporated into the present disclosure.

mit der Trägheits- bzw. Massenmatrix M(q), dem Vektor der Zentrifugal- und Corioliskräfte c(q,q̇), dem Vektor der Gravitationskräfte g(q) und den generalisierten Kräften, insbesondere Antriebskräften, τ im Gelenkkoordinatenraum. Antiparallele Kräftepaare bzw. (Dreh)Momente werden vorliegend verallgemeinernd ebenfalls als Kräfte bezeichnet.In the joint coordinate space, the dynamic behavior of the robot can be described in particular by: $$\tau =M\left(q\right)\ddot{q}+c\left(q,\dot{q}\right)+g\left(q\right)$$

with the inertia or mass matrix M(q), the vector of the centrifugal and Coriolis forces c(q,q̇), the vector of the gravitational forces g(q) and the generalized forces, in particular driving forces, τ in the joint coordinate space. Antiparallel force pairs or (torsion) moments are also generally referred to as forces.

Using the Jacobi matrix J _t , the relationship between the operational forces f _t and the joint or drive forces, in particular torques τ, is: $$\tau =J_t^T{f}_t$$

hinzugefügt werden, ohne die resultierenden Kräfte am operational point zu beeinflussen: $$\tau =J_t^T{f}_t+N_t{\tau }_0$$

mit $$N_t=I-J_j^T{\overline{J}}_t^T$$

mit der Projektion N_t in den Nullraum der (Haupt)Aufgabe, wobei τ₀ ein beliebiger generalisierter Gelenk-, insbesondere Antriebskraft-, insbesondere -momentenvektor ist, der in den Nullraum der (Haupt)Aufgabe projiziert wird, und ${\overline{J}}_t^T$

die dynamisch konsistente generalisierte Inverse („dynamically consistent generalized inverse“), die die momentane kinetische Energie des Roboters minimiert: $${\overline{J}}_t=M^{-1}{J}_j^T{\Lambda }_t$$

If the robot is redundant with respect to the task or the dimension of the minimum or joint coordinates q is larger than the dimension of the task coordinates x _t (dim(q) > dim(x _t )), movements or (drive) forces or moments in the null space of $J_t^T,$

added without affecting the resulting forces at the operational point: $$\tau =J_t^T{f}_t+{\mathrm{<figure-callout\; id="0"\; label="N\; t\; \tau "\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">N</figure-callout>}}_t{\tau }_{}{}_0$$

with $$N_t=I-J_j^T{\overline{J}}_t^T$$

with the projection N _t into the null space of the (main) task, where τ ₀ is an arbitrary generalized joint, in particular driving force, in particular moment vector, which is projected into the null space of the (main) task, and ${\overline{J}}_t^T$

the dynamically consistent generalized inverse, which minimizes the instantaneous kinetic energy of the robot: $${\overline{J}}_t=M^{-1}{J}_j^T{\Lambda }_t$$

$${\mathrm{\mu }}_t={\overline{J}}_t^{T}c-{\Lambda }_t{\dot{J}}_t\dot{q}$$

$$p_t={\overline{J}}_t^{T}g$$

This results in: $${\Lambda }_t={\left(J_t{M}^{-1}{J}_t^T\right)}^{-1}$$

$${\mathrm{\mu }}_t={\overline{J}}_t^{T}c-{\Lambda }_t{\dot{J}}_t\dot{q}$$

$$p_t={\overline{J}}_t^{T}g$$

wobei der Index von der höchsten Priorität (Index₁) zur niedrigsten Priorität (Index_k) läuft, d.h. τ₁ der Kraftvektor für die Aufgabe mit der höchsten Priorität, die ausgeführt wird, sofern die Kinematik bzw. Dynamik des Roboters dies gestattet, und τ_k der Kraftvektor für die Aufgabe mit der niedrigsten Priorität ist, die nur vollständig durchgeführt wird, wenn genug Koordinaten im Nullraum zur Verfügung stehen.The generalized joint, in particular driving force, in particular moment vector τ ₀ in (4) can be used for a second or lower-ranking task, which is (only) fully executed in the null space if it does not collide or interfere with the higher-ranking task, which is carried out by the force vector f _t , i.e. which has a higher priority. This makes it possible to realize a prioritization or prioritized structure for several tasks: $$\begin{array}{c}{\mathrm{<figure-callout\; id="1"\; label="\tau "\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">\tau </figure-callout>}}_1=J_1^{\mathrm{<figure-callout\; id="1"\; label="T\; f"\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">T</figure-callout>}}{f}_{}{}_1+{\mathrm{<figure-callout\; id="1"\; label="N"\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">N</figure-callout>}}_1{\mathrm{<figure-callout\; id="2"\; label="\tau "\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">\tau </figure-callout>}}_2\\ {\tau }_2=J_1^{\mathrm{<figure-callout\; id="2"\; label="T\; f"\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">T</figure-callout>}}{f}_{}{}_2+{\mathrm{<figure-callout\; id="2"\; label="N"\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">N</figure-callout>}}_2{\mathrm{<figure-callout\; id="3"\; label="\tau "\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">\tau </figure-callout>}}_3\\ ⋮\\ {\tau }_{k-1}=J_{k-1}^T{f}_{k-1}+N_{k-1}{\tau }_k\\ {\tau }_k=J_k^T{f}_k\end{array}$$

where the index runs from the highest priority (index ₁ ) to the lowest priority (index _k ), i.e. τ ₁ is the force vector for the task with the highest priority, which is executed if the kinematics or dynamics of the robot allow it, and τ _k is the force vector for the task with the lowest priority, which is only completely executed if enough coordinates are available in the null space.

mit der Differenz Δx_E zwischen der gewünschten bzw. Soll-Position und der aktuellen kartesischen Position des Endeffektors (x_E,d - x_E), der Massen-Matrix M , Dämpfungs-Matrix D und Steifigkeits- bzw. Federmatrix K des virtuellen Feder-Dämpfer-Systems, wobei M insbesondere aufgrund von Rauschen gleich Null gesetzt werden kann, so dass: $$f_{pos}^{*}=k_p\left(x_{E,d}-x_E\right)+k_d\left({\dot{x}}_{E,d}-{\dot{x}}_E\right),$$

mit dem kommandierten Kraftvektor $f_t^{*},$

der der Aufgabe entspricht bzw. diese repräsentiert, der Federkonstanten k_p und der Dämpfungskonstanten k_d.The task of keeping the end effector or a fixed point, in particular a viewing or alignment point, in particular a focus point of the camera, at the Cartesian (target) position x _E = [x,y,z] ^T , can be implemented in particular by a virtual spring-damper system: $$f*=M\Delta {\ddot{x}}_E+D\Delta {\dot{x}}_E+K\Delta x_E,$$

with the difference Δx _E between the desired or target position and the current Cartesian position of the end effector (x _E,d - x _E ), the mass matrix M , damping matrix D and stiffness or spring matrix K of the virtual spring-damper system, where M can be set equal to zero, in particular due to noise, so that: $$f_{pos}^{*}=k_p\left(x_{E,d}-x_E\right)+k_d\left({\dot{x}}_{E,d}-{\dot{x}}_E\right),$$

with the commanded force vector $f_t^{*},$

which corresponds to or represents the task, the spring constant k _p and the damping constant k _d .

The corresponding driving forces or moments for commanding or controlling the robot are obtained by projection into the joint coordinate space: $${\tau }_{pos}=J_{pos}^T\left({\Lambda }_{pos}{f}_{pos}^{*}+{\mathrm{\mu }}_{pos}+p_{pos}\right).$$

In one embodiment, one or more (target) views of the object to be captured, in one embodiment, one or more (target) views that are required for a complete capture or representation of all (currently) visible surfaces of the object are determined, and one or more of the poses are determined on the basis of these determined view(s). In one embodiment, these determined poses are automatically approached or the robot-guided camera is automatically moved into these poses. In one embodiment, this allows a model of the object to be advantageously determined, in particular quickly and/or precisely.

In one embodiment, the target point is determined on the basis of at least one image taken with the camera, in one embodiment on the basis of a center, in one embodiment on the basis of a center of gravity, of a point cloud that is determined on the basis of the image(s) taken. In one embodiment, this allows a model of the object to be determined advantageously, in particular quickly and/or precisely.

In one embodiment, when determining image data, a background in the recorded image is at least partially hidden, in one embodiment by means of segmentation, for example a RANSAC-based plane fitting. In one embodiment, this allows a model of the object to be advantageously determined, in particular quickly and/or precisely.

The camera is either a 2D or 3D camera, and the images taken are respectively 2D or 3D images of the object.

In one implementation, limited hand guidance can be used, maintaining orientation to the target point or keeping the object in the view of the camera. The operator can move the robot manually and the robot automatically adjusts the orientation, always keeping the object in the center of the camera image and maintaining the minimum distance to the object.

In one embodiment, movements around the object can be planned and automated in such a way that only one Cartesian point (the flange or the wrist) needs to be programmed and the orientation is automatically adjusted.

• - der Bediener muss nur den Ort des Objekts bzw. den Zielpunkt definieren, was auch automatisiert mittels 3D-Bildverarbeitung erfolgen kann;
• - Arbeitsraumbeschränkungen können automatisch bzw. implizit berücksichtigt bzw. eingehalten werden. Dies kann Programmieraufwand, insbesondere -zeit sparen, indem die Komplexität der Planung reduziert wird;
• - verwendete Robotertrajektorien können abgespeichert und wieder verwendet werden, was eine Offline-Programmierung unnötig machen kann;
• - der Bediener muss sich nicht um kollisionsfreie Bahnplanung kümmern, indem Gelenkbegrenzungen festgelegt werden.

The invention may, in one embodiment, offer one or more of the following advantages:

• - the operator only has to define the location of the object or the target point, which can also be done automatically using 3D image processing;
• - Workspace restrictions can be automatically or implicitly taken into account or complied with. This can save programming effort, especially time, by reducing the complexity of planning;
• - used robot trajectories can be saved and reused, which can make offline programming unnecessary;
• - the operator does not have to worry about collision-free path planning by setting joint limits.

Using the robot can improve the accuracy of modeling because the camera poses are determined using very precise forward kinematics and depend only on the camera calibration, which can also be very precise.

A means in the sense of the present invention can be designed in terms of hardware and/or software, in particular a processing unit, in particular a microprocessor unit (CPU), graphics card (GPU) or the like, preferably connected to a memory and/or bus system for data or signals, in particular digital, and/or one or more programs or program modules. The processing unit can be designed to process commands that are implemented as a program stored in a memory system, to detect input signals from a data bus and/or to output output signals to a data bus. A memory system can have one or more, in particular different, storage media, in particular optical, magnetic, solid-state and/or other non-volatile media. The program can be designed in such a way that it embodies or is capable of carrying out the methods described here, so that the processing unit can carry out steps of such methods and thus in particular model an object, in particular command or move the camera-operating robot accordingly. In one embodiment, a computer program product can have, in particular be, a storage medium, in particular a non-volatile one, for storing a program or with a program stored thereon, wherein executing this program causes a system or a controller, in particular a computer, to carry out a method described here or one or more of its steps.

In one embodiment, one or more, in particular all, steps of the method are carried out completely or partially automatically, in particular by the system or its means.

In one embodiment, the system comprises the robot.

• 1: ein System beim Modellieren eines Objekts mittels einer durch einen Roboter geführten Kamera nach einer Ausführung der vorliegenden Erfindung; und
• 2: ein Verfahren zum Modellieren des Objekts mittels der durch den Roboter geführten Kamera nach einer Ausführung der vorliegenden Erfindung.

Further advantages and features emerge from the subclaims and the embodiments. This shows, partly schematically:

• 1 : a system for modeling an object using a camera guided by a robot according to an embodiment of the present invention; and
• 2 : a method for modeling the object by means of the camera guided by the robot according to an embodiment of the present invention.

1 shows a system when modeling an object 1 using a camera 3 guided by a robot 2 according to an embodiment of the present invention. The robot is controlled by a controller 4, which carries out the steps of the method, in particular commands the robot 2 or its drives.

In 1 a robot base-fixed coordinate system 0 is indicated by its x- and z-coordinate axes °x, °z and a robot end-effector-fixed coordinate system EE is indicated by its y- and z-coordinate axes ^EE y, ^EE z.

When guiding by hand, the operator should be able to change the orientation, whereby a target point in the center of the object 1 is held in the coordinate system EE fixed to the robot end effector in such a way that the robot end effector or the camera 3 attached to it can also move in ^{the EE} z-direction, ie is also "free" in this direction. For this purpose, the Jacobi matrix and the Cartesian positions and velocities are described in the coordinate system fixed to the robot end effector and the row of the Jacobi matrix in the z-direction is deleted:

1. 1. Berechnen der Drehung ⁰R_E vom roboterbasisfesten Koordinatensystem 0 in das roboterendeffektorfeste Koordinatensystem EE unter Verwendung der Vorwärtskinematik bzw. -transformation des Roboters.
2. 2. Rotieren der Jacobi-Matrix unter Verwendung der Gleichung: $${}^{EE}{J}_{pos}=\left[\begin{array}{cc}{}^0{R}_E^T& 0\\ 0& {}^0{R}_E^T\end{array}\right]{}^{\text{0}}{J}_{pos}$$
   
3. 3. Verwendung der ersten und zweiten Zeile von ^EEJ_pos zum Definieren der Aufgabe.
4. 4. Bestimmen des Kommando-Vektors im roboterendeffektorfesten Koordinatensystem: $$f_{pos}^{*}=k_p\left({}^{EE}{x}_{E,d}{-}^{EE}{x}_E\right)+k_d\left({}^{EE}{\dot{x}}_{E,d}{-}^{EE}{\dot{x}}_E\right)$$
   
5. 5. Berechnen des Kommando-Drehmoments für die Aufgabe: $${\tau }_{pos}{=}^{EE}{J}_{pos}^T\left({\Lambda }_{pos}{f}_{pos}^{*}+{\mu }_{pos}+p_{pos}\right).$$
   

If the Jacobian matrix is initially described in the robot-based coordinate system 0, the Jacobian matrix must be rotated into the robot end-effector coordinates. To do this, the following steps are carried out:

1. 1. Calculate the rotation ⁰ R _E from the robot base-fixed coordinate system 0 into the robot end-effector-fixed coordinate system EE using the forward kinematics or transformation of the robot.
2. 2. Rotate the Jacobian matrix using the equation: $${}^{EE}{J}_{pos}=\left[\begin{array}{cc}{}^0{R}_E^T& 0\\ 0& {}^0{R}_E^T\end{array}\right]{}^{\text{0}}{J}_{pos}$$
   
3. 3. Use the first and second lines of ^EE J _pos to define the task.
4. 4. Determining the command vector in the robot end effector-fixed coordinate system: $$f_{pos}^{*}=k_p\left({}^{EE}{x}_{E,d}{-}^{EE}{x}_E\right)+k_d\left({}^{EE}{\dot{x}}_{E,d}{-}^{EE}{\dot{x}}_E\right)$$
   
5. 5. Calculate the command torque for the task: $${\tau }_{pos}{=}^{EE}{J}_{pos}^T\left({\Lambda }_{pos}{f}_{pos}^{*}+{\mu }_{pos}+p_{pos}\right).$$
   

Here, ^EE x _E,d , ^EE x _E , ^EE ẋ _E are [2x1] vectors in the x and y coordinates of the end effector, ^EE x _E,d is a [2×1] zero vector, and Λ _pos , µ _pos and p _pos are calculated using only the first two rows of ^EE J _pos .

$${\tau }_2={\tau }_{damp}$$

mit ${\tau }_{damp}=J_{damp}^T\left(M{f}_{pos}^{*}+c+g\right),\text{ wobei }{J}_{damp}^T$

eine [nxn]-Identitäts-Matrix mit der Anzahl n der Freiheitsgrade des Roboters ist.A joint damping task with lowest priority is used to dissipate energy and keep the system passive. This results in the final stack of tasks: $${\tau }_{stack}={\tau }_{pos}+N_{pos}{\mathrm{<figure-callout\; id="2"\; label="\tau "\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">\tau </figure-callout>}}_2$$

$${\mathrm{<figure-callout\; id="2"\; label="\tau "\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">\tau </figure-callout>}}_2={\tau }_{damp}$$

with ${\tau }_{damp}=J_{damp}^T\left(M{f}_{pos}^{*}+c+g\right),\text{ wobei }{J}_{damp}^T$

is an [nxn] identity matrix with the number n of degrees of freedom of the robot.

• • eine vorgegebene minimale Distanz der Kamera zu dem Zielpunkt bzw. Objekt einzuhalten bzw. nicht zu unterschreiten; und
• • vorgegebene Gelenkgrenzen einzuhalten bzw. nicht zu überschreiten, vorzugsweise zu vermeiden.

Two higher priority tasks are included:

• • to maintain or not to fall below a specified minimum distance between the camera and the target point or object; and
• • to comply with or not to exceed specified joint limits, preferably to avoid them.

The highest priority is assigned to the task of maintaining or not exceeding the joint limits, preferably avoiding them, and is calculated, for example, using a saturation in the joint coordinate space (“Saturation in Joint Space algorithm”, SJS). In addition, Osorio, Juan D. Muñoz; Fiore, Mario D.; Allmendinger, Felix: Operational Space Formulation Under Joint Constraints, ASME 2018 International Design Engineering Technical Conferences and Computers and Infor mation in Engineering Conference, American Society of Mechanical Engineers, 2018. Pages V05BT07A022-V05BT07A022 reference and the contents of this article are fully incorporated into the present disclosure.

The task of maintaining the given minimum distance can be implemented as a Cartesian constraint, for example using analogous Saturation in Cartesian Space algorithm (SCS) and given the second highest priority (by using the rotated Jacobian matrix in SCS it is possible to constrain the coordinates along the axis from the camera to the object). Thus, τ _stack is sent to the SCS algorithm and the torque output of the SCS is sent to the SJS algorithm.

This allows the operator to manually guide the robot while it performs the main task of maintaining the camera alignment on the target point.

wobei: $$v=\text{min}\left(\mathrm{1,}\frac{V_{max}}{\sqrt{{\dot{X}}_{op,d}^T{\dot{X}}_{op,d}}}\right)$$

und $${\dot{X}}_{op,d}=\frac{k_p}{k_d}\left(X_{op,d}-X_{op}\right)$$

In the null space of this main task, a further task can be defined to approach a Cartesian point at a given maximum speed, preferably as described in the above-mentioned article by Khatib. The operational point moves in a straight line to a desired point at maximum speed, except in the acceleration and deceleration sections. The force is calculated as follows: $$f_{goToPoint}^{*}=-k_d\left({\dot{X}}_{op}-v{\dot{X}}_{op,d}\right),$$

where: $$v=\text{min}\left(\mathrm{1,}\frac{V_{max}}{\sqrt{{\dot{X}}_{op,d}^T{\dot{X}}_{op,d}}}\right)$$

and $${\dot{X}}_{op,d}=\frac{k_p}{k_d}\left(X_{op,d}-X_{op}\right)$$

wobei J_goToPoint bis zum operationellen Punkt ausgedrückt bzw. entwickelt ist und nur die Zeilen der Positionskoordinaten verwendet werden. Der neue Aufgabenstapel („stack of tasks“) ist: $${\tau }_{stack}={\tau }_{pos}{N}_{pos}{\tau }_2$$

$${\tau }_2={\tau }_{goToPoint}+N_{goToPoint}{\tau }_3$$

$${\tau }_3={\tau }_{damp}$$

The index op indicates that these positions and speeds are those of the operational point, which for example in the LBR iiwa can be the sixth joint axis. The end effector cannot be used for this task to avoid conflicts with the main task. The commanded torques for this task are: $${\tau }_{goToPoint}=J_{goToPoint}^T\left({\Lambda }_{goToPoint}{f}_{goToPoint}^{*}+{\mathrm{\mu }}_{goToPoint}+p_{goToPoint}\right).$$

where J _goToPoint is expressed or expanded to the operational point and only the rows of position coordinates are used. The new stack of tasks is: $${\tau }_{stack}={\tau }_{pos}{N}_{pos}{\mathrm{<figure-callout\; id="2"\; label="\tau "\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">\tau </figure-callout>}}_2$$

$${\mathrm{<figure-callout\; id="2"\; label="\tau "\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">\tau </figure-callout>}}_2={\tau }_{goToPoint}+N_{goToPoin\mathrm{<figure-callout\; id="3"\; label="t\; \tau "\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">t</figure-callout>}}{\tau }_{}{}_3$$

$${\mathrm{<figure-callout\; id="3"\; label="\tau "\; filenames="DE102020204829B4\_0030.png"\; state="\{\{state\}\}">\tau </figure-callout>}}_3={\tau }_{damp}$$

Now the operational point moves to the desired point while the end effector points to a target point or a target position. This improves programming because only the desired points, for example for the sixth joint axis, need to be specified and the operator does not have to worry about the desired orientation. The end effector or camera always points to the target point or object or is aligned with it. This task stack allows robot movements to be carried out or programmed while images are being taken for modeling. The desired Cartesian points can also be predefined or entered using the hand guidance described.

1. S10: das Objekt 1 wird (bzw. ist) auf einem Tisch 5 oder einer anderen flachen Oberfläche platziert, so dass das gesamte Objekt in dem aufgenommenen zwei- oder dreidimensionalen Bild sichtbar ist.
2. S20: unter Verwendung eines Segmentierungs-Algorithmus, z.B.. RANSAC-basiertem Ebenen-Fitting, wird die Tisch- bzw. flache Oberfläche aus der 3D-Punktewolke entfernt, so dass Daten nur noch 3D-Punkte enthalten, die zu dem Objekt 1 gehören.
3. S30: ein 3D-Zentrum des Objekts wird berechnet und als Zielposition bzw. -punkt für die (Ausrichtung der) Kamera 3 verwendet.
4. S40: ein Bediener (nicht dargestellt) führt den Roboter 2 per Hand um das Objekt und versucht, dessen gesamte (aktuell sichtbare) Oberfläche zu erfassen.
5. S50: zu verschiedenen Zeitpunkten wird die aktuelle Punktewolke (die aus den Bildern der Kamera ermittelt wird) und die zugehörige bzw. aktuelle Kamerapose im Welt- bzw. Roboterbasiskoordinatensystem abgespeichert.
6. S60: alle abgespeicherten Punktewolken werden aggregiert, beispielsweise mittels TSDF, wie es in Curles, B., Levoy, M.: A Volumetric Method for Building Complex Models from Range Images, 1996 beschrieben ist, worauf ergänzend Bezug genommen und dessen Inhalt vollumfänglich in die vorliegende Offenbarung einbezogen wird.
7. S70: aus der resultierenden Punktewolke wird ein Polygongitter („polygon mesh“) berechnet und, gegebenenfalls nachbearbeitet, als Modell des Objekts 1 verwendet bzw. bereitgestellt, beispielsweise abgespeichert.

The procedure for modeling object 1 comprises the following steps (see 2 ):

1. S10: the object 1 is placed on a table 5 or other flat surface so that the entire object is visible in the recorded two- or three-dimensional image.
2. S20: using a segmentation algorithm, e.g. RANSAC-based plane fitting, the table or flat surface is removed from the 3D point cloud so that data only contains 3D points belonging to object 1.
3. S30: a 3D center of the object is calculated and used as target position or point for the (orientation of) camera 3.
4. S40: an operator (not shown) guides the robot 2 manually around the object and attempts to capture its entire (currently visible) surface.
5. S50: At different times, the current point cloud (which is determined from the camera images) and the corresponding or current camera pose are saved in the world or robot base coordinate system.
6. S60: all stored point clouds are aggregated, for example by means of TSDF, as described in Curles, B., Levoy, M.: A Volumetric Method for Building Complex Models from Range Images, 1996, to which reference is made by way of addition and the content of which is fully incorporated into the present disclosure.
7. S70: A polygon mesh is calculated from the resulting point cloud and, after further processing if necessary, is used as a model of object 1 or is made available, for example saved.

Claims (9)

Method for modeling an object (1) by means of a camera (3) guided by a robot (2), comprising the steps: - Aligning (S30) the robot-guided camera to a target point; - Moving (S40) the robot-guided camera into various poses relative to the object while maintaining the alignment to the target point; - Recording (S50) images of the object with the camera in the various poses; - Storing (S50) pose data that depend on the respective pose and image data that depend on an image taken in this pose; and - Determining (S60, S70) a model of the object on the basis of the stored pose and image data. procedure according to claim 1 , characterized in that the robot-guided camera is moved into at least one of the poses by applying forces to the robot. Method according to one of the preceding claims, characterized in that the robot-guided camera is automatically moved into at least one of the poses. Method according to one of the preceding claims, characterized in that when moving the robot-guided camera into at least one of the poses, drives of the robot are controlled on the basis of the Operational Space Framework and/or a predetermined minimum distance of the robot and/or the camera to the target point and/or predetermined joint limits of the robot. Method according to one of the preceding claims, characterized in that at least one view of the object to be captured is determined and at least one of the poses is determined on the basis of this determined view. Method according to one of the preceding claims, characterized in that the target point is determined on the basis of at least one image taken with the camera (S30). Method according to one of the preceding claims, characterized in that when determining image data, a background in the recorded image is at least partially hidden (S20). System (4) arranged to carry out a method according to one of the preceding claims. Computer program product with a program code stored on a computer-readable medium for carrying out a method according to one of the preceding claims.

Images