DE102009009569B4 - Method for determining a partial surface of a component - Google Patents

Abstract

Method for determining at least a partial surface of a component located in a container as an engagement surface (18) for attaching a gripping unit, in which
a) an image of the component is captured by means of a sensor, which comprises a three-dimensional cloud (10) with a plurality of captured image points (12);
b) the image is examined by means of an evaluation unit for the presence of the at least one partial surface of the component that can be used as an attack surface (18), pixels (12) being assigned to at least one three-dimensional grid cell (14) and the grid cell (14) using the pixels (12) ) an area piece (16) of the at least one partial area is determined as the contact area (18);
c) at least one gripping point for attaching a gripping unit as a function of a size of a contact surface of the gripping unit is determined on the engagement surface (18).

Description

The invention relates to a method for determining at least one partial surface, in particular an attack surface for attaching a gripping unit, of a component which is located in a container. In this method, an image of the component is captured by means of a sensor, which comprises a three-dimensional cloud with a plurality of captured image points. The image is examined by means of an evaluation unit for the presence of the at least one partial surface, in particular the attack surface, of the component.

The determination of partial areas of a component located in a container is of particular interest if this partial area is to serve as a potential contact surface for attaching a gripping unit, so that a component can be specifically removed from the container, for example a lattice box, by means of the gripping unit.

The PCT registration WO 2008/014960 A1 describes a process for automated 3D object detection and position determination. The method is based on a 3D measurement data record ("point cloud") of a work area in which (at least) one object is located. With the aid of the method, the object located in the area is detected and its position is recognized. For this purpose, geometrical elements that represent a “representation” of the object to be selected and are used to identify the object and determine its position are adapted to the 3D measurement data set; this is done, for example, using a best fit process. A treatment device, for example a gripper for grasping the object, can be controlled on the basis of the complete information regarding the position of the object to be selected in the work area.

The PCT registration WO 2007/083039 A2 describes a method for automated gripping of components that are located in a work area. Based on a 3D measurement data record (“point cloud”) of the work area, the components are recognized based on their 3D CAD data, and a virtual 3D CAD model of the work area is created. In this 3D CAD model of the work area, the component to be gripped virtually is determined and the virtual gripping positions are determined. The virtual gripping positions are transmitted to a robot that grips the real component with a real gripper.

The patent US 5,978,504 A discloses a method for segmenting a point cloud from 3D measurement data and for recognizing and extracting flat surfaces from this point cloud. Object detection is mentioned as a possible field of application of this method, in which the areas determined from the measurement data are adapted to a model of the object to be recognized.

The writing by Richtsfeld, M; Zillich, M .: "Grasping Unknown Objects Based on 2½ D Range Data" in 4th IEEE Conference on Automotive Science and Engineering 2008, pp. 691-696 abstracts, Sect. I u. V deals with the problem of calculating gripping points from a point cloud of 2½-dimensional measuring points. For this purpose, regions with large curvature are detected in the point cloud, which are identified as edges and mapped in the form of lines. Based on these border lines, the topology of the associated objects is determined and gripping points are defined.

The writing by Fransens, J., Reeth, F. in IEEE Proceedings of third International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT06), 2006, pp. 591-598 Abstr., Para. 4 u. 4: "Hierarchical PCA Decomposition of Point Clouds" describes a process for extracting flat partial areas from a 3D point cloud using the main component analysis. The method is based on a segmentation of the 3D point cloud with the help of an iterative tree structure (octree) and then grouping the extracted flat partial areas into larger flat areas.

It is known from the prior art to use a sensor to capture an image of the component which comprises a three-dimensional cloud with a plurality of captured pixels. However, evaluating this three-dimensional cloud of pixels for the presence of the at least one partial area is associated with a very high computing effort in methods known from the prior art.

Therefore, the examination for the presence of the attack surface is particularly time-consuming. A sufficiently short cycle time, sufficient for efficient handling or further processing, for gripping the component by means of the gripping unit cannot be achieved with such a method. In addition, known methods in which the three-dimensional cloud is evaluated at image points generally do not meet the requirements for permissible tolerances and the accuracy of the determination of the attack surface is unsatisfactory.

The object of the present invention is therefore to provide an improved method of the type mentioned at the outset.

This object is achieved by a method having the features of patent claim 1. As advantageous configurations with appropriate Developments of the invention are specified in the dependent claims.

• a) mittels eines Sensors ein Bild des Bauteils erfasst wird, welches eine dreidimensionale Wolke mit einer Mehrzahl von erfassten Bildpunkten umfasst, und bei welchem
• b) das Bild mittels einer Auswerteeinheit auf ein Vorliegen der wenigstens einen Teilfläche, insbesondere Angriffsfläche, des Bauteils untersucht wird,

werden beim Untersuchen gemäß Schritt b) Bildpunkte wenigstens einer dreidimensionalen Gitterzelle zugeordnet. Anhand der Bildpunkte der Gitterzelle wird ein Flächenstück der wenigstens einen Teilfläche als Angriffsfläche ermittelt. Als weiter vorteilhaft hat es sich gezeigt, auf der Teilfläche wenigstens einen Greifpunkt zum Ansetzen der Greifeinheit in Abhängigkeit von einer Größe einer Anlagefläche der Greifeinheit, insbesondere als binäres Bild, zu ermitteln. Ist ein Bereich der Teilfläche ausreichend groß, um die Anlagefläche der Greifeinheit vollständig mit dem Bereich in Anlage zu bringen, so kann dies in dem binären Bild besonders einfach kenntlich gemacht werden. Dies ermöglicht ein besonders einfaches Auswerten des binären Bildes.In the method according to the invention for determining at least one partial surface, in particular a contact surface for attaching a gripping unit, a component which is in a container in which

• a) by means of a sensor an image of the component is captured, which comprises a three-dimensional cloud with a plurality of captured image points, and in which
• b) the image is examined by means of an evaluation unit for the presence of the at least one partial surface, in particular the contact surface, of the component,

When examining according to step b), pixels are assigned to at least one three-dimensional grid cell. On the basis of the pixels of the grid cell, a surface piece of the at least one partial surface is determined as the attack surface. It has also proven to be advantageous to determine at least one gripping point on the partial surface for attaching the gripping unit as a function of a size of a contact surface of the gripping unit, in particular as a binary image. If an area of the partial area is sufficiently large to bring the contact surface of the gripping unit completely into contact with the area, this can be identified particularly easily in the binary image. This enables a particularly simple evaluation of the binary image.

The examination of the image for the presence of the at least one partial area can thus be carried out with comparatively little computational effort and consequently particularly quickly, since the evaluation unit does not use a large number of recorded image points, but rather only uses area parts determined on the basis of the image points within a three-dimensional grid cell to find the partial area. The complexity of the evaluation is so greatly reduced that a two-dimensional evaluation method can be used when examining the surface pieces for the presence of the at least one partial surface.

The combination of a large number of pixels within a respective three-dimensional grid cell to form a single piece of area that can be described using fewer parameters leads to a significant reduction in the data to be processed. The rapid processing of the data enables the partial area of a component to be determined quickly. If this partial surface is a contact surface for attaching a gripping unit, this enables fully automated gripping of the component, which can be, for example, a sheet metal part. Such sheet metal parts can in particular include steel parts for an integral support or a rear axle support, as well as aluminum parts, in particular for a rear axle support, of a motor vehicle.

The method can be used within the scope of an automated, rational further processing of components to be removed from containers, for example from mesh boxes, within a predetermined cycle time. In a further step, the component separated from the container by means of the gripping unit can be fed to a device for recognizing a position, position and orientation of the component located on the gripping unit. Once the position, orientation and position of the component on the gripping unit has been determined, the component can then be stored in a defined manner or fed to a corresponding further processing device. The correct detection of the point of attack at which the gripping unit has gripped the component makes it possible to cause the gripping unit to convert the component into a desired spatial orientation. This enables an orderly, correct storage of the component.

In an advantageous embodiment of the invention, when capturing the image of the component according to step a), a light-cutting method, in particular using more than one camera, in particular a laser light-cutting method is used. Such a light section method based on a triangulation approach has proven to be particularly suitable for determining the at least one partial area of the component.

By simultaneously using two or more cameras when using only one light section line, in particular laser light section line, potential attack surfaces can be determined with great accuracy and particularly quickly, despite inevitable shadowing of the components in the container. If only one camera and a light section line, in particular a laser light section line, are used, the components in the container can be scanned several times in order to capture the image. Likewise, two or more than two cameras, each with a light section line assigned to the camera, in particular a laser light section line, can be used to determine the target surface.

Neighboring relationships between pixels are further advantageously defined by assigning the pixels to at least one three-dimensional grid cell. This makes it possible to use different sensor data acquisitions, for example scans of the laser light section sensor, between which there are no defined neighborhood relationships to create the image comprising the three-dimensional cloud of pixels. The use of several, independent of each other Sensor data acquisition enables a very precise image of the component to be created.

In a further advantageous embodiment of the invention, when determining the area per three-dimensional grid cell, a main component analysis of the pixels of a respective grid cell is carried out. By reducing the pixels of a respective three-dimensional grid cell to the main components of a surface piece, partial areas, in particular attack surfaces for attaching the gripping unit, can be found in a particularly computation-free and thus quick manner. Furthermore, the sensor noise is almost completely suppressed by the main component analysis.

It has also been shown to be advantageous to combine surface pieces that are different from one another depending on a distance from a reference plane and / or depending on a spatial orientation of a surface normal to a common partial surface. Individual patches of a respective three-dimensional grid cell are thus fused into a larger unit. If the patches are essentially planar patches, the merger results in larger, flat patches.

By taking into account the respective distance of the surface piece from a reference plane, it can be ensured that only those surface pieces are combined into a common partial surface whose distance from the reference plane lies within a predefinable tolerance interval.

If the spatial orientations of the respective surface normal are taken into account when combining the individual surface pieces, it can be ensured that only the surface pieces are combined into a common partial surface, the respective spatial orientations of which do not differ from one another by more than a predeterminable value. The larger unit formed from individual surface pieces then has a comparatively uniform spatial orientation.

In a further advantageous manner, different surface sections with different spatial orientations of the surface normal are combined to form a partial surface with a predefinable curvature. This allows the joining of surface pieces into a curved partial surface. By specifying the maximum permissible tolerance of the deviations in the spatial orientations of the surface normal, a maximum radius of curvature can be specified, which enables the gripping unit to be securely attached.

It is also advantageous here if the at least one gripping point is determined as a function of a distance from the contact surface to an edge of the partial surface and / or an opening in the component and / or as a function of an orientation of the contact surface. Any holes or recesses in the component are therefore taken into account when determining the gripping point. This is particularly advantageous if a suction gripper is used instead of a magnetic gripper as the gripping unit. Furthermore, the orientation of the contact surface of the gripping unit can be changed such that there is a maximum distance between the contact surface of the gripping unit and the edge of the partial surface or the opening in the component. If at least one suitable gripping point is determined on the partial surface, then a secure gripping of the component is made possible.

Finally, it has proven to be advantageous to create a ranking of gripping points of different partial areas depending on a size of the respective partial area and / or a distance of the contact surface from an edge of the partial area and / or depending on a distance of the partial area from a reference plane , If such a ranking of the gripping points has been established, then that component can preferably be removed from the container which has the most suitable, ie highest-ranking, gripping point for attaching the gripping unit. In this way, collisions of components with one another when removed from the container can be avoided. Furthermore, a particularly secure gripping of the component is made possible.

• 1 einen Bildausschnitt von sich in einer Gitterbox befindenden Bauteilen, wobei das Bild eine dreidimensionale Wolke von mittels eines Lichtschnittsensors erfassten Bildpunkten umfasst;
• 2 in jeweiligen dreidimensionalen Gitterzellen der dreidimensionalen Wolke ermittelte, planare Flächenstücke auf Oberflächen der Bauteile;
• 3 Angriffsflächen an Bauteilen, welche durch Zusammenfassen einer Mehrzahl voneinander verschiedener Flächenstücke gemäß 2 gewonnen wurden; und
• 4 Angriffsflächen unterschiedlicher Bauteile mit möglichen Greifpunkten zum Ansetzen einer Greifeinheit an dem jeweiligen Bauteil.

Further advantages, features and details of the invention result from the following description of a preferred embodiment and from the drawings. Show:

• 1 an image detail of components located in a lattice box, the image comprising a three-dimensional cloud of image points detected by means of a light section sensor;
• 2 planar surface pieces on surfaces of the components determined in respective three-dimensional grid cells of the three-dimensional cloud;
• 3 Attack surfaces on components, which by combining a plurality of mutually different surface pieces according 2 were won; and
• 4 Attack surfaces of different components with possible gripping points for attaching a gripping unit to the respective component.

That with reference to 1 to 4 described method is used to determine potential areas of attack on components that are located in a mesh box. If a gripping point for attaching a gripping unit is determined on such an attack surface, the component can be removed from the grid box by means of the gripping unit. After this separation of the component, a position detection of the component is carried out. The component is then fed to a component-specific storage device, where it is defined and stored in the correct position.

The entire process of fully automatic gripping and depositing of the components when separating them from the mesh box is divided into two phases. In the first phase, an attack surface of the component in the lattice box is determined using 3D detection and a gripping point is determined on the attack surface. This enables the component to be gripped at the gripping point. After gripping, the component is fed to a 2D component position detection in the second phase. The exact position, position and orientation of the component on the gripping unit is determined in order to finally be able to store the component in a defined manner.

To do that in 1 To obtain the image shown, the grid box with the components located therein is scanned and a three-dimensional cloud is scanned using a light section sensor based on a triangulation method 10 with a plurality of captured pixels 12 created. Here, the light section sensor travels with a certain scan angle and along a certain travel path over the lattice box. The cloud 10 at pixels 12 may contain regions without information (missing data) due to reflections and shadowing. This missing information can be obtained by scanning these regions again with changed scanning angles and / or other travel paths of the light section sensor.

By repeatedly scanning the grid box, there are no neighborhood relationships between pixels in the image of the scan profiles 12 of the individual scans available. For this reason, the neighborhood has to be redefined. A three-dimensional grid is used for this. The grid comprises three-dimensional grid cells 14 , All grid cells 14 are the same in extent. The grid cells 14 can, as in 1 shown, arranged adjacent or overlapping. Every pixel 12 the three-dimensional cloud 10 undergoes a transformation to convert scan coordinates (polar coordinates) into Cartesian coordinates. Then the pixel 12 the associated grid cell 14 assigned. Multiple classifications for overlapping grid cells 14 are possible.

After all the pixels 12 of the scan is transformed into Cartesian coordinates and into the associated grid cell 14 sorted, the next step is to find small planar patches 16 (Patches), which in 2 are shown.

die Menge aller Bildpunkte 12 in der Gitterzelle i. $$c_i=\frac{1}{\left|G_i\right|}{\displaystyle \sum _{p\in G_i}p}$$

mit p = kartesische Koordinaten des Bildpunkts 12.For this, for each grid cell i, the minimum number of pixels 12 contains, the center of mass c, calculated according to equation (1). Here is $$G_i\in R^3$$

the amount of all pixels 12 in the grid cell i. $$c_i=\frac{1}{\left|G_i\right|}{\displaystyle \underset{p\in G_i}{\Sigma }p}$$

with p = Cartesian coordinates of the pixel 12 ,

If the center of gravity has been calculated, i (grid cell 14 in 1 ) a 3 × 3 correlation matrix C _i can be set up according to equation (2). $$C_i=\frac{1}{\left|G_i\right|}{\displaystyle \underset{p\in G_i}{\Sigma }\left(\left(p-c_i\right){\left(p-c_i\right)}^T\right)}$$

Here, (p - c _i ) ^{T stands} for the transposed column vector.

The correlation matrix contains the variances of all pixels 12 the grid cell 14 regarding their center of mass. A correlation ellipsoid is created by calculating the eigenvectors e, (1 ≤ i ≤ 3) and the associated eigenvalues λ _i . In this principal component analysis (PCA) of a planar patch 16 the eigenvectors and eigenvalues determine the correlation ellipsoid. The eigenvector of the smallest eigenvalue starting in the center of mass corresponds to the normal direction of the patch 16 ,

$${\lambda }_1\approx 0$$

$${\lambda }_2>>0$$

$${\lambda }_3>>0$$

The condition of the surface piece extraction can be defined as follows: If the eigenvalues λ _i with the associated eigenvectors e are sorted according to size according to equation (3), then the eigenvalues satisfy equations (4), (5) and (6). $$\text{i}<\text{j}\to {\lambda }_i\le {\lambda }_j$$

$${\lambda }_1\approx 0$$

$${\lambda }_2>>0$$

$${\lambda }_3>>0$$

The equations can be explained geometrically as follows: The variances of the pixels 12 are in the direction of the plane spanned by clamping vectors for planar patches 16 large. The eigenvectors of the large eigenvalues correspond to the clamping vectors of the planar patch 16 , In the direction of the surface normal, the variances are very small. The associated eigenvector thus corresponds to the surface normal. If the criteria are not met, the pixels can 12 Edges, spheres or the like form geometric figures.

After a plurality of planar patches 16 different areas can be determined 16 to a larger target 18 , some of which are in 3 are shown as an example for respective partial areas of the component.

For this purpose, a 2D gray value image of the distance information and a 2D color image of the normal directions are generated. Each pixel of the gray-scale image or the color image corresponds to a patch 16 , which was determined in the previous step. The X and Y rasterization is through the grid cells 14 Are defined. A distance of the respective area piece is shown in the 2D gray value image 16 to a reference plane, for example to a floor of the lattice box in which the components are located, when they are combined with the respectively adjacent surface piece 16 considered. Thus are patches 16 then to an attack surface 18 summarized if their respective distance to the reference plane by no more than a predetermined value from the distance of the adjacent surface piece 16 deviates from the reference plane.

In an analogous manner, the spatial orientations of the surface area become normal of neighboring surface areas 16 taken into account when summarizing. The normal directions of the individual patches 16 are color-coded in the 2D color image. For the color coding of the normalized normals, blue = X, green = Y and red = Z (Z = vertical direction, direction of the distance to the reference plane).

Adjacent patches 16 become a larger target 18 summarized if both the deviation of the distances and the spatial orientations of the surface normal lie within certain, predetermined tolerances. If the tolerance of the normal deviation is chosen less restrictively, surfaces with curvatures within this tolerance can be combined.

In 3 are in this way from combined patches 16 formed attack surfaces 18 of the components shown in the grid box.

Each merged attack surface is used to determine grip points 18 subjected to a template matching. For this it is checked whether a two-dimensional template of a holding means 20 the gripper unit (gripper template) on the attack surface 18 fits.

The holding device 20 the gripping unit is in 4 on a respective target 18 of a component and includes, for example, a sucker and / or magnet. Flat suction devices and / or bellows suction devices can in particular be used as suction devices, the flat suction device having a contact surface made of foam or the like also making it possible to hold the component when the flat suction device is partially attached to the component to cover a hole in the component. The two-dimensional template of the holding device 20 is generated by projection, either along the Z direction or along the surface normal.

When comparing stencils, it is therefore checked whether the stencil is attacking the surface that is also two-dimensional due to the projection 18 fits. As a result, a binary image can be created, in which a "1" indicates whether the template is completely on the attack surface 18 fits. The remaining points of the binary image are assigned "0".

This binary image is further evaluated. For each gripping point there is a distance between the contact surface of the holding means 20 to an edge of the attack surface 18 , for example to a nearest edge of the attack surface 18 or to a hole in the component.

Furthermore, there are different orientations of the contact surface in relation to the attack surface 18 thereupon examined at which orientation of the contact surface a particularly large distance from the edge of the attack surface 18 consists. The two-dimensional template of the holding device 20 the gripping unit can be rotated about an axis of rotation. Different rotations of the template and the determination of the maximum distance of all tested rotations of the template thus provide an optimal gripping point per attack surface 18 ,

In 4 are per attack surface 18 the associated optimal gripping points by displaying the position and orientation of the holding means 20 the gripping unit on the attack surface 18 illustrated. Such an optimal gripping point per attack surface 18 shows maximum distances to the nearest edges of the attack surface 18 on and is due to the associated rotation of the holding means 20 , for example of the sucker, defined.

After determining the optimal gripping point for each attack surface 18 , the quality of the respective gripping point is evaluated by a fitness function, and the gripping point of the highest quality from the total of the gripping points considered per attack surface 18 selected. A ranking of the gripping points is thus created. To create the ranking, the size of the merged attack surface 18 , the distance of the holding means 20 to the nearest edge of the attack surface 18 , an empirical value of previous gripping attempts in the region, and the distance of the attack surface 18 from a reference plane, i.e. the Z value of the gripping point. In order to avoid gripping the crate bottom, levels whose expansion is greater than a maximum and part-specific value and levels whose area is greater than a maximum part-specific area are also ignored, since these levels cannot represent any components. A highest gripping point 22 is in 4 by displaying the corresponding associated holding means on the attack surface 18 illustrated.

This gripping point 22 The gripper then uses it to remove the component from the grid box. After recognizing the exact position, position and orientation of the component on the gripping unit, the component can finally be stored in a defined manner.

Claims (8)

Method for determining at least a partial surface of a component located in a container as an engagement surface (18) for attaching a gripping unit, in which a) an image of the component is captured by means of a sensor, which comprises a three-dimensional cloud (10) with a plurality of captured image points (12); b) the image is examined by means of an evaluation unit for the presence of the at least one partial surface of the component that can be used as an attack surface (18), pixels (12) being assigned to at least one three-dimensional grid cell (14) and the grid cell (14) using the pixels (12) ) an area piece (16) of the at least one partial area is determined as the contact area (18); c) at least one gripping point for attaching a gripping unit as a function of a size of a contact surface of the gripping unit is determined on the engagement surface (18). Procedure according to Claim 1 , characterized in that when capturing the image of the component according to step a), a light section method, in particular using more than one camera, in particular a laser light section method, is used. Procedure according to Claim 1 or 2 , characterized in that neighborhood relationships between pixels (12) are defined by assigning the pixels (12) to at least one of the three-dimensional grid cells (14). Procedure according to one of the Claims 1 to 3 , characterized in that a main component analysis of the pixels (12) of a grid cell (14) is carried out when determining the surface piece (16). Procedure according to one of the Claims 1 to 4 , characterized in that surface areas (16) which are different from one another are combined as a function of a distance to a reference plane and / or as a function of a spatial orientation of a surface normal to a partial area (18). Procedure according to Claim 5 , characterized in that different areas (16) with different spatial orientations of the area norms are combined to form a partial area (18) with a predefinable curvature. Procedure according to one of the Claims 1 to 6 characterized in that when determining the gripping point according to step c), the at least one gripping point is determined as a function of a distance from the contact surface to an edge of the partial surface (18) and / or an opening in the component and / or as a function of an orientation of the contact surface , Procedure according to one of the Claims 1 to 7 , characterized in that a ranking of gripping points of different partial surfaces (18) depending on a size of the respective partial surface (18) and / or a distance of the contact surface from an edge of the partial surface (18) and / or a distance of the partial surface (18 ) is created from a reference plane.

Images