DE102024101127A1 - Method for providing combined load carriers for unstacking, computer program and data carrier - Google Patents

Abstract

The invention relates to a method for preparing combined load carriers (1) for unstacking, comprising the steps of: - detecting depth information and image information of an area (4) in which the load carriers (1) are positioned, by means of at least one sensor device; (S1) - classifying the image information by an object recognition device, wherein a class characterizes the presence of at least one transport packaging (5) of one of the load carriers (1); (S2) - depending on the classification: creating a bounding box (8) around at least one object classified as transport packaging (5); (S3) - comparing a point cloud contained in the depth information corresponding to the bounding box (8) with at least one threshold value; (S4) - validating the classification depending on the comparison; (S5) and - releasing the load carriers (1) for unstacking depending on the validation. (S6) Furthermore, the invention relates to a computer program and a load carrier.

Description

The invention relates to a method for preparing combined load carriers, for example stackable boxes, located on a transport system such as a pallet, for unstacking or detaching from the combination, according to patent claim 1. Furthermore, the invention relates to a computer program and an electronically readable data carrier according to patent claims 9 and 10.

For logistics, for example, in the series production of motor vehicles, it is advantageous if it can be operated as automatically and autonomously as possible. One task of logistics, for example, for series production is to remove delivered parts, which are delivered in load carriers stacked on pallets, from the pallets and distribute them accordingly. Due to automation, robots are now being used more frequently for this unstacking or depalletizing process.

Such a robot is shown by the DE 10 2021 115 950 A1 , in the form of a depalletizing device for depalletizing load carriers stacked on top of each other.

In order for the load carriers to be depalletized, an upper cover that surrounds the combined load carriers or can at least be arranged on one of the load carriers is usually removed manually, for example by a worker, after which the depalletizing device can take over the work.

If a corresponding lid is not removed, for example due to carelessness, and is fed into the depalletizing device, this can lead to damage to the depalletizing device.

The object of the invention is therefore to provide a method, a computer program and a data carrier by means of which load carriers which are still combined and/or their transport packaging, in particular lids, can be detected in a particularly advantageous manner in order to prepare them for depalletizing.

This object is achieved according to the invention by the subject matter of the independent patent claims. Advantageous embodiments and further developments are the subject matter of the dependent claims, the description, and the drawings.

• In einem ersten Schritt erfolgt ein Erfassen von Tiefeninformation, beispielsweise in Form einer Punktwolke und Bildinformation, beispielsweise in Form eines Farbfotos, eines Bereichs, in welchem die zusammengefassten Ladungsträger positioniert beziehungsweise angeordnet sind, mittels wenigstens einer Sensoreinrichtung.
• In einem zweiten Schritt erfolgt ein Klassifizieren der Bildinformation durch eine Objekterkennungseinrichtung, wobei eine Klasse ein Vorhandensein wenigstens einer Transportverpackung, insbesondere beispielsweise in Form einer Abdeckung oder Deckels, eines der Ladungsträger charakterisiert.
• In einem dritten Schritt erfolgt in Abhängigkeit von der Klassifizierung ein Erstellen eines Begrenzungsrahmens (Englisch Bounding Box) um wenigstens ein als Transportverpackung klassifiziertes Objekt. Das Erstellen des Begrenzungsrahmens kann insbesondere durch die Objekterkennungseinrichtung anhand der Bildinformation erfolgen. Es können somit ein Objekt oder mehrere Objekte klassifiziert werden. Ob es sich bei dem wenigstens einen als Transportverpackung klassifizierten Objekts tatsächlich um eine Transportverpackung handelt, wird im weiteren Verlauf des Verfahrens verifiziert.
• In einem vierten Schritt erfolgt ein Abgleichen einer Punktwolke, welche die Tiefeninformation charakterisiert, und den Teil der Tiefeninformation betrifft, welcher mit dem Begrenzungsrahmen korrespondiert, mit wenigstens einem Schwellenwert.
• In einem fünften Schritt erfolgt ein Validieren der Klassifizierung, in Abhängigkeit von dem Abgleich, wobei somit quasi das Ergebnis des Abgleichs mit dem Ergebnis der Klassifizierung verglichen wird und somit eine gegenseitige Kontrollfunktion vorhanden sein kann.

A first aspect of the invention relates to a method for preparing combined load carriers for unstacking. In other words, the method releases load carriers, in particular small load carriers, such as crates, on a pallet, such as a Euro pallet, on which they are combined, for unstacking or depalletizing, and thus for removal from the combination. Depalletizing can be carried out, for example, by means of a suitable robot. In order for the combined load carriers to be released or prepared, the method according to the invention comprises the following steps:

• In a first step, depth information, for example in the form of a point cloud and image information, for example in the form of a color photo, of an area in which the combined charge carriers are positioned or arranged, is acquired by means of at least one sensor device.
• In a second step, the image information is classified by an object recognition device, wherein a class characterizes the presence of at least one transport packaging, in particular for example in the form of a cover or lid, of one of the load carriers.
• In a third step, depending on the classification, a bounding box is created around at least one object classified as transport packaging. The bounding box can be created, in particular, by the object recognition device based on the image information. Thus, one or more objects can be classified. Whether the at least one object classified as transport packaging is actually transport packaging is verified in the further course of the method.
• In a fourth step, a point cloud characterizing the depth information and concerning the part of the depth information corresponding to the bounding box is compared with at least one threshold value.
• In a fifth step, the classification is validated depending on the comparison, whereby the result of the comparison is compared with the result of the classification and thus a mutual control function can be present.

Finally, in a sixth step, the release or provision of the load carriers for destacking takes place depending on the validation, whereby, for example, a control signal for a robot which De-stacking or depalletizing can be provided or, depending on the result of the validation or classification and comparison, a warning signal can be provided to a worker because, for example, de-stacking is not possible due to transport packaging not being removed, without the risk of damaging the robot.

In other words, a method is provided whose goal is to detect transport packaging of load carriers, in particular small load carriers, in the logistics process for automation using robots. A sensor device and an object recognition device, as well as a three-dimensional measurement based on depth information, are intended to detect transport packaging that has not been removed, such as plastic lids, which hinder the automation.

The method uses AI-based or machine learning in image recognition, combined with mathematical approaches, in the area of processing three-dimensional point clouds. Logistics-specific decisions and validation rules can be applied, for example, to robustly detect plastic lids. In particular, the use of a cost-effective RGB camera can be enabled. This ensures redundancy through validation, based on the recognition of the transport packaging using depth information, and based on the recognition of the transport packaging in the image information.

The invention is based on the realization that recognizing transport packaging, or in particular a lid designed as a plastic lid, which can be attached to combined load carriers, presents a challenge. On the one hand, there is a high level of product and container complexity, for example, a large number of different plastic lids as well as other transport packaging in logistics, such as plastic lids with banding. Furthermore, the method should scale well to enable cost-effective and low-maintenance complementation. Furthermore, factory conditions in logistics, for example in a warehouse or at a series production site, vary. For example, different lighting conditions may prevail for a detection device or the sensor device and thus the camera, which can cause sensor noise, for example. Furthermore, the nature of the transport packaging, or plastic lids, varies; they may have different colors and also contain damage, stickers, and other individual markings. Furthermore, the presented method should be adaptable to, or support, an increasing degree of automation. An increasing level of automation can mean that supported secondary activities, such as checking whether transport packaging is resting on the load carrier, are also automated, and such processes must also be secured to prevent damage, for example to the robot. Furthermore, overly sensitive security, for example due to frequent error alarms, can lead to system malfunctions and/or failures, which can incur costs.

Due to the already mentioned clever combination of mathematical methods and object recognition based in particular on machine learning methods, both the product or container complexity as well as the scaling and the factory conditions can be taken into account, thereby promoting an increasing degree of automation.

In an advantageous embodiment of the invention, a camera is used as the sensor device, which can in particular, for example, be a component of the robot intended for destacking, and by means of which at least one recording, in particular an RGB-D image, is created, comprising at least one piece of depth information and image information. In other words, the image information used by the object recognition device and the point clouds or depth information used for mathematical determination, for example of a position, can be captured by a joint recording in the form of the RGB-D image, whereby only one camera needs to be used for this purpose. This results in the advantage that the method can be implemented particularly inexpensively due to the simplicity of the sensor device, which is already attached, for example, to the robot used for destacking.

In a further advantageous embodiment of the invention, for comparison with the at least one threshold value, a measurement of the bounding box is carried out and/or a plane is determined based on the point cloud and a distance to this plane is determined. In other words, the threshold value can be set in an XY plane, i.e., in the image plane of the image information without the depth information, in order to determine, for example, whether the dimension of the bounding box corresponds to a size expected from the perspective of the image, such as a transport packaging. can, or the threshold value indicates how far the bounding box can deviate from the size of a transport packaging. Alternatively, and in particular additionally, a further threshold value can be compared, in which a plane is determined from the point cloud in the bounding box using the depth information, in particular in the image. This plane should, for example, correspond to the front of the transport packaging and a distance to it can be determined, whereby the distance can be used to determine, for example, whether the plane is in a work area or the area in which the combined load carriers are to be positioned. This is particularly advantageous in a logistics hall, where a large number of small load carriers and pallet lids may be present and the verification must be carried out on the small load carriers to be depalletized. This has the advantage that the method can particularly advantageously validate the classification, since the comparison with the fewest threshold value enables particularly precise determination of possible transport packaging, in particular plastic lids.

In a further advantageous embodiment of the invention, a statistical resampling or resampling algorithm, in particular the RANSAC algorithm, is used to determine the level. In other words, a sample is repeated or, for example, several levels with multiple points are determined for level determination, whereby, in particular, a match with a random sample is used to estimate a model or the fill level. RANSAC stands for Random Sample Consensus. This offers the advantage that statistical noise, for example, can be particularly advantageously minimized.

In a further advantageous embodiment of the invention, the object recognition device is provided by or by means of a machine learning method, which can be designed as a neural network, in particular a convolutional neural network (CNN). In other words, the object recognition device is formed by a self-learning algorithm and/or by an artificial neural network that has several layers, so-called convolution layers, followed by a pooling layer. These units can, in principle, be repeated any number of times; with sufficient repetition, a so-called deep convolution neural network can be provided, which can enable deep learning. This results in the advantage that the object recognition device can be used particularly advantageously for detecting transport packaging that may be located on the load carriers.

In order to be able to advantageously use the object recognition device provided by the machine learning method, in an advantageous embodiment of the method it is trained on different types of transport packaging, or types of lids, in particular designed as plastic lids, and/or a variance of stacking factors of the load carriers and/or, for example, transport packaging, such as the plastic lid, in the background, i.e. outside the workspace of the robot, in order to recognize them. In other words, corresponding training data can be provided which contain different types of transport packaging and are labeled, and/or training data which include differently stacked load carriers or have different backgrounds can be used. This results in the advantage that the machine learning method can be used particularly advantageously for recognizing the transport packaging.

In a further advantageous embodiment of the invention, additional validation is performed by detecting the load carriers, in particular small load carriers, using the object detection device. In other words, the object detection device is capable of detecting the load carriers themselves in addition to the transport packaging, so that, for example, validation can be performed in such a way that if a transport packaging is detected but no load carrier is detected, a false positive result can be assumed. This results in the advantage that the method can be used particularly advantageously for unstacking stacked load carriers.

In a further advantageous embodiment of the invention, a value range is specified for the at least one threshold value. Additionally and alternatively, the at least one threshold value is determined in relation to the area, i.e. the stacking area, or adapted to a transport system, such as a Euro pallet and/or a work space. In other words, the threshold value is not a fixed range, but allows for a certain amount of flexibility, for example, when measuring the boundary frame. For example, depending on the orientation of the load carriers, the transport packaging can have a minimum length in the field of view of the sensor device or a maximum length in the field of view of the sensor device, which can thereby be taken into account. Additionally or furthermore, for example For example, in the workspace of the destacking device, i.e., the robot, the pallet on which the combined load carriers are arranged can be positioned further forward or further back, so that this can be taken into account, for example, for the distance, if this is to be determined as a threshold value. This has the advantage that the method can be used for a large number of different transport packaging and for a wide variety of load carrier placements in the workspace.

A second aspect of the invention comprises a computer program. The computer program can be loaded, for example, into an electronic computing device of a system and comprises program means for executing the steps of the method according to the first aspect of the invention when the computer program is executed in the electronic computing device or, for example, a control device.

Advantages and advantageous embodiments of the first aspect of the invention are to be regarded as advantages and advantageous embodiments of the second aspect of the invention and vice versa.

A third aspect of the invention relates to an electronically readable data carrier. The electronically readable data carrier comprises electronically readable control information stored thereon, which includes at least one computer program according to the second aspect of the invention and is configured such that, when the data carrier is used in an electronic computing device, it can execute a method presented here according to the first aspect of the invention.

Advantages and advantageous embodiments of the third aspect of the invention are to be regarded as advantages and advantageous embodiments of both the second aspect and the first aspect of the invention and vice versa.

Further features of the invention emerge from the claims, the figures, and the description of the figures. The features and combinations of features mentioned above in the description, as well as the features and combinations of features mentioned below in the description of the figures and/or shown alone in the figures, can be used not only in the respective combinations specified, but also in other combinations or on their own.

• 1 ein schematisches Ablaufdiagramm für ein Verfahren zum Bereitstellen von zusammengefassten Ladungsträgern für ein Entstapeln;
• 2 verschiedene schematische Frontansichten verschiedener, auf Paletten zusammengefasster Ladungsträger;
• 3 schematische Ansicht der Verfahrensschritte anhand eines Ausführungsbeispiels einer Vorrichtung zum Entstapeln der Ladungsträger.

The invention will now be explained in more detail using a preferred embodiment and with reference to the drawings. It shows:

• 1 a schematic flow diagram for a method for providing combined load carriers for destacking;
• 2 various schematic front views of different load carriers grouped on pallets;
• 3 schematic view of the method steps based on an embodiment of a device for unstacking the load carriers.

1 shows a schematic flow diagram for a method for providing combined load carriers 1, in particular located on a transport system 2, such as a pallet, for unstacking, i.e. depalletizing, or releasing from the combination, for example by means of a robot 3, or a depalletizing device.

The combined load carriers 1, which are located, for example, on a transport system 2 designed as a Euro pallet, have transport packaging 5 that is to be removed upon delivery and before unstacking. This transport packaging can be designed, in particular, as a plastic lid covering several load carriers 1, which are designed, in particular, as small load carriers. Furthermore, the transport packaging 5 can additionally have a transport banding or banding 7.

For transporting the load carriers 1 and thus for transport logistics, a transport packaging 5 may be present, which must be removed for unstacking or depalletizing the load carriers 1. The transport packaging 5 is designed in particular as a plastic lid so that, for example, pallets can be stacked as transport systems 2. For vehicle production, parts delivered in the load carriers 1 on the pallets are automatically sorted by robots. For this purpose, workers remove the plastic lid before the automated unstacking by the robot 3. It may happen that, for example, a plastic lid is not removed, which can lead to a crash of the robot 3 during automated unloading. Therefore, a particularly robust, automated detection of the transport packaging 5, in particular the plastic lid, is desired.

• Im ersten Schritt S1 erfolgt ein Erfassen von Tiefeninformation, insbesondere in Form einer Punktwolke und Bildinformation, insbesondere in Form eines Farbfotos, eines Bereichs 4 in welchem die Ladungsträger 1 positioniert sind, mittels wenigstens einer Sensoreinrichtung, welche insbesondere als Kamera 6 ausgebildet ist.
• Im zweiten Schritt S2 erfolgt ein Klassifizieren, insbesondere anhand von zwei Klassen, das Vorhandensein oder Nichtvorhandensein der Transportverpackung 5, aus der Bildinformation durch eine Objekterkennungseinrichtung, wobei eine Klasse ein Vorhandensein wenigstens einer Transportverpackung 5, insbesondere in Form einer Abdeckung, beziehungsweise eines Deckels, insbesondere eines Plastikdeckels, wenigstens eines der Ladungsträger 1, beziehungsweise mehrere der zusammengefassten Ladungsträger 1 charakterisiert.
• Im dritten Schritt S3 erfolgt in Abhängigkeit von der Klassifizierung, also dem Ermitteln, ob eine Transportverpackung 5 vorhanden ist, ein Erstellen eines Begrenzungsrahmens 8, um wenigstens eines als Transportverpackung 5 klassifizierten Objekts, welches in der Regel die Transportverpackung 5 selbst ist. Jedoch kann beispielsweise im Fall einer Fehldetektion, beispielsweise auch eine Transportverpackung 5 in einem Hintergrund oder ein falsch erkannter Ladungsträger 1 das Objekt sein.
• Im vierten Schritt S4 erfolgt das Abgleichen einer Punktwolke, welche in der, zum Begrenzungsrahmen 8 korrespondierenden Tiefeninformation enthalten ist, mit wenigstens einem Schwellenwert.
• Im fünften Schritt S5 erfolgt das Validieren der Klassifizierung in Abhängigkeit von dem Abgleich, beziehungsweise wird das Ergebnis der Klassifizierung und des Abgleichs verglichen, um so für den sechsten Schritt S6 das Freigeben der Ladungsträger 1 zum Entstapeln, in Abhängigkeit von der Validierung zu ermöglichen.

The presented procedure comprises six steps S1 to S6, which are shown below:

• In the first step S1, depth information is acquired, in particular in the form of a point cloud and image information, in particular in the form of a color photo, of an area 4 in which the charge carriers 1 are positioned, by means of at least one sensor device, which is designed in particular as a camera 6.
• In the second step S2, a classification, in particular based on two classes, of the presence or absence of the transport packaging 5, is carried out from the image information by an object recognition device, wherein a class characterizes the presence of at least one transport packaging 5, in particular in the form of a cover or a lid, in particular a plastic lid, of at least one of the load carriers 1, or of several of the combined load carriers 1.
• In the third step S3, depending on the classification, i.e., the determination of whether a transport packaging 5 is present, a bounding box 8 is created around at least one object classified as a transport packaging 5, which is usually the transport packaging 5 itself. However, in the case of a false detection, for example, the object may also be a transport packaging 5 in a background or an incorrectly detected load carrier 1.
• In the fourth step S4, a point cloud contained in the depth information corresponding to the bounding box 8 is compared with at least one threshold value.
• In the fifth step S5, the classification is validated depending on the comparison, or the result of the classification and the comparison is compared in order to enable the release of the load carriers 1 for destacking in the sixth step S6, depending on the validation.

The release can be achieved, for example, by providing a control signal for the robot 3. Alternatively, if the validation shows that a transport packaging 5 prevents the robot 3 from unstacking the combined load carriers 1, a warning signal can be issued to a worker so that the worker can remove the transport packaging 5.

2 shows several load carriers 1, which are combined on three pallets used as a transport system 2. For example, a different number of load carriers 1 can be combined, especially stacked differently, so that different stacking factors pose a challenge for object recognition. Due to different order quantities from a supplier, a full load unit or pallet cannot always be delivered, which is why the different stacking factors on load carriers 1 arise.

Furthermore, the pallets, especially Euro pallets, can be shifted forward or backward and left or right within a tolerance range, which presents a further challenge for object detection. This means that there is no static, fixed position of the transport packaging 5, so a dynamic inspection may be necessary, which should be performed without a laser scanner, particularly due to low costs.

In addition, 2 A variety of plastic lids can occur. For example, they can have different shapes. The left and middle plastic lids have rounded corners, while the right plastic lid, the right transport packaging 5, has more rectangular corners. Furthermore, different sizes and colors, as well as damage, labels, and/or stickers, can be present. Furthermore, not every pallet or every load carrier 1 must have a plastic lid.

For example, a further challenge in detecting the plastic lid is the color if it has little contrast to a background, for example to objects in the area 4, which in particular describes a work area 9, or in the background. Furthermore, the surface quality of the plastic lid can, for example, enable corresponding reflections from overhead lighting. Furthermore, the plastic lid can have the same color as the load carriers 1 so that it does not stand out from them. Furthermore, remnants of stickers, or stickers or the like can be attached to the lids. Furthermore, damage, for example in the form of a breakout, is possible, so that, for example, measuring only the front side of the plastic lid facing the camera 6 does not allow reliable detection. Furthermore, as can be seen on the left, a banding 7 or transport banding may not have been removed, so that, for example, no continuous surface is created for object detection, which presents a further challenge.

When using robots, such as Robot 3, in goods receipt, it is not always possible in practice to have a fenced or isolated operation, for example with a partition walls. This can result in load units with plastic lids being visible even in the background, which is why relevant plastic lids must be distinguished from irrelevant ones, which can be done in particular by determining at least one threshold value, in particular by means of a distance from the plane. Therefore, it is useful to define the aforementioned working space 9, in particular for area 4.

3 schematically shows the procedure. The first part, marked with 1, shows the 3 , the robot 3 with the area 4 in which combined load carriers 1 are located, whereby the camera 6 (of the robot 3) generates an RGB-D image, or recording, the image information of which is, for example, in the second part, marked with 2, of the 3 , is shown.

The object recognition device, which is implemented in particular as a neural network on an electronic computing device, performs the classification and then, in step S3, creates the bounding boxes 8. In the example, two bounding boxes 8 are applied. The left, upper bounding box 9, for example, is a detection of a plastic lid in the background and is therefore not located in the area of the workspace 9.

This is followed by an evaluation of the depth information associated with the bounding boxes 8, which is indicated by the arrow 10. In the third part of the 3 The bounding boxes, or rather the points of depth information contained therein, are measured and compared with at least one threshold value according to step S4. The values for the bounding boxes 8 can be seen. The AI results, or rather the results of a neural network, which is used in particular for object recognition, are then validated with the comparison according to step S5, which results in the fourth part of the 3 Subsequently, a distance measurement 11 is carried out, in which a plane is formed from the point clouds of the depth information in the respective area of the bounding box 8 and the distance between camera 6 and the respective plane is determined.

As a further step of the at least one threshold value comparison 12, the result of the distance measurement 11 is validated in an alternative step S5 to check whether the plastic lid is located within the workspace 9, whereupon a selection 13 is made. In this step, the logistics process with the robot 3 is designed based on the result of the verification, i.e. whether the object recognition and the mathematical methods carried out on the depth information have detected no plastic lid or a plastic lid. Thus, if the AI or the neural network does not detect a plastic lid and the verification is carried out using the depth information, the logistics process of unstacking can be started. If the verification fails, a worker can be called upon to remove a possible plastic lid in the form of transport packaging 5.

For comparison, the method shown here can be used to measure the bounding box, and additionally, a plane can be determined and distance determined based on the point cloud. A resampling algorithm such as RANSAC is advantageously used to determine the plane.

In this case, an interval or value range is specified for the at least one threshold value or the threshold value is defined in relation to the area 4 or the working space 9.

In addition, for example, to supplement steps S5 or S6, a validation can be carried out by detecting the load carriers 1 themselves by the object detection device.

Thus, the presented method can also be described as follows: in the first step, an RGB-D image is captured. The image is advantageously generated using a low-cost camera 6 and can therefore include sensor noise. The RGB image thus includes associated depth information in the sense of a 3D point cloud. Using the example of the 3 In the robot 3 shown, the camera 6 can be mounted thereon and thus be a component of the robot 3. In the second step, object recognition takes place using a convolutional neural network in particular. A learning-based approach is therefore used for object recognition, in particular in the form of deep learning for object recognition, which can be carried out, for example, using Faster RCNN or Yolo V5. Alternatively, segmentation is possible, for example, using Mask RCNN, which can recognize the two objects - plastic lid and no plastic lid. If necessary, the load carriers 1 can also be recognized for validation purposes.

In a third step, an XY measurement is performed. This involves selecting the 3D depth information or point cloud within the bounding box or mask, i.e. the limiting frame 8. During the XY measurement, the length (x _max - x _min ) and height (y _max -y _{min )} are calculated for each detected plastic lid. A comparison with parameterized Values of a value range, for example of Euro pallets or transport system 2.

Finally, in a fourth step of the process, a 3-plane determination, distance measurement, and threshold testing can be performed. A robust approach for 3D plane determination (ax ₁ + bx ₂ + cx ₃ + d = 0) can be performed using RANSAC. Distance measurements along the plane normal (= distance from camera 6 to the side surface of the plastic lid) can be performed. These measurements can be compared with parameterized values within a value range (minimum and maximum) of the workspace 9 of robot 3.

Finally, in a fifth step or fifth stage, validation and decision-making take place. During validation, the consistency between the Classification-based classification or the classification using the neural network (in particular, step S2) and the threshold value check (steps 3 and 4) is determined. This is followed by the release of load carriers 1 for destacking based on a logistics-specific decision rule or a corresponding warning signal: For example, as soon as a valid plastic lid is visible in the work area of robot 3, load carrier 1 is not stored automatically but is manually checked by employees to prevent a crash of robot 3.

The method presented here, a corresponding computer program that can execute the process steps, as well as an electronically readable data carrier with program steps for executing the computer program, can be integrated particularly flexibly into an existing object recognition (“retro-fitting”), which already recognizes the load carriers 1 and their pose (position and rotation in space).

The method offers a robust approach based on RGB images and depth images to ensure redundancy. A learning-based component with neural networks can be trained on logistics-specific data sets in any environment, such as warehouses and factories. In addition, a particularly mathematical distance measurement is performed, which enables validation based on logistics-specific or robot-specific thresholds. The use of different technologies results in high process stability and consequently, particularly few failures, for example, due to an accident, on robot 3, which is used to unstack the combined load carriers 1.

The machine learning method underlying object recognition, in particular, can be optionally expanded, for example, so that, depending on the training data, different plastic lids, such as different color variants, damage, masking, etc., can be recognized. Furthermore, the load carrier type, for example, KLT6410, of load carrier 1 can be recognized, thus ensuring an optional step for checking the number of containers or load carriers 1. Furthermore, specific data can be read and compared using OCR (Optical Character Recognition) on stickers, which are designed as VDA labels, for example.

In particular, the method demonstrates a combination of neural networks and 3D plane determination, which can be performed largely independently of perspective and therefore without precise pre-positioning. This allows the method to be implemented or installed in Robot 3 or in existing conveyor systems.

Furthermore, the method is not susceptible to interfering contours in the field of view of camera 6, such as other corners, edges, or cables. Furthermore, distance measurement based on 3D plane determination is particularly robust, for example, through mathematical optimization such as RANSAC, so that individual, erroneous points in the depth information can be reliably removed.

Furthermore, the method offers economies of scale, such as cost advantages, over traditional computer vision, since end users can learn the training data without any programming knowledge. Furthermore, transfer between different plastic lids is possible, thus reducing the training effort.

Thus, a system for robust detection and validation of plastic covers in logistics and environments using RGB depth cameras is shown.

List of reference symbols

1

load carrier

2

transport system

3

robot

4

Area

5

Transport packaging

6

camera

7

Banding

8

bounding box

9

workspace

10

Arrow

11

Distance measurement

12

Threshold adjustment

13

Selection

S1

first step

S2

second step

S3

third step

S4

fourth step

S5

fifth step

S6

sixth step

QUOTES CONTAINED IN THE DESCRIPTION

This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 10 2021 115 950 A1 [0003]

Claims (10)

A method for preparing combined load carriers (1) for destacking, comprising the steps of: - detecting depth information and image information of an area (4) in which the load carriers (1) are positioned using at least one sensor device; (S1) - classifying the image information using an object recognition device, wherein a class characterizes the presence of at least one transport packaging (5) of one of the load carriers (1); (S2) - depending on the classification: creating a bounding box (8) around at least one object classified as a transport packaging (5); (S3) - comparing a point cloud contained in the depth information corresponding to the bounding box (8) with at least one threshold value; (S4) - validating the classification based on the comparison; (S5) and - releasing the load carriers (1) for destacking based on the validation. (S6) Procedure according to Claim 1 , characterized in that a camera (6) is used as the sensor device, by means of which at least one recording comprising the depth information and the image information is created. Procedure according to Claim 1 or 2 , characterized in that for the comparison with the at least one threshold value, a measurement of the boundary frame (8) is carried out and/or a plane is determined on the basis of the point cloud and a distance to this is determined. Procedure according to Claim 3 , characterized in that a resampling algorithm, in particular the RANSAC algorithm, is used to determine the plane. Method according to one of the preceding claims, characterized in that the object recognition device is provided by a machine learning method. Procedure according to Claim 5 , characterized in that the machine learning method is trained to recognize, as the transport packaging (5), different types of lids and/or a variance of stacking factors and/or lids in a background. Method according to one of the preceding claims, characterized in that an additional validation is carried out by means of recognition of the load carriers (1) by the object recognition device. Method according to one of the preceding claims, characterized in that a value range is specified for the at least one threshold value and/or the threshold value is determined with respect to the area (4) and/or a working space (9). Computer program which can be loaded directly into a memory of an electronic computing device, with program means for carrying out the steps of the method according to one of the Claims 1 until 8 to be executed when the program is executed in an electronic computing device. Electronically readable data carrier with electronically readable control information stored thereon, which contains at least one computer program according to Claim 9 and are designed in such a way that when the data carrier is used in an electronic computing device, they implement a method according to one of the Claims 1 until 8 carry out.