NL2036065B1

NL2036065B1 - Method and device for automated packing

Info

Publication number: NL2036065B1
Application number: NL2036065A
Authority: NL
Inventors: Paul Emerentiana Welling Niels; Sabine Van Der Vliet Mariska; De Bruijn Stephanus
Original assignee: R Solution Medical B V
Priority date: 2023-10-17
Filing date: 2023-10-17
Publication date: 2025-05-01
Also published as: WO2025084926A1

Abstract

The invention provides, amongst other aspects, a method for generating first and second instructions for manipulating a physical structure comprising a first portion 5 and a second portion, comprising: receiving data on a reference volume, the reference volume comprising the first portion and the second portion of the physical structure; repeatedly generating, with respect to the reference volume, first instructions for a first robot arm for manipulating the physical structure at the first portion; repeatedly generating, with respect to the reference volume, second 10 instructions for a second robot arm for manipulating the physical structure at the second portion; wherein the first and second instructions comprise programmable logic controller, PLC, instructions; and wherein the first and second instructions are generated based at least on a measured physical interaction between the first and second robot arm measured by means of a feedback means.

Description

METHOD AND DEVICE FOR AUTOMATED PACKING

FIELD OF THE INVENTION

[0001] The present invention relates to improved automation and industrial computers involving PLC. Particularly, the invention relates to automation involving manipulations via a combination of two or more robot arms.

BACKGROUND ART

[0002] With the advent of industrial computers, many tasks that were performed manually by operators have been partially or fully automated. This may improve ergonomics for operators, lead to higher accuracy, and lead to faster task handling, especially in the case of tasks that involve multiple manual operations.

Opportunities for automation may be especially high in cases of manipulation- intensive tasks, carried out by human operators which involve many manipulation steps, as the automation may provide for a particularly large speed-up in those cases.

[0003] One example of such a manipulation-intensive task is the folding of an instrument tray into a packaging material. The packing relates to ensuring that medical tools and instruments which are used during a medical intervention are supplied in a sterile fashion. Such folding is inherently manipulation-intensive, which may entail physical complaints for operators doing the folding routineously.

Also other challenges occur, relating to, e.g., incorrect folding or imperfections of the material. For these reasons, the automation of the folding is highly desirable.

[0004] Other examples of manipulation-intensive tasks may relate to the repeated folding of sheet-like material (also for applications beyond packing), or the collecting and depositing of multiple objects in a container (e.g., the collecting of instruments and depositing it in an instrument tray). Also for such examples, automation is highly desirable for reasons of ergonomy and/or reliability and/or speed.

[0005] EP3648714 discloses related methods and systems, but only discloses consecutive manipulations, without direct interaction between robot arms, which may be suboptimal in terms of speed.

[0006] US20140158141A1 describes a setup with four robotic arms in a medical context, and is related to packaging. However, the control of the robot arms is not described. Moreover, it apparently concerns a particular form of robot arm control, see US11185380, which appears complex to implement.

[0007] CN105729472 describes a PLC-based control of a robot arm for packaging.

CN105729472 thereby describes a PLC control with a 2D camera and vision software. However, CN1057 merely describes the control of a single robot arm, and does not describe a set-up with more than one robot arm.

[0008] WO2019121571A1 describes a gripping tool for packing. However,

WO2019121571A1 is thereby directed to using a single gripping tool, being limited in speed by having the gripping tool carrying out all the manipulation steps consecutively, which may lead to slow manipulation.

[0009] The present invention aims at addressing issues such as the issues mentioned above.

SUMMARY OF THE INVENTION

[0010] According to a first aspect, the present invention provides a method for generating first and second instructions for manipulating a physical structure comprising a first portion and a second portion, comprising: receiving data on a reference volume, the reference volume comprising the first portion and the second portion of the physical structure; repeatedly generating, with respect to the reference volume, first instructions for a first robot arm for manipulating the physical structure at the first portion; repeatedly generating, with respect to the reference volume, second instructions for a second robot arm for manipulating the physical structure at the second portion; wherein the first and second instructions comprise programmable logic controller, PLC, instructions; and wherein the first and second instructions are generated based at least on a measured physical interaction between the first and second robot arm measured by means of a feedback means.

[0011] Such a method may advantageously enable faster and/or more reliable manipulation, by taking into account measured physical interaction between robot arms. This may be particularly advantageous for manipulation-intensive tasks. For such tasks, prior art methods with multiple robot arms and manipulation based on

PLC instructions typically rely on steps that are consecutive, typically only slightly overlapping in time, or even non-overlapping in time, whereby one robot arm carries out a first manipulation step according to a first pre-determined program, and another robot arm only initiates the next manipulation step according to its proper pre-determined program when the first program is nearly or entirely finished, according to some pre-determined schedule. In contrast, by measuring physical interaction and acting thereupon, a more concurrent approach is enabled, wherein, e.g., robot arms may head toward different coordinates in the same reference volume, and carry out manipulation steps concurrently in each others’ direct vicinity. This, in turn, may enable faster manipulation and/or more robustness in cases of imperfections or unpredictability, where the distance between robot arms may be deviating from what is expected. Indeed, by detecting such deviations, in embodiments, the method may be carried on in view of this information in an adaptive fashion, rather than merely detecting a deviation and going to some default response action (such as, merely for the sake of example, stalling all robot arms, or withdrawing all robot arms).

[0012] According to a second aspect, the invention provides a device for generating first and second instructions each comprising PLC instructions for manipulating a physical structure comprising a first portion and a second portion, the device comprising a PLC module, preferably being a PLC module, comprising a processor for executing a method according to the invention.

[0013] According to a further aspect, the invention provides a system for generating first and second instructions for manipulating a physical structure comprising a first portion and a second portion, the system comprising: a device, preferably the device according to the invention; a first and second robot arm connected to said device and preferably positioned at different respective angles with respect to said physical structure; a feedback means for measuring a physical interaction between the first and the second robot arm and connected to said device; wherein said device is configured for: receiving data on a reference volume, the reference volume comprising the first portion and the second portion of the physical structure; repeatedly receiving, from said feedback means, a measured physical interaction between the first and second robot arm;

repeatedly generating, with respect to the reference volume and based on the measured physical interaction, first instructions for the first robot arm for manipulating the physical structure at the first portion; repeatedly generating, with respect to the reference volume and based on the measured physical interaction, second instructions for the second robot arm for manipulating the physical structure at the second portion; wherein the first and second robot arm, respectively, are configured for: repeatedly receiving, from the device, the first and the second instructions, respectively; repeatedly manipulating, based on the first and second instructions, respectively, said physical structure at said first and second portion, respectively; wherein the feedback means is configured for: repeatedly measuring the physical interaction between the first and the second robot arm; and repeatedly sending the measurements of physical interaction to the device; wherein the instructions comprise programmable logic controller,

PLC, instructions.

[0014] According to a further aspect, the invention offers a computer program product program product comprising a medium for storing instructions for carrying out a method according to the invention.

[0015] Preferred embodiments and their advantages are provided in the description and the dependent claims.

BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The present invention will be discussed in more detail below, with reference to the attached drawings.

[0017] Figures 1-5 show different perspective views relating to an example system according to the invention. Thereby, Figures 1 and 2 show a rear view and a side view, respectively, while Figures 3-5 show three frontal close-up views.

[0018] Figures 6-7 illustrate two different example types of labyrinth-like packing relating to aspects of the invention. Thereby, Figure 6 illustrates labyrinth-like packing according to a modified package method, whereas Figure 7 illustrates labyrinth-like packing according to an envelope packing method.

DESCRIPTION OF EMBODIMENTS

[0019] The following descriptions depict only example embodiments and are not considered limiting in scope. Any reference herein to the disclosure is not intended to restrict or limit the disclosure to exact features of any one or more of the exemplary embodiments disclosed in the present specification. 5 [0020] Furthermore, the terms first, second, third and the like in the description and in the claims are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order. The terms are interchangeable under appropriate circumstances and the embodiments of the invention can operate in other sequences than described or illustrated herein.

[0021] Furthermore, the various embodiments, although referred to as “preferred” are to be construed as exemplary manners in which the invention may be implemented rather than as limiting the scope of the invention.

[0022] The term “comprising”, used in the claims, should not be interpreted as being restricted to the elements or steps listed thereafter; it does not exclude other elements or steps. lt needs to be interpreted as specifying the presence of the stated features, integers, steps or componenis as referred to, but does not preclude the presence or addition of one or more other features, integers, steps or components, or groups thereof. Thus, the scope of the expression “a device comprising A and B” should not be limited to devices consisting only of components

A and B, rather with respect to the present invention, the only enumerated components of the device are A and B, and further the claim should be interpreted as including equivalents of those components.

[0023] In this document, the term "packing" and "packaging" are interchangeable.

Likewise, the terms "packing material" and "packaging material” are interchangeable.

[0024] In this document, the term "PLC" refers to Programmable Logic Controller.

In preferred embodiments, PLC thereby refers to devices, methods, programming languages and systems that comply with IEC 61131.

[0025] In this document, the term "controller” and "PLC module" are interchangeable. In embodiments, the device comprises the PLC module and further modules. In embodiments, the device is the PLC module.

[0026] In embodiments, the device, preferably the PLC module, comprises input/output (1/0) points for connecting to further devices, such as the robot arms and/or sensors and/or actuators. In embodiments where the feedback means comprise external feedback means, the 1/O points may allow connecting to the external feedback means.

[0027] In embodiments, the input/output points allow input from any or any combination of the feedback means, one or more cameras for computer vision, an analog variable from a process sensor such as a weight sensor or a sensor for measuring a physical dimension of a portion of the physical structure, such as a depth sensor or a surface sensor or a volume sensor. In embodiments, the input/output points allow output toward any of or any combination of actuation means for causing the robot arms to manipulate the physical structure, indicator lamps, sirens, pneumatic or hydraulic cylinders, magnetic relays, solenoids, or analog outputs. The actuation means may relate, e.g., to electric motors and/or pneumatic motors and/or variable frequency drives (VFD) and/or servos and/or steppers.

[0028] In embodiments, the device, preferably the PLC module, comprises an

Ethernet interface.

[0029] in this document, the term "PLC instructions" relates to any instructions either provided as input or as output, either directly or indirectly, of the PLC module comprised in the device. The term may thereby relate to, e.g., instructions formatted according to a PLC programming language. The term may also relate to instructions with data formatting compatible to be read as input via one of the I/O points of the PLC module, i.e., instructions entering the PLC module, without being limited to instructions written in a PLC programming language. The term may equally relate to instructions generated as output, via one of the I/O points, of the

PLC module, i.e., instructions originating from the PLC module, without being limited to instructions written in a PLC programming language.

[0030] In this document, the term "instrument tray" refers to an example of a physical structure (physical object + packing material) from a medical context, wherein the physical object is an instrument tray and the manipulation relates 10 packing the physical object in the packing material. The “instruments” present in the instrument tray are merely example medical instruments not involved in the packing, and are not to be confused with the first, second and third instrument which, in embodiments, are involved in the actual manipulations according to the invention.

[0031] In embodiments, the device of the invention comprises a PLC module, preferably is a PLC module. In embodiments, this PLC module may relate to a controller for industrial computers, comprising a CPU and memory comprising PLC instructions which, when carried out on the processor, cause the PLC module to send control instructions the two or more robot arms comprising actuation means.

In embodiments, sending control instructions to the robot arms relates to sending control instructions to the actuation means comprised in the robot arms.

[0032] In embodiments, the device, preferably at least the controller, complies with

IEC 61131. In embodiments, the device, preferably at least the controller, complies with at least two parts of Part 1-10 of IEC 61131. In embodiments, the device, preferably at least the controller, complies with Part 3 of IEC 61131, hereafter "61131-3", said Part 3 relating to PLC programming languages.

[0033] In embodiments, preferably embodiments wherein the device complies with

Part 3 of IEC 61131, the PLC instructions relate to a PLC-related programming language compliant with IEC 61131-3. This may relate to any of Ladder diagram (LD), Function block diagram (FBD), Structured text (ST), Instruction list (IL),

Sequential function chart (SFC). In preferred embodiments the PCL instructions relate to Structured Text comprising SCL definition.

[0034] In embodiments, the device comprises or is a controller belonging to any of the Siemens® Simatic® S7-1500 series (e.g., S7-1510, S7-1511, §7-1512, S7- 1513, 87-1514, S7-1515, S7-1516, S7-1516, S7-1517, S7-1518) or S7-1200 series (e.g., S7-1210, S71211, 87-1212, S7-1213).

[0035] In embodiments, the feedback means comprises or is an external feedback means not belonging to the device. In embodiments, the feedback means comprises or is an internal feedback means belonging to the device. In embodiments, the feedback means comprises an internal and an external feedback means.

[0036] In embodiments, the feedback means, either internal or external, relates to a sensor specific to a single robot arm and present on (and preferably, belonging to) the robot arm or provided in the vicinity of the robot arm, wherein the sensor provides information on a position (e.g., on a coordinate) of the robot arm and/or a speed of the robot arm and/or an acceleration of the robot arm. In such embodiments, the measured physical interaction may relate to receiving, by the device, data from respective feedback means of different robot arms, and measuring, from this information, a physical interaction between the first and the second robot arm. This may relate, e.g., to repeatedly receiving 3D position information from each of the first and the second robot arm and determining, based on how the 3D position information of the first and the second robot arm relate, that a physical interaction is taking place or will take place.

[0037] In embodiments, the feedback means, either internal or external, is specific to more than one robot arm and provided in the vicinity of the more than one robot arm (e.g., provided on one of the more than one robot arms), wherein the sensor provides information, with respect io each of the more than one robot arm, on a position {e.g., on a coordinate) of the robot arms and/or a speed of the robot arms and/or an acceleration of the robot arms. In such embodiments, the measured physical interaction may relate to receiving, by the device, data from the single feedback means and measuring, from this information, the physical interaction between the first and the second robot arm. This may relate, e.g., to repeatedly receiving 3D position information regarding each of the first and the second robot arm and determining, based on how the 3D position information of the first and the second robot arm relate, that a physical interaction is taking place or will take place.

[0038] In embodiments, the feedback means and the measured physical interaction relate to calculating, based on desired positions of the robot arms, a 3D path for each of the robot arms. In embodiments, the repeatedly receiving of information of the feedback means allows for correcting the 3D path of one of the robot arms. This may be suitable for applications for which correction in view of unexpected manipulations or, e.g., a collision between arms is desirable.

[0039] In embodiments, the feedback means and the measured physical interaction relate to calculating, based on desired positions of the robot arms, a 3D path for each of the robot arms, thereby realizing feedback between robot arms, without any receiving of further information during execution of the path. This may be suitable for applications with high predictability, wherein some concurrent action of robot arms is desirable (e.g., for increasing speed), yet the feedback between robot arms according to predetermined 3D paths may be sufficient, without any (real-time) correction being needed.

[0040] In embodiments, the first and second instructions consist of PLC instructions.

[0041] In embodiments, the measured physical interaction relates to a detected physical proximity between the first and the second robot arm, preferably relates to a distance between the first and second robot arm being lower than a pre- determined threshold. This may relate, e.g., to a distance being lower than 50 mm or 40 mm or 30 mm or 20 mm or 15 mm or 10 mm or 5 mm.

[0042] In embodiments, the method further comprises the repeatedly generating, with respect to the reference volume, of third instructions for a third robot arm for manipulating the physical structure at a third portion comprised in the reference volume; wherein the third instructions are generated based at least on a measured interaction between the third robot arm and one of the first and second robot arm.

[0043] In embodiments, the method further comprises the repeatedly generating, with respect to the reference volume, of fourth instructions for a fourth robot arm for manipulating the physical structure at a fourth portion comprised in the reference volume; wherein the fourth instructions are generated based at least on a measured interaction between the fourth robot arm and one of the first, second and third robot arm.

[0044] In embodiments, the receiving of the data on the reference volume relates to automatically detecting a two-dimensional or three-dimensional bounding box comprising the first and second portion of the physical structure. This may advantageously allow for automated initiation when a new physical structure is provided for manipulation.

[0045] In embodiments, the bounding box is automatically detected as 3D bounding box surrounding the physical structure. This may allows to define the reference volume dynamically, as a cuboid with a portion of a working area as base (depth D and width W) and a height H, wherein each of D, W and H are defined relative to the detected bounding box dimensions DB, WB, and HB. For instance, the reference volume may be centered such that the middle of the bounding box and the base of the reference volume coincide, with D= p*DB, W= p*WB, and H=

H“HB, whereinp may be 1.2 or 1.3 or 1.5 0r1.60r1.70r1.8o0r1980r20r250r3 orá4orborbor7or8or9Sor 10.

[0046] In embodiments, the reference volume may be defined statically as a cuboid with some portion of a working area as base and a height H, wherein the height may be expressed in absolute terms (e.g., 20 mm or 30 mm or 40 mm or 50 mm or 100 mm or 150 mm or 200 mm or 300 mm or 400 mm or 500 mm or 600 mm or 700 mm), but may also be expressed relative to the pre-determined maximum height HT of the physical structure, e.g., 4/3*HT or 3/2*HT or 2*HT or 3*HT or 4*HT.

[0047] In embodiments, the automatically detecting of the two-dimensional or three-dimensional bounding box relates to acquiring sensor data from at least two sensors and applying a bounding box detection algorithm to said sensor data. In related embodiments, the at least two sensors comprise at least two cameras, wherein the sensor data relates to image data, and wherein the bounding box detection algorithm relates to a computer vision detection algorithm.

[0048] In embodiments, the PLC instructions relate to a PLC module comprising means for controlling the first robot arm to manipulate the first portion and the second robot arm to manipulate the second portion concurrently, said controlling including the calculating of 3D paths such that the robot arms do not collide during the concurrent manipulation.

[0049] In embodiments, the physical structure relates to a physical object and packing material, and the manipulating relates to manipulating the packing material at its first and second portion being end portions for realizing a packing with several turns, preferably according to a labyrinth-like packing. This relates to the manipulation-intensive task of folding of an instrument tray into a packaging material. The packing relates to ensuring that medical tools and instruments which are used during a medical intervention are supplied in a sterile fashion. Thereby, instrument trays comprising cleaned instruments are packed into packaging material, wherein the instrument tray may be sterilised either before packing or after packing. The packing thereby serves to ensure that the sterilized condition can be maintained as long as possible over time, typically being done with a labyrinth-like packing pattern, requiring many folds. Such folding is inherently manipulation-intensive, which may entail physical complaints for operators doing the folding routineously. Such folding may furthermore be complicated by practical considerations. For instance, it may occur that an instrument tray is not manually positioned in the correct place on the packaging material, whereby during folding it becomes clear that the required overlap of packaging parts cannot be achieved.

There is also a risk that the correct folding pattern is not achieved due 10 imperfections, wherein for example the minimum number of turns for the envisaged labyrinth-like packing is not achieved. For these reasons, accurate automation of the folding is highly desirable.

[0050] In embodiments, the labyrinth-like packing relates to a "modified package” method (Dutch: gemodificeerde pakket methode). This is further illustrated by

Example 2. In further example embodiments, the folding is done as illustrated by

Figure 2a-2i and corresponding disclosure of NL2019198B1. As may be clear, realizing packing according to the modified package method involves many consecutive manipulations, and the present invention may provide a significant speed-up over what is disclosed, e.g., in NL2019198B1, by allowing operation within the same reference volume by multiple arms at the same time, as enabled by the measured physical interaction between robot arms measured by means of the feedback means.

[0051] In embodiments, the labyrinth-like packing relates to an "envelope" packing method (Dutch: enveloppe techniek). This is further illustrated by Example 2.

[0052] In embodiments, the manipulating at the first and second portion relates to a respective first and second instrument, wherein the method further comprises repeatedly generating instructions for exchanging one of the first and second instrument for a third instrument, and repeatedly generating further instructions for manipulating the packing material by means of said third instrument. This may be advantageous, e.g., for cases where the manipulation involves more different instruments than robot arms. The first, second and third instrument may thereby relate to any means with functionality for tooling and/or gripping. In example embodiments, the instruments relate to tooling and/or grippers.

[0053] In embodiments, at least one of the first robot arm and the second robot arm is a 6-axis robot arm, preferably wherein each of the robot arms is a 6-axis robot arm. However, the robot arms may be provided with any suitable number of axes.

In example embodiments, at least on the robot arm and the second robot arm is a 3-axis or 4-axis or 5-axis or 7-axis robot arm.

[0054] Example embodiments of the invention will be described with reference to

Figs. 1-7.

[0055] Example 1: example system according to the invention

[0056] Fig. 1-5 show different perspective views relating to an example system 10 according to the invention. Thereby, Fig. 1 and 2 show a rear view and a side view, respectively, while Fig. 3-5 show three frontal close-up views.

[0057] The system 10 is provided with four robot arms (1-4) which are essentially identical, in variant examples (not shown) they may differ. Each of the robot arms is a 6-axis robot arm.

[0058] The robot arms (1-4) comprise different parts, as illustrated on Figure 5 for robot arm 1, with at least a shoulder blade 1a, an upper arm 1b, a lower arm 1c, and a wrist 1d. When an instrument is provided at the end (e.g., the instrument 7) this may be considered as the hand, whereas the part connecting the upper arm 1b and the shoulder blade 1a may be considered as the shoulder. Thereby, the first robot arm 1 is provided on a surface portion lying in a first non-horizontal plane 11,

whereas the second, third and fourth robot arm 2-4 are provided surface portions lying in a shared second plane 12 being an essentially horizontal plane, which furthermore comprises an essentially rectangle work area 5 extending within said second plane. Thereby, the surface portion of the first robot arm is situated higher than the second plane, and the second plane and first plane define an angle a between them, with a between 30 and 60 degrees.

[0059] The system comprises a PLC module (not shown) that controls the motion of each of the robot arms 1-4 with corresponding PLC instructions.

[0060] Figures 1 and 2 show the system in non-operational mode, with none of the arms involved in any manipulation, and are each in a resting position (wherein the shown resting position is merely one example resting position). Figures 3-5 provide views relating to operational modes. Particularly, Figures 3-5 illustrate manipulating by means of an instrument 7 comprising a gripper. Further elements 6, 8 and 9 are iflustrated on Fig. 3 and 5.

[0061] The system is provided for manipulating a physical structure (not shown) by means of each of the robot arms, wherein the manipulation involves manipulating first and second portions of the physical structure. The physical structure relates to a physical object being an instrument tray and packing material, and the manipulating relates to manipulating the packing material at its first and second portion being end portions for realizing a packing with several turns, preferably according to a labyrinth-like packing. Particularly, the packing relates to folding of an instrument tray into a packaging material, to ensure that medical tools and instruments which are used during a medical intervention are supplied in a sterile fashion.

[0062] In this example, the labyrinth-like packing relates to a "modified package" method (Dutch: gemodificeerde pakket methode), whereof one example is discussed in Example 2 (without being limited thereto). As may be clear, realizing packing according to the modified package method involves many consecutive manipulations, and the present invention may provide a significant speed-up over what is disclosed, e.g., in NL2019198B1, by allowing operation within the same reference volume by multiple arms at the same time.

[0063] Particularly, with the method of the invention, a physical interaction between the robot arms 1-4 is measured by means of a feedback means, in this example feedback means that are provided within the PLC module and the robot arms.

[0064] Particularly, the feedback means and the measured physical interaction are provided by having the PLC module calculate, based on desired positions of the robot arms, a 3D path for each of the robot arms. Thereby, the repeatedly receiving of information of the feedback means allows for correcting the 3D path of the robot arms, to prevent collisions or otherwise undesirable behavior. The feedback means relates to sensors specific to each robot arm, providing real-time information on the position (i.e., the coordinate) of the robot arm.

[0065] The measured physical interaction is realized by receiving, by the PLC module, data from the sensors of the different robot arms, and measuring, from this information, the physical interaction between the robot arms. This may be advantageous as it allows two or more robot arms to work within the same small reference volume at the same time, rather than having to work in consecutive steps. Also, this may be advantageous as it allows the packing to take place with corrections in real-time. For instance, the measured physical interaction may indicate that an instrument tray is not aligned correctly with respect to the packaging material, whereby during folding it becomes clear, from the measured physical interaction, that the required overlap of packaging parts cannot be achieved. This may cause instructions to be generated to halt one, or each of the robot arms, and may furthermore cause an alert to be generated, e.g., an alert for the attention of an operator. in other examples, the measured physical interaction may reveal that the correct folding pattern is not achieved due to imperfections, wherein for example the minimum number of turns for the envisaged labyrinth-like packing is not achieved. This may equally cause instructions to be generated to halt one, or each of the robot arms, and may furthermore cause an alert to be generated, e.g., an alert for the attention of an operator.

[0066] In this example, the system comprises a weight sensor measuring the weight of any given instrument tray and sending it to the PLC modules via one of the I/O points of the PLC module.

[0067] In this example, the bounding box is automatically detected as 3D bounding box surrounding the instrument tray. The detection involves two cameras connected to a further device carrying out bounding box detection through computer vision, sending the detected bounding box dimensions to an I/O point of the PLC module. The PLC module then calculates the reference volume as a function of the bounding box, taking into account the space needed for carrying out manipulations during the folding. This allows to define the reference volume dynamically, as a cuboid with a portion of the working area 5 as base (depth D and width W) and a height H, wherein each of D, W and H are defined relative to the detected bounding box dimensions DB, WB, and HB. In this example, the reference volume is centered such that the middle of the bounding box and the base of the reference volume coincide, with D= p*DB, W= WWB, and H= p*HB, wherein p is 2 or3or4dor5or6or7or8or9 or 10 depending on the type and dimensions of the instrument trays, the detected weight detected by a weight sensor, and the type of packing.

[0068] In this example, the system comprises a supply means for supplying packing material (not shown) and a supply means for supplying an instrument tray (not shown). The packing material is supplied from a roll and must be cut off at predefined lengths to form a substantially rectangular packaging material. To realize this, the packing material is clamped by means of the gripper belonging to instrument 7, allowing to align the packing material with cutting means. The cutting means may relate, e.g, to a blade and/or an ultrasonic blade.

[0069] Instrument 7 relates to only a single instrument provided on only one of the robot arms. In variants of this example (not shown), the manipulating at the first and second portion relates to a respective first and second instrument, wherein the method further comprises repeatedly generating instructions for exchanging one of the first and second instrument for a third instrument, and repeatedly generating further instructions for manipulating the packing material by means of said third instrument.

[0070] Example 2: various examples of labyrinth-like packing relating to aspects of the invention

[0071] Figures 6-7 illustrate two different example types of labyrinth-like packing relating to aspects of the invention. Thereby, Figure 6 illustrates labyrinth-like packing according to one example of a modified package method, with steps 61-66.

On the other hand, Figure 7 illustrates labyrinth-like packing according to one example of an envelope packing method, with steps 71-79. As may be understood from the figures, some of the steps may be performed slightly differently or may be swapped in sequence, leading to many variant examples of the modified package model, on the one hand, and the envelope packing method, on the other hand, respectively. As is also clear from these figures, such packing is manipulation- intensive, and entails many packing steps. The present invention, with measured physical interaction between the first and second robot arm measured by means of a feedback means, may advantageously allow to do at least some of the steps in cycles that partially overlap in time, rather than doing the steps in a strict sequence.

Also, the invention may advantageously enable to detect, and correct for, the many unexpected factors that may arise in such a complex operation, which may relate, e.g. to material imperfections or imperfect alignment or problems relating to the instrument tray having one or more instruments sticking out.

[0072] (End of example 2)

[0073] in the above, several examples are given wherein the physical structure relates to a physical object and packing material, and the manipulating relates to manipulating the packing material at its first and second portion being end portions for realizing a packing with several turns according to a labyrinth-like packing.

However, the invention is not limited thereto. As may be clear to the skilled person, the invention may be applied to a whole range of manipulation-intensive tasks, which may relate to any of the repeated folding of sheet-like material (also for applications beyond packing), or the collecting and depositing of multiple objects in a container (e.g., the collecting of instruments and depositing it in an instrument tray), or any other application involving manipulating a physical structure at several portions by means of two or more robot arms.

Claims

Conclusions

1. A method for generating first and second instructions for machining a physical structure comprising a first portion and a second portion, comprising: receiving data at a reference volume, the reference volume comprising the first portion and the second portion of the physical structure; repeatedly generating first instructions for a first robotic arm for machining the physical structure on the first portion, relative to the reference volume; repeatedly generating second instructions for a second robotic arm for machining the physical structure on the second portion, relative to the reference volume; wherein the first and second instructions comprise programmable logic controller, PLC, instructions; and wherein the first and second instructions are generated based on at least one measured physical interaction between the first and second robotic arms measured by a feedback means.

2. The method of claim 1, wherein the first and second instructions are PLC instructions.

Method according to claim 1 or 2, wherein the measured physical interaction relates to a detected physical proximity between the first and second robot arms, preferably relates to a distance between the first and second robot arms being lower than a predetermined threshold.

4. The method of claims 1 to 3, further comprising repeatedly generating, relative to the reference volume, third instructions for a third robotic arm to process the physical structure on a third portion located in the reference volume; wherein the third instructions are generated based on at least one measured interaction between the third robotic arm and one of the first and second robotic arms.

5. The method of claim 4, further comprising repeatedly generating fourth instructions, relative to the reference volume, for a fourth robotic arm to process the physical structure on a fourth portion located in the reference volume; wherein the fourth instructions are generated based on at least one measured interaction between the fourth robotic arm and one of the first, second, and third robotic arms.

6. A method according to claims 1 to 5, wherein receiving the data on the reference volume involves automatically detecting a two-dimensional or three-dimensional bounding box comprising the first and second portions of the physical structure.

7. A method according to claim 6, wherein automatically detecting the two-dimensional or three-dimensional bounding box involves acquiring sensor data from at least two sensors and applying a bounding box detection algorithm to said sensor data.

The method of claim 7, wherein the at least two sensors comprise at least two cameras, the sensor data relates to image data, and the bounding box detection algorithm relates to a computer vision detection algorithm.

9. Method according to claims 1-8, preferably claims 2-8, wherein the PLC instructions relate to a PLC module comprising means for simultaneously controlling the first robot arm to machine the first part and the second robot arm to machine the second part, said control comprising calculating 3D paths such that the robot arms do not collide during simultaneous machining.

10. Method according to claims 1-9, wherein the physical structure relates to a physical object and packaging material, and the processing relates to processing packaging material on its first and second part being end portions to realize a package with multiple bends, preferably according to a labyrinth-like package.

11. A method according to claim 10, wherein the processing of the first and second portions relates to a first and second instrument respectively, and wherein the method further comprises repeatedly generating instructions for exchanging one of the first and second instruments for a third instrument, and repeatedly generating further instructions for processing the packaging material by means of said third instrument.

12. Method according to claims 1-11, wherein at least one of the first robot arm and the second robot arm is a 6-axis robot arm, preferably wherein each of the robot arms is a 6-axis robot arm.

13. Apparatus for generating first and second instructions each comprising PLC instructions for operating a physical structure comprising a first portion and a second portion, the apparatus comprising a PLC module comprising a processor for performing a method according to claims 1-12.

14. A system for generating first and second instructions for processing a physical structure comprising a first portion and a second portion, the system comprising: an apparatus, preferably the apparatus according to claim 13; a first and second robot arm connected to said apparatus and preferably positioned at different respective angles relative to said physical structure; a feedback means for measuring a physical interaction between the first and second robot arms and connected to said apparatus; said apparatus being adapted to: receive data on the reference volume, the reference volume comprising the first portion and the second portion of the physical structure; repeatedly receiving, from said feedback means, a measured physical interaction between the first and second robot arms;

repeatedly generating, relative to the reference volume and based on the measured physical interaction, first instructions for the first robot arm to process the physical structure on the first portion; repeatedly generating, relative to the reference volume and based on the measured physical interaction, second instructions for the second robot arm to process the physical structure on the second portion; wherein the first and second robot arms, respectively, are configured for: repeatedly receiving, from the apparatus, the first and second instructions, respectively; repeatedly processing, based on the first and second instructions, respectively, said physical structure on said first and second portions, respectively; wherein the feedback means are configured for: repeatedly measuring the physical interaction between the first and second robot arms; and repeatedly transmitting the measurements of physical interaction from the apparatus; wherein the instructions comprise programmable logic controller, PLC, instructions.

15. A computer program product comprising a medium for storing instructions for performing a method according to claims 1-12.