DE102022124000A1 - Method for determining boundaries of and immovable objects in deployment areas for autonomous and/or partially autonomous all-terrain vehicles for deploying the all-terrain vehicles in the deployment areas - Google Patents

Abstract

The invention relates to a method for determining boundaries (3) of and immovable objects in areas of use (4) for autonomous and/or partially autonomous all-terrain vehicles (1) for use of the all-terrain vehicles (1) in the areas of use (4), comprising the steps: - (20) providing images of the area of application (4) from a first viewing direction, the images comprising geo-coordinates;- (40) determining boundaries (3) of the area of application (4) using the images;- (50) calculating geo-coordinates of the Boundaries (3) based on the images and the determined boundaries (3); - (70) providing a 3-dimensional representation of the application area (4), the 3-dimensional representation comprising views from other viewing directions for at least parts of the application area (4). - (80) determining immovable objects within the operational area (4) based on the 3-dimensional representation using a neural network;- (90) classifying the determined immovable objects using the neural network;- (100) determining geo-coordinates of the determined boundaries (3) and/or the immovable objects;

Description

The invention relates to a method for determining the boundaries of and immovable objects in areas of use for autonomous and/or partially autonomous all-terrain vehicles for use of the all-terrain vehicles in the areas of use, an all-terrain vehicle and a computer program product.

The US2006095171 discloses a method for generating a route for an autonomous all-terrain vehicle.

• - Bereitstellen von Bildern des Einsatzgebietes aus einer ersten Ansichtsrichtung, wobei die Bilder Geokoordinaten umfassen;
• - Ermitteln von Begrenzungen des Einsatzgebietes anhand der Bilder;
• - Berechnen von Geokoordinaten der Begrenzungen anhand der Bilder und der ermittelten Begrenzungen;
• - Bereitstellen einer 3-dimensionalen Repräsentation des Einsatzgebietes, wobei die 3-dimensionale Repräsentation Ansichten aus weiteren Ansichtsrichtungen für wenigstens Teile des Einsatzgebietes umfasst;
• - Ermitteln von unbeweglichen Objekten innerhalb des Einsatzgebietes anhand der 3-dimensionalen Repräsentation mittels eines neuronalen Netzes;
• - Klassifizieren der ermittelten unbeweglichen Objekte mittels des neuronalen Netzes;
• - Ermitteln von Geokoordinaten der ermittelten Begrenzungen und/oder der unbeweglichen Objekte;

A method according to the invention for determining boundaries of and immovable objects in areas of application for autonomous and/or partially autonomous all-terrain vehicles for use of the all-terrain vehicles in the areas of use comprises the steps:

• - Providing images of the deployment area from a first viewing direction, the images comprising geo-coordinates;
• - Determination of limitations of the area of application based on the images;
• - Calculation of geo-coordinates of the boundaries based on the images and the determined boundaries;
• - Providing a 3-dimensional representation of the area of application, wherein the 3-dimensional representation includes views from other viewing directions for at least parts of the area of application;
• - Determination of immovable objects within the deployment area based on the 3-dimensional representation using a neural network;
• - Classifying the determined stationary objects by means of the neural network;
• - Determining geo-coordinates of the determined boundaries and/or the immovable objects;

Because a 3-dimensional representation of the area of application is used to determine immovable objects, and because the 3-dimensional representation includes views from other viewing directions, it is possible for the immovable objects to be classified precisely. This makes it possible for the immovable objects, including the geo-coordinates, to be known before the all-terrain vehicle is used. This makes it possible to determine missions based on the immovable objects.

Areas of use preferably include agricultural fields and/or cultivated areas, mining areas for mining raw materials such as coal and/or ores, or military areas of use.

Off-road vehicles preferably include agricultural vehicles, mining vehicles and/or military vehicles.

Boundaries preferably include edges of fields to be cultivated and/or mine areas in which the all-terrain vehicle is intended to operate.

Immovable objects preferably include trees, buildings, bridges, permanently installed devices and/or infrastructure components.

As a rule, immobile objects in deployment areas are determined manually by driving through the deployment area with a vehicle that includes sensors for detecting the deployment area. The vehicle can be the emergency vehicle. Human intervention is essential for this, for example to assess whether and how immovable objects have to be taken into account in operations. It is often essential that a human determines the type of immovable object. Then the geo-coordinates of the objects must also be determined manually. As soon as boundaries and/or immovable objects change within the area of operation, this manual process has to be repeated.

In contrast to this, the method according to the invention enables an automated determination of boundaries and immovable objects. This saves an enormous amount of time and money compared to manually capturing the boundaries and immovable objects.

In addition, the method according to the invention is scalable in terms of the size of the areas of use and is therefore suitable for large mining areas as well as for small agricultural fields. The method according to the invention is also scalable with regard to the accuracy of the determined boundaries and the immovable objects. Greater accuracy goes hand in hand with a longer execution time.

• Ermitteln eines ersten Einsatzes für ein Geländefahrzeug unter Berücksichtigung der ermittelten Begrenzungen und/oder ermittelten und klassifizierten unbeweglichen Objekte.

The method according to the invention preferably comprises the step:

• Determining a first deployment for an all-terrain vehicle taking into account the determined boundaries and/or determined and classified immovable objects.

This allows for determining the first bet to overcome the limitations and the immovable objects are taken into account. As a result, the field of application can be limited, for example, by the limitations. In addition, this enables the all-terrain vehicle to avoid immovable objects during the first deployment. This prevents the emergency vehicle from colliding with the stationary objects.

• - Durchführen eines zweiten Einsatzes durch das Geländefahrzeug;
• - Erfassen von Sensordaten von Teilen des Einsatzgebietes mittels Sensoren des Geländefahrzeugs während des zweiten Einsatzes des Geländefahrzeugs,
• - Analysieren der Sensordaten zum Ermitteln von zweiten Begrenzungen des und/oder unbeweglichen Objekte innerhalb des Einsatzgebietes anhand der Sensordaten;
• - Vergleichen der Begrenzungen des und/oder unbeweglichen Objekten innerhalb des Einsatzgebietes und der zweiten Begrenzungen des und/oder unbeweglichen Objekte innerhalb des Einsatzgebietes;
• - Verifizieren der Begrenzungen des und/oder unbeweglichen Objekten innerhalb des Einsatzgebietes für den Fall, dass der Vergleich eine Übereinstimmung der Begrenzungen des und/oder unbeweglichen Objekten innerhalb des Einsatzgebietes und der zweiten Begrenzungen des und/oder unbeweglichen Objekte innerhalb des Einsatzgebietes ergeben hat.

The method according to the invention preferably comprises the steps:

• - performing a second mission by the off-road vehicle;
• - Acquisition of sensor data from parts of the deployment area using sensors of the all-terrain vehicle during the second deployment of the all-terrain vehicle,
• - Analyzing the sensor data to determine second boundaries of the and / or immovable objects within the area of use based on the sensor data;
• - comparing the boundaries of the and/or immovable objects within the deployment area and the second boundaries of the and/or immovable objects within the deployment area;
• - verifying the boundaries of the and/or immovable objects within the deployment area in the event that the comparison has revealed a match between the boundaries of the and/or immovable objects within the deployment area and the second boundaries of the and/or immovable objects within the deployment area.

Missions preferably include moving the mission vehicle along a path through the mission area.

Because the boundaries are verified on the second mission, it is made possible for the boundaries and immovable objects to always be up to date and to be taken into account for determining further missions.

An all-terrain vehicle according to the invention is designed and set up to carry out an operation that was determined using a method according to the invention and/or to carry out a method according to the invention. The all-terrain vehicle is partially and/or fully autonomous. The all-terrain vehicle includes sensors for acquiring sensor data from parts of a deployment area.

A computer program product according to the invention comprises a program which, when it is executed by a computer, causes the computer to carry out a method according to the invention for determining boundaries of and immovable objects in areas of application for autonomous and/or partially autonomous all-terrain vehicles for use of the all-terrain vehicles in the areas of application.

The advantages of the method according to the invention result in the same way for the off-road vehicle according to the invention and the computer program product according to the invention.

The dependent claims describe advantageous embodiments of the invention.

• 1 ein Ausführungsbeispiel eines erfindungsgemäßen Verfahrens zum Ermitteln von Begrenzungen von und unbeweglichen Objekten in Einsatzgebieten für autonome und/oder teilweise autonome Geländefahrzeuge für Einsätze der Geländefahrzeuge in den Einsatzgebieten durchzuführen,
• 2 ein Ausführungsbeispiel eines erfindungsgemäßen Geländefahrzeugs und
• 3 zeigt ein schematisches Ausführungsbeispiels eines Einsatzgebietes mit Begrenzungen.

Preferred exemplary embodiments are explained in more detail with reference to the following figures. while showing

• 1 to carry out an exemplary embodiment of a method according to the invention for determining boundaries of and immovable objects in areas of application for autonomous and/or partially autonomous all-terrain vehicles for use of the all-terrain vehicles in the areas of application,
• 2 an embodiment of an all-terrain vehicle according to the invention and
• 3 shows a schematic embodiment of an application area with limitations.

2 1 shows an exemplary embodiment of an all-terrain vehicle 1 according to the invention and an inventive computer program product 2. The all-terrain vehicle 1 includes sensors 5 for acquiring sensor data from at least parts of an application area 4. The all-terrain vehicle 1 is designed as an agricultural utility vehicle.

The computer program product 2 according to the invention comprises a program which, when it is executed by a computer, causes the computer to carry out a method according to the invention for determining boundaries 3 of and immovable objects in areas of application 4 for autonomous and/or partially autonomous all-terrain vehicles 1 for use of the all-terrain vehicles 1 to be carried out in the operational areas 4.

1 shows an exemplary embodiment of a method according to the invention for determining boundaries 3 of and immovable objects in operational areas 4 for autonomous and/or partially autonomous all-terrain vehicles 1 for deployments of the all-terrain vehicles 1 in the operational areas 4.

Operational area 4 is designed as an agricultural field on which potatoes are grown are. The all-terrain vehicle 1 is designed as an autonomous harvesting machine.

In a first step 10, images of the deployment area 4 are created using a drone to provide the images.

The drone is equipped with a high-resolution camera. When flying over the operational area 4 , the drone takes pictures of the operational area from the air, as a result of which these images of the operational area 4 show a plan view of the operational area 4 . This makes it possible for small objects, which are located within the deployment area 4, to be captured by the images.

In a subsequent step 20, images of the deployment area 4 are provided from a first viewing direction, the images including geo-coordinates.

The drone includes a device for determining geo-coordinates of recorded images. The determined geo-coordinates are included in the images.

As an alternative to step 10, the images in step 20 are high-resolution satellite images. The satellite images also include geo-coordinates of the images. This makes it possible to dispense with the use of a drone. This enables a significant time reduction for the provision of the images, since they do not have to be created in advance. However, the images provided only allow detection of smaller objects within the operational area 4 to a limited extent.

In a subsequent step 30, a Sobel filter is used to determine the boundaries 3 of the area of application 4.

3 shows a Sobel filter applied to a satellite image of the operational area 4. This filter makes it possible to reliably determine field boundaries and thus boundaries 3 . The boundaries are clearly visible as white markings against the black field.

Alternatively, a Prewitt filter or a Lapace filter is used.

In a subsequent step 40, delimitations 3 of the area of use 4 are determined using the images.

In this exemplary embodiment, a rule-based analysis of the white markings is carried out using a computer. In particular, the course and the width of the white markings are taken into account in order to determine the boundaries.

In a subsequent step 50, geo-coordinates of the boundaries 3 are calculated using the images and the determined boundaries 3.

Geocoordinates in the form of GPS coordinates are included both from satellite images and from the images created by the drone. Using these GPS coordinates and the determined boundaries, the GPS coordinates of the boundaries are now determined by comparison.

In a subsequent step 60, additional images of the deployment area 4 are created from additional viewing directions using a drone to provide the 3-dimensional representation of the deployment area 4.

For this purpose, the drone moves at a low altitude of up to 20 m above the operational area 4. The images therefore show at least partial side views of the operational area 4. In this exemplary embodiment, the side views are created from different directions. The side views and the top views are now processed into a 3-dimensional representation of the application area 4. Common algorithms for creating 3-dimensional representations are used.

In a subsequent step 70, a 3-dimensional representation of the area of use 4 is provided. The 3-dimensional representation includes views from other viewing directions for at least parts of the area of use 4.

As an alternative to step 60, existing 3-dimensional representations of the operational area 4 are used, such as are available on maps on the Internet. This enables enormous time savings compared to carrying out step 60. However, the resolution of the 3-dimensional representation of the operational area 4 can be significantly increased if step 60 is carried out.

In a subsequent step 80, immobile objects within the operational area 4 are determined based on the 3-dimensional representation using a neural network.

There are now images from different viewing directions, also of objects within the operational area 4. This enables a neural network to detect objects from these images. The neural network was trained to detect and classify objects in images.

In a subsequent step 90, the identified immovable objects are classified using the neural network.

In this exemplary embodiment, there is a tree as an immovable object within the application area 4. The neural network determines the tree and classifies it as a tree.

In a subsequent step 100, geo-coordinates of the determined boundaries 3 and/or the immovable objects are determined.

By comparing the GPS coordinates of the tree are now determined using the images. The boundaries of the tree are also determined. This allows for an accurate determination of the size of the tree.

In a subsequent step 110, a first use for an all-terrain vehicle 1 is determined, taking into account the determined boundaries 3 and/or determined and classified immovable objects.

The tree would not allow the harvester to run over it. However, the harvesting machine should cover as large a part of the field as possible in order to maximize a crop yield. Therefore, the tree should be avoided as closely as possible. This is made possible by determining the size of the tree. As a first step, a path for driving down the field is now planned so that the harvesting machine allows a complete harvest of the field. The tree is excluded from the path. However, this does not reduce the crop yield, since potatoes cannot be present at the location of the tree.

In a subsequent step 120, a second operation is carried out by the all-terrain vehicle 1.

The first and second inserts are identical in this embodiment. During the second use, the harvesting machine follows the planned path and automatically harvests potatoes.

In step 130, which is executed at the same time, sensor data is recorded from parts of the deployment area 4 using sensors 5 of the all-terrain vehicle 1 during the second deployment of the all-terrain vehicle 1.

In this exemplary embodiment, the sensors include 5 cameras and a module for determining GPS coordinates. While driving along the planned path, the camera takes pictures while the module determines the GPS coordinates of the pictures. A number of the images show the tree from different directions.

In a subsequent step 140, the sensor data is analyzed to determine second boundaries of the and/or immovable objects within the deployment area 4 using the sensor data.

In this exemplary embodiment, the analysis is carried out analogously to steps 80 and 90. The tree is again classified as such.

In a subsequent step 150, the boundaries 3 of the objects and/or immovable objects within the area of use 4 and the second boundaries of the objects and/or immovable objects within the area of use 4 are compared.

In this exemplary embodiment, the two determined data sets of the tree are compared and they match.

In a subsequent step 160, the boundaries 3 of the and/or immovable objects within the deployment area 4 are verified in the event that the comparison shows a match between the boundaries 3 of the and/or immovable objects within the deployment area 4 and the second delimitations of and/or or immovable objects within Area of Operation 4.

In this embodiment, therefore, the position and class of the tree is verified.

As an alternative to step 160, in a step 170 the boundaries 3 of the objects and/or immovable objects within the operational area 4 are changed to the second boundaries of the objects and/or immovable objects within the operational area, in the event that the comparison does not show that the boundaries 3 of the and/or immovable objects within the operational area 4 and the second boundaries of the and/or immovable objects within the operational area 4.

In this alternative exemplary embodiment, a new building was erected within operational area 4. This is now recorded and classified according to the steps described. The second mission will now be adjusted so that the building is also bypassed.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• US2006095171 [0002]

Claims (10)

Method for determining boundaries (3) of and immovable objects in areas of use (4) for autonomous and/or partially autonomous all-terrain vehicles (1) for use of the all-terrain vehicles (1) in the areas of use (4), comprising the steps: - (20) providing images of the deployment area (4) from a first viewing direction, the images comprising geo-coordinates; - (40) Determination of boundaries (3) of the area of use (4) based on the images; - (50) calculating geo-coordinates of the boundaries (3) using the images and the determined boundaries (3); - (70) providing a 3-dimensional representation of the area of use (4), the 3-dimensional representation comprising views from other viewing directions for at least parts of the area of use (4); - (80) determining immobile objects within the deployment area (4) based on the 3-dimensional representation by means of a neural network; - (90) classifying the determined stationary objects by means of the neural network; - (100) determining geo-coordinates of the determined boundaries (3) and/or the immovable objects; procedure after claim 1 , comprising the step of: - (110) determining a first deployment for an all-terrain vehicle (1) taking into account the determined boundaries (3) and/or determined and classified immovable objects. Method according to one of the preceding claims, comprising the steps of: - (120) performing a second mission by the off-road vehicle (1); - (130) acquisition of sensor data from parts of the deployment area (4) using sensors (5) of the all-terrain vehicle (1) during the second deployment of the all-terrain vehicle (1), - (140) Analyzing the sensor data to determine second boundaries of the and/or immovable objects within the deployment area (4) based on the sensor data; - (150) comparing the boundaries (3) of the and/or immovable objects within the deployment area (4) and the second boundaries of the and/or immovable objects within the deployment area (4); - (160) Verifying the boundaries (3) of the and / or immovable objects within the area of operation (4) in the event that the comparison shows a match of the boundaries (3) of the and / or immovable objects within the area of \u200b\u200b(4) and the second limitations of and/or immovable objects within the operational area (4). Method according to one of the preceding claims, comprising the step: - (170) Changing the boundaries (3) of the and/or immovable objects within the area of operation (4) to the second boundaries of the and/or immovable objects within the area of operation in the event that the comparison does not indicate a match of the boundaries (3) of the and/or immovable objects within the deployment area (4) and the second boundaries of the and/or immovable objects within the deployment area (4). Method according to one of the preceding claims, comprising the step: - (10) Creation of images of the operational area (4) using a drone to provide the images. Method according to one of the preceding claims, comprising the step: - (30) Applying a Sobel filter, a Prewitt filter and/or a Laplace filter to determine the boundaries (3) of the field of application (4). Method according to one of the preceding claims, comprising the step: - (60) Creation of further images of the operational area (4) from other viewing directions using a drone to provide the 3-dimensional representation of the operational area (4). All-terrain vehicle (1), trained and equipped to carry out an operation that he using a method according to a claims 2 until 6 was determined and / or to carry out a method according to one of claims 2 or 3 , wherein the all-terrain vehicle is partially and/or fully autonomous, and wherein the all-terrain vehicle (1) has sensors (5) for acquiring sensor data from parts of a deployment area (4). All-terrain vehicle (1) after claim 8 , designed as an agricultural utility vehicle and/or as a mining vehicle. Computer program product (2), comprising a program which, when executed by a computer, causes the computer to perform an inventive method according to one of Claims 1 until 7 to perform.

Images