DE102022111179A1 - METHOD AND DEVICE FOR GENERATING AN ARTIFICIAL INTELLIGENCE-BASED PREDICTOR AND USE THEREOF, AND COMPUTER PROGRAM - Google Patents

Abstract

A method is provided for generating an artificial intelligence-based predictor, the method comprising: providing a digital map with landmarks obtained from sensor data; Determining a benefit of the respective landmark and/or the sensor data used to determine it for determining the position of a motor vehicle in the digital map based on predetermined criteria; Creating a training data set comprising the landmarks and/or the sensor data used to determine them and their respective specific benefits; and training the artificial intelligence-based predictor with the training data set, so that the predictor is designed after training to predict the benefit for determining the position of another motor vehicle in a digital map based on a landmark obtained from the recorded sensor data.

Description

The present disclosure relates to a method for generating an artificial intelligence-based predictor and a data processing device configured to at least partially carry out the method. Furthermore, an optionally automated motor vehicle can be provided with the data processing device. The disclosure further relates to a use of the predictor generated by the method in a motor vehicle. Additionally or alternatively, a computer program is provided which includes commands which, when the program is executed by a computer, cause it to at least partially carry out the method. Additionally or alternatively, a computer-readable medium may be provided that includes instructions that, when the instructions are executed by a computer, cause the computer to at least partially execute the method.

In motor vehicles, especially automobiles, the degree of automation is constantly increasing, i.e. more and more driving tasks are being carried out automatically by the motor vehicle itself. The driving tasks are carried out by so-called driving assistance systems, which sometimes require a position of the motor vehicle in order to fulfill the driving tasks. As the level of automation increases, the accuracy with which the position information is required often increases. For this reason, the accuracy of so-called GNSS is sometimes no longer sufficient and landmarks stored in a digital map are used to determine the position of the motor vehicle. Static objects that can be detected or recognized by one or more sensors of the motor vehicle can be referred to as landmarks. These are, for example, road markings, street signs and similar objects.

For example, describes the WO2019053013A1 in this context, a method for determining a position of a motor vehicle in an environment, wherein a respective position information of the landmark is provided for at least one landmark of the environment and it is recognized that the motor vehicle is currently passing the at least one landmark, and the position information of the respective current The landmark passed is output as the position of the motor vehicle. It is provided that a maneuver characteristic of a driving maneuver that occurs when passing the respective landmark is provided, the respective maneuver characteristic indicating predetermined characteristic features of the respective driving maneuver, and then descriptive data of a driving maneuver currently being driven by the motor vehicle is generated by means of at least one detection device of the motor vehicle and Predetermined features of the driving maneuver are extracted from the description data and the currently passed landmark is recognized by comparing the extracted features with the characteristic features from the respective maneuver characteristics of the at least one landmark.

However, in order to use these landmarks in the digital map, they must first be known.

To generate maps, driving environment data from multiple drives is collected and sent to the cloud or to a backend, where digital maps for automated driving functions are created with the help of the driving environment data.

Driving environment data includes, for example, object detections, visual keypoints or radar measurement points.

In the vehicle, the driving environment data is filtered according to certain rules in order to filter out, for example, short-term disturbances/noise (e.g. due to precipitation) or moving objects (e.g. other vehicles).

For robust localization and thus the reliable operation of automated driving functions, the map generated should contain landmarks that can be recognized by different sensor systems. These can differ depending on the sensor system, which is why sensor data from different sensors are required to generate the map. To increase robustness, as many landmarks as possible should also be provided. Both result in a large amount of sensor data having to be sent to the cloud per trip, and the effort and costs for processing it there increase.

To reduce the amount of data, a selection of data is made in the vehicle and sent to the cloud. In the vehicle, for example, multiple measurements of the same object can be aggregated, for example by averaging the measurements or selecting the best measurement based on distance, signal strength and/or other attributes.

However, this can only be done very roughly with expert knowledge alone. In addition, the optimal selection of sensor data to be sent can vary in time and place due to changes in the sensor technology used (e.g. sensor generations, software versions, manufacturers, etc.) and external circumstances (e.g. seasons, building regulations, etc.).

Against the background of this prior art, the object of the present invention is to provide a device and a method which are each suitable for overcoming at least the above-mentioned disadvantages of the prior art.

The task is solved by the features of the independent claim. The subclaims contain preferred developments of the invention.

The task is then solved by a method for generating a predictor based on artificial intelligence.

The method includes providing a digital map with landmarks obtained from sensor data. A landmark can be obtained from multiple landmark observations. This means that sensor data from various measurements in which a potential landmark was detected can be aggregated, for example in a backend, in order to obtain the landmark. In concrete terms, this can mean that when one or more motor vehicles drive through a road section several times, a static object, e.g. a road sign, and its position (e.g. relative to the motor vehicle and the position of the motor vehicle during detection) are recognized several times. These individual detections can each be referred to as a landmark observation. The sensor data relating to the recognized object, e.g. image data, and the recognized position of the object can be sent to the backend, for example, in which, for example, an average of all positions at which the object was recognized is adopted as a landmark in a digital map.

The method includes determining a benefit of the respective landmark for determining the position of a motor vehicle in the digital map based on predetermined criteria. Additionally or alternatively, the method includes determining a use of the sensor data that was used to determine or obtain the landmarks for determining the position of a motor vehicle in the digital map based on predetermined criteria. In particular, the benefit of the respective landmark can be determined based on the predetermined criteria (e.g. how many other landmarks already exist in the vicinity of the landmark) and this benefit of the landmark can be assigned to the sensor data of the respective landmark observation that led to the determination of the landmark . This means that the usefulness of the sensor data can be determined indirectly via the usefulness of the landmark. It is also conceivable, additionally or alternatively, to directly evaluate the benefit of individual landmark observations, i.e. the sensor data. For this purpose, for example, it can be determined at what distance from the motor vehicle to the landmark the object on which the landmark is based was recognized and with what reliability.

The method further includes creating a training data set comprising the landmarks and their respective specific benefits. Additionally or alternatively, the method includes creating a training data set comprising the sensor data that was used to determine or obtain the landmarks and their respective specific benefits.

The method is characterized in that it involves training the artificial intelligence-based predictor with the training data set, so that after training the predictor is designed to be useful for determining the position of another motor vehicle in a (further) digital map based on one to determine the landmark obtained from the recorded sensor data

This means that the predictor can predict the usefulness of sensor data for creating another digital map that is used to determine the position of another motor vehicle. In other words, the predictor can predict the usefulness of the sensor data for generating a digital map in such a way that it best supports position determination in the vehicle.

A predictor based on artificial intelligence can be understood as software that is designed to receive unknown sensor data or data based on sensor data as input data and to make a prediction or estimate based on this unknown data as to how great their benefit is for is a localization solution. Machine learning can be used here. This means that the artificial system or predictor learns from examples, here the already existing landmarks, the already existing sensor data (and possibly their feature vector) and their benefits determined based on the predetermined criteria, and can use the examples after the end of the learning phase or generalize after training. To do this, machine learning algorithms build a statistical model that is based on the training data. This means that the examples are not simply memorized, but patterns and regularities are recognized in the training data. The predictor can also assess unknown data, for example during use in a motor vehicle (so-called learning transfer). This means that there is no need to define a filter based on expert knowledge, which simply defines the captured sensor data based on it ter criteria Evaluate and thus filter can and cannot react to unknown scenarios.

A predictor is proposed that can be used to select sensor data to be sent to the cloud or to the backend. In particular, the predictor can always be used to determine the measurement data, even when the external conditions in the vehicle change, and then transfer it to the cloud, which has a certain minimum benefit for map creation and the driving functions based on it. This allows both the amount of data to be transferred to the cloud to be reduced, which, for example, reduces the costs for data transfer to the cloud, as well as the amount of data to be processed in the cloud, which reduces the required computing power and consequently the costs for the Data processing significantly reduced.

Possible or optional developments of the above method are described below.

The predetermined criteria for determining the usefulness of the respective landmark can include how reliably the respective landmark was detected in individual measurements of the sensor data, how high the spread of the individual measurements is, in particular with regard to their position information relative to a motor vehicle recording the sensor data, and/or how many other landmarks are present in a predetermined area of the respective landmark in the digital map.

The predetermined criteria for determining the usefulness of the sensor data that were used to determine a or the landmark include how (well) or to what extent they match the features with regard to their features, and in particular their position information relative to a motor vehicle receiving the sensor data , and in particular the position of the landmark determined from them and other sensor data. This means that different detections or landmark observations of the same landmark can differ in their usefulness for its creation, e.g. because they are recognizable as outliers with regard to the landmark position. The use of the respective sensor data is therefore not necessarily dependent on the use of the landmark.

The landmarks and/or the sensor data that were used to determine the landmarks can each comprise a feature vector that has at least one piece of information that provides a measurement of the respective landmark or the sensor data (or the observation/measurement/sensor data). that were used to determine the landmarks. The feature vector can be combined with the utility of the respective landmark. the sensor data used to determine the landmarks can be used to train the artificial intelligence-based predictor.

Providing the digital map may include creating the digital map based on the landmarks obtained from the sensor data.

It is conceivable that only or exclusively those landmarks from the digital map are used that lie in a predetermined area of the digital map, that were extracted from sensor data that were recorded at predetermined times of the year and/or day, and/or that Sensor data was extracted that was recorded by a predetermined sensor type.

The predictor can be generated by a backend in the manner described above and the method can have a step of transmitting the trained predictor, i.e. the predictor generated in the manner described above, to a motor vehicle.

Furthermore, a data processing device is provided which includes means for carrying out the method described above. What has been described above with reference to the method also applies analogously to the data processing device and vice versa.

Furthermore, the disclosure relates to a use of the predictor generated using the above method in a motor vehicle for predicting the use of sensor data recorded by means of the motor vehicle for determining the position of another motor vehicle in a digital map based on a landmark obtained from the recorded sensor data.

It is conceivable that only sensor data from the motor vehicle is transmitted to a backend whose benefit predicted by the predictor exceeds a predetermined limit value.

What has been described above with reference to the method and the data processing device also applies analogously to the use and vice versa.

Furthermore, a computer program comprising instructions is provided which, when the program is executed by a first computer, causes it to generate the artificial intelligence-based predictor by executing the above method and optionally to send the generated predictor to a second computer. The first computer can be a backend act. The second computer, which can also be designed to use the received predictor, can be a data processing device in the motor vehicle.

• Mit Hilfe von erfassten Sensordaten wird in der Cloud eine digitale Karte erzeugt. Für die darin enthalten Landmarken wird ausgewertet, wie zuverlässig diese von den einzelnen Fahrten detektiert werden, wie hoch dabei die Streuung der einzelnen Messungen, insbesondere hinsichtlich ihrer Position, ist und wie viele andere Landmarken es in der unmittelbaren Umgebung gibt. Daraus und ggf. anhand weiterer Metriken wird abgeleitet, welchen Einfluss jede Landmarke für eine Güte einer darauf basierenden Lokalisierungslösung hat. Dies dient als Grundlage, um den Nutzen jeder einzelnen Messung für die Erzeugung der digitalen Karte zu bewerten.

What has been described above can be summarized in other words and in relation to a specific embodiment, which is described as not limiting the present disclosure, as follows:

• Using captured sensor data, a digital map is created in the cloud. For the landmarks contained therein, it is evaluated how reliably they are detected by the individual trips, how high the spread of the individual measurements is, especially with regard to their position, and how many other landmarks there are in the immediate vicinity. From this and, if necessary, other metrics, it is derived what influence each landmark has on the quality of a localization solution based on it. This serves as a basis for evaluating the usefulness of each individual measurement for generating the digital map.

In addition to the position information relative to the vehicle, other features can be recorded per measurement and sent to the cloud - in the case of visual landmarks, for example, a SIFT feature vector or the result of semantic segmentation, in the case of RADAR reflections, for example, the strength of the reflection. The resulting feature vector can be used, for example, to train a predictor for the benefit of the measurement with the help of deep neural networks / deep learning. The training takes place using all measurements collected in the cloud, i.e. on any state of the art across a large number of trips and vehicles, with different measurements being made for different regions, times of year/day, sensor versions, etc. through appropriate selection of the measurement data Predictors can be trained.

The determined predictors can, for example, be imported into the vehicles via remote software update in order to enable data collection according to the second filter regime, in which the vehicle only transmits the measurement data to the cloud for which the predictor predicts a certain minimum benefit for map creation .

• 1 zeigt schematisch ein Ablaufdiagramm eines Verfahrens zur Erzeugung eines auf künstlicher Intelligenz beruhenden Prädiktors sowie eine Verwendung des mit dem Verfahren erzeugten Prädiktors.

An embodiment is described below with reference to 1 described.

• 1 schematically shows a flowchart of a method for generating a predictor based on artificial intelligence and a use of the predictor generated with the method.

The procedure for generating an artificial intelligence-based predictor is in 1 shown on the left and runs in a cloud or a backend 1. The procedure includes how 1 can be seen, essentially five steps S1 - S5.

In a first step S1 of the method, a digital map with landmarks obtained from sensor data is provided. As part of providing the digital map, the method can include creating the digital map in the backend based on the landmarks obtained from the sensor data. However, it is also conceivable that the digital map with the landmarks already exists and/or is received from the backend by an external data processing device.

In a second step S2 of the method, a use of the respective landmark for determining the position of a motor vehicle in the digital map is determined based on predetermined criteria. Additionally or alternatively, the use of the sensor data used to determine the landmarks is determined based on predetermined criteria.

The predetermined criteria for determining the usefulness of the respective landmark or the sensor data include how reliably the respective landmark was detected in individual measurements of the sensor data, how high the spread of the individual measurements is, in particular with regard to their position information relative to a motor vehicle recording the sensor data and/or how many other landmarks are present in a predetermined area of the respective landmark in the digital map.

The predetermined criteria for determining the usefulness of the sensor data that were used to determine a or the landmark include how (well) or to what extent they match the features with regard to their features, and in particular their position information relative to a motor vehicle receiving the sensor data , and in particular the position of the landmark determined from them and other sensor data.

In a third step S3 of the method, a training data set is created comprising the landmarks and/or the sensor data that were used to determine them and their respective specific benefits.

The landmarks and/or the sensor data and thus the training data set each have a feature vector that has at least one piece of information that characterizes a measurement of the respective landmark.

In a fourth step S4 of the method, the artificial intelligence-based predictor is trained with the training data set. For this purpose, the feature vector is used together with the benefit of the respective landmark or sensor data. After training or training, which can also be referred to as the learning phase, the predictor is designed to use further sensor data, i.e. previously unknown to the predictor, for determining the position of a motor vehicle in a further, i.e. yet to be created, digital map to predict. More specifically, the usefulness of the sensor data is predicted to determine the extent to which the landmark that can be obtained or extracted from the sensor data is useful as part of the digital map for determining the position of a motor vehicle or a localization solution.

It is conceivable that in the fourth step S4 of the method different predictors are generated or trained for different boundary conditions. For this purpose, only those landmarks from the digital map that lie in a predetermined area of the digital map, that were extracted from sensor data that were recorded at predetermined times of the year and/or day, and/or that from sensor data can be used for the respective predictor were extracted, which were recorded by a predetermined sensor type.

In a fifth step S5 of the method, the trained predictor is transferred to a motor vehicle 2, which uses the predictor generated as described above as part of a filter, which is explained in detail below.

The use (right part in 1 ) of the predictor therefore takes place in the motor vehicle 2, which is used to collect sensor data, for example comprising image data, LiDAR data and/or radar data, which in turn is used to determine landmarks for the digital map after filtering in the motor vehicle 2 from the motor vehicle 2 the backend 1 is transmitted.

The motor vehicle 2, in particular a data processing device of the motor vehicle 2 (not shown), uses the predictor in a first step S6 of using the predictor to determine the use of the sensor data recorded by the motor vehicle 2 for determining the position of another motor vehicle in a digital map based on a Landmark obtained from the recorded sensor data.

In a second step S7 of using the predictor, the determined or estimated benefit is compared with a limit value. If this is exceeded, i.e. if the specific benefit exceeds a minimum benefit, the sensor data are transmitted to the backend 1 in a third step S8 of using the predictor. Otherwise they will be discarded.

The method for generating and using the predictor have been described above as two sequential methods, with the disclosure relating to these two methods both in combination and independently of one another.

Furthermore, the method described above may include creating or supplementing a digital map based on landmarks obtained from sensor data that were filtered using the predictor described above. Furthermore, the method can include using the digital map created or supplemented in this way in a motor vehicle 2 for automated driving.

Reference symbol list

1

Backend or cloud

2

motor vehicle

S1 - S5

Steps of the process for generating the predictor

S6 - S8

Steps of using the predictor

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• WO 2019053013 A1 [0003]

Claims (11)

A method for generating an artificial intelligence-based predictor, the method comprising: - Providing a digital map with landmarks obtained from sensor data, - determining a benefit of the respective landmark and/or the sensor data used to determine it for determining the position of a motor vehicle (2) in the digital map based on predetermined criteria, - Creating a training data set comprising the landmarks and/or the sensor data used to determine them and their respective specific benefits, and - Training the artificial intelligence-based predictor with the training data set, so that the predictor is designed after training to predict the use of further sensor data for determining the position of another motor vehicle in a digital map based on a landmark obtained from the recorded sensor data. Procedure according to Claim 1 , wherein the predetermined criteria for determining the usefulness of the respective landmark or the sensor data that were used to determine the landmark include how reliably the respective landmark was detected in individual measurements of the sensor data, how high the scatter of the individual measurements, in particular with regard to their position information relative to a motor vehicle (2) receiving the sensor data, and/or how many other landmarks are present in a predetermined environment of the respective landmark in the digital map. Procedure according to Claim 1 or 2 , wherein the predetermined criteria for determining the usefulness of the sensor data that were used to determine a landmark include how these are in terms of their features, and in particular their position information relative to a motor vehicle (2) receiving the sensor data, with the features, and in particular the Position of the landmark determined from them and other sensor data match. Procedure according to one of the Claims 1 until 3 , wherein the landmarks and/or the sensor data that were used to determine the landmarks each comprise a feature vector that has at least one piece of information that characterizes a measurement of the respective landmark or the sensor data that was used to determine the landmarks, and the feature vector, together with the utility of the respective landmark or the sensor data used to determine the landmarks, is used to train the artificial intelligence-based predictor. Procedure according to one of the Claims 1 until 4 , wherein providing the digital map includes creating the digital map based on the landmarks obtained from the sensor data. Procedure according to one of the Claims 1 until 5 , wherein only those landmarks and / or sensor data that were used to determine the landmarks are used from the digital map that lie in a predetermined area of the digital map, which were extracted from sensor data at predetermined times of the year and / or time of day were recorded, and/or that were extracted from sensor data recorded by a predetermined sensor type. Procedure according to one of the Claims 1 until 6 , wherein the predictor is generated by a backend (1) and the method has a step of transmitting the trained predictor to a motor vehicle (2). Data processing device, comprising means for carrying out the method according to one of Claims 1 until 7 . Using the method according to one of the Claims 1 until 7 generated predictor in a motor vehicle for predicting the use of sensor data recorded by means of the motor vehicle (2) for determining the position of another motor vehicle in a digital map based on a landmark obtained from the recorded sensor data. Use after Claim 9 , whereby only sensor data from the motor vehicle is transmitted to a backend whose benefit predicted by the predictor exceeds a predetermined limit. Computer program, comprising commands which, when the program is executed by a first computer, optionally a backend, cause it to execute the method according to one of the Claims 1 until 7 to generate the artificial intelligence-based predictor and optionally to send the generated predictor to a second computer, optionally a data processing device in a motor vehicle (2).

Images