DE102022107338B3 - Method for testing automated vehicle functions - Google Patents

Abstract

Method for testing automated driving functions, which determines critical scenario-specific parameter combinations using a high-performance, scalable test environment and feeds them to a complex, trustworthy test environment in order to be able to decide on the release of the automated driving function based on the risk determined.

Description

The invention relates to a method for testing automated driving or vehicle functions, such as driver assistance systems or their functions, by linking the results of different test environments.

State of the art

The promise of increased safety in traffic is a major motivator for the development of automated driving functions through to autonomous vehicles. Proving that automated driving functions are at least as safe as a human driver remains a challenge. The question arises as to whether the testing of these functions should be purely virtual, which allows for a large number of tests in a fully controllable environment. Real field tests, on the other hand, are very complex and cannot be carried out in the same number. Both methods have advantages and disadvantages, which is why the combination of both methods makes sense.

ISO 21448 requires the definition of explicit release criteria to address the risk posed by automated driving functions. A socially recognized release criterion is that the automated vehicle function does not cause more fatal accidents than a human-driven vehicle. Fulfilling this requires estimating the rate of fatal accidents caused by the automated vehicle function, which is non-trivial.

Since accidents occur rather seldom, several million test kilometers would have to be completed in real driving tests in order to achieve statistical representativeness. Another disadvantage of real driving tests in road traffic is the very limited control over general conditions, such as the weather, the number of other road users and the specific scenario to be tested. There is always a risk of endangering other road users.

So-called scenario-based testing is often used to test these systems in order to improve risk assessment by focusing on dangerous situations. Certain situations from road traffic are isolated and tested individually. Such a situation can be, for example, that another vehicle cuts in just in front of an automated vehicle or that a pedestrian crosses the road in front of the vehicle to be tested (functional scenario).

In scenario-based testing, the scenario is described with different parameters. Each of these parameters characterizes a certain property of the scenario. Examples of these parameters can be the differential speed of the vehicles or the time interval at which cutting in is initiated. An interval is set for each of the parameters described, in which tests can be completed. This interval usually describes the lower and upper limit in which the system can operate. The limits are selected on the basis of different criteria such as physical conditions or what is known as the operational domain of the function, i.e. the entire functional area within which the automated function can be used or operated. This process is carried out for each parameter and together this results in a multi-dimensional parameter space in which possible tests can be placed (logical scenario). The number of possible attempts here is theoretically infinite. For this reason, various methods for designing experiments are used in practice. These range from a selection of experiments based on expert knowledge, to methods such as Design of Experiments (DoE) and advanced methods such as iterative processes. The derived tests are called concrete scenarios.

Scenario-based testing is a well-known approach for the systematic investigation of the behavior of an automated vehicle (WINKELMANN et al.: Probabilistic Metamodels for an Efficient Characterization of Complex Driving Scenarios). Complex situations are divided into finite sets of concrete scenarios. Due to the rarity of critical scenarios, an efficient translation of the logical into the concrete scenarios is required. For this purpose, different meta models are examined, which are trained with a limited number of test cases in order to be used for the analysis or a targeted selection of test cases.

During the execution of the tests, information (KPls - Key Performance Indicators) is collected, on the basis of which decisions can be made regarding the further development process or the release of the tested function. An example of a KPI for the above example could be the smallest distance between the two vehicles throughout the scenario. The expected damage in the event of an accident can also be a KPI.

When analyzing scenarios, the aim is often to determine the probability of occurrence of events such as accidents, injuries or deaths. For this purpose, a distribution of the probability of occurrence is determined for the parameters that parameterize the scenario. In the case of the lane change scenario mentioned above, the parameter would be the distance to the vehicle in front at the start of the lane change. The focus of the distribution would be at a moderate to high distance with a decreasing probability of occurrence as the distance decreases. In other words, most lane changes in reality occur at a medium to high distance from the vehicle in front. The expected probability distribution of the scenario-specific parameters in the functional domain of the automated function is denoted by p(x), where x is the parameter vector of the scenario-specific parameters. p(x) can also be modeled as a frequency distribution or density function.

In addition to the probability of the absolute occurrence of the situation, the severity that would cause incorrect behavior for the corresponding parameter setting can also be determined. In this case, intuition suggests that misbehaving at very small distances will result in potentially greater damage. In other words, the expected damage is highest when changing lanes at a very small distance from the vehicle in front. At the same time, the probability of occurrence there is low.

In order to determine the risk of the scenario, both the distribution of the specific scenarios and the damage or hazard within the specific scenarios must be taken into account. Ultimately, the distribution can be described as p(x) and the damage as f (x), the risk then results in f f(x)p (x) dx.

In order to be able to assess the risk and thus the functional safety of automated driving functions as realistically as possible, the selection of the test environment, i.e. the specific test setup, is of great importance. Each test scenario can be tested differently in different test environments. For example, 3D simulations are very well suited to examining the influence of weather conditions at a reasonable cost. In contrast, highly dynamic driving maneuvers can be better investigated on a real test site. Depending on the development stage, test environments of different abstraction can be used along the development process. Examples include software in the loop (SiL) or hardware in the loop (HiL). In general, it can be said that the higher the abstraction of the test environment, the fewer real parts of the vehicle are examined, the fewer resources are required, but the lower the information content.

Great efforts are made to compensate for the respective disadvantages of the test environments, for example through more realistic simulations. Nevertheless, there will probably always be both abstract but highly scalable test environments (e.g. virtual tests) as well as reliable but expensive test environments (e.g. tests in the real world or complex 3D simulations).

With Monte Carlo sampling, the parameters are randomly selected over the entire parameter space following the distribution p(x) and fed into a model that estimates the occurrence of a damage event depending on the respective parameter combination. A probability of occurrence, for example for collisions, is calculated from the total number of tests and the number of test results with damage. A specific scenario is drawn from the distribution p(x) and examined to determine f(x). This process is carried out repeatedly and the integral is determined. However, this method is very inefficient since the scenarios in which the hazard f(x) is high are very rare. For example, a first accident could only be observed after 1000 attempts and it takes around 100,000 attempts before the risk can be assessed with sufficient accuracy.

An alternative method is the so-called importance sampling. Here the trials are not drawn from p(x), but a q(x) is defined from which trials are drawn. This q(x) is chosen so that more accidents/damages are observed. In order to be able to assess the risk, the damage observed for the parameter vector x is then always multiplied by p(x)/q(x) and thus scaled to the actual distribution p(x). In this way, the risk can be assessed much more quickly, since it converges with fewer tests carried out. When using importance sampling, the central challenge is the choice of q(x). q(x) can be calculated on the basis of historical data (usually based on human drivers related) can be determined or estimated by experts. In complex scenarios, however, even experts are often unable to say exactly when a test run will be critical and when not.

Adaptive importance sampling is an iterative process in which the so-called cross entropy method can be used. First, a distribution function q(x) is chosen for the critical parameter space. This can be a parametric or non-parametric probability distribution or a combination of both. Then z. A large number of tests are carried out, for example using Monte Carlo sampling, in order to obtain information about the system behavior. After that, the cross entropy between the selected distribution and the sought distribution is minimized. A sufficiently precise distribution of the critical parameter space can be found by iterative repetition.

Realistic methods such as tests on proving grounds are currently indispensable for reliable risk determination. The determination of q(x) is the decisive factor for how much the risk determination can be accelerated. If q(x) is defined on the basis of expert knowledge or selected on the basis of historical data from other systems (e.g. human drivers), there is no evidence and no reference to the actual system to be examined. Risks can arise in automated vehicles that neither experts are aware of nor can be found in historical data. The determination of q(x) in its current form therefore harbors the risk of not optimally selecting tests and thus not being able to identify critical situations caused by automated driving functions. Ultimately, this creates risks for people and machines.

The DE 10 2020 005 507 A1 discloses a method for testing an automated vehicle function, in particular an emergency braking function. Measurement data from field tests are clustered according to driving situations, criticality metrics (KPI, e.g. time interval between vehicle and vehicle in front) are identified by correlating a criticality threshold determined from synthetic driving scenarios (HiL tests) and the clustered driving situations and then the tests for individual grouped logical Scenarios executed. The method runs iteratively using an adaptive importance sampling algorithm.

object of the invention

The object of the present invention is to reduce the effort and the use of resources when testing automated driving functions and still obtain valid test results that enable the automated driving function to be released.

Presentation and advantages of the invention

The object is solved by a method for testing automated vehicle functions in accordance with the measures of independent claim 1 . In particular, the task is solved by an evidence-based risk determination by linking different test environments. Advantageous configurations of the invention are the subject matter of the dependent claims.

Automated vehicle functions are functions for driving the vehicle, such as steering, accelerating and decelerating, which are implemented independently by the vehicle without driver intervention. They are also frequently referred to as driver assistance systems. Examples of this are adaptive cruise control, lane departure warning systems, emergency brake assistants and fully autonomous vehicles.

In a first step of the method according to the invention, logical test scenarios are defined using scenario-specific parameters x and a preferably multi-dimensional parameter space of the scenario-specific parameters is created. Each parameter, e.g. B. the vehicle speed or the distance to the vehicle in front is defined by an upper and lower bounded interval (parameter interval). For example, the perceived distance to the vehicle in front cannot be less than zero and cannot be greater than the sensor range. The upper and lower limits can therefore be of a physical nature or given by the scope of the vehicle function to be tested.

In a next step, the distribution p(x) of the scenario-specific parameters is determined over the entire parameter space. p(x) describes the distribution of each individual parameter or parameter combination in the test scenario. In other words, p(x) is proportional to the frequency of each parameter expression in the parameter interval, as it occurs in reality in the corresponding scenario.

The next step is to define a scenario-specific hazard. A hazard can be a collision with another object, the severity of the collision, i.e. the extent of damage to the vehicle or people, or falling below critical distances.

In a next step, a hazard characteristic f(x) is determined depending on the scenario-specific parameters across the entire parameter space using a high-performance test environment. The nature of the hazard can be the frequency of occurrence of a binary event (collision or not) or the (continuous or discrete) nature of the hazard, for example the amount of damage. An example would be the probability or the frequency of collisions to be observed during a lane change or a cutting-in process. The distribution of this hazard characteristic over the parameter space could be a decreasing frequency of collisions with increasing distance to the vehicle in front.

In a next step, critical scenario-specific parameter combinations are determined within the entire parameter space depending on the determined probability of occurrence p(x) of the scenario-specific parameters and the determined risk level f(x) and the determination of a distribution q(x) of the critical scenario-specific parameter combinations. By mapping the hazard level f(x) to the parameter space, critical parameter levels can be identified. In the event that a vehicle driving in front cuts in, for example, the value of the parameter distance to the vehicle in front would be critical if this falls below a distance threshold value derived from the risk value. In other words, the distance parameter is critical if it is less than, for example, 10 m at the time of cutting in.

In a next step, the automated driving function is tested depending on the identified critical scenario-specific parameter combinations using a trustworthy test environment. By focusing the tests on the critical parameter combinations, time-consuming and complex test procedures that produce very realistic test results, such as complex 3D simulations or tests in the real world, can be reduced to a minimum.

In a next step, a scenario-specific risk is determined depending on the testing of the automated driving function. This can be done on the basis of KPIs that are determined with each test. Depending on the risk determined, the automated driving function can finally be enabled.

The method according to the invention is characterized in that the risk characteristics are determined as a function of the scenario-specific parameters over the entire parameter space using a high-performance, scalable test environment. The testing of the automated driving function and the actual risk determination then take place in a complex, trustworthy test environment depending on the determined critical scenario-specific parameter combinations. In this way, an understanding of the system can be obtained with little effort and the risks of the automated driving function can then be determined very precisely. The high-performance, scalable test environment can be a 2D or 3D simulation and is preferably purely virtual. In contrast, the complex, trustworthy test environment is preferably not purely virtual, but contains real components of the driving function to be tested, such as control units, sensors, actuators or the entire vehicle. This can include hardware-in-the-loop testing, vehicle-in-the-loop testing, proving ground testing, or real-world road testing. In contrast to the prior art, in which the same test environment is used for the example of a meta-model-based importance sampling for both the meta-models and the actual risk determination, the method according to the invention is able to combine different test environments and to optimally balance the advantages and disadvantages of different test environments and thus to deliver faster and more accurate risk assessments.

In a preferred embodiment, the high-performance, scalable test environment is modeled using a meta-model. A meta-model is a model whose original, i.e. what is to be modeled, is not part of the real world but part of a model. In other words, a meta-model is a model of a model. The meta-model does not bring particularly realistic results, partly because the model structure is very simple and only a few essential parameters are included. On the other hand, the metal model has a high computational efficiency, which is why a large number of estimates can be made over the entire parameter space. The results of the meta-model can therefore not be used for the final risk assessment of the automated driving function but provide a statistically adequately secured critical set of parameters. Meta-models are usually parameterized on the basis of virtual test environments, such as SiL tests.

Another advantage of using a meta-model to determine the level of risk is that no complex probabilities have to be determined. A binary statement, for example whether a collision has occurred or not, can be sufficient here. In addition, the scenario parameterization that is known, for example, from field tests or previous development stages or products can simply be adopted. In this way, distortions or systematic errors (bias) of estimating functions can be avoided.

The actual testing of the automated driving function then takes place with a complex test procedure that is very trustworthy due to its complexity. This can be a model, for example a complex 3D simulation with a large number of input parameters. This can also be a HiL test in which the original hardware of the vehicle to be tested is tested. Sensors can be represented by complex sensor models. However, the original sensors can also be used and the object representation, e.g. B. dummy vehicle, are simulated complex. Finally, testing can also take place in the real world on a test track with corresponding object representations of other road users or real other road users.

In an advantageous embodiment of the invention, the meta-model is a trained classification model. Such classifiers are known in the field of machine learning and are correspondingly easy to implement. Here, for example, a Bayesian classifier, a support vector machine (SVM), a random forest classifier, an extra trees classifier, a Gaussian process classifier, a deep Gaussian process classifier or a classifier based on a Bayesian neural network can be used. The training of the classifiers usually takes place by means of supervised learning. A classifier is particularly useful in predicting a binary statement such as whether or not a collision has occurred.

In a further advantageous embodiment of the invention, the meta-model is a trained regression model. Mathematically, regression models are not very different from classifiers. Artificial neural networks, e.g. B. a Bayesian neural network, decision trees, Gaussian process regressors or deep Gaussian process regressors can be used.

The use of a Gaussian process is particularly advantageous. The training of the regression model usually takes place by means of supervised learning. A regression model can be used particularly advantageously in the prediction of continuous values, such as the minimum distance to the vehicle in front.

The effect of the method according to the invention will now be described again in other words. It is known for importance sampling which q(x), i.e. which critical parameter combinations, would theoretically be optimal as a starting point for the complex, trustworthy test procedure (prerequisite f(x) ≥ 0): $$q\left(\text{x}\right)=f\left(x\right)*p\left(x\right)/{\displaystyle \int \left(f\left(x\right)*p\left(x\right)\right)}\mathrm{i.e}x$$

A single attempt would determine the risk: $$f\left(x\right)*p\left(x\right)/q\left(\text{x}\right)=f\left(x\right)*p\left(x\right)*{\displaystyle \int \left(f\left(x\right)*p\left(x\right)\right)\mathrm{i.e}x/f\left(x\right)*p\left(x\right)=}{\displaystyle \int \left(f\left(x\right)\ast p\left(x\right)\right)\mathrm{i.e}x}$$

In practice, of course, the optimal q(x) cannot be chosen, since the solution to the problem (the risk) already appears in the definition. However, the risk can be determined very precisely using scalable, virtual methods.

In the solution proposed according to the invention, the advantages of different test environments are used in order to be able to carry out an efficient and precise risk determination. p(x) (the probability distribution of the scenarios) can still be derived from existing data or macroscopic simulations. Because when automated vehicles are introduced, it can be assumed that they will have to be able to handle the same situations as human drivers.

However, f(x) is determined on the basis of efficient (usually virtual) test environments. This process can be automated. First, the desired scenario is modeled in a test environment with low complexity. The behavior of the automated vehicle is then determined for the spanned parameter space. Current methods of planning experiments can be used for this. It is important here that the entire parameter space is observed and tested across the function. Particular attention should be paid to the areas of the parameter space in which dangerous behavior occurs. Iterative methods of test planning can also be used to determine this question. In this step of testing, the advantages of less complex test environments can be used in particular. Experiments can be modeled quickly and a large number of tests can be completed within a short time.

The knowledge obtained in this way regarding f(x) can then be combined with the knowledge about p(x) to determine q(x) and the actual tests for risk assessment can be carried out using complex methods (usually real). For combination, q(x) is determined according to the equation above.

The linking of the test environments and the transfer of knowledge has a particularly positive effect in the following points.

The expectation of the outcome of the test cases can be based on evidence related to the system to be examined, resulting in a very good q(x) (before: either historical data from human drivers or other systems, sometimes even simple guessing).

The transfer of knowledge within the framework of importance sampling is particularly secure, since no bias occurs in the evaluation, even if behavior in simulation and reality differ. With many other types of transmission, it would be very complicated to combine the different behaviors in different test environments in such a way that a result without systematic error (bias) can be obtained in the end.

The advantages and disadvantages of the different test environments can each be balanced by transferring knowledge between the test environments (abstract test environments can allow extensive investigation, realistic test environments can precisely identify risks).

Overall, risks can be determined very efficiently and precisely, which creates the basis for the release of automated driving functions.

It can be expected that many weak points in the automated vehicle function will be uncovered, thereby protecting man and machine.

example

Further features, application possibilities and advantages of the invention result from the following description of an exemplary embodiment of the invention with reference to the diagrammatic drawing.

Here shows 1 a scenario in which a pedestrian 2 crosses the street and an ego vehicle 1 has the automated driving function to be tested. This can be an emergency brake assistant. In order to assess the driving function with regard to its functional safety, the risk of the scenario must be determined as a function of a large number of starting parameters and environmental influences. The risk can be quantified on the basis of a collision that has occurred or the minimum distance between the ego vehicle and the pedestrian after the emergency brake assistant has intervened.

With some starting parameters, there will be no significant risk, for example with a defensive own speed of ego vehicle 1 of 6 m/s and a longitudinal distance between ego vehicle 1 and pedestrian 2 at the beginning of crossing the road of more than 50 m Parameter combination would therefore not be critical. If the vehicle's own speed is increased, for example on freeways, it is no longer possible to make a simple intuitive statement about the criticality of the parameter combination. The vehicle 1 then drives onto the pedestrian much more quickly, but the emergency brake assistant has more warning time and can accordingly intervene sooner.

In order to test the crossing scenario with regard to the emergency braking function, the environmental conditions are first parameterized. For this purpose, pedestrian models are combined with weather data. Pedestrian data relate, for example, to the frequency of occurrence of pedestrians, e.g. B. on an hourly basis. This results in a probability that the scenario will occur, which is proportional to the frequency of occurrence of pedestrians. To make the environmental conditions applicable, fog (ω _fοg ), wind (ω _wind ), and rain (ω _rain ) parameters are parameterized, typically in an interval from zero to one. Based on this, a coefficient of friction μ _fric between the road and the vehicle tires is determined, which in turn influences the braking distance of the ego vehicle 1 . The data is pulled directly from traffic and weather databases reflecting real-life situations over a period of 8 to 10 years.

In the case of defensive driving, the vehicle's own speed is normally distributed with an expected value of 6 m/s and a standard deviation of 0.2 m/s. A model describes at which gaps between vehicles a pedestrian 2 starts crossing. These are mapped using the longitudinal distance between ego vehicle 1 and pedestrian 2 . The speed v _ped of the pedestrian 2 is also dependent on the rain intensity ω _rain .

Testing all parameters in reality would be extremely complex or almost impossible to carry out. Even complex simulations of all of these parameters would be very time-consuming, since the majority of the starting parameters would not lead to any critical situations and therefore each test run with non-critical parameter combinations would be time-consuming and costly without enabling a risk assessment. A classic modeling of existing parameters in a 3D simulation (3D-SiL), for example using the simulation tool CARLA and a modular stack of automated driving functions using Pylot, would take approx. 90 s per test. A majority of the simulations would involve seed parameters that do not lead to collisions. In the scenario considered, it is to be expected that the proportion of collisions will be in the range of 10 ^-3 to 10 ^-6 . Accordingly, simulation tests in the mid to high five-digit range would have to be carried out in order to obtain statistically relevant statements.

Therefore, the behavior of the complex 3D simulation is represented by a meta model. In the present example, this is a Gaussian process model that is trained with a number of 200 test runs. The training lasts about 5 s. The risk is determined using the minimum distance d* _min between ego vehicle 1 and pedestrian 2 after activation of the emergency brake assistant. If a collision occurs, the risk is determined based on the theoretical remaining braking distance. $${\mathrm{i.e}}_{min}^{*}=\{\begin{array}{ll}{\mathrm{i.e}}_{min},\hfill & if{\mathrm{i.e}}_{min}>0\hfill \\ -v_{AVcOl/}^2\left(2G{\mathrm{\mu }}_{f\mathrm{right}ic}\right),\hfill & OtHe\mathrm{right}wise\hfill \end{array}$$

The meta model is further simplified and more resource efficient when a perceptual model is introduced. This determines whether a pedestrian 2 has been detected at all. To do this, an extra-tree classifier is used, which is trained with individually rendered images of pedestrians and varying weather conditions. If a pedestrian 2 is detected, an extra tree regressor follows, which determines the perceived position of pedestrian 2 from the point of view of the automated driving function (ego vehicle 1).

In this way, a complex sensor model can be replaced by two simple and scalable meta-models. Although these do not allow the scenario risk to be determined with sufficient accuracy, they do provide sufficiently accurate critical parameter combinations for further testing. In a particularly advantageous embodiment, a random classification and regression model can then be selected from a large number of models trained in this way for each simulation run (frame). For example, a sensor model can be mapped by a trained model ensemble, with an ensemble model being selected for each frame at runtime. This allows systematic and random sensor noise to be simulated very efficiently.

The dynamic model of the complex 3D simulation for a HiL test can also be modeled using a meta model. The airspeed, acceleration, and overrun and braking behavior of four consecutive time steps (frames) are used to parameterize an extra-tree regression model for predicting the acceleration/deceleration for the next time step.

The results of the high-performance, scalable test environment, i.e. the critical parameter combinations, can be transferred to the complex, trustworthy test environment if necessary. Transfer learning methods can be used for this. If the same parameters are used in both test environments, the so-called homogeneous transfer learning takes place. In the event that the scenario-specific parameters are not selected identically for both test environments, so-called heterogeneous transfer learning can be used to transfer the parameters between the two test environments. This can affect certain weather conditions, for example. Rain can be simulated directly in a 3D simulation, whereas it is just noise in the 2D simulation.

The method according to the invention can be referred to as transfer importance sampling, which combines methods from different test environments. Distortions (bias and deviations from reality) of individual estimation methods from the prior art, which can only be reduced by a large number of tests, can be avoided.

Reference List

1

ego vehicle

2

pedestrian

d0

longitudinal distance ego vehicle and pedestrian

i.e

perceived distance between ego vehicle and pedestrian

dped

lateral distance of ego vehicle and pedestrian

vAV

speed ego vehicle

vped

speed pedestrians

Claims (4)

Method for testing automated driving functions with the following steps: - Defining logical test scenarios using scenario-specific parameters and creating a parameter space of the scenario-specific parameters, - Determining the distribution of the scenario-specific parameters over the entire parameter space, - Defining scenario-specific hazards, - Determining a hazard characteristic depending on the scenario-specific parameters over the entire parameter space, - determining critical scenario-specific parameter combinations within the entire parameter space depending on the determined probability of occurrence of the scenario-specific parameters and the determined hazard characteristics as well as determining a distribution of the critical scenario-specific parameter combinations, - testing the automated driving function depending on the determined critical scenario-specific parameter combinations and - Determining a scenario-specific risk depending on the testing of the automated vehicle function, characterized in that - determining the risk characteristics depending on the scenario-specific parameters over the entire parameter space using a high-performance, scalable test environment and - testing the automated driving function depending on the determined critical scenario-specific Parameter combinations are carried out using a complex, trustworthy test environment. Procedure for testing automated driving functions claim 1 , whereby the high-performance, scalable test environment is modeled using a meta model. Procedure for testing automated vehicle functions claim 2 , where the meta-model is a trained classification model. Procedure for testing automated vehicle functions claim 2 , where the meta-model is a trained regression model.

Images