JP2022550058A - Safety analysis framework - Google Patents

Abstract

Techniques for determining safety metrics associated with vehicle controllers are discussed herein. Various operating regimes (scenarios) can be identified based on operational data and adjusted in relation to scenario parameters to determine whether complex systems (which may be untestable) can operate safely. . Scenarios may be identified for testing in order to verify the safe operation of such systems. Ability to quantify error metrics for system subsystems. Error metrics can be introduced into scenarios in addition to other system/subsystem probabilistic errors. Scenario parameters can also be perturbed. Any number of such perturbations can be instantiated in a simulation to test, for example, a vehicle controller. A safety metric associated with the vehicle controller can be determined based on the simulation as well as the cause of any faults.

Description

The present invention relates to a safety analysis framework.

[Related Application]
This patent application claims priority to a U.S. utility patent application entitled "SAFETY ANALYSIS FRAMEWORK" with serial number 16/586,838 filed on September 27, 2019 and No. 16/586,853, entitled "ERROR MODELING FRAMEWORK," filed in U.S. Utility Patent Application No. 16/586,853. Application Serial Nos. 16/586,838 and 16/586,853 are fully incorporated herein by reference.

An autonomous vehicle can use an autonomous vehicle controller to guide the autonomous vehicle through the environment. For example, autonomous vehicle controllers use planning methods, devices, and systems to determine driving paths, and control dynamic objects (e.g., vehicles, pedestrians, animals, etc.) and static objects (e.g., buildings, signs, It can guide an autonomous vehicle through an environment, including stationary vehicles, etc.). However, it is important to verify the safety of the controller to ensure the safety of the occupants.

The detailed description is described with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different drawings indicates similar or identical components or features.

FIG. 1 illustrates generating vehicle performance data associated with a vehicle controller based on a parameterized scenario. FIG. 2 illustrates a computing device that generates scenario data based at least in part on log data generated by a vehicle, where the scenario data indicates one or more variations of the scenario. FIG. 3 illustrates generating error model data based at least in part on vehicle data and ground truth data. FIG. 4 illustrates using error model data to perturb the simulation by providing at least one of the errors or uncertainties associated with the simulated environment. FIG. 5 illustrates a computing device that generates perceptual error model data based at least in part on log data and ground truth data generated by a vehicle. FIG. 6 illustrates a computing device that generates simulation data based at least in part on the parameterized scenario data and generates safety metric data based at least in part on the simulation data. FIG. 7 shows a block diagram of an exemplary system for implementing the techniques described herein. FIG. 8 depicts a flow diagram of an exemplary process for determining safety metrics associated with a vehicle controller, according to examples of this disclosure. FIG. 9 depicts a flow diagram of an exemplary process for determining statistical models associated with subsystems of an autonomous vehicle.

The techniques described herein are directed to various aspects of determining system performance metrics. In at least some examples described herein, such performance metrics may be determined using simulations in conjunction with determining other performance metrics, for example. Simulations are used to validate software (e.g. vehicle controllers) running on vehicles (e.g. autonomous vehicles) and collect safety metrics to determine how the software can safely operate such vehicles in various scenarios. You can be sure that you can control it. In an additional or alternative example, simulation can be used to learn about the constraints of autonomous vehicles using autonomous controllers. For example, simulations can be used to understand the operating space of an autonomous vehicle (eg, the envelope within which an autonomous controller effectively controls the autonomous vehicle) considering road conditions, ambient noise, failed components, etc. Simulation can also be useful in generating feedback for improving the operation and design of autonomous vehicles. For example, in some instances, simulation may be useful for determining the amount of redundancy required in the autonomous controller or how to modify the behavior of the autonomous controller based on what is learned through simulation. . Further, in additional or alternative examples, simulations may be useful in informing the hardware design of autonomous vehicles, such as optimizing the placement of sensors on autonomous vehicles.

When creating a simulation environment to perform testing and verification, it is possible to specifically enumerate the environment with various specific examples. Each instantiation of such an environment is unique and definable. Manually enumerating all possible scenarios can take excessive time, and different scenarios may not be tested if all possible scenarios are not constructed. . Scenario parameters can be used to parameterize properties and/or attributes of objects within a scenario to provide scenario variations.

For example, a vehicle or vehicles can traverse an environment and generate log data associated with the environment. Log data includes sensor data captured by one or more sensors of the vehicle, sensory data indicative of objects identified (or generated in a post-processing stage) by one or more systems onboard the vehicle, It may include predictive data indicating object intent (whether generated during or after recording), and/or status data indicating diagnostic information, trajectory information, and other information generated by the vehicle. The vehicle can transmit log data over the network to a database that stores the log data and/or to a computing device that analyzes the log data.

Based on the log data, the computing device can determine different scenarios, frequencies of different scenarios, and areas of the environment associated with different scenarios. In some examples, the computing device can group similar scenarios represented in the log data. For example, scenarios can be analyzed using, for example, k-means clustering and/or between parameters of the environment (e.g., daytime, nighttime, precipitation, vehicle position/speed, object position/speed, road segment, etc.) Grouping can be done by evaluating weighted distances (eg Euclidean). As mentioned above, clustering similar or similar scenarios can help simulate an autonomous controller in its own scenario, rather than simulating an autonomous vehicle in a nearly similar scenario, which would result in redundant simulation data/results. reduces the amount of computational resources required to simulate autonomous controllers in the environment. As can be appreciated, autonomous controllers may be expected to perform similarly (and/or may apparently perform) in similar scenarios.

For example, the computing device can determine the rate at which pedestrians appear in crosswalks based on the number of pedestrians represented in the log data. In some examples, the computing device can determine the probability of detecting a pedestrian in a crosswalk based on the percentage and the time period of operating the autonomous vehicle. Based on the log data, the computing device can determine scenarios and identify scenario parameters based on the scenarios that can be used in simulations.

In some examples, the simulation is used to determine the autonomous vehicle controller's response to defective (and/or defective) vehicle sensors and/or defective (and/or defective) processing of sensor data. can be tested and verified. In such examples, the computing device can be configured to introduce inconsistencies in the object's scenario parameters. For example, an error model can indicate errors and/or error percentages associated with scenario parameters. Scenarios can incorporate errors and/or error percentages into simulated scenarios to simulate the response of an autonomous vehicle controller. Such errors are determined based, without limitation, on statistical aggregation using ground truth data, functions (e.g., errors based on input parameters), or any other model that maps parameters to specific errors. can be represented by a lookup table. In at least some examples, such error models may map specific errors to their probability/frequency of occurrence.

By way of example, and not limitation, an error model may indicate that scenario parameters such as velocities associated with objects in the simulated environment are associated with error percentages. For example, an object may traverse in a simulated scenario at a speed of 10 meters per second and the error percentage may be 20% resulting in a range of speeds from 8 meters per second to 12 meters per second. . In some examples, a range of velocities may be associated with a probability distribution that indicates that some parts of the range are more likely to occur than others. (For example, 8 meters per second and 12 meters per second are associated with a 15% probability, 9 meters per second and 11 meters per second are associated with a 30% probability, and 10 meters per second are associated with a 10% probability).

A parameterized scenario can be generated based on the error model and/or scenario. A parameterized scenario can provide a set of scenario variations. Therefore, instantiating an autonomous vehicle controller in a parameterized scenario and simulating the parameterized scenario can efficiently cover a wide variety of scenarios without requiring manual enumeration of variations. Additionally, based at least in part on executing the parameterized scenario, the simulation data indicates how the autonomous vehicle controller responded (or will respond) to the parameterized scenario, and the simulation data Based at least in part, a successful outcome or an unsuccessful outcome can be determined.

Aggregating simulation data associated with parameterized scenarios can provide safety metrics associated with the parameterized scenarios. For example, simulation data can indicate success and/or failure rates of autonomous vehicle controllers and parameterized scenarios. In some examples, meeting or exceeding the success rate can indicate successful verification of the autonomous vehicle controller, which is then downloaded by the vehicle (or otherwise) for further vehicle control and operation. forwarded).

For example, parameterized scenarios can be associated with outcomes. Simulation data may indicate that the autonomous vehicle controller responded consistently or inconsistently to the results. By way of example, and not limitation, a parameterized scenario may represent a simulated environment that includes a vehicle controlled by an autonomous vehicle controller traversing at some speed and performing a stopping action in front of an object in front of the vehicle. can be done. Velocity can be associated with a scenario parameter that indicates a range of vehicle velocities. A parameterized scenario can be simulated based at least in part on the range of velocities and can generate simulation data indicative of the distance between the vehicle and the object when the vehicle completes the stopping action. A parameterized scenario can be associated with results indicating that the distance between the vehicle and the object meets or exceeds a distance threshold. Based on the simulation data and scenario parameters, the success rate is compared to the total number of times the vehicle completes the stop action, and the distance between the vehicle and the object when the vehicle completes the stop action satisfies the distance threshold or The number of times exceeded can be indicated.

The techniques described herein provide various computational efficiencies. For example, by using the techniques described herein, a computing device may require fewer computational resources, enabling multiple simulated scenarios than is available via conventional techniques. can be generated quickly. Conventional technology is not scalable. For example, training, testing, and/or validating a system (e.g., one or more components of an AI stack) onboard an autonomous vehicle (e.g., deploying it in a new real-world environment to which such an autonomous vehicle is addressed). Generating the required number of sets of unique simulated environments (before testing) can take a significant amount of time, which makes it difficult to adapt to real-life scenarios and/or environments. Limited ability to train, test, and/or validate such systems (eg, one or more components of an AI stack) onboard an autonomous vehicle prior to entry. The techniques described herein leverage sensor data collected from real environments and supplement that data with additional data to provide substantially more accurate simulations than are available with prior art techniques. It is unconventional in that it more efficiently generates simulated environments (eg, relative to corresponding real environments). Furthermore, the techniques described herein, such as the various aspects of scenarios, enable the generation of many scalable simulated scenarios in less time and less computational resources than is available with conventional techniques. do.

Additionally, the technology described herein is directed to improving safety. That is, the simulated environments resulting from the generation techniques described herein are used to test, train, and validate systems on board autonomous vehicles, and such systems are used in real-world environments. can ensure that autonomous vehicles can operate safely when deployed in That is, the simulated environment resulting from the generation techniques described herein can be used to test, train, and validate planner and/or predictive systems of autonomous vehicle controllers, which , can be used by autonomous vehicles to navigate them along trajectories in real environments. Accordingly, such training, testing, and validation enabled by the techniques described herein can provide an opportunity to ensure that autonomous vehicles can operate safely in real-world environments. Accordingly, the techniques described herein improve safety and affect navigation.

FIG. 1 shows an example 100 of generating vehicle performance data associated with a vehicle controller based on a parameterized scenario. Input data 102 can be used to generate scenarios. Input data 102 may include vehicle data 104 and/or additional context data 106 . Vehicle data 104 may include log data captured by vehicles traversing through the environment. As noted above, log data can be used to identify scenarios for simulating autonomous controllers. For the purposes of illustration, the vehicle is published by the U.S. National Highway Traffic Safety Administration describing a vehicle capable of performing all safety-critical functions throughout its journey without the expectation of constant control of the vehicle by the driver (or occupant). It may be an autonomous vehicle configured to operate according to a level 5 classification that In such instances, the vehicle can be configured to control all functions from start to stop, including all parking functions, and thus can be unmanned. This is only an example, and the systems and methods described herein can be used on any surface, including vehicles that require manual control by the driver at all times, to those that are partially or fully autonomously controlled. , aerial or water vehicles.

Vehicles may include computing devices that include perception engines and/or planners to perform operations such as detecting, identifying, segmenting, classifying, and/or tracking objects from sensor data collected from the environment. For example, objects such as pedestrians, cyclists/bicyclists, motorcyclists/motorcyclists, buses, trams, trucks, animals, etc. can be present in the environment.

As the vehicle traverses through the environment, the sensors can capture sensor data associated with the environment. For example, some of the sensor data can be associated with objects (eg, vehicles, cyclists, and/or pedestrians). In some examples, sensor data can be associated with other objects including, but not limited to, buildings, road surfaces, signs, barriers, and the like. Thus, in some examples, sensor data can be associated with dynamic and/or static objects. A dynamic object, as described above, is associated with movement (e.g., vehicles, motorcycles, cyclists, pedestrians, animals, etc.) or capable of movement within the environment (e.g., parked vehicles , a standing pedestrian, etc.). Static objects can be objects associated with the environment, such as buildings/structures, road surfaces, road markers, signs, barriers, trees, sidewalks, etc., as described above. In some examples, the vehicle computing device can determine information about objects in the environment such as bounding boxes, classifications, segmentation information, and the like.

A vehicle computing device can use the sensor data to generate a trajectory for the vehicle. In some examples, the vehicle computing device can also determine pose data associated with the position of the vehicle. For example, a vehicle computing device can use sensor data to determine position, coordinate, and/or orientation data for the vehicle within the environment. In some examples, the pose data may include xyz coordinates and/or may include pitch, roll, and yaw data associated with the vehicle.

The vehicle computing device can generate vehicle data 104, which can include the data described above. For example, vehicle data 104 may include sensor data, perception data, planning data, vehicle status data, speed data, intent data, and/or other data generated by vehicle computing devices. In some examples, sensor data may include time-of-flight sensors, position sensors (GPS, compass, etc.), inertial sensors (inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.). ), lidar sensor, radar sensor, sonar sensor, infrared sensor, camera (e.g., RGB, IR, intensity, depth, etc.), microphone sensor, environment sensor (e.g., temperature sensor, humidity sensor, light sensor, pressure sensor, etc.), It can include data captured by sensors such as ultrasonic transducers, wheel encoders, and the like. Sensor data can be data captured by such sensors, such as time-of-flight data, position data, lidar data, radar data, sonar data, image data, audio data, etc. . Such log data includes, but is not limited to, messages indicating the detection of objects, object tracks, predictions of future object locations, multiple trajectories generated in response to such detections, command execution It may further include intermediate output by any one or more systems or subsystems of the vehicle, including control signals passed to one or more systems or subsystems used for the vehicle. In some examples, vehicle data 104 may include temporal data associated with other data generated by the vehicle computing device.

In some examples, the input data 102 can be used to generate scenarios. Input data 102 may include vehicle data 104 and/or additional context data 106 . By way of example, and not limitation, additional context data 106 may include data such as incident reports from third party sources. Third-party sources may include law enforcement agencies, the motor vehicle department, and/or security administration, which may publish and/or store reports of activities and/or incidents. For example, the report may include the type of activity (eg, debris on the road, traffic hazards such as localized flooding), location, and/or description of the activity. By way of example, and not limitation, a report may describe that a driver, while maneuvering a vehicle, hit a fallen tree branch in the roadway while crossing at a speed of 15 meters per second. Reports can be used to generate similar scenarios that can be used in simulations.

In some examples, additional context data 106 may include captured sensor data (eg, image data). By way of example, and not limitation, a vehicle driver may use a camera to capture image data while the driver operates the vehicle. In some examples, image data can capture activities such as incidents. By way of example, and not limitation, a driver can use a dashboard camera (eg, a camera mounted on the vehicle's internal dashboard) to capture image data while the driver is operating the vehicle. As the driver maneuvers the vehicle, animals may run across the roadway and the driver may quickly apply the brakes to slow the vehicle. A dashboard camera can capture image data of an animal running across the roadway, slowing the vehicle down. Image data can be used to generate scenarios of animals running across the roadway. As described above, probabilities associated with scenarios can be determined and scenarios can be identified for simulation based on the probabilities of meeting or exceeding probability thresholds. As an example, and not by way of limitation, if the probability of encountering a scenario is less than 0.001%, a probability threshold of 0.001% can be used, and then prioritize scenarios with higher probability for simulation. They can be ranked to determine the safety metric associated with the probable scenarios.

Input data 102 , such as vehicle data 104 and/or additional situation data 106 , can be used by scenario editor component 108 to generate initial scenario 110 . For example, input data 102 can be input to scenario editor component 108 that can generate a synthesized environment that represents at least a portion of input data 102 within the synthesized environment. Examples of generated scenarios, such as vehicle-generated data that can be included in initial scenario 110 and vehicle data 104, are found in, for example, U.S. patent application entitled "Scenario Editor and Simulator," filed Apr. 23, 2019. 16/392,094, which is incorporated by reference in its entirety.

Scenario editor component 108 may be configured to scan input data 102 and identify one or more scenarios represented in input data 102 . By way of example, and not limitation, scenario editor component 108 may determine that a portion of input data 102 represents a pedestrian crossing a street with no right-of-way (eg, no crosswalk, no pedestrian sign, etc.). The scenario editor component 108 can identify this as a scenario (eg, jaywalking parameters) and label (and/or classify) the scenario as, for example, a jaywalking scenario. For example, scenario editor component 108 can generate initial scenario 110 using rules that define actions. By way of example, and not limitation, a rule may define that a pedestrian crossing a road in an area not associated with a crosswalk is a jaywalker. In some examples, scenario editor component 108 can receive label data from a user of scenario editor component 108 and associate portions of input data 102 with labels to generate initial scenario 110 .

In some examples, scenario editor component 108 may scan other portions of input data 102 to identify similar scenarios and label similar scenarios with the same jaywalking label. In some examples, the scenario editor component 108 can identify scenarios that do not correspond to (or are excluded from) existing labels and generate new labels for those scenarios. In some examples, scenario editor component 108 can generate a library of scenarios and store the library of scenarios in a database within scenario editor component 108 . By way of example, and not limitation, the library of scenarios may include crosswalk scenarios, merge scenarios, lane change scenarios, and the like.

In at least some examples, such initial scenarios 110 may be manually specified. For example, when one or more users specify a particular scenario to test and run such a scenario, the vehicle may be tested even though it has never (or rarely) encountered the scenario before. can ensure that it can operate safely.

A parameters component 112 can determine scenario parameters associated with the initial scenario 110 identified by the scenario editor component 108 . By way of example, and not limitation, the parameters component 112 may analyze a jay-walking scenario to determine pedestrian position, pedestrian pose, pedestrian size, pedestrian speed, pedestrian footprint, distance between vehicle and pedestrian. Scenario parameters associated with the jaywalking scenario can be determined, including distance, vehicle speed, road width, and the like.

In some examples, the parameter component 112 can determine ranges or sets of values associated with scenario parameters. For example, parameter component 112 can determine the classification associated with an object represented in initial scenario 110 (eg, a pedestrian) and determine other objects in input data 102 of the same classification. Parameter component 112 can then determine the range of values associated with the scenario parameters, as represented by initial scenario 110 . By way of example, and not limitation, scenario parameters may indicate that a pedestrian may have a velocity in the range of 0.5 to 1.5 meters per second.

In some examples, the parameter component 112 can determine probabilities associated with scenario parameters. By way of example, and not limitation, the parameters component 112 can associate probability distributions such as Gaussian distributions (also called normal distributions) with scenario parameters. In some examples, parameter component 112 can determine probabilities associated with scenario parameters based on input data 102 . As discussed above, the parameter component 112 can determine the classification associated with objects represented in the input data 102 and can determine other objects of the same classification within the input data 102 and/or other log data. The parameter component 112 can then determine probability distributions of scenario parameters associated with the objects as represented by the input data 102 and/or other log data.

By way of example, and not limitation, the parameter component 112 may indicate that 30% of pedestrians are walking at speeds less than 0.3 meters per second, 30% of pedestrians are walking at speeds greater than 1.2 meters per second, and 40% of pedestrians walking at a speed between 0.3 and 1.2 meters per second. The parameters component 112 can use the distribution as the probability that a pedestrian in a jaywalking scenario will walk at a particular speed. As an additional example, without limitation, parameter component 112 may indicate that a vehicle traversing the environment encounters a jaywalker 5% of the time while traversing the environment, a jaywalking scenario probability of 1%. can be determined to be In some examples, scenario probabilities may be used during simulation of an autonomous vehicle controller to include scenarios at a rate associated with the scenario probabilities.

In some examples, the parameter component 112 can receive supplemental data 114 embedded in the distribution. By way of example, and not limitation, the parameter component 112 indicates that a pedestrian may have a distance to the vehicle in the range of 30-60 meters while the vehicle is traveling at a speed of 15 meters per second, or alternatively Scenario parameters can be determined, expressed as time to collision of 2-4 seconds. Complementary data 114 (e.g., regulations or guidelines) may indicate that the vehicle must handle the scenario with a time-to-crash of 1.5 seconds, which can be a lower bound (also called a parameter threshold). . The parameter component 112 can incorporate the imputed data 114 and can determine scenario parameters to have a time to impact of 1.5 to 4 seconds (although any time range can be specified). In some examples, parameter component 112 can use the probability distributions described above to determine probabilities (using interpolation and/or extrapolation techniques) associated with imputed data 114 .

Error model component 116 can determine an error model that can indicate errors associated with scenario parameters. For example, a perceptual error model can generate perceptual errors associated with the simulated object's perceptual parameters, a prediction error model can generate prediction errors associated with the simulated object's prediction parameters, and so on. . In some examples, as the vehicle traverses the environment, the sensors may generate erroneous sensor data and/or the vehicle's computing devices may erroneously generate sensor data that may result in perceptual errors. may be processed by Testing and simulating perceptual errors can help indicate vehicle operating margins associated with potential perceptual errors. For example, scenario parameters, such as perceptual parameters, can indicate the size of an object or the range of positions of an object within the environment. The error model component 116 uses the perceptual error model to identify potential errors associated with the size of objects that can yield vehicle perception data indicating that objects are larger or smaller than actual objects in the environment. can be shown.

Error model component 116 can determine a perceptual error model, for example, by comparing input data 102 to ground truth data. In some examples, ground truth data can be determined from manual labeling and/or other validated machine learning components. For example, input data 102 may include sensor data and/or sensory data generated by the vehicle. The error model component 116 can compare the input data 102 with ground truth data that can indicate the actual parameters of objects in the environment. By comparing input data 102 to ground truth data, error model component 116 can determine perceptual errors. By way of example, and not limitation, the input data 102 may indicate that the pedestrian is 1.8 meters tall, while the ground truth data indicates that the pedestrian is 1.75 meters tall. , so the perceptual error model can indicate a perceptual error of about 3% (eg, [(1.8-1.75)/1.8]*100).

In some examples, the error model component 116 can determine the classification associated with objects represented in the input data 102 and determine other objects within the input data 102 of the same classification. The error model component 116 can then determine a probability distribution (also referred to as an error distribution) associated with the extent of the object's error and associate the probability distribution with the objects in the initial scenario 110 . By way of example, and not limitation, the error model component 116 determines that objects with pedestrian classification have a perceptual error of 4% to 6% and objects with vehicle classification have a perceptual error of 3% to 5%. can be determined. In some examples, the error model component 116 can, for example, determine a probability distribution indicating that objects larger than a threshold size are more or less likely to have errors (eg, in classification).

In some examples, the parameter component 112 can use the region data 118 to determine the set of regions of the environment that are compatible with the scenario and scenario parameters. By way of example and not limitation, a scenario is that the vehicle has a speed of 15 meters per second, the pedestrian has a speed of 1.5 meters per second, and the distance between the vehicle and the pedestrian is 30 meters. It can be shown that The parameter component 112 can determine the region based on region data 118 of the environment that fits the scenario. By way of example, and not limitation, the parameter component 112 may exclude school zones because the speed of vehicles in the scenario may exceed the speed limit associated with the school zone and thus the scenario is not valid in the school zone. can. However, such vehicle speeds (eg, 15 m/s) may be reasonable on county roads adjacent to farmland, so such areas may be considered within the set of areas.

In some examples, parameter component 112 can store area data 118 that includes a partition of the drivable area of the environment. Techniques for identifying drivable road surface segments and similar segments, and segment classifications and/or stereotypes, for example, entitled "Extension of Autonomous Driving Functionality to New Regions", published March 19, 2019 U.S. Patent Application No. 16/370,696, filed and U.S. Patent Application No. 16/376,842, entitled "Simulating Autonomous Driving Using Map Data and Driving Data," filed April 5, 2019; , which are incorporated herein by reference in their entireties.

For example, environmental region data 118 can be parsed into segments and similar segments can be identified. In some examples, the segments may include junction segments, eg, intersections, junctions, etc., or connecting road segments, eg, lengths and/or widths of roads between junctions. By way of example, and not limitation, all two-lane road segments with speed limits within the 10 mph range can be associated with the same stereotype. In some examples, data can be associated with each individual segment. For example, a junction division may be a junction type, e.g., a junction, a "T", a roundabout, the number of roads meeting at the junction, the relative position of those roads, e.g., the angle between the roads meeting at the junction, the junction may include information about traffic control signals in the , and/or other features. Data associated with connecting road segments may include a number of lanes, widths of those lanes, direction of crossing in each of the lanes, identification of parking lanes, speed limits for road segments, and/or other characteristics. can.

In some examples, sections of drivable surfaces may be grouped according to section classifications or section stereotypes. By way of example, and not limitation, some or all junction partitions that fit some range of metrics or attributes can be grouped together (e.g., weighted distance between partition parameters using k-means (e.g. Euclidean) or otherwise cluster such partitions based on partition parameters).

Scenarios containing the same or similar stereotypes can be used to validate the functionality of the autonomous vehicle controller. For example, autonomous vehicles may be expected (and/or apparently do) to do the same things in the same or similar stereotypes. In some instances, the use of stereotypes can reduce the number of comparisons that have to be made. For example, identifying similar regions reduces the number of simulation scenarios that can provide useful information. The techniques described herein can reduce computational complexity, memory requirements, and processing time by optimizing over specific scenarios that provide useful information for verification and testing.

Parameterization scenario component 120 can use data determined by parameter component 112 (eg, initial scenario 110 , scenario parameters, a set of regions, and/or error model data) to generate parameterization scenario 122 . For example, initial scenario 110 may represent scenarios such as a lane change scenario, a right turn scenario, a left turn scenario, an emergency stop scenario, and the like. Scenario parameters may indicate the speed associated with the vehicle controlled by the autonomous vehicle controller, the pose of the vehicle, the distance between the vehicle and objects, and the like. In some examples, the scenario parameters may indicate objects, positions associated with the objects, velocities associated with the objects, and the like. Additionally, error models (eg, perceptual error models, predictive error models, etc.) can indicate errors associated with scenario parameters and provide ranges of values and/or probabilities associated with scenario parameters. By way of example, and not limitation, scenario parameters such as vehicle speed may be associated with a range of speeds, such as 8-12 meters per second. As noted above, the range of velocities can be associated with a probability distribution that indicates the probability of velocities within the range of velocities occurring.

A set of regions can represent a portion of the environment that can be used to place objects within the simulated environment. For example, initial scenario 110 may represent a scenario involving a two-way multi-lane drive surface associated with a speed limit of 35 miles per hour. Based on the initial scenario 110, the set of regions can exclude regions that would not include a two-way multi-lane drive surface associated with a speed limit of 35 miles per hour, such as parking lots. Parameterized scenarios 122 can be used to cover variations provided by scenario parameters, error models, domain data 118, and the like.

By way of example, and not limitation, scenario parameters may include a vehicle traversing the environment at a speed of 10, 11, or 12 meters per second (or any speed) while approaching a junction. The set of regions may include uncontrolled junctions, junctions with four-way stops, and junctions with traffic lights. Additionally, the perceptual error model can indicate a perceptual error of 1.34%, as can be provided by the perceptual metric determined for the perceptual system being tested. Thus, parameterized scenarios 122 can allow a total of 9 different scenarios by varying scenario parameters and regions (eg, 3 speeds*3 regions=9 scenarios/permutations). Additionally, when the simulation component 124 simulates a parameterized scenario, the simulation component 124 can use the perceptual error model to introduce perceptual errors associated with the perceptual data determined by the vehicle. As can be appreciated, this is merely an example, and the parameterized scenario 122 can include more or fewer permutations, as well as different types, domains, and/or perceptual errors of scenario parameters.

Simulation component 124 can execute parameterized scenario 122 as a set of simulation instructions to generate simulation data 126 . For example, simulation component 124 can instantiate a vehicle controller in a simulated scenario. In some examples, simulation component 124 can run multiple simulated scenarios simultaneously and/or in parallel. Additionally, simulation component 124 can determine the outcome of parameterized scenario 122 . For example, simulation component 124 can run variations of parameterized scenarios 122 for use in simulations for testing and verification. The simulation component 124 can generate simulation data 126 that indicates how the autonomous vehicle controller performed (eg, responded), compares the simulation data 126 to predetermined results, and/or responds to any predetermined It can determine if a rule/assertion has been violated/triggered.

In some examples, variations for simulation can be selected based on generalized intervals of scenario parameters. By way of example, and not limitation, scenario parameters may be related to vehicle speed. In addition, scenario parameters can be associated with vehicle value ranges. Variations for simulation can be selected based on generalized intervals to increase coverage of the range of values (eg, select 25th percentile, 50th percentile, 75th percentile, etc.). In some examples, variations can be randomly selected and/or variations can be randomly selected within a standard deviation of a range of values.

In some examples, predetermined rules/assertions can be based on parameterized scenarios 122 (e.g., traffic rules for pedestrian crossings can be activated based on pedestrian crossing scenarios, or traffic rules for crossing lane markers can be enabled). The rule can be disabled for stationary vehicle scenarios). In some examples, the simulation component 124 can dynamically enable and disable rules/assertions as the simulation progresses. For example, rules/assertions associated with the school zone can be enabled or disabled when the simulated object approaches the school zone and when the simulated object leaves the school zone. In some examples, the rules/assertions can include, for example, comfort metrics related to how fast an object can accelerate given a simulated scenario. In at least some examples, rules may include, for example, obeying road rules, leaving safety buffers between objects, and the like.

determining that the autonomous vehicle controller performed consistent with a predetermined result (i.e., it did everything it was supposed to do) and/or that no rules were broken; Alternatively, based at least in part on determining that the assertion was not triggered, the simulation component 124 can determine that the autonomous vehicle controller was successful. determining that the performance of the autonomous vehicle controller did not match a predetermined result (i.e., the autonomous vehicle controller did something it was not supposed to do) and/or that a rule was broken or an assertion was Based at least in part on determining that it has been triggered, the simulation component 124 can determine that the autonomous vehicle controller has failed. Therefore, based at least in part on executing the parameterized scenario 122, the simulation data 126 indicates how the autonomous vehicle controller will respond to each variation of the parameterized scenario 122, and the simulation data 126, as described above. Successful or unsuccessful results can be determined based at least in part on data 126 .

Analysis component 128 can be configured to determine the degree of success or failure. By way of example, and not limitation, a rule may indicate that a vehicle controlled by an autonomous vehicle controller must stop within a threshold distance of an object. Simulation data 126 may indicate that in a first variation of parameterization scenario 122, the simulated vehicle stopped more than 5 meters from the threshold distance. In a second variation of parameterization scenario 122, simulation data 126 may indicate that the simulated vehicle has stopped more than 10 meters from the threshold distance. Analysis component 128 can indicate that the simulated vehicle performed better in the second variation compared to the simulated vehicle in the first variation. For example, analysis component 128 can determine an ordered list (eg, ordered according to relative success scale) containing associated variations of simulated vehicles and parameterized scenarios 122 . Such variations may also be used to determine limits for various components of the simulated system.

Analysis component 128 can determine additional variations of parameterized scenario 122 based on simulation data 126 . For example, simulation data 126 output by simulation component 124 may indicate variations (which may be expressed as continuous likelihoods) of parameterized scenarios 122 associated with success or failure. Analysis component 128 can determine additional variations based on variations associated with failures. By way of example, and not limitation, a variation of parameterized scenario 122 associated with failure may represent a vehicle crossing the driving surface at a speed of 15 meters per second and an animal crossing the driving surface at a distance of 20 meters in front of the vehicle. can be done. Analysis component 128 can determine additional variations of scenarios to determine additional simulation data 126 for analysis. By way of example, and not limitation, analysis component 128 may determine additional variations including vehicles traversing at 10 meters per second, 12.5 meters per second, 17.5 meters per second, 20 meters per second, and so on. Additionally, the analysis component 128 can determine additional variations including animals crossing the driving surface at distances such as 15 meters, 17.5 meters, 22.5 meters, and 25 meters. Additional variations can be input to simulation component 124 to generate additional simulation data. Such additional variations can be determined, for example, based on perturbations to scenario parameters of scenarios run in the simulation.

In some examples, the analysis component 128 can determine additional variations of scenarios by overriding scenario parameters. For example, a parameterized scenario may include a first scenario parameter associated with object velocity and a second scenario parameter associated with object position. Parameterized scenario 122 may include a first range of values associated with velocity and a second range of values associated with position. In some examples, after simulating parameterized scenario 122, simulation data 126 indicates that some variations of parameterized scenario 122 yielded successful results and some variations yielded unsuccessful results. can be shown. Analysis component 128 then determines to override the first scenario parameter (eg, sets a fixed value associated with the first scenario parameter) and parameterizes based on the second scenario parameter. Scenario 122 can vary. By disabling one of the scenario parameters, analysis component 128 can determine whether the scenario parameter and/or the value of the scenario parameter are associated with a successful outcome or a failed outcome. Such parameters may be disabled randomly or otherwise based on the probability of failure. By way of example, and not limitation, simulation data 126 as a result of disabling all of the scenario parameters can indicate problems with planning components of autonomous vehicles.

In some examples, the analysis component 128 can be used to perform sensitivity analysis. For example, analysis component 128 disables scenario parameters and, based on simulation data 126 generated by simulation component 124, determines how disabling scenario parameters affects simulation data 126 (e.g., success rate (increase the success rate, decrease the success rate, or have a minimal impact on the success rate). In some examples, analysis component 128 can disable scenario parameters individually to determine how disabling each scenario parameter affects simulation data 126 . The analysis component 128 can collect statistical data showing how individual scenario parameters affect the simulation data 126 over the course of many simulations. In some examples, analysis component 128 can be configured to override a set of scenario parameters (eg, override nighttime environmental parameters and override wet conditions environmental parameters). As noted above, the analysis component can collect statistical data that indicates how a set of scenario parameters affect simulation data 126 . The statistical data can be used to determine scenario parameters that increase or decrease the likelihood of resulting in a successful simulation, and sub-systems of the autonomous vehicle associated with scenario parameters that increase or decrease the success rate of the simulation data 126. can be used for identification.

In some examples, analysis component 128 can adjust the degree to which scenario parameters are adjusted. By way of example, and not limitation, scenario parameters may indicate wet environmental conditions (eg, rainy conditions). Scenario parameters can be adjusted over a range (eg, quarter inch rainfall, 1 inch rainfall, etc.). The analysis component 128 adjusts the magnitude of the scenario parameters and performs sensitivity analysis based on the magnitudes of the scenario parameters associated with the scenario parameters that can result in success or failure of the simulation. A threshold can be determined. In some examples, binary search algorithms, particle filter algorithms, and/or Monte Carlo methods may be used to determine the threshold, although other suitable algorithms are contemplated.

Vehicle performance component 130 can determine vehicle performance data 132 based on simulation data 126 (and/or additional simulation data based on additional variations from analysis component 128) and the type of failure. In some examples, vehicle performance data 132 may indicate how a vehicle performs within an environment. By way of example, and not limitation, vehicle performance data 132 may indicate that a vehicle traversing at a speed of 15 meters per second has a stopping distance of 15 meters. In some examples, vehicle performance data may indicate safety metrics. By way of example, and not limitation, vehicle performance data 132 may indicate an event (eg, failure) and the cause of the event. In at least some examples, such indications are binary (with or without fault), coarse (levels of fault, e.g., "critical," "non-critical," and "passing"), or continuous (e.g., fault ), but any other representation is contemplated. For example, for event type 1 and cause type 1, data 134(1) can indicate a safety rating, as can data 134(2)-134(4). In some examples, Cause Type 1 and Cause Type 2 may indicate failures such as vehicle failures or object (eg, bicyclist) failures. Vehicle performance data 132 may then indicate safety metrics associated with parameterized scenario 122 . In some examples, the vehicle performance component 130 can use the target metric and compare the vehicle performance data 132 to the target metric to determine if the safety metric meets or exceeds the target metric. In some examples, the target metric can be based on standards and/or regulations associated with autonomous vehicles.

In some examples, vehicle performance data 132 may be input to filter component 134 to determine filtered data 136 based on vehicle performance data 132 . For example, filter component 134 can be used to determine filtered data 136 that identifies regions that do not meet a coverage threshold. By way of example, and not limitation, initial scenario 110 may indicate a speed limit of 35 mph and a two-way multi-lane drive surface associated with region data 118 . Based on initial scenario 110 and region data 118 , parameter component 112 can identify five regions of the environment that can be used to simulate initial scenario 110 . After simulating parameterized scenario 122, vehicle performance data 132 may indicate that simulation data 126 is associated with three of the five regions. For example, simulation component 124 can simulate a scenario and generate simulation data 126 based on running the scenario in three of the five regions identified by parameter component 112 . A filter component 134, based on one or more filters, produces filtered data 136 indicating that the remaining two of the five regions do not meet a coverage threshold (eg, minimum number of simulations associated with the regions). can decide.

In some examples, filter component 134 can determine filtered data 136 indicative of the occurrence of an event based on vehicle performance data 132 . By way of example, and not limitation, simulation data 126 may include the occurrence of events such as emergency stops, flat tires, animals crossing a driving surface, and the like. A filter component 134 can determine filtered data 136 indicative of an emergency stop occurrence based on one or more filters. Additionally, filtered data 136 may include portions of simulation data 126 associated with occurrences of emergency stops, such as, for example, stopping distances associated with emergency stops.

FIG. 2 illustrates an example 200 of a vehicle 202, which is similar to the vehicle generating vehicle data 104 described with reference to FIG. It can be sent to device 204 . As described above, scenario editor component 108 is configured to scan input data 102 (eg, vehicle data 104 and/or additional situation data 106) and identify one or more scenarios represented in input data 102. can be configured to As a non-limiting example, such scenarios can be determined, for example, based on clustering (eg, using k-means, etc.) parameterization of the log data. In some examples, scenario editor component 108 can use scenario definition data 206 to identify one or more scenarios represented in input data 102 . For example, scenario definition data 206 can identify characteristics associated with types of scenarios. By way of example, and not limitation, scenario definition data 206 may identify jaywalking scenarios that include features such as a pedestrian crossing a portion of a driving surface that is not associated with a crosswalk. A scenario editor component 108 scans the input data 102 to identify portions of the input data that include characteristics of pedestrians crossing portions of the driving surface not associated with crosswalks to determine jaywalking scenarios. can. In at least some examples, such scenarios may also be manually entered and/or derived from third-party data (eg, police reports, publicly available video clips, etc.).

Additionally, as described above, the parameter component 112 can determine scenario parameters that can indicate values or ranges of values associated with parameters of objects in the scenario. The parameter component 112 can generate scenario data 208, as depicted in FIG.

Scenario data 208 may represent a base scenario that includes a vehicle 210 traversing along a driving surface and an object 212 (which may be a different vehicle) traversing near vehicle 210 in the same direction as vehicle 210 . Vehicles 210 and objects 212 can approach junctions with pedestrian crossings. The parameters component 112 can determine scenario parameters that describe the distance between the vehicle 210 and the object 212 as having a range of distances. Scenario S1 may thus represent the first scenario of a set of scenarios having a first distance between vehicle 210 and object 212, and scenario S2 may represent a second distance between vehicle 210 and object 212. and scenario _SN may represent the Nth scenario of the set of scenarios with the Nth distance between the vehicle 210 and the object 212. can. Examples of additional types of parameters (also referred to as attributes) are found, for example, in U.S. patent application Ser. can be found and is incorporated herein by reference in its entirety.

For example, the scenario parameters may be the velocity of object 212, the acceleration of object 212, the x-position of object 212 (eg, global position, local position, and/or position relative to any other frame of reference), the y-position of object 212 (eg, , size, pose, local position, global position, and/or position relative to any other frame of reference), bounding box associated with object 212 (e.g., extent (length, width, and/or height), yaw , pitch, roll, etc.), lighting conditions (e.g., brake lights, blinker lights, hazard lights, headlights, reverse lights, etc.), direction of wheels of object 212, map elements (e.g., distance between object 212 and traffic lights, stop sign, speed bump, intersection, yield sign, etc.); object 212 classification (e.g., vehicle, car, truck, bicycle, motorcycle, pedestrian, animal, etc.); object characteristics (e.g., object changes lane); proximity to one or more objects (in any coordinate frame), lane type (eg, lane direction, parking lane, etc.) , road markings (eg, indicating whether passing or lane changes are permitted, etc.), object density, and the like.

As described above, the parameter component 112 can determine ranges of values associated with scenario parameters as represented by the vehicle data 104 and/or other input data. Thus, each of the exemplary scenario parameters identified above, as well as other scenario parameters, can be used to generate a set of scenarios in which the scenarios of a set of scenarios differ by one or more values of the scenario parameters. can be associated with a set or range of

FIG. 3 illustrates an example 300 of generating error model data based at least in part on vehicle data and ground truth data. As shown in FIG. 3, vehicle 202 may generate vehicle data 104 and transmit vehicle data 104 to error model component 116 . As described above, the error model component 116 can determine error models that can indicate errors associated with scenario parameters. For example, vehicle data 104 may be data associated with subsystems of vehicle 202, such as perception systems, planning systems, tracking systems (also called tracker systems), prediction systems, and the like. By way of example, and not limitation, vehicle data 104 may be associated with a sensory system, and vehicle data 104 may include bounding boxes associated with objects detected by vehicle 202 in the environment.

The error model component 116 can receive ground truth data 302 that can be determined from manually labeled and/or other validated machine learning components. By way of example, and not limitation, ground truth data 302 can include verified bounding boxes associated with objects in the environment. By comparing the bounding box of vehicle data 104 to the bounding box of ground truth data 302 , error model component 116 can determine errors associated with subsystems of vehicle 202 . In some examples, vehicle data 104 may include one or more characteristics (also called parameters) associated with the detected entity and/or the environment in which the entity is located. In some examples, properties associated with an entity include x-position (global position), y-position (global position), z-position (global position), orientation, entity type (e.g., classification, etc.), entity velocity, It can include, but is not limited to, the extent (size) of the entity, and the like. Characteristics associated with an environment can include the presence of other entities in the environment, the state of other entities in the environment, the time of day, the day of the week, the season, weather conditions, darkness/light indications, etc. Not limited. Therefore, errors can be related to other characteristics (eg environmental parameters).

Error model component 116 can process multiple vehicle data 104 and multiple ground truth data 302 to determine error model data 304 . Error model data 304 may include errors computed by error model component 116, which may be represented as errors 306(1)-(3). In addition, the error model component 116 can generate errors 306(1)-(3) represented as probabilities 308(1)-(3) that can be associated with environmental parameters for presenting the error models 310(1)-(3). The probability associated with (3) can be determined. By way of example, and not limitation, vehicle data 104 may include bounding boxes associated with objects that are 50 meters from vehicle 202 in an environment that includes rainfall. Ground truth data 302 can provide verified bounding boxes associated with objects. Error model component 116 can determine error model data 304 that determines which errors are associated with the sensory system of vehicle 202 . A distance of 50 meters and rainfall can be used as environmental parameters to determine which of the error models 310(1)-(3) to use. Once the error model is identified, the error model 310(1)-(3) can provide the error 306(1)-(3) based on the probability 308(1)-(3), where the higher probability 308 Errors 306(1)-(3) associated with (1)-(3) are preferred over errors 306(1)-(3) associated with lower probabilities 308(1)-(3). likely to be

FIG. 4 illustrates an example 400 of using error model data to perturb a simulation by providing at least one of errors or uncertainties associated with the simulated environment. As noted above, the error model data can include error models that relate errors 306 to probabilities 308, and the error models can relate to environmental parameters. The simulation component 124 can use the error model data 304 to inject errors, which can provide perturbed parameterization scenarios for perturbing the simulation. Based on the injected error, the simulation data can show how the autonomous controller responds to the injected error.

For example, the simulation component 124 can perturb the simulation by continuously injecting errors into the simulation. By way of example, and not limitation, example 402 shows bounding box 404 associated with an object at time t ₀ . Bounding box 404 may represent detection of an object by a vehicle that includes errors such as errors in bounding box size and/or bounding box position. At time t ₁ , the simulation component 124 can use a bounding box 406 representing the object and containing different errors. For example, at each simulation time (eg, t ₀ , t ₁ , or t ₂ ), simulation component 124 can use different error 306 based on probability 308 associated with error 306 . At time _t2 , the simulation component 124 can use a bounding box 408 representing the object and containing different errors.

In some examples, the simulation component 124 can perturb the simulation by injecting uncertainty associated with bounding boxes representing objects in the environment. By way of example, and not limitation, example 410 shows bounding box 412 associated with an object at time t ₀ . The bounding box may contain a 5% uncertainty, which may indicate the uncertainty of the size and/or position of the object by the 5% amount. Additionally, the uncertainty can persist with the object through times t ₁ and t ₂ rather than injecting different errors at different simulation times, as depicted in example 402 .

FIG. 5 illustrates an example 500 of vehicle 202 generating vehicle data 104 and transmitting vehicle data 104 to computing device 204 . As noted above, the error model component 116 can determine a perceptual error model that can indicate errors associated with scenario parameters. As noted above, vehicle data 104 may include sensor data generated by sensors of vehicle 202 and/or sensory data generated by a sensory system of vehicle 202 . A perceptual error model can be determined by comparing vehicle data 104 with ground truth data 302 . Ground truth data 302 can be manually labeled, can be related to the environment, and can represent known results. Accordingly, deviations in vehicle data 104 from ground truth data 302 can be identified as errors in the sensor and/or perception systems of vehicle 202 . By way of example, and not limitation, a perceptual system may identify an object as a cyclist, where ground truth data 302 indicates that the object is a pedestrian. As another example, and not by way of limitation, the sensor system can generate sensor data representing an object as having a width of 2 meters, where the ground truth data 302 indicates that the object has a width of 1.75 meters. .

As discussed above, the error model component 116 can determine the classification associated with objects represented in the vehicle data 104 and determine other objects of the same classification within the vehicle data 104 and/or other log data. Error model component 116 can then determine a probability distribution associated with the extent of error associated with the object. Based on the error comparison and extent, error model component 116 can determine perceptual error model data 502 .

As shown in FIG. 5, environment 504 may include objects 506(1)-(3) represented as bounding boxes generated by the perceptual system. Perceptual error model data 502 may indicate scenario parameters as 508(1)-(3) and errors associated with the scenario parameters as 510(1)-(3). As shown in FIG. 5, the error associated with object 508(1) can be visualized in environment 504 as a larger bounding box 512 that indicates uncertainty regarding the dimensions of object 508(1).

FIG. 6 illustrates an example computing device 204 600 that generates simulation data 126 and determines vehicle performance data 132 . A parameterized scenario component 120 can determine a parameterized scenario based on the scenario parameters, the set of regions, and the perceptual error model. For example, the scenario parameters may indicate objects, positions associated with the objects, velocities associated with the objects, etc. in the parameterized scenario. Additionally, the scenario parameters may indicate ranges indicating ranges of values and/or probabilities associated with the scenario parameters. A set of regions can represent a portion of the environment that can be used to place objects within the simulated environment. Additionally, the perceptual error model can indicate errors associated with the scenario parameters. These can be combined to create parameterized scenarios that can cover variations provided by scenario parameters, sets of regions, and/or perceptual error models, as detailed herein.

A parameterized scenario can be used by the simulation component 124 to simulate variations of the parameterized scenario. For example, the simulation component 124 can run variations of parameterized scenarios for use in simulations for testing and verification. The simulation component 124 can generate simulation data 126 that indicates how the autonomous vehicle controller performed (eg, responded), compares the simulation data 126 to predetermined results, and/or follows any predetermined rules. / can determine if an assertion has been broken/triggered.

As illustrated in FIG. 6, simulation data 126 may indicate a number of simulations (eg, simulation 1, simulation 2, etc.) and simulation results (eg, result 1, result 2). For example, as described above, the results can indicate pass or fail based on the rules/assertions that were broken/triggered. Additionally, simulation data 126 can indicate the probability of encountering a scenario. By way of example, and not limitation, simulation component 124 can simulate a scenario involving a jaywalking pedestrian. The input data may indicate that the vehicle encounters jaywalking pedestrians at a rate of one minute per hour of driving. This can be used to determine the probability of encountering a particular simulation associated with variations of parameterized scenarios. In some examples, the simulation component 124 can identify variations of parameterized scenarios with low probabilities and run simulations corresponding to those variations. This can enable testing and validation of autonomous vehicle controllers in more unique situations.

Additionally, the simulation component 124 can identify variations of parameterized scenarios for additional simulations based on the results. By way of example, and not limitation, a simulation result may be an obstacle with scenario parameters associated with a vehicle speed of 15 meters per second. The simulation component 124 can identify velocities close to 15 meters per second to determine a threshold that the simulation will pass, which can further aid in the development of safer vehicle controllers.

Based on simulation data 126 , vehicle performance component 130 can generate vehicle performance data 132 . As described above, for event type 1 and cause type 1, for example, data 134(1) can indicate a safety rating, as can data 134(2)-134(4). In some examples, the event type may indicate that the cost meets or exceeds the cost threshold, although other event types are contemplated. For example, costs can include, but are not limited to, reference costs, obstacle costs, horizontal costs, vertical costs, and the like.

A reference cost can include a cost associated with a difference between a point on a reference trajectory (also called a reference point) and a corresponding point on a target trajectory (also called a point or target point), whereby The difference represents one or more differences in yaw, lateral offset, velocity, acceleration, curvature, curvature rate, and the like. In some examples, reducing the weight associated with the reference cost can reduce the penalty associated with the target trajectory being located at a distance from the reference trajectory, which is safer and/or It can provide a smoother transition leading to more comfortable vehicle operation.

In some examples, an obstacle cost may include a cost associated with the distance between a point on the reference or target trajectory and a point associated with the obstacle in the environment. As examples, the points associated with the obstacles can correspond to points on the boundary of the drivable area, or can correspond to points associated with obstacles in the environment. In some examples, obstacles in the environment can be static objects (e.g., buildings, curbs, sidewalks, lane markings, signs, traffic lights, trees, etc.) or dynamic objects (e.g., vehicles, cyclists, pedestrians, animals, etc.). In some examples, dynamic objects can also be called agents. In some examples, static or dynamic objects may be generally referred to as objects or obstacles.

In some examples, lateral costs can refer to costs associated with steering inputs to the vehicle, such as maximum steering input versus vehicle speed. In some examples, longitudinal costs may refer to costs associated with vehicle speed and/or acceleration (eg, maximum braking and/or acceleration). Such costs can be used to ensure that the vehicle is operating within the practicable and/or comfort limits of the passengers operating the ferry.

In some examples, Cause Type 1 and Cause Type 2 may indicate failures such as vehicle failures or object (eg, bicyclist) failures. Vehicle performance component 130 can determine faults using predetermined rules/assertions. By way of example, and not limitation, a rule may indicate that if a vehicle is impacted by an object behind the vehicle, the failure can be associated with the object. In some examples, additional rules can be used to indicate that the vehicle must traverse the environment in a forward direction when impacted by an object behind it. In some examples, cause types (eg, cause type 1 and/or cause type 2) may be associated with components of the autonomous vehicle controller. As non-limiting examples, such causes may include perceptual systems, predictive systems, planner systems, network latency, torque/acceleration failures, and/or failures of any other component or subcomponent of the vehicle.

As described above, the analysis component determines, based on simulation data 126, to override scenario parameters (eg, set fixed values associated with scenario parameters) and to vary other scenario parameters. can be done. Separating the scenario parameters allows the analysis component to determine the scenario parameters associated with a successful or unsuccessful outcome. Vehicle performance data 132 can then indicate safety metrics associated with the parameterized scenario. Additionally, the analysis component can perform sensitivity analysis to determine the cause of failure. For example, the analysis component may individually disable scenario parameters to isolate one or more scenario parameters, determine how disabling the scenario parameters affects the response of the autonomous vehicle; Statistical data associated with disabling one or more scenario parameters can be captured and results captured as success or failure results. Statistical data can show how a set of scenario parameters affects results, can be used to determine which scenario parameters increase or decrease the likelihood of resulting in a successful simulation, and can be used to determine the success of simulation data. It can be used to identify autonomous vehicle subsystems associated with scenario parameters as increasing or decreasing rates.

FIG. 7 shows a block diagram of an exemplary system 700 for implementing the techniques described herein. In at least one example, system 700 can include vehicle 202 . In the illustrated example 700, the vehicle 202 is an autonomous vehicle, although the vehicle 202 may be any other type of vehicle (eg, a driver-controlled vehicle that may provide an indication of whether it is safe to perform various maneuvers). vehicle).

Vehicle 202 includes a computing device 702, one or more sensor systems 704, one or more emitters 706, one or more communication connections 708 (also called communication devices and/or modems), at least one direct Connections 710 (eg, for physically coupling with vehicle 202 to exchange data and/or provide power) and one or more drive systems 712 may be included. One or more sensor systems 704 can be configured to capture sensor data associated with the environment.

Sensor system 704 may include time-of-flight sensors, position sensors (eg, GPS, compass, etc.), inertial sensors (eg, inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes, etc.), lidar sensors, radar sensors, sonar sensors, infrared sensors, cameras (e.g. RGB, IR, intensity, depth, etc.), microphone sensors, environmental sensors (e.g. temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), ultrasonic transducers, wheel encoders, etc. can contain. Sensor system 704 may include multiple instances of each of these or other types of sensors. For example, the time-of-flight sensors may include individual time-of-flight sensors located at the corners, front, rear, sides, and/or top of vehicle 202 . As another example, camera sensors may include multiple cameras positioned at various locations around the exterior and/or interior of vehicle 202 . Sensor system 704 can provide input to computing device 702 .

Vehicle 202 may also include one or more emitters 706 for emitting light and/or sound. One or more emitters 706 in this example include internal audio and visual emitters for communicating with passengers of vehicle 202 . By way of example and not limitation, internal emitters may include speakers, lights, symbols, display screens, touch screens, tactile emitters (e.g. vibration and/or force feedback), mechanical actuators (e.g. seat belt tensioners, seat positioners). , headrest positioners, etc.). The one or more emitters 706 in this example also include external emitters. By way of example and not limitation, the external emitters in this example include lights for signaling direction of travel or other indicators of vehicle operation (e.g., indicator lights, signs, light arrays, etc.), and acoustic beam steering technology. It includes one or more audio emitters (eg, speakers, speaker arrays, horns, etc.) for audibly communicating with one or more pedestrians or other nearby vehicles that may be provided.

Vehicle 202 also has one or more communication connections that enable communication between vehicle 202 and one or more other local or remote computing devices (e.g., remote teleoperation computing devices) or remote services. A portion 708 can be included. For example, communication connection 708 can facilitate communication with other local computing devices on vehicle 202 and/or drive system 712 . Communication connection 708 may also allow vehicle 202 to communicate with other nearby computing devices (eg, other nearby vehicles, traffic lights, etc.).

Communications connection 708 may include physical and/or logical interfaces for connecting computing device 702 to another computing device or one or more external networks 714 (eg, the Internet). For example, communication connection 708 may support Wi-Fi-based communications, such as via frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G etc.), satellite communications, dedicated short range communications (DSRC), or any suitable wired or wireless communications protocol that allows each computing device to interface with other computing devices. In at least some examples, communication connection 708 may comprise one or more modems, as described in detail above.

In at least one example, vehicle 202 may include one or more drive systems 712 . In some examples, vehicle 202 may have a single drive system 712 . In at least one example, where vehicle 202 has multiple drive systems 712, individual drive systems 712 may be located at opposite ends of vehicle 202 (eg, front and rear, etc.). In at least one example, drive system 712 may include one or more sensor systems 704 for detecting conditions around drive system 712 and/or vehicle 202 . By way of example and not limitation, sensor system 704 includes one or more wheel encoders (e.g., rotary encoders) for sensing rotation of the wheels of the drive system, inertial sensors (e.g., , inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.), cameras or other image sensors, ultrasonic sensors for acoustically detecting objects around the drive system, lidar sensors, radar sensors, etc. can be done. Some sensors, wheel encoders, etc. may be specific to drive system 712 . In some cases, sensor system 704 on drive system 712 may overlap or supplement a corresponding system (eg, sensor system 704) on vehicle 202 .

The drive system 712 includes a high voltage battery, a motor that propels the vehicle, an inverter that converts direct current from the battery to alternating current for use in other vehicle systems, a steering motor and a steering rack (which may be electric). , braking systems including hydraulic or electric actuators, suspension systems including hydraulic and/or pneumatic components, stability control systems for braking force distribution to reduce loss of traction and maintain control, HVAC systems, lighting (e.g., vehicle lighting such as head/tail lights that illuminate the external environment), and one or more other systems (e.g. cooling system, safety system, onboard charging system, DC/DC converter, high voltage junction, high voltage cable, charging system) , other electrical components such as charging ports). Additionally, drive system 712 can include a drive system controller that can receive and preprocess data from sensor system 704 and control the operation of various vehicle systems. In some examples, the drive system controller may include one or more processors and memory communicatively coupled to the one or more processors. The memory can store one or more modules for performing various functions of drive system 712 . In addition, drive system 712 also includes one or more communication connections that enable the respective drive system to communicate with one or more other local or remote computing devices.

Computing device 702 can include one or more processors 516 and memory 518 communicatively coupled to one or more processors 716 . In the illustrated example, memory 718 of computing device 702 stores localization component 720 , perception component 722 , prediction component 724 , planning component 726 , and one or more system controllers 728 . Although shown as residing in memory 718 for purposes of illustration, localization component 720, perception component 722, prediction component 724, planning component 726, and one or more system controllers 728 may additionally or Alternatively, computing device 702 may be accessible (eg, stored in a different component of vehicle 202) and/or vehicle 202 may be accessible (eg, stored remotely). It is contemplated that it can.

In memory 718 of computing device 702 , localization component 720 may include functionality for receiving data from sensor system 704 to determine the location of vehicle 202 . For example, the localization component 720 can include and/or request/receive a three-dimensional map of the environment and can continuously determine the position of the autonomous vehicle within the map. In some examples, the localization component 720 uses SLAM (simultaneous localization and mapping) or CLAMS (simultaneous calibration, localization and mapping) to provide time-of-flight data, image data, lidar data, radar data, Sonar data, IMU data, GPS data, wheel encoder data, or any combination thereof, or the like, may be received to accurately determine the position of the autonomous vehicle. In some examples, localization component 720 can provide data to various components of vehicle 202 to determine an initial position of the autonomous vehicle for generating a trajectory, as discussed herein.

Perception component 722 may include functionality for performing object detection, segmentation, and/or classification. In some examples, the perceptual component 722 detects the presence of entities in proximity to the vehicle 202 and/or their presence as entity types (e.g., cars, pedestrians, bicyclists, buildings, trees, road surfaces, curbs, sidewalks, unknown, etc.). It can provide processed sensor data that indicates the entity's classification. In additional and/or alternative examples, the sensory component 722 uses processed sensor data indicative of one or more characteristics (also called parameters) associated with the detected entity and/or the environment in which the entity is located. can provide In some examples, properties associated with an entity include x-position (global position), y-position (global position), z-position (global position), orientation, entity type (e.g., classification, etc.), entity velocity, It can include, but is not limited to, the extent (size) of the entity, and the like. Characteristics associated with an environment may include the presence of other entities in the environment, the state of other entities in the environment, time of day, day of the week, season, weather conditions, geographic location, darkness/light indications, etc. but not limited to these.

Sensory component 722 may include functionality for storing sensory data generated by sensory component 722 . In some examples, the perceptual component 722 can determine tracks corresponding to objects categorized as object types. By way of example only, sensory component 722 using sensor system 704 can capture one or more images of the environment. The sensor system 704 can capture images of the environment including objects such as pedestrians. A pedestrian can be in a first position at time T and in a second position at time T+t (eg, moving during a span of time t after time T). In other words, the pedestrian can move from the first location to the second location during this time period. Such movements can be logged, for example, as stored sensory data associated with the object.

The stored sensory data, in some examples, may include fused sensory data captured by the vehicle. Fused sensory data may be the fusion of sensor data from sensor system 704 such as image sensors, lidar sensors, radar sensors, time-of-flight sensors, sonar sensors, global positioning system sensors, internal sensors, and/or any combination thereof or other sensor data. can include a combination of The stored sensory data may additionally or alternatively include classification data including semantic classifications of objects (eg, pedestrians, vehicles, buildings, road surfaces, etc.) represented in the sensor data. The stored sensory data may additionally or alternatively be track data (historical position, orientation, sensor features associated with the object over time) corresponding to movement of the object through the environment classified as a dynamic object. etc.). Track data may include multiple tracks of multiple different objects over time. This track data is used to generate images of a particular type of object (e.g. pedestrian, animal, etc.) when the object is stationary (e.g. stationary) or moving (e.g. walking, running, etc.). It can be mined for identification. In this example, the computing device determines the track corresponding to the pedestrian.

The prediction component 724 can generate one or more probability maps representing predicted probabilities of possible locations of one or more objects in the environment. For example, prediction component 724 can generate one or more probability maps for vehicles, pedestrians, animals, etc. within a threshold distance of vehicle 202 . In some examples, the prediction component 724 measures the track of the object and generates discretized prediction probability maps, heat maps, probability distributions, discretized probability distributions and/or trajectories can be generated. In some examples, one or more probability maps may represent intents of one or more objects in the environment.

The planning component 726 can determine the route that the vehicle 202 follows to traverse through the environment. For example, planning component 726 can determine different routes and paths and different levels of detail. In some examples, planning component 726 can determine a route to travel from a first location (eg, current location) to a second location (eg, target location). For the purposes of this description, a route can be a series of waypoints for traveling between two locations. Non-limiting examples of waypoints include roads, intersections, global positioning system (GPS) coordinates, and the like. Additionally, the planning component 726 can generate instructions for guiding the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 726 can determine how the autonomous vehicle should be guided from a first waypoint of the series of waypoints to a second waypoint of the series of waypoints. In some examples, an instruction may be a path or part of a path. In some examples, multiple paths can be generated substantially simultaneously (ie, within technological tolerances) according to the receding horizon technique. A single path of multiple paths in the receding data horizon with the highest confidence level can be selected to steer the vehicle.

In other examples, the planning component 726 can alternatively or additionally use data from the perception component 722 to determine the path that the vehicle 202 takes to traverse through the environment. For example, planning component 726 can receive data from sensory component 722 regarding objects associated with the environment. Using this data, planning component 726 determines a path to travel from a first location (eg, current location) to a second location (eg, target location) to avoid objects in the environment. can. In at least some examples, such planning component 726 determines that there are no such collision-free paths, and then directs vehicle 202 to avoid all collisions and/or otherwise damage-mitigating safety routes. can provide a path leading to a catastrophic outage.

In at least one example, computing device 702 may include one or more system controllers 728, which control steering, propulsion, braking, safety, emitters, communications, and other systems of vehicle 202. can be configured to These system controllers 728 may communicate with and/or control corresponding systems of drive system 712 and/or other components of vehicle 202 , which may be configured to operate according to the routes provided by planning component 726 .

Vehicle 202 may be connected to computing device 204 via network 514 and may include one or more processors 730 and memory 732 communicatively coupled to one or more processors 730 . In at least one example, one or more processors 730 can be similar to processor 716 and memory 732 can be similar to memory 718 . In the illustrated example, memory 732 of computing device 204 stores scenario editor component 108 , parameter component 112 , error model component 116 , parameterization scenario component 120 , simulation component 124 , analysis component 128 , and vehicle performance component 130 . do. Although depicted as residing in memory 732 for illustrative purposes, scenario editor component 108, parameter component 112, error model component 116, parameterization scenario component 120, simulation component 124, analysis component 128, and vehicle performance component 130 may additionally or alternatively be accessible to the computing device 204 (eg, stored in different components of the computing device 204) and/or be accessible to the computing device 204. (eg, stored remotely). Scenario editor component 108, parameter component 112, error model component 116, parameterized scenario component 120, simulation component 124, analysis component 128, and vehicle performance component 130 are similar to scenario editor component 108, parameter component 112, and error model component 112 of FIG. 116 , can be substantially similar to parameterization scenario component 120 , simulation component 124 , analysis component 128 , and vehicle performance component 130 .

Processor 716 of computing device 702 and processor 730 of computing device 204 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. can be done. By way of example and not limitation, processors 716 and 730 may be one or more central processing units (CPUs), graphics processing units (GPUs), or other devices capable of processing electronic data and storing the electronic data in registers or memory. It can include any other device or part of a device that converts to electronic data. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices are also processors and processors, so long as they are configured to implement the encoded instructions. can be regarded as

Memory 718 on computing device 702 and memory 732 on computing device 204 are examples of non-transitory computer-readable media. Memories 718 and 732 can store an operating system and one or more software applications, instructions, programs, and/or data that implement the methods and functions attributed to the various systems described herein. In various implementations, memories 718 and 732 may be any memory such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile/flash type memory, or any other type of memory capable of storing information. can be implemented using any suitable memory technology. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, illustrated herein in the accompanying drawings. is merely an example relevant to the description of

In some examples, aspects of some or all of the components described herein may include any models, algorithms, and/or machine learning algorithms. For example, in some examples the components in memories 718 and 732 may be implemented as neural networks.

FIG. 8 shows an exemplary process 800 for determining safety metrics associated with vehicle controllers. Some or all of process 800 may be performed by one or more components of FIGS. 1-7, as described herein. For example, some or all of process 800 may be performed by computing device 204 and/or computing device 702 .

At operation 802 of exemplary process 800, process 800 can include receiving log data associated with operations in an autonomous vehicle within an environment. In some examples, log data can be generated by the vehicle capturing at least environmental sensor data.

At operation 804 of the exemplary process 800, the process 800 can include determining a set of scenarios based on the log data (or other data), the scenarios of the set of scenarios depending on aspects of the environment. Contains associated scenario parameters. In some examples, the computing device can group similar scenarios represented in the log data. For example, scenarios can be grouped using, for example, k-means clustering and/or evaluating weighted distances (eg, Euclidean) between parameters of the environment. Additionally, the scenario parameters may represent environmental parameters such as night time environmental parameters or wet condition environmental parameters. In some examples, scenario parameters may be associated with vehicles or objects (eg, pose, speed, etc.).

At operation 806 of exemplary process 800, process 800 may include determining an error model associated with a subsystem of the autonomous vehicle. An error model component can compare vehicle data (eg, log data) to ground truth data to determine differences between the vehicle data and the ground truth data. In some examples, vehicle data can represent estimates associated with objects in the environment, such as estimated position, estimated direction, and estimated range, while ground truth data can represent the object's actual position, actual direction, etc. , or can represent the actual range. Based on the differences, the error model component can determine errors associated with vehicle subsystems (eg, perception system, tracking system, prediction system, etc.).

At operation 808 of exemplary process 800, process 800 can include determining a parameterized scenario based on the scenario parameters and the error model. These can be combined to create parameterized scenarios that can cover variations provided by scenario parameters and/or error models. In some examples, scenario parameters can be randomly selected and combined to create a parameterized scenario. In some examples, scenario parameters can be combined based on their probabilities of co-occurrence. By way of example, and not limitation, the log data may indicate that 5% of the driving experiences include pedestrian encounters, and the parameterized scenario component generates , can include pedestrians as scenario parameters. In some examples, the parameterized scenario component can validate parameterized scenarios to reduce unlikely or improbable combinations of scenario parameters. By way of example, and not limitation, the vehicle is not located on the lake and the pedestrian is not walking at 30 meters per second. As a non-limiting example, such a parameterized scenario is a specific defined scenario with various Gaussian (or other distributions) of errors in perceptual models, predictive models, etc., based at least in part on the scenario parameters. range of vehicle and jaywalker distances, speeds, lighting conditions, weather conditions, etc. on roads.

At operation 810 of the exemplary process 800, the process 800 modifies at least one of the parameterized scenario, the scenario parameters, or the components of the simulated vehicle based at least in part on the error. perturbing the scenario. In some examples, uncertainty can be associated with scenario parameters. As an example, and not by way of limitation, an object's position may be associated with a 5% uncertainty and the autonomous controller may be allowed to traverse the environment while accounting for the 5% uncertainty. In some examples, the simulator can determine from the error model which errors to incorporate into the simulation when the simulator runs the simulation.

At operation 812 of exemplary process 800, process 800 may include instantiating the simulated vehicle in the perturbed parameterized scenario. The simulator uses a simulated vehicle that can be associated with an autonomous controller and allows the autonomous controller to traverse a simulated environment. Instantiating an autonomous vehicle controller in a parameterized scenario and simulating the parameterized scenario can efficiently cover a wide variety of scenarios without requiring manual enumeration of variations. Additionally, based at least in part on executing the parameterized scenario, the simulation data indicates how the autonomous vehicle controller responded to the parameterized scenario; You can determine the outcome of a failed or unsuccessful attempt.

At operation 814 of exemplary process 800, the process may include receiving simulation data indicating how the simulated vehicle will respond to the perturbed parameterized scenario. After simulation, the results may indicate a pass (eg, a successful result), a failure, and/or a degree of success or failure associated with the vehicle controller.

At operation 816 of exemplary process 800, the process may include determining a safety metric associated with the vehicle controller based on the simulation data. For example, each simulation can have a success or failure outcome. Additionally, as described above, the vehicle performance component can determine vehicle performance data that can indicate how the vehicle performs in the environment based on the simulation data. Based on the sensitivity analysis, the vehicle performance data may indicate scenario parameter boundaries that indicate scenarios for which the simulation did not result in success, reasons for the simulation failure, and/or scenario parameter values for which the simulation resulted in success. can. Thus, the safety metric can indicate the pass/fail rate of vehicle controllers in various simulated scenarios.

FIG. 9 depicts a flow diagram of an exemplary process for determining statistical models associated with subsystems of an autonomous vehicle. Some or all of process 900 may be performed by one or more components of FIGS. 1-7, as described herein. For example, some or all of process 900 may be performed by computing device 204 and/or computing device 702 .

At operation 902 of exemplary process 900, process 900 may include receiving vehicle data (or other data) associated with a subsystem of the autonomous vehicle. Vehicle data may include log data captured by vehicles traversing through the environment. In some examples, vehicle data may include control data (eg, data used to control systems such as steering, braking, etc.) and/or sensor data (eg, lidar data, radar data, etc.). can.

At operation 904 of exemplary process 900, process 900 may include determining output data associated with the subsystem based on the vehicle data. By way of example, and not limitation, the subsystem may be a perceptual system and the output data may be bounding boxes associated with objects in the environment.

At operation 906 of exemplary process 900, process 900 can include receiving ground truth data associated with the subsystem. In some examples, ground truth data can be determined from manual labeling and/or other validated machine learning components. By way of example, and not limitation, ground truth data can include verified bounding boxes associated with objects in the environment.

At operation 908 of the exemplary process 900, the process 900 can include determining a difference between the first portion of the output data and the second portion of the ground truth data, the difference being sent to the subsystem. Represents the associated error. As noted above, the output data may include bounding boxes associated with objects in the environment detected by the vehicle's perception system, and the ground truth data may include validated bounding boxes associated with the objects. can contain. The difference between the two bounding boxes can indicate errors associated with the vehicle's perceptual system. By way of example, and not limitation, the output data bounding box can be larger than the validated bounding box, indicating that the perceptual system is detecting that the object is larger than in the environment.

At operation 910 of exemplary process 900, process 900 can include determining a statistical model associated with the subsystem based on the difference. for example,

[Exemplary Content of the Invention]
A: one or more processors and, when executed, causing the system to receive log data associated with operating an autonomous vehicle within an environment; , determining a set of scenarios, the scenarios of the set of scenarios including scenario parameters associated with aspects of the environment; and determining error models associated with subsystems of the autonomous vehicle. determining a parameterized scenario based at least in part on said scenario parameters and said error model; said scenario parameters or simulated vehicle instantiated in said perturbed parameterized scenario; wherein the simulated vehicle is controlled by a vehicle controller; instantiating the simulated vehicle in a parameterized scenario; and receiving simulation data indicating how the simulated vehicle will respond to the perturbed parameterized scenario. and determining a safety metric associated with the vehicle controller based at least in part on the simulation data. A system that includes a readable medium.

B: Determining the set of scenarios is clustering the log data to determine a first set of clusters, wherein individual clusters of the first set of clusters are individual scenarios determining a probability associated with the individual cluster based at least in part on the first set of clusters; and determining a probability threshold and the first set of clusters from and determining a second set of clusters based at least in part.

C. Determining the error model determines an error based at least in part on receiving ground truth data associated with the environment and comparing the ground truth data to the log data. and determining an error distribution based at least in part on the error, wherein the error model includes the error distribution.

D: the parameterized scenario is a first parameterized scenario, the perturbed parameterized scenario is a first perturbed parameterized scenario, and the simulation data is the first simulation data A, wherein the operation determines a second parameterized scenario including at least one of a first subset of the scenario parameters or a second subset of the error model based on the first simulation data. perturbing said second parameterized scenario as a second perturbed parameterized scenario; and said simulated vehicle in said second perturbed parameterized scenario , receiving second simulation data, and updating the safety metric based at least in part on the second simulation data. system.

E: determining a scenario including scenario parameters describing a portion of an environment; receiving an error model associated with a subsystem of the vehicle; determining a parameterized scenario based on an objective; perturbing said parameterized scenario as a perturbed parameterized scenario; and determining said subsystem of said vehicle in said perturbed parameterized scenario. and determining a safety metric associated with the subsystem of the vehicle based at least in part on the simulation data. .

F: The method of paragraph E, wherein the scenario parameters are associated with at least one of object size, object velocity, object pose, object density, vehicle velocity, vehicle trajectory.

G: determining the scenario comprises receiving log data associated with an autonomous vehicle; clustering the log data to determine a first set of clusters; a set of individual clusters associated with the scenario; determining probabilities associated with the individual clusters based at least in part on the first set of clusters; and determining that the probability meets or exceeds a probability threshold.

H: The error model determines errors based at least in part on receiving ground truth data associated with the environment and comparing the ground truth data to log data associated with the vehicle. and determining an error distribution based at least in part on said error, said error model comprising said error distribution.

I: said parameterized scenario is a first parameterized scenario, said perturbed parameterized scenario is a first perturbed parameterized scenario, and said simulation data is a first simulation data; A second parameterized scenario comprising at least one of a first subset of the scenario parameters or a second subset of the error model based on the first simulation data. perturbing the second parameterized scenario; receiving second simulation data; and updating the safety metric based at least in part on the second simulation data. and further comprising the method of paragraph E.

J: invalidating at least a first portion of one of said scenario parameters or said error model; and transmitting said second simulation data to at least a second portion of said one of said scenario parameters or said error model not invalidated; The method of paragraph I further comprising associating with the portion of

K: The method of paragraph E, wherein the safety metric indicates the probability of meeting or exceeding a cost threshold.

L: said portion is a first portion and said method is to receive map data, said second portion of said map data being associated with said first portion of said environment; and determining that the second portion of the map data is associated with scenarios associated with probabilities that meet or exceed threshold probabilities associated with the scenario parameters. The method described in .

M: when the instructions are executed, determining a scenario including scenario parameters that describe a portion of the environment to the processor and receiving one or more of the error models associated with the vehicle's subsystems; determining; determining a parameterized scenario based at least in part on said scenario, said scenario parameters, and said error model; and perturbing said parameterized scenario as a perturbed parameterized scenario. and receiving simulation data indicating how the subsystem of the vehicle will respond to the perturbed parameterized scenario; and based at least in part on the simulation data, the A non-transitory computer-readable medium storing processor-executable instructions for performing operations including: determining a safety metric associated with a subsystem;

N: The non-transient computer readable of paragraph M, wherein the scenario parameters are associated with at least one of object size, object velocity, object pose, object density, vehicle velocity, vehicle trajectory. medium.

O: determining the scenario comprises receiving log data associated with an autonomous vehicle; clustering the log data to determine a first set of clusters; a set of individual clusters associated with the scenario; determining probabilities associated with the individual clusters based at least in part on the first set of clusters; determining that the probability meets or exceeds a probability threshold.

P: the error model determines errors based at least in part on receiving ground truth data associated with the environment and comparing the ground truth data to log data associated with the vehicle; and determining an error distribution based at least in part on said error, said error model comprising said error distribution. readable medium.

Q: The parameterized scenario is the first parameterized scenario, the perturbed parameterized scenario is the first perturbed parameterized scenario, and the simulation data is the first simulation data. A, wherein the operation determines a second parameterized scenario including at least one of a first subset of the scenario parameters or a second subset of the error model based on the first simulation data. perturbing the second parameterized scenario; receiving second simulation data; and updating the safety metric based at least in part on the second simulation data. The non-transitory computer-readable medium of paragraph M, further comprising:

R: said operation invalidates at least a first portion of said one of said scenario parameters or said error model; Associating with at least the second portion. The non-transitory computer-readable medium of paragraph Q.

S: The non-transitory computer-readable medium of paragraph M, wherein the safety metric indicates a probability of meeting or exceeding a cost threshold.

T: The non-transitory computer-readable medium of paragraph M, wherein the error model is associated with one or more of the vehicle's perception system, the vehicle's prediction system, or the vehicle's planner system.

U: one or more processors, when executed to said system, receiving vehicle data; and inputting at least a first portion of said vehicle data to a subsystem of an autonomous vehicle; wherein the subsystem is associated with at least one of a perception system, a planning system, a tracking system, or a prediction system; and based at least in part on a second portion of the vehicle data, an environmental parameter receiving an estimate from the subsystem; receiving ground truth data associated with the subsystem; and determining a difference between the estimate and the ground truth data. wherein the difference represents an error associated with the subsystem; and based at least in part on the difference, determining a statistical model associated with the subsystem indicative of the probability of the error. and one or more computer-readable media storing computer-executable instructions for performing an operation comprising: said probability being associated with said environmental parameter.

V: the vehicle data comprises sensor data from sensors on the autonomous vehicle, the environmental parameters comprise one or more of speed or weather conditions of the autonomous vehicle, and the subsystem is a perception subsystem; wherein the estimate is one or more of an estimated position, an estimated orientation, or an estimated range of an object represented in the vehicle data, and the ground truth data is an actual position, actual orientation, or Or the system of paragraph U representing an actual range.

W: Determining the statistical model determines a first frequency associated with the environmental parameter and a second frequency associated with the difference based at least in part on the vehicle data; and determining the probability based at least in part on the first frequency and the second frequency.

X: the operation determines a simulated environmental parameter based at least in part on simulated vehicle data; and determining that the simulated environmental parameter corresponds to the environmental parameter. determining a simulated estimate based at least in part on the simulated vehicle data and the subsystem; and at least in part on the error based at least in part on the probability. and perturbing the simulated estimates by modifying a portion of the corresponding simulated scenario based on .

Y: receiving data associated with a vehicle; determining an environmental parameter based at least in part on a first portion of said data; and based at least in part on a second portion of said data. determining output data associated with a system of the vehicle; receiving ground truth data associated with the system and the data; and determining a difference between the output data and the ground truth data. determining that the difference represents an error associated with the system; and based at least in part on the difference, determining a statistical model associated with the system indicative of the probability of the error. and wherein said probability is associated with said environmental parameter.

Z: The method of paragraph Y, wherein determining the statistical model comprises determining frequencies associated with the errors based at least in part on the data.

AA: The method of paragraph Y, wherein the environmental parameters include one or more of speed of the vehicle, weather conditions, geographic location of the vehicle, or time of day.

AB: generating a simulation; determining that simulated environmental parameters of said simulation correspond to said environmental parameters; inputting simulated data into said system; Y. The method of paragraph Y, further comprising: receiving a predicted output; and perturbing a simulation based at least in part on said probability and said error.

AC: the system is a perceptual system, the output data includes a first bounding box associated with an object, the ground truth data includes a second bounding box associated with the object, and the difference is Determining a first extent of the first bounding box and a second extent of the second bounding box, or a first pose of the first bounding box and a second extent of the second bounding box The method of paragraph Y comprising determining the difference between at least one of the second poses.

AD: said system is a tracker system, said output data comprises planned trajectory data of said vehicle, said ground truth data comprises measured trajectory of said vehicle, and determining said difference comprises said planned trajectory data and The method of paragraph Y, comprising determining the difference between the measured trajectory.

AE: said system is associated with a prediction system, said data includes predicted trajectories of objects in an environment, said ground truth data includes observed trajectories of said objects, and said determining the difference comprises: and the observation trajectory.

AF: said data is a first data, said environmental parameter is a first environmental parameter, said difference is a first difference, said error is a first error, said probability is a first probability, the method comprising: receiving second data associated with the system of the vehicle; and determining a second environmental parameter based at least in part on the second data. , determining a second difference between a third portion of the output data and a fourth portion of the ground truth data, the second difference being a second and updating the statistical model associated with the system, the statistical model indicating a second probability of the second error, the second probability being the second The method of paragraph Y, further comprising: being associated with two environmental parameters.

AG: when the instructions are executed, receiving data to a processor; determining an environmental parameter based at least in part on said data; based at least in part on said data and a system of the vehicle; determining output data; receiving ground truth data associated with the system and the data; determining a difference between a first portion of the output data and a second portion of the ground truth data; determining that the difference represents an error associated with the system; and based at least in part on the difference, determining a statistical model associated with the system indicative of the probability of the error. and relating said probability to said environmental parameter.

AH: The non-transitory computer-readable medium of paragraph AG, wherein determining the statistical model includes determining a frequency associated with the difference based at least in part on the data.

AI: The non-transitory computer-readable medium of paragraph AG, wherein the environmental parameters include one or more of speed of the vehicle, weather conditions, or time of day.

AJ: generating a simulation including a vehicle in which said manipulation is simulated; receiving simulated data; determining that simulated environmental parameters correspond to said environmental parameters; inputting at least a portion of said simulated data into said system; receiving simulated output data from said system; modifying the generated output data. The non-transitory computer-readable medium of paragraph AG.

AK: said system is a perceptual system, said data comprises a first bounding box associated with an object, said ground truth data comprises a second bounding box associated with said object, and determining said difference a first extent of the first bounding box and a second extent of the second bounding box, or a first pose of the first bounding box and a second extent of the second bounding box; The non-transitory computer-readable medium of paragraph AG, comprising determining the difference between at least one of two poses.

AL: The system is a tracker system, the data includes planned trajectory data of the vehicle, the ground truth data includes measured trajectories of the vehicle, and determining the difference comprises combining the planned trajectory data and the The non-transitory computer-readable medium of paragraph AG, comprising determining the difference between a measured trajectory.

AM: said system is associated with a prediction system, said data includes predicted trajectories of objects in an environment, said ground truth data includes observed trajectories of said objects, and determining said difference is associated with said predicted trajectories; and the observed trajectory.

AN: the data is the first data, the environmental parameter is the first environmental parameter, the difference is the first difference, the error is the first error, and the probability is the first probability, wherein the operation includes receiving second data associated with the system of the vehicle and determining a second environmental parameter based at least in part on the second data. , determining a second difference between a third portion of the output data and a fourth portion of the ground truth data, the second difference being a second and updating the statistical model associated with the system, the statistical model indicating a second probability of the second error, the second probability being the second The non-transitory computer-readable medium of paragraph AG, further comprising: associated with two environmental parameters.

Although the above example subject matter has been described in terms of one particular implementation, in the context of this document the example subject matter can also be implemented through methods, devices, systems, computer-readable media, and/or other implementations. It should be understood that it can be implemented Moreover, any of the examples A-AN may be implemented alone or in combination with any other one or more of the examples A-AN.

[Conclusion]
Although one or more examples of the techniques described herein have been described, various modifications,
Additions, permutations, and their equivalents are included within the scope of the technology described herein.

The illustrative description refers to the accompanying drawings, which form a part hereof and which show, by way of illustration, specific examples of the claimed subject matter. It is to be understood that other examples may be used and modifications or substitutions, such as structural changes, may be made. Such illustrations, modifications or substitutions do not necessarily depart from the scope of the intended claimed subject matter. Although the steps herein may be presented in a particular order, in some cases the order may be changed so as to provide certain inputs at different times or in a different order without changing the functionality of the systems and methods described. can be changed. The disclosed procedures can also be performed in different orders. Moreover, the various computations presented herein need not be performed in the order disclosed, and other examples using alternate orders of computations can be readily implemented. In addition to reordering, computations can also be decomposed into subcomputations with the same result.

Claims (15)

determining a scenario including scenario parameters describing portions of the environment;
receiving an error model associated with a vehicle subsystem;
determining a parameterized scenario based at least in part on the scenario, the scenario parameters, and the error model;
perturbing the parameterized scenario as a perturbed parameterized scenario;
receiving simulation data indicating how the subsystems of the vehicle will respond to the perturbed parameterized scenario;
determining a safety metric associated with the subsystem of the vehicle based at least in part on the simulation data;
method including. 2. The method of claim 1, wherein the scenario parameters are associated with at least one of object size, object speed, object pose, object density, vehicle speed, and vehicle trajectory. Determining the scenario includes:
receiving log data associated with the autonomous vehicle;
clustering the log data to determine a first set of clusters, wherein individual clusters of the first set of clusters are associated with the scenario;
determining probabilities associated with the individual clusters based at least in part on the first set of clusters;
determining that the probability meets or exceeds a probability threshold;
3. The method of claim 1 or 2, comprising: The error model is
receiving ground truth data associated with the environment;
determining an error based at least in part on comparing the ground truth data to log data associated with a vehicle;
determining an error distribution based at least in part on the error;
determined, at least in part, based on
4. The method of any one of claims 1-3, wherein the error model comprises the error distribution. the parameterized scenario is a first parameterized scenario, the perturbed parameterized scenario is a first perturbed parameterized scenario, the simulation data is a first simulation data, and
The method includes:
determining a second parameterized scenario including at least one of a first subset of the scenario parameters or a second subset of the error model based on the first simulation data;
perturbing the second parameterized scenario;
receiving second simulation data;
updating the safety metric based at least in part on the second simulation data;
5. The method of any one of claims 1-4, further comprising. disabling at least a first portion of one of the scenario parameters or the error model;
6. The method of claim 5, further comprising associating the second simulation data with at least a second portion of one of the scenario parameters or the error model that are not overridden. 7. A method according to any preceding claim, wherein said safety metric indicates a probability of meeting or exceeding a cost threshold. said portion is a first portion;
said method comprising:
receiving map data, wherein a second portion of the map data is associated with the first portion of the environment;
determining that the second portion of the map data is associated with a scenario associated with a probability of meeting or exceeding a threshold probability associated with the scenario parameter;
8. The method of any one of claims 1-7, further comprising: A computer program product comprising coded instructions which, when run on a computer, implements the method of any of claims 1-8. one or more processors;
one or more non-transitory computer-readable media storing instructions;
A system comprising
The instructions, when executed, cause the one or more processors to:
determining a scenario including scenario parameters that describe a portion of the environment;
receiving or determining one or more of the error models associated with vehicle subsystems;
determining a parameterized scenario based at least in part on the scenario, the scenario parameters, and the error model;
perturbing the parameterized scenario as a perturbed parameterized scenario;
receiving simulation data indicating how the subsystems of the vehicle will respond to the perturbed parameterized scenario;
determining a safety metric associated with the subsystem of the vehicle based at least in part on the simulation data;
A system that allows you to perform operations, including the scenario parameters are associated with at least one of object size, object velocity, object pose, object density, vehicle velocity, vehicle trajectory;
the safety metric indicates a probability of meeting or exceeding a cost threshold; or the error model is associated with one or more of the vehicle's perception system, the vehicle's prediction system, or the vehicle's planner system. ,
11. The system of claim 10, wherein the system is at least one of: Determining the scenario includes:
receiving log data associated with the autonomous vehicle;
clustering the log data to determine a first set of clusters, wherein individual clusters of the first set of clusters are associated with the scenario;
determining probabilities associated with the individual clusters based at least in part on the first set of clusters;
determining that the probability meets or exceeds a probability threshold;
12. A system according to claim 10 or 11, comprising: The error model is
receiving ground truth data associated with the environment;
determining an error based at least in part on comparing the ground truth data to log data associated with a vehicle;
determining an error distribution based at least in part on the error;
determined based at least in part on
13. The system of any one of claims 10-12, wherein the error model comprises the error distribution. the parameterized scenario is a first parameterized scenario, the perturbed parameterized scenario is a first perturbed parameterized scenario, the simulation data is a first simulation data, and Said operation is
determining a second parameterized scenario including at least one of a first subset of the scenario parameters or a second subset of the error model based on the first simulation data;
perturbing the second parameterized scenario;
receiving second simulation data;
updating the safety metric based at least in part on the second simulation data;
14. The system of any one of claims 10-13, further comprising: said operation invalidating at least a first portion of one of said scenario parameters or said error model;
associating the second simulation data with at least a second portion of one of the non-overridden scenario parameters or the error model;
15. The system of claim 14, further comprising:

Images