CN112816954A - Road side perception system evaluation method and system based on truth value - Google Patents

Abstract

The present application discloses a ground truth-based roadside sensing system evaluation method, which includes the following steps: establishing a truth sensing device group, and synchronizing with the sensing equipment of the roadside sensing system RSS to be tested in a selected test time interval. Roadside perception data collection; process the original data returned by the truth perception device group, complete target type identification and target trajectory identification, and complete perception data labeling; generate true values based on the labeled data, and the true value data includes traffic participant targets Type, position, speed, acceleration, trajectory; in the selected test time interval, compare the structured perception data output by the RSS to be tested and the true value data, and output the statistical evaluation result of the perception performance. The present application also proposes a ground truth system for roadside perception system evaluation.

Description

A ground truth-based roadside perception system evaluation method and system

technical field

The present application relates to the technical field of automatic driving, and in particular, to a method and device for evaluating a roadside perception system based on ground truth.

Background technique

Roadside Sensing System (RSS) is an important means to support connected autonomous driving, improve traffic operation efficiency and relieve congestion. Providing information such as over-the-horizon perception, blind spot warning, and driving intent for autonomous vehicles through the RSS system is one of the important technical means to make up for the limitations of single-vehicle autonomous driving perception. The current composition of RSS is different, including configuration schemes such as lidar + camera, millimeter-wave radar + camera, etc. Each scheme is better than the other, and there is a lack of systematic evaluation methods for RSS as a whole, especially the structured perception of RSS output. Test specifications for data type, accuracy, and quality have not yet been established, making it impossible for the vehicle to directly accept roadside information. At the current stage of autonomous driving development, RSS can only be used as a redundant source of information for vehicle integration at the perception level, which cannot be achieved. Quantitative evaluation of RSS is one of the key factors restricting collaborative decision-making and control and realizing fully autonomous driving.

SUMMARY OF THE INVENTION

The present application proposes a ground truth-based roadside perception system evaluation method and system, which solves the problem that the current vehicle-road collaborative roadside perception system lacks a systematic evaluation method, and especially solves the problem that the quality of roadside perception messages cannot be quantitatively evaluated.

On the one hand, an embodiment of the present application proposes a ground truth-based roadside perception system evaluation method, which includes the following steps:

Establish a true value sensing device group, and collect roadside sensing data synchronously with the sensing equipment of the roadside sensing system RSS to be tested in the selected test time interval;

Process the original data returned by the truth-value sensing device group, complete target type identification and target trajectory identification, and complete sensing data labeling;

Generate ground truth based on labeled data, ground truth data includes target type, position, speed, acceleration, trajectory of traffic participants;

In the selected test time interval, the structured sensing data output by the RSS to be tested and the true value data are compared, and the statistical evaluation result of the sensing performance is output.

Preferably, the ground truth perception device group includes a high-beam lidar, a high-definition camera, and a millimeter-wave radar.

Preferably, the processing of the original data returned by the truth-sensing device group to complete target type identification and target trajectory identification further includes:

Based on the point cloud data returned by the high-speed laser radar, the target type identification and target trajectory tracking of traffic participants are completed. The data collected by the camera is used to correct the target type, and the millimeter wave radar data is used to correct the target trajectory.

Preferably, the processing of the original data returned by the truth-sensing device group further includes:

Data cleaning is performed on the original data returned by the truth-value sensing device group, and data from different sensing devices is time-aligned.

In any one of the method embodiments of the present application, preferably, the comparing the structured sensory data output by the RSS to be tested and the true value data further includes, calculating:

Target recognition accuracy = the number of targets or events correctly detected by the RSS to be tested/the number of targets or events in the true value data.

In any one of the method embodiments of the present application, preferably, the comparing the structured sensory data output by the RSS to be tested and the true value data further includes, calculating:

Target missed detection rate=1-number of targets or events detected by the RSS to be detected/number of targets or events in the true value data.

In any one of the method embodiments of the present application, preferably, the comparing the structured sensory data output by the RSS to be tested and the true value data further includes, calculating:

The distance between the discrete time series of state parameters in the structured sensory data output by RSS and the discrete time series of state parameters in the ground-truth data.

The state parameters include at least one of the following: target size, position, speed, heading angle, acceleration, and trajectory.

On the other hand, the present application also proposes a truth value system for roadside perception system evaluation, which implements the method described in any one of the embodiments of the present application, and the truth value system includes a truth value perception device group and a server;

The truth value perception device group at least includes high-line-speed lidar, high-definition camera, and millimeter-wave radar;

The server includes a data acquisition module, an intelligent processing module, a true value storage module, and an RSS evaluation module;

the data acquisition module, configured to fuse the image, video and point cloud data output by the truth perception device group;

The intelligent processing module is used to process the original data returned by the truth-value sensing device group, complete target type identification and target trajectory identification, and complete sensing data labeling;

The truth value storage module is used for backing up the truth value data;

The RSS evaluation module is used to compare the structured perception data output by the RSS to be tested and the true value data, and output the statistical evaluation result of the perception performance.

Preferably, the sensing device group and the RSS to be tested are multiplexed with pole frame resource deployment.

The above-mentioned at least one technical solution adopted in the embodiments of the present application can achieve the following beneficial effects:

The solution of the present invention will provide feasible support for the function and performance evaluation of the roadside perception system, and also provide a test basis for the product selection of the roadside perception system, promote the technological evolution and product iterative upgrade in the field of vehicle-road collaboration, and improve the Industry normative effects.

Description of drawings

The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the attached image:

Figure 1 is a technical architecture diagram of RSS;

Figure 2 is the overall architecture diagram of the truth system;

Fig. 3 is the test method of the roadside perception system based on RS;

Figure 4 is a sample space representation of detection accuracy calculation.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below with reference to the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

The invention proposes a roadside perception system evaluation method based on the truth value system. First, the composition structure and deployment principle of the truth value system are proposed, and then the test logic and the test logic of the roadside perception system are described in detail on the basis of the truth value system. Finally, an evaluation index system for roadside perception system is proposed.

The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

Figure 1 is a technical architecture diagram of RSS.

The basic components of RSS are roadside sensing equipment and roadside computing units. As shown in Figure 1, roadside sensing equipment includes but is not limited to cameras, lidars, millimeter-wave radars and other equipment, which can collect images of the currently covered traffic environment in real time. , video, point cloud and other original perception data, the roadside computing unit includes but is not limited to edge computing servers, industrial computers and other computing equipment, through real-time fusion calculation of the original perception data collected by the roadside perception equipment, to realize the traffic participants in the traffic environment. Obtain full information such as state information, road condition information, and traffic events, and then send perception messages to local/global traffic participants through the roadside unit RSU and central subsystem.

Regardless of the composition of the RSS architecture, the final output structured perception data format is defined by the "Application Layer and Application Data Interaction Standard for the Vehicle Communication System of the Cooperative Intelligent Transportation System" T/CSAE 53-2017, but the quality of the perception data is closely related to the RSS. The composition is closely related, for example, the RSS equipped with lidar can obtain more accurate information such as the speed, acceleration, and trajectory tracking of the participants, and the RSS equipped with the camera has stronger ability to identify target types, etc., and establish a set of equipment with complete configuration and perception performance. A superior data reference system is of great significance for the detailed description of the traffic environment and the quantitative evaluation of the system performance of RSS.

Figure 2 is the overall architecture diagram of the truth system.

The present invention proposes a true value system (Reference System, RS) oriented to RSS test and evaluation, and a block diagram of the system is shown in FIG. 2 . This RS system includes a perception equipment group composed of high-performance sensors and an offline truth value system server that meets the needs of big data processing. The perception equipment group includes but is not limited to high-beam lidar, high-definition cameras, and millimeter-wave radar; offline truth value system server It has professional storage, processing and analysis capabilities of PB-level data, and carries four functional modules: data acquisition module, intelligent processing module, true value storage module, and RSS evaluation module. The intelligent processing module mainly completes the original data association, automatic labeling and other links, generates long-term environmental truth values, and realizes the storage and redundancy of PB-level data through the truth value storage module. The RSS evaluation module passes The set evaluation dimensions and index system output statistical analysis results.

The RSS to be tested is generally deployed in key traffic monitoring areas such as urban intersections, expressway ramp entrances/exits, bridges and tunnels, and reuses pole frame resources such as signal light gantry, highway gantry, and roadside light poles. The equipment group can be deployed with the RSS multiplexing pole frame resources to be tested. In view of the fact that the offline true value system server uses offline calculation to generate the true value of the collected environmental information, it can be flexibly deployed on the roadside, central computer room, etc. according to the actual situation. The wireless network implements the data backhaul from the sensing device group to the server.

Fig. 3 is the test method of the roadside perception system based on RS.

The present invention proposes a method for testing and evaluating the roadside perception system based on the true value system RS. The test score is divided into two parts: roadside original perception data collection and server-side offline processing. The specific test steps are shown in FIG. 3 .

Roadside raw perception data collection:

Step 1: Reuse the RSS pole frame resources to be tested and deploy the RS sensing equipment group;

Step 2: Perform global sensor calibration on the RS sensing device group, and set the true value collection area according to the sensing area of the RSS to be tested;

Step 3: Select the test time interval, the RSS and RS sensing device groups to be tested start the roadside sensing data collection synchronously, and send the data back to the server through the wired/wireless network.

Server-side offline processing:

Step 1: Perform data cleaning on the original sensing data returned by the RS sensing device group to ensure data consistency, and complete the time alignment of lidar point cloud, millimeter wave radar point cloud, image, video and other data;

Step 2: Automatic labeling of the data returned by the RS sensing equipment group - based on the point cloud data returned by the high-beam lidar, based on algorithms such as machine learning and deep learning, to complete the identification and detection of the target type of traffic participants, and Offline trajectory tracking of multiple targets. The data collected by the camera is fused, the target type is corrected twice, and the millimeter wave radar data is integrated to perform the second correction on the target trajectory data (including speed, acceleration, position, etc.). Input the automatically labeled data into the correction module (allowing manual annotation to intervene in the correction) to complete all kinds of perceptual data labeling;

Step 3: Generate static and dynamic true values based on the marked data, including but not limited to the true values of the target type, position, speed, acceleration, trajectory, etc. of the traffic participants, and complete the true value storage and RS establishment;

Step 4: Extract the true value in the test time interval, complete the time alignment with the output structured perception data of the RSS to be tested, set the evaluation dimension, and output the statistical evaluation result of the perception performance.

The invention also proposes a perceptual data quality evaluation algorithm based on a multi-dimensional index system, which is applied to the design and development of the RSS evaluation module. The design of the multi-dimensional index system is based on the roadside message body defined in T/CSAE 53-2017 of the "Application Layer and Application Data Interaction Standard for Vehicle Communication System of Cooperative Intelligent Transportation System" and the perception data category output by the current mainstream RSS. The specific evaluation indicators include Target recognition accuracy, target missed detection rate, detection accuracy (target size, position, speed, heading angle, acceleration, trajectory), among which, target recognition accuracy and target missed detection rate are indicators to measure the performance of RSS to identify traffic participants , its calculation method can be expressed as:

FIG. 4 is a schematic diagram of the sample space for detection accuracy calculation. The detection accuracy is an indicator to measure the ability of RSS to track the state of any traffic participant. The sample space for statistical evaluation is the target state data output by the RSS to be tested and the true value of the target state stored by the RS (completed time alignment). The two sets of data can represent The polyline structure presented for Figure 4:

1,2,…,n代表离线时间序列，构成轨迹的点可以代表任一状态参数(目标大小、位置、速度、航向角、加速度、轨迹)，进而通过轨迹相似性分析得出待测RSS输出的状态信息与RS状态真值的距离大小，以距离度量表征各个状态参数的检测精度。轨迹相似性度量方法大致可分为基于点的距离、基于形状的距离、以及基于分段的距离三大类，可综合考虑轨迹长度、噪点敏感度、计算复杂度等因素弹性选择轨迹距离的度量方式。in,

1,2,…,n represents the offline time series, the points constituting the trajectory can represent any state parameter (target size, position, speed, heading angle, acceleration, trajectory), and then the RSS output to be tested is obtained through trajectory similarity analysis The distance between the state information of , and the true value of the RS state is represented by the distance metric to characterize the detection accuracy of each state parameter. The trajectory similarity measurement methods can be roughly divided into three categories: point-based distance, shape-based distance, and segment-based distance. The measurement of trajectory distance can be selected flexibly considering factors such as trajectory length, noise sensitivity, and computational complexity. Way.

The roadside perception system evaluation method based on the truth value system proposed by the present invention is based on the high-performance truth value system deployed synchronously with the RSS to be tested, and realizes the refined data collection of the real complex traffic environment, and then passes offline post-processing. The objective truth data is generated, and finally the truth data can be used to realize the test and evaluation of the perceptual performance of the RSS to be tested.

It should also be noted that the terms "comprising", "comprising" or any other variation thereof are intended to encompass a non-exclusive inclusion such that a process, method, article or device comprising a series of elements includes not only those elements, but also Other elements not expressly listed, or which are inherent to such a process, method, article of manufacture, or apparatus are also included.

The above descriptions are merely examples of the present application, and are not intended to limit the present application. Various modifications and variations of this application are possible for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims (10)

1. a true value-based roadside perception system evaluation method, is characterized in that, comprises the following steps: Establish a true value sensing device group, and collect roadside sensing data synchronously with the sensing equipment of the roadside sensing system RSS to be tested in the selected test time interval; Process the original data returned by the truth-value sensing device group, complete target type identification and target trajectory identification, and complete sensing data labeling; Generate ground truth based on labeled data, ground truth data includes target type, position, speed, acceleration, trajectory of traffic participants; In the selected test time interval, the structured sensing data output by the RSS to be tested and the true value data are compared, and the statistical evaluation result of the sensing performance is output. 2. The method of claim 1, wherein Ground-truth perception device group, including high-beam lidar, high-definition cameras, and millimeter-wave radar. 3. The method of claim 1, wherein The processing of the original data returned by the truth-sensing device group to complete target type identification and target trajectory identification further includes: Based on the point cloud data returned by the high-speed laser radar, the target type identification and target trajectory tracking of traffic participants are completed. The data collected by the camera is used to correct the target type, and the millimeter wave radar data is used to correct the target trajectory. 4. The method of claim 1, wherein The processing of the original data returned by the truth-sensing device group further includes: Data cleaning is performed on the original data returned by the truth-value sensing device group, and data from different sensing devices is time-aligned. 5. The method according to any one of claims 1 to 4, wherein, Described comparing the structured sensory data output by the RSS to be tested and the true value data, further comprising, calculating: Target recognition accuracy = the number of targets or events correctly detected by the RSS to be tested/the number of targets or events in the true value data. 6. The method according to any one of claims 1 to 4, wherein, Described comparing the structured sensory data output by the RSS to be tested and the true value data, further comprising, calculating: Target missed detection rate=1-number of targets or events detected by the RSS to be detected/number of targets or events in the true value data. 7. The method according to any one of claims 1 to 4, wherein, Described comparing the structured sensory data output by the RSS to be tested and the true value data, further comprising, calculating: The distance between the discrete time series of state parameters in the structured sensory data output by RSS and the discrete time series of state parameters in the ground-truth data. 8. The method of claim 7, wherein The state parameters include at least one of the following: target size, position, speed, heading angle, acceleration, and trajectory. 9 . A truth value system for roadside perception system evaluation, implementing the method according to any one of claims 1 to 8 , characterized in that it comprises a truth value perception device group and a server; The truth value perception device group at least includes high-line-speed lidar, high-definition camera, and millimeter-wave radar; The server includes a data acquisition module, an intelligent processing module, a true value storage module, and an RSS evaluation module; the data acquisition module, configured to fuse the image, video and point cloud data output by the truth perception device group; The intelligent processing module is used to process the original data returned by the truth-value sensing device group, complete target type identification and target trajectory identification, and complete sensing data labeling; The truth value storage module is used for backing up the truth value data; The RSS evaluation module is used to compare the structured perception data output by the RSS to be measured and the true value data, and output the statistical evaluation result of the perception performance. 10. The truth value system for roadside perception system evaluation according to claim 9, characterized in that, The sensing device group and the RSS to be tested are multiplexed with pole frame resources.

Images