Simulation-Based Testing of Automated Driving Systems

Automated Driving Systems (ADS) require extensive safety testing before receiving a road permit. To gain public trust, ADSs must be as safe as a Human Driven Vehicle (HDV) or even safer. Simulation-based safety testing is a cost-effective way to check the safety of ADS. My goal is to compare the safety behavior of ADS with HDV via simulation and to develop a process of selecting testing scenarios that could be useful to build trust and reliability in simulations. Additionally, I aim to translate the performance advantages and disadvantages observed in simulated ADS behavior into real-world safety-critical traffic situations.

My Ph.D. project aims to evaluate ADS safety through simulationbased tests and compare its advantages and disadvantages against a typical human-driven vehicle (HDV).The outcomes from my research could guide transportation regulators in addressing any identified ADS disadvantages.For example, in some cases, making infrastructure and traffic rule adjustments might be more feasible than enhancing ADS performance.An in-depth analysis of available quantitative data can shed light on potential strategies for neutralizing identified ADS disadvantages.
Simulation-based testing provides a controlled, cost-efficient method for evaluating ADS safety compared to on-road trials.However, there are infinitely many potential scenarios, and testing every scenario is unfeasible [5,7].Therefore, defining a process for scenario prioritization and selecting scenarios is essential, which is not covered in the existing literature.Additionally, deriving precise conclusions from simulation-based testing can be complex as it is essential to ensure that simulated elements closely resemble their real-world counterparts.Measuring ADS' disadvantages against a typical HDV poses another challenge, given the limited availability of pre-crash HDV data and the lack of knowledge about the precise behavior of a well-driven HDV in both crash and non-crash situations.

RESEARCH GOALS
The research goals of my Ph.D. thesis are as follows: • RG1 [Scenarios]: To develop a process for prioritizing and selecting scenarios for ADS safety testing.• RG2 [Simulation Environment]: To create a simulation environment and implement the selected scenarios within a simulator.

• RG3 [Experiments]:
To execute the experiments in the simulator and assess the behavior and performance of an ADS compared to an HDV in scenarios.

RESEARCH APPROACH & CONTRIBUTIONS
The research approach consists of five main steps, as shown in Figure 1: (i) prioritizing and selecting test scenarios, (ii) creating simulation models capturing the behavior of an ADS and an HDV, (iii) running simulations against the selected test scenarios, (iv) defining criteria for performance evaluation (v) and evaluating the performance of both ADS and HDV.
To initiate my research, I conducted a comprehensive literature survey to understand the current state-of-the-art from research and industry experts regarding the safety testing methods employed for ADS [10].This review aims to provide an overview of (i) types of ADS, (ii) safety features, (iii) testing methods, and (iv) tools and datasets utilized in ADS safety testing.I outline the planned contributions to fulfill the mentioned research goals.Contribution1 -aiming to achieve RG1: Since ADS encounters infinite scenarios, it is crucial to prioritize and choose which ones to focus on.Therefore, I defined a process to prioritize and select scenarios using pre-existing textual scenario catalogs and real-world autonomous car video data 4 (in progress).I started with the pre-existing scenario catalog from a reputable organization.A set of scenarios is selected that aligns with the specific Operational Design Domain (ODD) of ego vehicle 5 .These scenarios are grouped based on the similar critical action of the ego vehicle and target object (vehicles, pedestrians, cyclists, etc.) and then prioritized using accident statistics.Scenarios that are duplicates or fall outside the limitations of simulators are excluded.The remaining scenarios are scored based on the occurrence frequency of elements like actors, driving maneuvers, weather, and lighting conditions from accident datasets; the highest-scoring scenario from each prioritized group is chosen for simulation.Figure 2 shows the proposed process for prioritizing ADS safety testing scenarios.Additionally, I aim to identify critical scenarios where safety drivers intervene in autonomous mode (particularly when uncertain of the Bolt car's response), using the Autonomous Driving Lab's dataset from the University of Tartu, Estonia 4 .These contributions aim to refine scenario selection for enhanced ADS safety assessment.
Contribution 2 -aiming to achieve RG2: I will use CARLA 6 simulator and ScenarioRunner 7 for the implementation of selected scenarios, as it can depict complex scenarios and the movements of entities such as vehicles and pedestrians.Furthermore, I will use 4 https://adl.cs.ut.ee/This dataset comprises 20 service bags, each containing a video recording of a single loop where a Bolt car operates in autonomous mode along a designated route 5 The ADS under test 6 https://carla.org/ 7https://carla-scenariorunner.readthedocs.io/en/latest/Python API8 for the simulation of ADS as it offers diverse vehicle control models, allowing manipulation of both the environment and the vehicles themselves.The CARLA ROS bridge 9 makes it easy to link CARLA with a third-party ROS-based control system for the ego vehicle.I will use the ROS bridge to connect with Autoware Mini 10 .This contribution will provide a controlled simulation setting and findings about how ADS will behave/respond to specific scenarios that could further be compared to HDV.This ultimately enhances understanding of where ADS excels and where it may face challenges in diverse real-world driving scenarios.
Contribution 3 -aiming to achieve RG3: I will define the criteria and metrics for evaluating and assessing ADS and HDV simulated behavior and performance in Contribution 2, specifically on safety.

FIRST RESULTS (CONTRIBUTION 1)
I applied the proposed process for prioritizing and selecting scenarios to two existing scenario catalogs for ADS safety testing.The first catalog is published by the Land Transport Authority of Singapore 11 and contains 67 real-world traffic scenarios.The second catalog is published by the US Department of Transportation 12 and contains 44 pre-crash scenarios precisely capturing the situations and conditions leading up to an accident or collision.The total number of scenarios in both selected catalogs is 111.Twenty-one scenarios are excluded based on the ODD specific to ADS.The remaining scenarios are categorized into fifteen distinct groups.Nine duplicated scenarios are removed, and eight scenario groups are prioritized based on crash statistics retrieved from real-world data.Considering CARLA simulators' limitations, six scenario groups from the prioritized containing fifty-one scenarios for testing ADS in the CARLA simulator.

RELATED WORK
Several studies performed the safety testing of ADS using openroad [3,4,8,15] test beds [1,13] simulators [2,9,11,14,17] and [6,12] are related to scenario prioritization and selection.However, we only include the studies that perform ADS safety testing using simulation due to space limitations.Son et al. [16] introduced a co-simulation platform that integrates vehicle dynamics, sensors, and traffic modeling, exemplified through use cases e.g., adaptive cruise control.Matthew et al. [14] proposed a simulation framework that employs an adaptive sampling method to test an entire ADS.Jha et al. [9] presented a fault injection tool that systematically injects faults into the hardware and software of an ADS to evaluate safety and reliability.Ben et al. [2] presented an approach to test ADS via Simulink using multi-objective search and surrogate models to identify critical test cases regarding an ADS behavior.My approach to simulation-based ADS safety testing differs from current methods as it takes a black-box perspective when analyzing the ADS's behavior.The primary focus of ADS is pinpointing ADS behavior that deviates from HDVs, considering both positive and negative distinctions.My Ph.D. project is expected to be completed by January 2026.
217 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

Figure 2 :
Figure 2: Overview of proposed process for prioritization and selection of scenarios for safety testing of ADS