Scenario-based Safety Evaluation of Highly Automated Vehicles

Abstract

A highly automated vehicle (HAV) is a safety-critical system. Therefore, a verification and validation (V&V) process that rigorously evaluates the safety of HAVs is necessary before their mass deployment on public roads. This dissertation will present the methodology and implementation procedure of a scenario-based evaluation framework for HAVs.

First, an evaluation framework for reactive scenarios is proposed, where the risk level of test cases could be objectively categorized in advance. The pedestrian crossing scenario is used as a case study. We first build a statistical model for the pedestrian scenario based on naturalistic data. Next, reachability analysis is applied to partition the scenario testing space into different risk level sets, which are then combined with importance sampling to generate test cases efficiently and realistically. The proposed method achieves unbiased crash rate estimation in an accelerated fashion, while all the test cases are feasible and have controlled risk levels.

Then, a novel evaluation framework for interactive scenarios is proposed, including highway merging and roundabout entering. Instead of assuming that the primary other vehicle (POV) takes predetermined maneuvers, we model the POVs as game-theoretic agents. To capture a wide variety of interactions between the POV and the vehicle under test (VUT), we use level-k game theory and the social value orientation (SVO) concept to model the POV, and generate a diverse library of POV policies using reinforcement learning. On the other hand, an adaptive test case generation method is developed based on adaptive sampling, stochastic optimization and upper confidence bound (UCB) algorithm to generate customized challenging cases for the VUT from the testing space. In simulations, the proposed POV library captures a wide range of interactive patterns for both highway merge and roundabout entering scenarios. The proposed test case generation method covers the failure modes of a black-box VUT more effectively compared to other approaches.

In addition, the problem of generating corner cases for interactive scenarios systematically is considered. an ambiguity-guided adversarial planning algorithm is developed to generate confusing behaviors for the primary other road user (PORU) in interactive scenarios. The PORU is modeled as a cost-minimizing agent with hierarchical intentions. The adversarial PORU plans actions to confuse the HAVs by maximizing the ambiguity with respect to its intentions, while also taking nominal behavior planning goals into consideration. Two interactive scenarios are studied: highway merging and pedestrian crossing. A corner case testing scheme is designed and implemented for both scenarios to evaluate the performance of different HAVs comprehensively and objectively.

Finally, the procedure of implementing behavior competence testing in the real world is presented for reactive scenarios. Speed planning algorithms for the POV are developed to synchronize its motion with the VUT. A speed tracking controller for experimental vehicles is designed based on the preview control algorithm. It is demonstrated that tests can be executed for multiple reactive scenarios with real vehicles on the Mcity test track in an accurate, repeatable and automated fashion. In addition, a digital twin of Mcity in the CARLA simulator is created, and the same testing capability in the simulation is demonstrated.