A Framework for Test & Evaluation of Autonomous Systems Along the Virtuality-Reality Spectrum

Abstract

Test & Evaluation of autonomous vehicles presents a challenge as the vehicles may have emergent behavior and it is frequently difficult to ascertain the reason for software decisions. Current Test & Evaluation approaches for autonomous systems place the vehicles in various operating scenarios to observe their behavior. However, this introduces dependencies between design and development lifecycle of the autonomous software and physical vehicle hardware. Simulation-based testing can alleviate the necessity to have physical hardware; however, it can be costly when transitioning the autonomous software to and from a simulation testing environment. The objective of this thesis is to develop a reusable framework for testing autonomous software such that testing can be conducted at various levels of mixed reality provided the framework components are sufficient to support data required by the autonomous software. The paper describes the design of the software framework and explores its application through use cases