A Machine Learning Environment for Evaluating Autonomous Driving Software

Autonomous vehicles need safe development and testing environments. Many traffic scenarios are such that they cannot be tested in real life. We see photorealistic game technology based hybrid simulation as a viable tool for developing AI software for autonomous driving. We present a machine learning environment for detecting autonomous vehicle corner case behavior. Our system is based connecting Carla simulation software to Tensorflow machine learning framework and custom AI client software. The AI client software receives data from the model world via virtual sensors and transforms data into information using multiple CNN models. Multiple instances of the AI software control vehicles in the virtual world. We use our system to drive virtual vehicles in the model world while monitoring the state assumed by the vehicle AIs to the ground truth state derived from the simulation model. Our system can search for corner cases where the AI models are unable to correctly understand the situation. Corner case search is computationally very demanding and slow to perform. In our paper, we present the overall system architecture and compare different system configurations. We present performance measurements from real setups, and outline the main parameters affecting the system performance. Our paper contributes to the fields of autonomous driving and hybrid simulation research by presenting a AI system evaluation methodology and the related performance measurements.