Scenario based V&V automation using simulator
Machine learning, in particular deep learning, is a critical enabling technology for many of the highly automated applications today. Typical examples include Intelligent Transport Systems (ITS) where ML solutions are used to extract a digital representation of the traffic context from the highly dimensional sensor inputs.
The major bulk of system-level testing of autonomous features in the automotive industry is carried out through on-road testing or using naturalistic field operational tests. These activities, however, are expensive, dangerous, and ineffective.
A feasible and efficient alternative is to conduct system-level testing through computer simulations that can capture the entire self-driving vehicle and its operational environment using effective and high-fidelity physics-based simulators. There is a growing number of public-domain and commercial simulators that have been developed over the past few years to support realistic simulation of self-driving systems, e.g., TASS/Siemens PreScan, ESI Pro-SiVIC, CARLA, LGSVL, SUMO, AirSim, and BeamNG. Simulators will play an important role in the future of automotive V&V, as simulation is recognized as one of the main techniques in ISO/PAS 21448.
Still there is an open research area to be explored to develop tools and methods that can be applicable for V&V of ML-based systems to be used in safety critical applications. Within this project, we propose a customization of safety assurance process of ML in autonomous system [VVM10] to be used in conjunction with a set of simulation and supporting tools. The process diagram is illustrated in Figure 2.13.
As the possible input space when testing automotive systems is practically infinite, attempts to design test cases for comprehensive testing over the space of all possible simulation scenarios are futile. Hence,
Search-based software testing has been advocated as an effective and efficient strategy to generate test scenarios in simulators [VVM5, VVM6]. Another line of research proposes techniques to generate test oracles, i.e., mechanisms for determining whether a test case has passed or failed [VVM7]. Related to the oracle problem, several authors proposed using metamorphic testing of ML-based perception systems [VVM8, VVM9], i.e., executing transformed test cases while expecting the same output. Such transformations are suitable to test in simulated environments, e.g., applying filters on camera input or modifying images using generative adversarial networks.
- Cost efficient: Using simulations for scenario’s data generation for V&V of ML systems reduces the cost of using a real-world data capture.
- Time: Having an immediate response from a simulator shortens the software development cycle, i.e., it enables quicker feedback.
- Safety: Currently, testing many vehicle collisions and accident scenarios are done using safe dedicated test and assessment protocols, however, testing an incomplete system always exposes the testers to unpredictable dangers. Using simulators, the risks of test driving of an autonomous vehicle in urban areas will be substantially reduced.
- Edge cases: Many low probability safety critical situations and hazards that would not be encountered on a test track can be generated in simulated environments.
Because of the opaque nature of ML systems and informal formulation of safety requirements, this method only reduces the gap between testing by simulation and testing in real-world, i.e., the method does not close the gap entirely.
- [VVM5] Abdessalem, R.B., Nejati, S., Briand, L.C. and Stifter, T., 2018, May. Testing vision
- based control systems using learnable evolutionary algorithms. In 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE) (pp. 1016-1026). IEEE.
- [VVM6] Gambi, A., Mueller, M. and Fraser, G., 2019, July. Automatically testing self-driving cars with search-based procedural content generation. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (pp. 318-328).
- [VVM7] Stocco, A., Weiss, M., Calzana, M. and Tonella, P., 2020, June. Misbehaviour prediction for autonomous driving systems. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering (pp. 359-371).
- [VVM8] Tian, Y., Pei, K., Jana, S. and Ray, B., 2018, May. DeepTest: Automated testing of deep neural-network-driven autonomous cars. In Proceedings of the 40th international conference on software engineering (pp. 303-314).
- [VVM9] Zhang, M., Zhang, Y., Zhang, L., Liu, C., & Khurshid, S. (2018). DeepRoad: Gan-based metamorphic autonomous driving system testing. arXiv preprint arXiv:1802.02295. Improvements
- [VVM10] Hawkins, R.; Paterson, C.; Picardi, C.; Jia, Y.; Calinescu, R.; Habli, I. Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS). arXiv:2102.01564 [cs] 2021.