Simulation-Based Fault Injection at System-level
System-level simulation is comprised of hardware and software models of a cyber physical system (CPS). The fault injection could be performed on different abstraction layers such as logical, functional, hardware, software, or system level etc [SFI1]. In this case, we focus on the simulation-based fault injection at system level.
In our method, simulation-based fault injection at system level, the faults are injected on wireless communication systems modelled in Veins and Veins-INET simulators together with SUMO traffic simulator [SFI3] [SFI4]. The simulation-based fault injection at system level using simulators may be used for safety testing of automated systems such as autonomous vehicles [SFI2].
Simulation-based fault injection is a V&V method where faults are injected into system model or system software in a simulation environment. This type of fault injection is applicable when:
- A system model is available at early stages of system development.
- A software is available to run in a simulation environment. This type of testing is called Software in the Loop (SiL) testing and the software under evaluation is called SiL component*1 [SFI5]
- The hardware is not available.
- The software needs to be verified and validated in a simulation environment.
Simulation-based fault injection is useful for both development and deployment stages to identify and resolve different types of vulnerabilities relevant for each stage.
*1 A SiL component is an executable code written for a specific system, adjusted to run only in a simulation environment for software testing. This type of testing is useful especially when the hardware is either not existing or when it is in the development phase or when the verification results are required in short span of time. The latter could be facilitated by parallel execution of the tests in a cluster. Hardware requirements are taken away (e.g., end-to-end protection) from the SiL component so that it can run in a completely simulated or model-based environment. Note that the SiL testing is complimented by hardware in the loop (HiL) testing, when the hardware is available, in order to also evaluate the system when the software resides in the intended hardware, such as a particular mechatronic system.
The simulation-based fault injection at the system level can be useful for:
- End-to-end resilience assessment of a complete system specially in edge case scenarios*1
- Introducing faults in different parts of a system such as sensors, functions, and actuators to evaluate that specific part or even a complete system behaviour.
- Introducing faults in automated systems, which may be hard to do through other verification methods.
- It is possible to introduce multiple faults by using this method.
- Measurements from simulation-based faults injection may be useful in later V&V activities.
*1The edge cases are realised by injecting attacks in the system to create a test scenario which is otherwise rarely tested or testable in the real-world.
- The simulation-based fault injection at system level is limited to the injection of faults in simulations only, so it may not be possible to accurately evaluate the actual physical system.
- The use of simulation-based fault injection techniques for Machine Learning (ML) based systems has showed promising results in previous experiments [SFI2]. However, there is a need to further explore this test technique for ML or deep learning-based systems.
- [SFI1] M.-C. Hsueh, T.K. Tsai, and R.K. Iyer, “Fault Injection Techniques and Tools,” Computer, vol. 40, no. 4, pp. 75-82, Apr. 1997.
- [SFI2] S. Jha et al., “AVFI: Fault Injection for Autonomous Vehicles,” in Proc. 2018 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks Workshops (DSN-W), pp. 55–56.
- [SFI3] Michael Behrisch, Laura Bieker et al., “SUMO – Simulation of Urban Mobility, An Overview”, Institute of Transportation Systems, German Aerospace Center, Rutherfordstr. 2, 12489 Berlin, Germany.
- [SFI4] D. Eckhoff and C. Sommer, “A Multi-channel IEEE 1609.4 and 802.11p EDCA model for the Veins framework,” in Proceedings of 5th ACM/ICST international conference on simulation tools and techniques for communications, networks and systems: 5th ACM/ICST international workshop on OMNet++.(Desenzano, Italy, 19-23 March, 2012). OMNeT, 2012.[SFI5]https://www.add2.co.uk/applications/sil/#:~:text=The%20term%20'software%2Din%2D,prove%20or%20test%20the%20software.