Simulation-Based Verification

Simulation-based verification is the transfer of systems from the real environment to the simulation environment for secure and low cost verification of autonomous systems. Given spesific inputs, it allows the execution of meaningful scenarios designed to check that the system complies with the standards. It enables a test environment to be created to functionally validate the system.

Method Purpose Short description of the method's purpose and main benefits

Method description A detailed description of the method

In industrial operations, the failure of an autonomous system can cause a significant hazard to these systems. Verifying the security of autonomous systems is critical to prevent such accidents and prevent possible loss of life and property. Currently, most systems are tested through field testing, which is costly, time consuming, limited in repeatable scenarios, and risky in case of unacceptable behavior. To mitigate these issues, their software can be pre-validated using simulation-based testing.

Simulation-based verification allows verification in a virtual environment similar to the real environment in which systems will operate, that is, in operating simulations of these systems. In the simulation environment, the behavior of an autonomous system can be monitored and the security of the systems can be verified without causing any environmental hazards [SBV1].

Today, detailed computer simulations, sometimes called “digital twins”, are created for virtual prototypes of the systems under investigation [SBV2]. These simulations often have detailed physics models of the system and its environment. The term simulation-based verification is based on these simulations and is useful for examining critical autonomous system tests. The main advantage of Simulation Verification compared to traditional V&V techniques is cost and security [SBV3]. Using simulators for V&V processes can eliminate dangerous situations and leave only extreme states for physical testing. In addition, simulation models and environments can provide exactly the same conditions for different test cases. Flight simulation software (FLIGHTLAB, X-Plane, etc.) can be cited as sample applications for simulation-based verification [SBV4]. These applications contain physical models and control systems in a virtual 3D environment. This way development and validation can be done relatively cheaply and quickly.

Method Strengths A listof the strengths of the method

Simulation-based verification can be scalable if models, scenarios and simulations are created properly. In simulation-based testing, verification activities can be done for different robots without changing other models or tools. Also, system testing can be done without producing any physical item and adding risk to the environment.
The fault injection function for system and monitoring in simulation environment can provide powerful monitoring.

Method Limitations The limitations of the method

Most of the applications do not consider physical models in verification, as simulation-based applications mostly run-on hierarchical models. This narrows availability of both academic and industrial resources in development.
Simulation tools for simulation-based fault injection can require much computational power and limit real-time applications.

Method References Bibliography references to papers about the method.

[SBV1] Timperley, C. S., Afzal, A., Katz, D. S., Hernandez, J. M., & Le Goues, C. (2018, April). Crashing simulated planes is cheap: Can simulation detect robotics bugs early? In 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) (pp. 331-342). IEEE. doi: 10.1109/ICST.2018.00040
[SBV2] Xiao, A., & Bryden, K. M. (2004, January). Virtual Engineering: A vision of the next-generation product realization using virtual reality technologies. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference (Vol. 46970, pp. 461-469). doi: 10.1115/DETC2004-57698
[SBV3] Robert, C., Guiochet, J., & Waeselynck, H. (2020,November). Testing a non-deterministic robot in simulation-How many repeated runs?. In 2020 Fourth IEEE International Conference on Robotic Computing (IRC) (pp. 263-270). IEEE. doi: 10.1109/IRC.2020.00048
[SBV4] Bonner, M.C.; Taylor, R.M.; Miller, C.A. Tasking Interface Manager: Affording Pilot Control of Adaptive Automation and Aiding. In Contemporary Ergonomics 2000; CRC Press: Boca Raton, FL, USA, 2004; pp. 70–74.

Method Dimensions

Evaluation Environment Type Type of environment where the method is applied. It can be more than one type of environment. (Dimension 1)

Evaluation Type Type of evaluation performed with that method. If a method is hybrid, more than one type can be selected. (Dimension 2)

Type of Component Under Evaluation Type of components that can be evaluated with that method. It can be more than one type. (Dimension 3)

Evaluation Stage The evaluation stage where the method is used. (Dimension 5)

Purpose of the Component Under Evaluation The type of purpose that can be evaluated by a method. It can be more than one type. (Dimension 6)

Type of Requirement Under Evaluation The type of requirements that a method is able to evaluate. It can be more than one type. (Dimension 7)

Evaluation Performance Indicator The type of evaluation criteria or KPI that a method is able to improve. It can be more than one type. (Dimension 8)

Relations

Related Methods A method could be related to other methods.

Tools A method can use zero or more tools.

Part Method A method (when it is a combination: Combination of method artefact that is a specialization of a V&V Method) could be composed of a set of V&V Methods.

Test Case or Verification and Validation activity Test Case or V&V activity performed in a Use Case. We will link to the method to the test cases where the method is used.

Standards A method can be linked with standards.

Context

Workflow

Contents

There are currently no items in this folder.