Optimize Fault Injection Experiments Using Model-Based Mutation Testing

Combine fault injection with model-based mutation testing

Method Purpose Short description of the method's purpose and main benefits

Method description A detailed description of the method

For fault injection experiments, system stimuli are needed to operate the system under test while faults are injected. This is needed to verify that a fault-tolerant system is gracefully handling a fault in all (relevant) situations. While the number of possible injected faults grows linearly with the size of the system, the number of possible input stimuli combinations, system states, and paths over the system states (i.e., test sequences) tends to grow exponentially with the size of the system.

By naively combining all injected faults with all tests, the time to run fault injection experiments would also grow exponentially with the size of the system. Therefore, ways to limit the size of the test suite, ensure the quality of the test suite, and select relevant test sequences per injected fault are crucial.

The combination with model-based mutation testing can support this by generating tests that:

cover the system behaviour as completely as possible,
do so with a low number of test sequences,
ensure to propagate a problem caused by the injected fault long enough to become observable at the test interface.

If the system under evaluation can be instrumented to show which test sequences exercise the part of the system where the fault is injected, this can generally be used to limit the tests that need to be run per an injected fault.

In this case, if model-based testing is used to create the test sequences, the tests can be used to establish correlations between model elements and parts of the system where faults are injected. By only mutating these model elements during model-based mutation testing, a specific, small test suite can be generated to be run for all fault injections into the related system part. Since the tests are optimized for problems in the specific part of the system, the overall execution time of the fault injection experiments can be substantially lower than when using only selected tests of a standard test suite. Tests for the complete system with regression testing as the test goal would try to achieve as much coverage as possible with a test, independently of whether the exercised system parts are of interest for the fault injection or not. Tests for the complete system with debug testing as the test goal would try to reach each coverage goal with as little steps as possible, but running all relevant tests out of such a test suite would re-run shared prefixes of the test sequences numerous times.

Optimize Fault Injection Experiments Using Model-Based Mutation Testing.png

Method Strengths A listof the strengths of the method

Fault injection uses optimized stimuli/test sequences
- Effective: situations where the respective fault can lead to an error and failure are contained in the test sequences
- Efficient: The test sequences avoid unnecessarily running experiments that cannot be affected by a certain fault.
In complex automated systems, time in physical test environments is often much more expensive than pure computation time. Combining the methods allows one to shift efforts and time from physical fault injection experiments towards preparative test-case generation, without falling back to pure simulation solutions.

Method Limitations The limitations of the method

Both fault injection and test case generation are computationally expensive methods. Even when combining them, complex systems might not be analysable in depth.

Method References Bibliography references to papers about the method.

Aichernig, H. Brandl, E. Jöbstl, W. Krenn, R. Schlick, and S. Tiran, “MoMuT::UML model-based mutation testing for UML,” in Software testing, verification and validation (icst), 2015 ieee 8th international conference on, 2015, pp. 1-8
Krenn, R. Schlick, and B. K. Aichernig, “Mapping UML to labeled transition systems for test-case generation: a translation via object-oriented action systems,” in Proceedings of the 8th international conference on formal methods for components and objects, Berlin, Heidelberg, 2010, p. 186–207
V. Carreira, D. Costa, and S. J. G, "Fault Injection Spot-Checks Computer System Dependability," IEEE Spectrum, pp. 50–55, 1999.

Method Dimensions

Evaluation Environment Type Type of environment where the method is applied. It can be more than one type of environment. (Dimension 1)

Evaluation Type Type of evaluation performed with that method. If a method is hybrid, more than one type can be selected. (Dimension 2)

Type of Component Under Evaluation Type of components that can be evaluated with that method. It can be more than one type. (Dimension 3)

Evaluation Stage The evaluation stage where the method is used. (Dimension 5)

Purpose of the Component Under Evaluation The type of purpose that can be evaluated by a method. It can be more than one type. (Dimension 6)

Type of Requirement Under Evaluation The type of requirements that a method is able to evaluate. It can be more than one type. (Dimension 7)

Evaluation Performance Indicator The type of evaluation criteria or KPI that a method is able to improve. It can be more than one type. (Dimension 8)

Relations

Related Methods A method could be related to other methods.

Tools A method can use zero or more tools.

Part Method A method (when it is a combination: Combination of method artefact that is a specialization of a V&V Method) could be composed of a set of V&V Methods.

Test Case or Verification and Validation activity Test Case or V&V activity performed in a Use Case. We will link to the method to the test cases where the method is used.

Standards A method can be linked with standards.

Context

Workflow

Contents

There are currently no items in this folder.