Simulation-based testing has been envisioned as an efficient means to test Automated Systems. Employing simulation models permits 1) the execution of more and larger test cases, 2) selection of critical scenarios, 3) specification of test oracles for the automated validation of the system, or 4) replication of safety-critical functions of a real system where it is expensive to execute test cases [TOS2]. However, although the use of simulation methods provides several advantages, testing Automated Systems is still expensive and time-consuming. Simulation models of Automated System could be very complex, and executing the simulations becomes computationally expensive, which often make it infeasible to execute all the test cases. 

For this reason, test optimization plays a crucial role when testing automated systems. The objective of test optimization is to cost-effectively test a system, i.e., reduce the cost of testing a system while the overall test quality is maintained.  

Test optimization could include test case selection, test case minimization, test case prioritization, etc.  

• Test case selection focuses on selecting a set of test cases from the test suite that tests a specific system version.  
• Test minimization aims to eliminate redundant test cases from the existing test suite in order to reduce cost (i.e., reduce the test execution time).  
• Test case prioritization techniques schedule test cases for execution in an order that attempts to increase their effectiveness at meeting some performance goal. 

Test optimization could be also obtained using automatic test case generation: the process of generating test suites for a particular system.  

Since these approaches are typically non-trivial, often search-based algorithms are employed. 

Test case generation: [TOS4] and [TOS2] present an approach for the generation of the optimal set of reactive test cases for simulation-based testing of Cyber-Physical Systems using search-based algorithms. The optimization is made taking into account the following cost-effectiveness measures: requirements coverage, test case similarity and test execution time.  

Test case selection: [TOS3] and [TOS5] present a cost-effective approach for test case selection that relies on black-box data related to inputs and outputs of the system. 

Test case prioritization: [TOS1] presents a search-based approach that aims to cost-effectively optimize the test process by prioritizing the test cases that are executed at different test levels (i.e., MiL, SiL and HiL). 

These approaches have recently been improved through the use of Machine Learning techniques, as in the case of optimising test case selection based on simulation models for self-driving cars [TOS6]. 

Method artefacts: 

• Inputs: 
  • Simulation-based model 
  • Test Case Inputs 

• Outputs: 
  • Generated Test Suite 
  • Selected Test Suite 
  • Prioritized Test Suite