The “Simulation based V&V of Computer Vision System” method is a combined method that enables the validation of a CV system focused on objects detection in a semi-automated way. It is based on synthetic images generated using a simulator. The activities that compose the workflow of the method are described below:

Simulated scenario design: The first activity comprises the design of the scenarios using the simulator. At this point, by means of a simulation environment the objects that CV system should detect are placed in different scenarios.

Scenario selection and execution: This activity will comprise the selection of a previously designed scenario that will be executed in the simulator. Setting the configuration parameters will enable to carry out a simulation in different conditions.

Scenario frames recording: During a simulation execution, the Dataset Generator for Validation (DaGe4V) tool records the frames and related metadata from the simulator. As a result, the validation datasets for testing the system under test are generated.

Ground truth template generation: This activity comprises semi-automatically labelling the validation datasets, using external tools, such as DarkLabel, to get the ground truth information with accurate information on objects location in each frame.

Accuracy Metric Calculation: In this activity, Validation Test Result Analyser (VaTRA) tool will analyse the results obtained in several tests carried out for the CV system. It will compare the results get on each test with the ground truth template, providing several metrics and identifying potential safety violations due to incorrect object detection during the tests, generating the V&V Results artifact.