Simulation-Based V&V of Computer Vision System

This method comprises the generation of image datasets for test cases to validate a computer vision system (which is the SUT) and the analysis of the results obtained in the performed tests.

Method Purpose Short description of the method's purpose and main benefits

Method description A detailed description of the method

This method comprises the generation of image datasets for test cases to validate a computer vision system (which is the SUT) and the analysis of the results obtained in the performed tests.

Generation of image datasets in field is often a very costly process (specially in domains such as railway). In order to speed up this process image datasets for system validation are obtained using a simulator, where scenarios are designed and run under different light and weather conditions. This enables to test system’s behaviour under different conditions, increasing the test coverage for the SUT.

The results obtained from the execution of the tests are analysed to obtain accuracy metrics, to identify potential safety violations and to evaluate the behaviour of the system in the different operation conditions.

Method Strengths A listof the strengths of the method

Cost efficient: The use of synthetic image datasets for testing reduces the cost of the validation process, as a set of validation tests can be performed in the laboratory without the need for costly field recordings.
Time: Obtaining image datasets using a simulation tool also reduces the time required for testing dataset generation. Time reduction on dataset generation can allow the execution of a higher number of tests.
Safety: The use of synthetic images that simulates different light and weather conditions enables to test computer vision system’s behaviour under distinct conditions. Obtaining images from recordings in the field for the same circumstances is a costly and time-consuming task that is not always possible to carry out. To carry out a larger number of tests, will allow to detect potential safety violations earlier.
Performance measures: Automatic performance measures provided by this method will enable to detect the weaknesses of the computer vision system to detect the signs and signals under specific working conditions.

Method Limitations The limitations of the method

The difference between synthetic images from simulators and images in the real world can lead to different results in the validation of the system in the laboratory and in the field.

Method References Bibliography references to papers about the method.

Berenguel-Baeta, Bruno; Bermudez-Cameo, Jesus; Guerrero, Jose J. (2020). "OmniSCV: An Omnidirectional Synthetic Image Generator for Computer Vision" Sensors 20, no. 7: 2066.
Davis, J., Goadrich, M. The relationship between Precision-Recall and ROC curves. (2006). ACM International Conference Proceeding Series, 148, pp. 233-240.

Method Dimensions

Evaluation Environment Type Type of environment where the method is applied. It can be more than one type of environment. (Dimension 1)

Evaluation Type Type of evaluation performed with that method. If a method is hybrid, more than one type can be selected. (Dimension 2)

Type of Component Under Evaluation Type of components that can be evaluated with that method. It can be more than one type. (Dimension 3)

Evaluation Stage The evaluation stage where the method is used. (Dimension 5)

Purpose of the Component Under Evaluation The type of purpose that can be evaluated by a method. It can be more than one type. (Dimension 6)

Type of Requirement Under Evaluation The type of requirements that a method is able to evaluate. It can be more than one type. (Dimension 7)

Evaluation Performance Indicator The type of evaluation criteria or KPI that a method is able to improve. It can be more than one type. (Dimension 8)

Relations

Related Methods A method could be related to other methods.

Tools A method can use zero or more tools.

Part Method A method (when it is a combination: Combination of method artefact that is a specialization of a V&V Method) could be composed of a set of V&V Methods.

Test Case or Verification and Validation activity Test Case or V&V activity performed in a Use Case. We will link to the method to the test cases where the method is used.

Standards A method can be linked with standards.

Context

Workflow

Contents

There are currently no items in this folder.