UC2_Eval_VV_2

Coverage of test set – The implementation of a test suite associated with this use case enhances the comprehensiveness of the test set in a simulation-based environment. Initially, tests were performed manually and targeted wireless communication of the teleoperated system. However, as anticipated, both the total number of tests, test quality, and the proportion of systematically verified requirements have been significantly improved. The trust in the teleoperation system also increased by simulation-based testing as increased coverage also means increased trust in the analyzed system.
Coverage of test set – The implementation of a test suite associated with this use case enhances the comprehensiveness of the test set in a simulation-based environment. Initially, tests were performed manually and targeted wireless communication of the teleoperated system. However, as anticipated, both the total number of tests, test quality, and the proportion of systematically verified requirements have been significantly improved. The trust in the teleoperation system also increased by simulation-based testing as increased coverage also means increased trust in the analyzed system.
UC2
Evaluation Criteria for V&V Processes
Coverage of test set

The UC2_Eval_VV_2 metric is linked to the comprehensiveness of the test set within a simulated environment. The ability to conduct tests in a simulation environment offers several advantages. Firstly, it enables the execution of a larger number of test campaigns within a shorter time frame (i.e., the test coverage). Additionally, it provides the opportunity to test edge cases that would otherwise be challenging to perform in a real-world setting due to safety concerns for the surrounding environment.

In order to validate the simulation environment integrating the UC2 mockup with the Veins_inet simulation framework, we conducted tests and compared the baseline scenario (involving only the mockup) with the Veins_inet scenario (i.e., the mockup combined with Veins_inet). The purpose of this comparison was to assess the impact of incorporating Veins_inet on the performance and accuracy of the simulation. Our evaluation yielded the following results:

Baseline testing results

We have disconnected the transmission link between the ECU and the remote station for both video and command and verified if the safe stop (i.e., a fallback mechanism) is triggered. Following are the few DoS attack test results that initiate the disconnection at different times.

Disconnection testing results

We have tested the teleoperation system provided by the UC2 providers. The aim was to test and validate the safety functionality implemented that should be activated in case of fault or attacks on the transmission link.

To this end, we have conducted experiments where a delay fault and DoS attacks were injected into the transmission link. The following description is equally valid for the UC2_Eval_SCP_1 and UC2_EVAL_SCP_5. 

For in-depth analysis, we created and executed five test suites which are as follows,

1- DoS attack where both video and command channels are disconnected.

2- DoS attack where only the command channel is disconnected.

3- DoS attack where only the video channel is disconnected.

4- Delay attack where the communication on both video and command channels is delayed.

5- Delay attack where the communication on command channels is delayed. 

6- Delay attack where the communication video channel is delayed.

The purpose of these test suites is to verify and validate the two main safety mechanisms (aka fallback mechanism) which are implemented in the teleoperation system.  Following are the safety mechanisms that we used for result classification,

1- Vehicle Safe Stop: when the communication is delayed to a certain duration, the teleoperated vehicles must reduce their speed for ensuring safety.  The safe stop can be a slight safe stop which low deceleration rates or the safe stop can trigger a high deceleration rate (emergency braking) depending on the delay introduced.  the threshold for safe stop activation is when the communication delay is between 150ms and 1500ms.

2- Vehicle Disconnect: If the communication delay exceeds the 1500ms threshold, the disconnection between the remote station and the ECU must be initiated. 

Contents

There are currently no items in this folder.