UC2_Eval_SCP_1

To assess the teleoperation system's ability to handle delayed or invalid data that is received due to injected faults and attacks into the system. This error coverage analysis is employed to evaluate the system's responses to these tests.
UC2
To assess the teleoperation system's ability to handle incorrect, invalid, or untrusted data, faults and attacks are deliberately introduced. This error coverage analysis is employed to evaluate the system's responses to these tests.
Evaluation Criteria for Safety, Cybersecurity, and Privacy (SCP)
Number of safety/security requirement violations

In order to evaluate the teleoperation system's ability to handle delayed data or connection loss, we conducted tests by introducing various delays in the transmission link. These delays were meant to simulate faults or congestion in the communication system. Additionally, we performed experiments where we intentionally emulated complete connection loss of the transmission link, representing a Denial-of-Service (DoS) attack on the communication system. The objective of these tests was to verify the system's resilience and performance under adverse conditions.

Upon analyzing the test results, we observed an interesting trend: as the duration of the attack increased, the vehicle's tolerance for delays decreased, resulting in an earlier triggering of the vehicle's safe stop mechanism. This indicates that with prolonged attack durations, the system becomes more susceptible to delays and disruptions in the communication link. Furthermore, as the delay in the transmission link increased, we observed a gradual disconnection between the remote station and the 'ecu,' as depicted in the figure below.

These findings highlight the critical role of maintaining a reliable and low-latency communication channel in ensuring the smooth operation of the teleoperation system. They also emphasize the need for robust countermeasures to mitigate the impact of potential attacks and disruptions in the communication infrastructure. By understanding the system's behavior under various attack scenarios, we can develop strategies to enhance its resilience and improve its ability to handle delayed data or connection loss effectively.

DelayAttack.jpg

For in-depth analysis, we created and executed five test suites which are as follows,

1- DoS attack where both video and command channels are disconnected.

2- DoS attack where only the command channel is disconnected.

3- DoS attack where only the video channel is disconnected.

4- Delay attack where the communication on both video and command channels is delayed.

5- Delay attack where the communication on command channels is delayed. 

6- Delay attack where the communication video channel is delayed.

The purpose of these test suites is to verify and validate the two main safety mechanisms (aka fallback mechanism) which are implemented in the teleoperation system.  Following are the safety mechanisms that we used for result classification,

1- Vehicle Safe Stop: when the communication is delayed to a certain duration, the teleoperated vehicles must reduce their speed for ensuring safety.  The safe stop can be a slight safe stop which low deceleration rates or the safe stop can trigger a high deceleration rate (emergency braking) depending on the delay introduced.  the threshold for safe stop activation is when the communication delay is between 150ms and 1500ms.

2- Vehicle Disconnect: If the communication delay exceeds the 1500ms threshold, the disconnection between the remote station and the ECU must be initiated. 

We also automated the test configuration and execution process to facilitate the simulation-based testing.

Following are the test configuration values we have used for detailed analysis and evaluation:

Test scenario: The "top speed" test scenario is selected for testing the safety mechanisms of the system. In this scenario, the speed starts to accelerate to achieve a maximum speed which is 50 km/h, and then decelerates until the vehicle is stopped. The total duration of the test is 15 seconds. See the figure below which depicts the test scenario.

Attack start time: We choose the attack start time from 1 second to 7 seconds with the step of 1 second. The reason behind this selection is that the speed of the vehicle stabilizes at 1 second and 7 seconds was selected for the better coverage of the test scenario. See the figure below which depicts the test scenario.

topspeedscen.JPG

 

Attack duration: The total of seven attack durations are selected for each attack start time. 

Attack value: There are 24 attack values selected in total. The minimum selected value is 50 ms and the maximum value is 1800 ms. The safe stop should activate between 50 ms and 150 ms. Delays of more than 150 ms should initiate the disconnection as mentioned above.

The total test cases that we ran are 1680 which takes approximately 32 hours to complete.

Contents