CAF Signalling is a company (technological subsidiary of CAF Group) that designs and provides Integral Signalling Solutions. It is focused on the design, development, manufacturing, supply and maintenance of Rail Signalling Systems.

European standardization group of Shift2Rail [18], in which CAF Signalling is involved, are currently working in a future GoA4 (driverless) automatic train operation (ATO) system definition. This solution is highly demanded for European railway sector users. CAF Signalling already has started exploring (developing and testing) computer vision (CV) and AI enhanced technologies for fully autonomous train operation (visual odometry, automatic object and traffic signal detection and identification, rolling stock automatic coupling…) in order to offer to its clients, the benefits of operation cost reduction, railway products life-cycles enlargement and safety increase.

However, as many companies across the sector, CAF Signalling is facing up different Validation and Verification (V&V) challenges for CV&AI-enhanced autonomous train operation which are based on non-deterministic algorithms. All AI-enhanced algorithm must verify and validate under diverse scenarios in order to get certified. However, it is not easy to collect a real database containing different real scenarios to validate computer vision-based AI techniques. CAF Signalling will use the VALU3S V&V approach on AI-enabled verification and validation processes to identify and simulate a large enough set of critical scenarios in a virtual environment to be able to show the product is sufficiently safe, especially those scenarios that are unprovable (not real imagery database to test them) but critical from a safety point of view (i.e. people crossing railways, reduced visibility due meteorological conditions).

The CV&AI-enhanced algorithms for (driverless) autonomous train operation will need a further substantial effort to increase the TRL before bringing it to the market. CV&AI-enhanced technology must fulfil strict standards and safety regulation in order to be certified. In addition, regarding the certification process of railway systems and according to EN-5012x standards, CV&AI-enhanced techniques are not currently recommended, so the adoption of this kind of solutions in such a domain is still a challenge. For this reason, virtual environment for V&V will reduce costs of AI-enhanced algorithms and it will support an easier marketing process avoiding important first barriers.

CAF Signalling has been working in CV&AI based railway signal detector/identifier techniques. After several data recorded in the field (real railway journeys), CAF Signalling trains different CV&AI based object detectors/identifiers. Light signals (green, red, orange), static speed restrictions panels, platform stopping point signals, and platform proximity signals have been labelled in different video databases in order to train these custom models. Although, the resulting models show accurate performances in nominal scenarios, they must be tested in higher variety of situations, extreme conditions and hazard situations in order to consider them really validated and verified.

Diverse and complete database creation is expensive task in terms of time and budget. Moreover, it could be almost unaffordable task due to hazard situation only happens very rarely or once in a long time. It is mandatory that well validated and verified system has been tested using databases containing different videos/clips representing all kind of a) visibility conditions (meteorological, daylight or occlusions issues), b) situations and behaviours of the static/dynamic object that are present in railway environment (pedestrian, vehicles…) or c) hazard combination of them.

The global aims of this UC9, is to set a semi-automatic V&V method, based on virtually generated scenarios to test the algorithm and AI model's robustness facing reduced visibility conditions, see Figure ‎1.17. They will test over same railway journey but under different meteorological, daylight or partial occlusion conditions.

Figure ‎1.17: Virtual environment simulating a scenario under sunny meteorological conditions (left) and rainy foggy journey's meteorological conditions (right).

In this UC, the following tasks will be carried out:

1. V&V framework requirements and scenario definition/design for reliable virtual certification of CV&AI-enhanced signalling systems.
2. Visual scenario database generation: UC9 will use a virtual environment for CV&AI-enhanced application V&V. It will output a demonstrator based on virtual train operation simulator to railway signalling (signals, speed limits…) detection/identification functionalities. The resulting database should contain a large variety of scenarios representing different visibility conditions in real train operation due to:

• Meteorological conditions (extreme situation)
  • (Dense) Fog
  • (Heavy) Rain
• Daytime periods
  • Dawn (facing sun)
  • Sunset (facing sun)
  • (Dark, without moon light) Night
• Partial occlusions
  • Vegetation (i.e. tree branches)
  • Signal deterioration (due time, inclement weather…)
  • Vandalism (i.e. broken or painted signals)

3. Metrics definition for CV&AI-enhanced application

• V&V: Precision and recall curves. Precision measures how accurate are predictions. Recall measures how good the system finds all the positives.
• Mean Average Precision (mAP) for Object Detection. AP (Average precision) is a popular metric in measuring the accuracy of object detectors. Average precision computes the average precision value for recall value over 0 to 1. mAP is the average of AP. In some context, the AP is computed for each class and average them. But in some context, they mean the same thing.

4. Semi-automatic V&V framework prototype generation. All generated virtual scenarios will contain same signals in same physical position. Furthermore, the camera recording the virtual journey will point to the same place keeping camera’s orientation and position inside train cabin. The only changes will be about daylight, meteorological conditions or partial occlusions of object we want to detect. This means that if the first nominal scenario is labelled manually, it can be used as ground truth template and perform all the rest validation and verification tests against it in an automatic way. The proposed workflow will contain following modules:

• Database manual labelling (ground truth template generation)
• Automatic test execution
• Automatic analysis and V&V of results

V&V procedures must consider that the system robustness may depend on a) Model training malfunctions (i.e. overfitting) or b) inference AI algorithm malfunctions