Simulation-Based Testing for Human-Robot Collaboration employing Constraint-Based Oracles

Improvement from the base method Simulation-Based Testing for Human-Robot Collaboration into a description of an improved method. Simulation-based testing, Human-robot collaboration, Constraint-based modelling, Knowledge modelling, Test oracles, Artificial intelligence

Method Purpose Short description of the method's purpose and main benefits

Method description A detailed description of the method

A key factor for enabling the industry of the future is the collaboration between humans and robots. However, human-robot interaction is a complex process that raises issues of compatibility and operational safety. Although simulation-based testing can address some of the main challenges [SBT-CBO1, SBT-CBO5, SBT-CBO6], such as the safety of human-robot collaboration, specifying validation oracles remains a complex task.

Constraint-based Modelling Oracle

The improvement proposed in this method consists of developing a new functionality into the ULISES framework to act as an oracle [SBT-CBO2, SBT-CBO3]. The ULISES framework was developed on the basis of constraint-based modelling techniques [SBT-CBO7] to interpret and diagnose procedural tasks carried out by users, including impaired people [SBT-CBO4]. This method provides an oracle for simulation-based testing of the interaction between impaired people and robots in a manufacturing and disassembly domain. The provided oracle functionality will be integrated in the existing framework in order to perform the diagnosis (test evaluation) of the human-robot interaction and provide a verdict.

The method workflow is described in the figure below.

The framework receives four inputs: On the one hand, the SimulationStream is received as input from the simulation system. On the other hand, the framework requires three models (observation, task, and interpretation models) that will be used to determine the diagnosis of the activities that carried out in the simulation.

The observation subsystem transforms SimulationStreams into observations based on the information that is defined in the observation model. These observations are the primitives for the higher interpretation and diagnosis subsystems. In the interpretation subsystem, observations and the interpretation models are used as input. As a result, the steps and situations being carried out are obtained. Lastly, those steps and situations are diagnosed by the diagnosis subsystem, which also is provided by observations from the observation subsystem. Only those steps and situations that are defined in the Task Model will be diagnosed. This last subsystem will generate (in real time) step and situation validations as result. The process is conducted until the SimulationStrean is finished (i.e., the simulation is finished).

Compared to the framework of the base method, this method contributes to the integration of new observers and the integration of the monitoring tool.

SBT-CBO method workflow: Simulation-Based Testing for Human-Robot Collaboration employing Constraint-Based Oracles.

Method artefacts:

Inputs:
- Diagnosis model
  - Observation model
  - Interpretation model
  - Task model
- Simulation stream
Intermediate results
- Observations
- Steps
- Situations

Outputs:
- Activity diagnosis results
  - Step validation result
  - Situation validation result

Method Strengths A listof the strengths of the method

In simulation-based testing, verification activities can be done for different robots without changing other models or tools. Also, system-testing can be done without producing any physical item and adding risk to environment.
Constraint-based modelling of oracles can provide powerful asserts in complex simulation testing.
Capability to integrate with ROS-based environments.

Method Limitations The limitations of the method

The generation of Constraint-based knowledge model is a complex and time-consuming task.
Time-related constraints must be satisfied and cannot be accelerated for time-saving purposes.

Addressed gaps:

Functionality:
- [GAPM_SBT01] In some cases, human intervention is necessary to determine the verdict of the execution of the simulation. This work requires an effort by the validation engineer. To address this gap an oracle for diagnosis of the Human-Robot interaction will be included.

Method References Bibliography references to papers about the method.

[SBT-CBO1] Koenig, N., & Howard, A. (2004, September). Design and use paradigms for gazebo, an open-source multi-robot simulator. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No. 04CH37566) (Vol. 3, pp. 2149-2154). IEEE. DOI: 10.1109/IROS.2004.1389727
[SBT-CBO2] Aguirre, A., Lozano-Rodero, A., Matey, L. M., Villamañe, M., & Ferrero, B. (2014). A novel approach to diagnosing motor skills. IEEE Transactions on Learning Technologies, 7(4), 304-318. DOI: 10.1109/TLT.2014.2340878
[SBT-CBO3] Aguirre, A., Lozano-Rodero, A., Villamañe, M., Ferrero, B., & Matey, L. M. (2012). OLYMPUS: An Intelligent Interactive Learning Platform for Procedural Tasks. In GRAPP/IVAPP (pp. 543-550). ISBN: 978-989856502-0
[SBT-CBO4] Ostiategui, F., Amundarain, A., Lozano, A., & Matey, L. (2010). Gardening Work Simulation Tool in Virtual Reality for Disabled People Tutorial. Proceedings of Integrated Design and Manufacturing-Virtual Concept (IDMME’10).
[SBT-CBO5] Webster, M., Western, D., Araiza-Illan, D., Dixon, C., Eder, K., Fisher, M., & Pipe, A. G. (2020). A corroborative approach to verification and validation of human–robot teams. The International Journal of Robotics Research, 39(1), 73-99. DOI:10.1177/0278364919883338
[SBT-CBO6] Takaya, K., Asai, T., Kroumov, V., & Smarandache, F. (2016, October). Simulation environment for mobile robots testing using ROS and Gazebo. In 2016 20th International Conference on System Theory, Control and Computing (ICSTCC) (pp. 96-101). IEEE. DOI: 10.1109/ICSTCC.2016.7790647
[SBT-CBO7] Kodaganallur, V., Weitz, R. R., & Rosenthal, D. (2005). A comparison of model-tracing and constraint-based intelligent tutoring paradigms. International Journal of Artificial Intelligence in Education, 15(2), 117-144.

Method Dimensions

Evaluation Environment Type Type of environment where the method is applied. It can be more than one type of environment. (Dimension 1)

Evaluation Type Type of evaluation performed with that method. If a method is hybrid, more than one type can be selected. (Dimension 2)

Type of Component Under Evaluation Type of components that can be evaluated with that method. It can be more than one type. (Dimension 3)

Evaluation Stage The evaluation stage where the method is used. (Dimension 5)

Purpose of the Component Under Evaluation The type of purpose that can be evaluated by a method. It can be more than one type. (Dimension 6)

Type of Requirement Under Evaluation The type of requirements that a method is able to evaluate. It can be more than one type. (Dimension 7)

Evaluation Performance Indicator The type of evaluation criteria or KPI that a method is able to improve. It can be more than one type. (Dimension 8)

Relations

Related Methods A method could be related to other methods.

Tools A method can use zero or more tools.

Part Method A method (when it is a combination: Combination of method artefact that is a specialization of a V&V Method) could be composed of a set of V&V Methods.

Test Case or Verification and Validation activity Test Case or V&V activity performed in a Use Case. We will link to the method to the test cases where the method is used.

Standards A method can be linked with standards.

Context

Workflow

Contents

There are currently no items in this folder.