Technical Summary: Engineering Reliable Autonomous Systems (ERAS) Workshop Report

Problem Statement

The proliferation of autonomous systems (AS) in safety-critical domains—ranging from nuclear decommissioning and space debris removal to healthcare triage and domestic assistance—has outpaced the development of rigorous engineering practices to ensure their reliability. While classical automated systems rely on predictable environments and deterministic logic, autonomous systems operate in open, dynamic, and partially observable environments. This introduces significant challenges in assurance, particularly regarding:

• The "Reality Gap": The disconnect between abstract models/simulations and the noisy, unpredictable nature of physical deployment.
• Heterogeneity: The integration of symbolic reasoning, machine learning (ML) components (often black-box), and physical processes creates complex interactions that are difficult to verify using traditional methods.
• Adaptability: The need for systems to adapt to changing conditions conflicts with the requirement for predictable, certifiable behavior.
• Community Silos: The field is fragmented between the Formal Methods community (focused on rigorous verification, e.g., FMAS) and the Cognitive Robotics/Multi-Agent Systems community (focused on agent architectures and real-world applications, e.g., AREA), leading to a lack of shared benchmarks, terminology, and integrated solutions.

Methodology

This paper is a workshop report summarizing the outcomes of the "Engineering Reliable Autonomous Systems" (ERAS) workshop held at the Lorentz Centre in June 2024. The methodology was collaborative and qualitative, designed to bridge the gap between the FMAS and AREA communities.

1. Participant Composition: The workshop gathered 30 participants, including organizers from both FMAS and AREA, industry practitioners, and researchers from diverse sectors (healthcare, defense, nuclear, aerospace).
2. Case Study-Driven Analysis: The discussions were anchored by 11 detailed case studies (e.g., ASUMI for mine inspection, DAISY for medical triage, DARPA AUV for underwater missions). These were analyzed to identify recurring challenges and the applicability of current verification techniques.
3. Breakout Sessions: Participants engaged in structured breakout sessions focused on three core research topics:
   • Techniques for verification and validation (V&V) of AS.
   • Engineering real-world autonomous systems.
   • Software architectures for safe AS.
4. Roadmap Construction: The group synthesized discussions into a research roadmap by answering three strategic questions: "What should the future look like?", "How will we get there?", and "What are the important milestones?"

Key Contributions

1. Synthesis of Challenges and Qualities

The report identifies specific quality dimensions and desirable capabilities required for reliable AS, extending beyond traditional safety and security:

• Qualities: Privacy, Robustness, Trustworthiness (respect for autonomy, harm prevention, fairness, explicability), Sustainability, Ethics/Accountability, Social/Cultural Norms, and Transparency.
• Capabilities: Configurability, Adaptability, Dependability, Autonomy, Interaction (human-system and system-system), Perception, Cognitive reasoning, and Explainability.

2. Analysis of Verification Techniques

The paper provides a framework for selecting verification techniques based on system characteristics (e.g., openness, risk, change, complexity, regulation).

• Technique Integration: It advocates for combining formal methods (model checking, theorem proving) with runtime verification, simulation, and testing.
• Neural Network Verification: It highlights the specific challenges of verifying learning-enabled components (e.g., using SMT, MILP, or abstract interpretation) and the need to integrate these results into system-level guarantees.
• Assumption/Guarantee: It proposes using assumption/guarantee specifications to link heterogeneous components, allowing different verification techniques to be applied to different subsystems while maintaining system-level coherence.

3. Engineering Real-World Systems

The report argues that "real-world" deployment often involves significant constraints (Operational Design Domains) rather than fully open environments. Key engineering challenges identified include:

• Explainability: Essential for trust, liability, and regulatory acceptance, particularly in human-centric domains.
• Requirements Elicitation: The need to capture non-functional requirements (SLEEC: Social, Legal, Ethical, Empathetic, Cultural) alongside functional ones.
• Uncertainty: The difficulty of reasoning about imperfect data, out-of-distribution scenarios, and the limitations of ground truth in real-world settings.

4. Software Architectures: MAPE-K and BDI

A significant contribution is the detailed analysis of two architectural paradigms and their potential integration:

• MAPE-K (Monitor-Analyse-Plan-Execute-Knowledge): A reference model for self-adaptation, useful for maintaining safety envelopes and handling runtime changes.
• BDI (Belief-Desire-Intention): An agent-based model for goal-directed deliberation, offering explicit reasoning about why a decision was made.
• Integration Patterns: The report proposes four patterns for combining these architectures (e.g., BDI-in-the-Loop, MAPE-K supervising BDI, Layered Control, BDI as Safety Shells) to achieve systems that are both adaptive and explainable.

5. Research Roadmap and Deliverables

The workshop produced a structured roadmap with near-term (0–10 years), medium-term (10–20 years), and long-term (20+ years) goals.

• Near-term: Curated benchmark case studies, standardized terminology, and scoring rubrics.
• Medium-term: Component-contract specification languages, monitor synthesis tools, and reusable assurance case patterns.
• Long-term: Self-monitoring/self-verifying AS, broad regulatory acceptance, and fully certified neuro-symbolic autonomy.

Results

The primary result of the workshop is the unification of the FMAS and AREA communities around a shared understanding of the challenges in engineering reliable AS.

• Catalogue of Challenges: A comprehensive list of technical and societal barriers, including the difficulty of modeling unpredictable environments, the "black-box" nature of ML, and the lack of standardized assurance arguments for adaptive systems.
• Pathway to Solutions: A clear identification of the steps required to move from research prototypes to deployable systems, emphasizing the need for interdisciplinary collaboration, better toolchains, and the development of "evidence templates" for assurance.
• Tangible Outcomes: The report itself serves as a foundational document for a special theme in the Philosophical Transactions of the Royal Society A, aiming to disseminate these findings to a broader audience.

Significance

The paper claims significance as a foundational step toward a joint research community that bridges the gap between formal rigor and practical engineering.

• Bridging Communities: It addresses the historical disconnect between formal methods researchers (who often lack large-scale use cases) and robotics/agent researchers (who often lack deep knowledge of formal verification).
• Holistic View: It moves the discourse beyond pure safety to include trustworthiness, ethics, and societal acceptance, recognizing that reliability is a socio-technical property.
• Actionable Guidance: By providing specific architectural patterns (MAPE-K/BDI integration) and a concrete roadmap with milestones, the report offers a practical guide for researchers and practitioners to align their efforts.
• Future Direction: It posits that the future of reliable autonomy lies in neuro-symbolic systems where high-level rational reasoning is mathematically proven to coexist with learning-enabled components, supported by continuous, runtime assurance.

The report concludes that while isolated advances exist, the field must now shift toward integrated methods, shared artefacts, and sustained collaboration to realize the benefits of autonomous systems while maintaining the confidence of regulators, users, and society.