Agentic AI for Safety-critical Multi-drone Systems: Challenges and Opportunities

Technical Summary: Agentic AI for Safety-critical Multi-drone Systems

Problem Statement
Multi-drone systems offer significant potential for safety-critical missions, including search and rescue (SAR), firefighting, and critical infrastructure monitoring. However, real-world adoption is hindered not merely by technical autonomy performance, but by the socio-technical challenge of integrating agentic behavior into professional workflows. Operators must understand, trust, and govern automation under conditions of uncertainty, time pressure, and high accountability. As fleet sizes increase, the traditional one-operator/one-vehicle model fails due to coordination overhead, degraded shared situation awareness, and the risk of small uncertainties cascading into safety issues (e.g., conflicting priorities or ambiguous responsibility). While recent advances in Large Language Models (LLMs) and agentic AI enable natural-language tasking and mixed-initiative planning, they introduce new interaction challenges: representing agent intentions, supervising decisions, and guaranteeing correct behavior in messy field environments.

Methodology and Research Approach
The paper adopts a human-centered, participatory, and iterative research approach, treating agentic AI as a "socio-technical design material" rather than a fixed technical solution. This methodology is grounded in two complementary ongoing projects:

1. NAMUR: Focuses on time-critical emergency response (SAR and firefighting), emphasizing rapid decision cycles, dynamic hazards, and the need for unified abstraction layers across heterogeneous robot platforms.
2. PERSIST: Focuses on long-horizon, persistent operations at critical infrastructure (e.g., power plants), emphasizing sustained activity, repeatability, shift handovers, and organizational integration.

The research process involves:

• Needs Discovery: Grounded in concrete scenarios and demonstrations with stakeholders (incident commanders, field responders, security personnel) to map work practices, decision points, and cognitive bottlenecks.
• Participatory Prototyping: Using successive functional prototypes as "boundary objects" to negotiate agent capabilities, authorization requirements, and uncertainty cues.
• System Design: Developing an architecture that decomposes natural-language intent into reviewable sub-tasks, constrains execution via deterministic tools, and enforces operator preview and approval.

Key Contributions: The LLM-MAS Architecture
The paper presents a functional prototype architecture: a Large Language Model Multi-Agent System (LLM-MAS) designed for multi-drone control. This system is composed of five specialized agents that interface with external deterministic tools (e.g., spatial computation, motion execution, memory storage, task scheduling):

1. Coordinator Agent: Acts as the central orchestrator, inferring operator intent and decomposing commands into sub-queries delegated to subordinate agents.
2. Events Agent: Manages periodic, delayed, and future-oriented directives (e.g., scheduled biomass estimation or return-to-base commands), ensuring tasks are executed or surfaced at the appropriate time.
3. Spatial Agent: Provides situational awareness by accessing GeoJSON maps and live drone/first-responder positions. It identifies semantic map features and computes spatial relationships.
4. Swarm Agent: Handles motion planning and task execution. It selects suitable drone platforms based on availability and context. Crucially, all requests generated by this agent are subject to operator preview and confirmation before execution. It publishes actions to a ROS topic to trigger flight control pipelines.
5. Summarizer Agent: Generates concise, human-readable summaries of executed actions and system states. These are stored in interaction memory to provide context for subsequent dialogue turns, enabling coherent multi-turn interactions and reference resolution (e.g., "move it to the nearest safe zone").

The system is deployment-dependent; for SAR, tools support compliant coverage path planning, while for infrastructure monitoring, tools support flight path generation for biomass estimation.

Results and Findings
The paper synthesizes lessons from prior field engagements and prototype evaluations to identify key opportunities and tensions:

• Opportunities: Agentic AI can reduce coordination overhead by providing selective attention/summarization, enabling mixed-initiative planning that maintains coverage as information changes, and supporting long-horizon orchestration (scheduling, anomaly triage).
• Tensions: The authors identify critical safety tensions that autonomy alone cannot resolve:
  • The need for systems that operate without constant attention but resist "black-box" behavior.
  • The risk that strong alerting improves short-term performance but causes "attention tunneling," reducing holistic oversight.
  • The necessity of calibrated trust under imperfect perception, requiring conservative defaults and verification workflows.
  • The requirement for operational fit, including alignment with roles, protocols, and auditable decision trails.

The functional prototype has been demonstrated in live control of Unmanned Ground Vehicles (UGVs) and Unmanned Aerial Vehicles (UAVs), validating the decomposition of natural language into reviewable sub-tasks and the enforcement of explicit operator approval.

Significance and Claims
The paper argues that agentic AI for safety-critical work must be approached as a socio-technical design problem where interfaces, oversight mechanisms, and evaluation practices are as critical as the algorithms themselves. The authors claim that their approach supports future research by operationalizing "governable agentic autonomy" through three key implications:

1. Governability as a Core Property: Authorization points, constraints, and intervention mechanisms must be explicit and auditable system properties, not UI add-ons.
2. Constrained Tool Use: Agentic behavior should be built from constrained, inspectable tool use rather than unconstrained free-form action to ensure predictable failure modes and reproducible debugging.
3. Real-World Evaluation: Evaluation must reflect actual operational use, measuring not just task success but also situation awareness, trust calibration, coordination overhead, and support for accountable after-action reviews.

The paper concludes that while agentic AI holds promise for scaling multi-drone operations, its successful deployment relies on iterative stakeholder engagement to shape systems that remain transparent, learnable, and aligned with local protocols and accountability practices.