💻 computer science

A Model for Mediating Multi-Modal Human Intent into Safe Maneuvers for UAVs

This paper presents a requirements-governed model that mediates multi-modal human intent into safe UAV maneuvers by treating operator inputs as bounded requests validated against environmental and flight constraints through a structured request-evaluate-execute pipeline with formal safety specifications.

Original authors: Sofia Nelson, Dalal Alrajeh, Pedro Antonio Alarcon Granadeno, Jane Cleland-Huang

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sofia Nelson, Dalal Alrajeh, Pedro Antonio Alarcon Granadeno, Jane Cleland-Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a tiny, super-smart drone. You want to tell it to "Go left!" or "Fly up!" using your voice, a hand wave, or a button on a screen. In the past, scientists thought, "Great! Let's just listen to the captain and do exactly what they say." But this paper says: Stop! That's a dangerous game.

If you're standing on a hill and shout "Go forward," but there's a steep cliff right in front of the drone, doing exactly what you said would mean a crash. Or if you wave your hand to "Go right," but another drone is hovering there, you'd cause a collision. Humans get distracted, lose their sense of direction, or make mistakes. The paper argues that we cannot treat human commands as direct orders that the drone must obey instantly. Instead, we need a safety referee in the middle.

The "Bounded Request" Idea

The authors, Sofia Nelson and her team, propose a new way to talk to drones. They call it the M3R model (Multi-Modal Maneuver Request). Think of it like this:

When you tell the drone to "Go left," you aren't giving a command to move. You are making a request to move, but with a strict limit. It's like asking a friend, "Can you walk over to that tree?" but adding, "But only if you can get there without tripping over a rock or bumping into someone."

The drone doesn't just listen; it checks the rules before it moves. It treats your input as a "bounded maneuver request." This means the drone says, "Okay, I hear you want to go left, but I can only go 1 meter left, and I can't go faster than a certain speed, and I have to make sure the ground is flat and no other drones are in the way."

The Safety Referee's Checklist

Before the drone actually moves, a digital "Safety Supervisor" runs a quick checklist. It looks at four big things:

The Ground: Is there a cliff or a tree in the way? The drone checks the terrain to make sure it won't crash into the ground.
The Neighbors: Are there other drones nearby? It checks to make sure it won't bump into them.
The Drone's Limits: Can the drone actually do this? It checks if the move is too fast or too far for the drone's engine to handle safely.
The Captain's Clarity: Did the drone understand you? If you shouted over loud wind or waved a hand in the dark, the drone might not be 100% sure what you meant. If it's not confident, it won't move.

If the request passes all these checks, the drone does a constrained version of what you asked. If you asked to go 5 meters but there's a wall 2 meters away, the drone will only go 2 meters. If the wall is right there, it won't go at all. It might even say, "Blocked!" and stay still.

The "Anchor" and the "Bubble"

To keep things safe, the drone picks a safe spot called an Anchor when it starts listening to you. It stays inside a 5-meter bubble around that spot. Every time you give a new command, the drone updates its anchor, but it never leaves this safe zone. It's like the drone is playing in a giant, invisible playpen that moves with it.

How They Tested It

The team built a prototype to see if this idea works. They didn't just write code; they actually tested it in a lab with a simulated drone. They used three ways to talk to the drone:

Voice: Using a tool called VOSK to listen to words like "Go forward" or "Fly up."
Gestures: Using a camera and a smart AI (GPT-5.o) to watch people wave their hands.
Buttons: A simple screen with buttons for each move.

They found that the system could reliably understand these inputs and, more importantly, safely stop or shrink the move if it wasn't safe. For example, if a human tried to command the drone to fly into a simulated wall, the system blocked it or made the drone stop short.

What They Didn't Solve Yet

It's important to know what this paper is not claiming. The authors are very honest about the limits:

It's not a finished product for the real world yet. They tested it in a lab with a simulated environment. They haven't flown this in a windy, messy outdoor field with real obstacles and other real drones yet.
The "Turn" problem is tricky. They admit that telling a drone to "Turn left" is hard because if the drone spins around, the human might lose track of which way is "left." They didn't fully solve this in the current version.
The "Eyes" need work. For the gesture part, they used a powerful AI model that works well in a lab, but they know it might struggle in bad weather or from far away. They plan to train a better model later.

The Big Takeaway

The main finding is that safety cannot depend on humans being perfect. Humans will make mistakes, get distracted, or give unclear commands. The paper suggests that the only way to let humans control drones safely is to put a smart safety filter between the human and the drone. This filter turns every "Go!" into a "Go, but only if it's safe."

The authors suggest this approach is a solid foundation for the future. They have shown it works in simulations and lab tests, and they have a clear set of rules (requirements) for how it should behave. But they are careful to say this is just the first step. The real test—flying in the wild with wind, rain, and real traffic—is still ahead. They are building the rules of the road before they let the cars drive on the highway.

Technical Summary: A Model for Mediating Multi-Modal Human Intent into Safe Maneuvers for UAVs

Problem Statement
Direct interaction between human operators and autonomous Uncrewed Aerial Vehicles (UAVs) via multi-modal inputs (speech, gestures, graphical interfaces) introduces significant safety risks in dynamic environments. While existing research focuses on accurately recognizing human intent using Computer Vision (CV) and Natural Language Processing (NLP), it often operates under the assumption that recognized commands can be executed directly. This approach fails to account for scenarios where operator requests conflict with terrain constraints, inter-UAV separation requirements, flight-envelope limitations, or environmental uncertainties. Current safety-oriented work primarily addresses physical separation distances and operator comfort but lacks mechanisms to constrain and mediate operator-issued commands prior to execution. Consequently, there is a critical need for a runtime framework that treats human inputs not as unconditional actuation commands, but as bounded maneuver requests subject to continuous safety evaluation.

Methodology
The authors propose the Multi-Modal Maneuver Request (M3R) model, a requirements-governed framework that mediates human intent into safe UAV maneuvers. The core methodology involves treating operator inputs as requests for "bounded motion primitives" rather than direct control signals.

Operational Modes: The UAV operates in distinct states, including NormalAutonomy, ManeuverResponseArmed (ready to receive commands), ManeuverResponseActive (executing a bounded primitive), Blocked (request rejected), EmergencyHold, and ReturnToAnchor.
Request–Evaluate–Execute Pipeline:
- Interpret: Inputs (gesture, speech, GUI) are processed to infer intent with an associated confidence score.
- Validate: The requested maneuver is checked against four constraint categories:
  - Terrain Constraints: Ensuring minimum altitude above ground level (AGL) and path clearance.
  - Inter-UAV Separation: Maintaining 3D separation from neighboring drones.
  - Flight Envelope: Bounding displacement, velocity, and rotational rates.
  - Context and Confidence: Requiring sufficient input confidence, localization accuracy, and fresh state data for neighbors.
- Constrain: If a request is unsafe, the system either rejects it or constrains it to a safe, bounded version (e.g., shortening the displacement).
- Execute & Monitor: The approved primitive is executed while a runtime monitor continuously checks for safety violations, triggering an EmergencyHold if necessary.
Anchor-Based Workspace: Upon pairing, the UAV establishes an "anchor state" (position and heading relative to the operator). All subsequent maneuvers are constrained within a bounded local workspace ( $R_{max}$ ) around this anchor, which updates dynamically.
Formal Specification: The model is operationalized through:
- Requirements Specification: Behavioral and safety properties defined via preconditions, invariants, guard conditions, and postconditions.
- State Machine: A formal representation of system transitions (e.g., from ManeuverRequestValidated to Blocked if safety guards fail).
- Contracts: Maneuver-specific contracts (e.g., for "GoForward") defining obligations regarding terrain clearance, separation, and speed limits.

Key Contributions

Constrained Multimodal Operating Model: A shift from direct command execution to a "bounded maneuver request" paradigm, where human inputs are filtered through safety constraints before actuation.
Requirements-Governed Framework: A formal specification model that links high-level safety requirements to executable controller behavior, supporting runtime verification and future reactive synthesis.
Safety Decision Function: A computational logic that evaluates candidate maneuvers against terrain, separation, envelope, and confidence constraints to determine admissibility.
Prototype Implementation: A lab-based prototype integrating three modalities (gesture, voice, GUI) with a safety supervisor that enforces the M3R model.

Results
The paper presents an initial validation of the M3R model using a lab-based prototype.

Modality Integration: The system successfully recognized gestures (using a Visual Large Language Model in the prototype), voice commands (using VOSK/Kaldi), and GUI inputs.
Safety Mediation: The prototype demonstrated the ability to interpret voice and GUI inputs as constrained maneuver requests. Crucially, the system successfully blocked or constrained maneuvers that would violate safety constraints (e.g., terrain proximity or workspace boundaries) and transitioned to an EmergencyHold state when violations were detected during execution.
Limitations: The validation was conducted in a controlled environment with simulated obstacles. The gesture recognition component relied on a general-purpose VLLM rather than a specialized CV model trained for field conditions, and the study did not yet fully address real-world uncertainties like wind, GPS drift, or communication latency.

Significance and Claims
The authors claim that this work establishes a foundational step toward runtime-assured and formally analyzable human–UAV interaction. The paper argues that safe multimodal interaction cannot rely solely on accurate intent recognition; it requires explicit runtime safety mediation. By formalizing maneuvers through requirements, safety contracts, and state-machine behavior, the framework provides a traceable path from safety requirements to controller execution. The authors position this not as a completed operational solution for all real-world conditions, but as a necessary abstraction and pipeline for systematically reasoning about multi-modal human intent prior to execution, enabling future integration with terrain-aware reasoning, dynamic airspace awareness, and reactive controller synthesis.