🤖 AI

End-to-End LLM Flight Planning with RAG-based Memory and Multi-modal Coach Agent

The paper introduces FRAMe, an end-to-end LLM-based flight planning system for eVTOL aircraft that integrates a multi-modal coach agent and RAG-based memory to translate natural language instructions into safe, efficient routes that align with human operator preferences, achieving up to 93.8% validity across diverse scenarios.

Original authors: Amin Tabrizian, Arsyi Aziz, Aarifah Ullah, Mahyar Ghazanfari, Pouria Razzaghi, Peng Wei

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amin Tabrizian, Arsyi Aziz, Aarifah Ullah, Mahyar Ghazanfari, Pouria Razzaghi, Peng Wei

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the captain of a futuristic, electric flying car (called an eVTOL). You need to fly from Point A to Point B, but the sky isn't empty. There are invisible "no-fly" zones—like storm clouds, restricted airspace, or dangerous wind pockets—that you must avoid.

Traditionally, computers plan these routes using strict math rules. They are great at finding the shortest path, but they are terrible at understanding human nuance. If you say, "I want to fly high and far away from the storms," a classic computer might just ignore that preference and take the mathematically shortest route, even if it skims dangerously close to the danger.

This paper introduces FRAMe, a new system designed to bridge the gap between human intent and computer planning. Think of FRAMe as a super-smart flight planning team made of three distinct characters working together:

1. The Planner (The "Dreamer")

This is a Large Language Model (LLM)—basically a very advanced AI that reads and writes like a human.

What it does: You tell it, "Fly from here to there, but stay as far away from the wind hazards as possible." The Planner translates your words into a list of GPS coordinates (waypoints) to create a flight path.
The Problem: Sometimes, the Planner gets creative but makes mistakes. It might draw a line that accidentally cuts through a "no-fly" zone or forgets to start at the right airport.

2. The Coach (The "Strict Referee")

This is a multi-modal AI agent (it can "see" images and read text) that acts as a quality control inspector.

What it does: Before a flight plan is approved, the Coach looks at the map the Planner drew.
- Step 1 (Geometry Check): It uses a ruler and protractor to ensure the path doesn't hit any "no-fly" zones and stays within the allowed airspace.
- Step 2 (Visual Check): It looks at a picture of the flight path to see if it actually follows your instructions (e.g., "Did it really stay far away from the storms?").
- Step 3 (Feedback): If the plan is bad, the Coach rejects it. If it's good, it gives a thumbs up.
The Analogy: Think of the Planner as a student writing an essay, and the Coach as the teacher grading it. The student might write a great story, but if they miss the prompt or break the rules, the teacher sends it back for corrections.

3. The Memory (The "Librarian")

This is a system called RAG (Retrieval-Augmented Generation).

What it does: Every time the team successfully plans a flight, the details are saved in a digital library. When you ask for a new flight plan, the Librarian quickly finds past flights that were similar (same starting point, same type of weather, same preferences) and shows them to the Planner.
The Analogy: Imagine you are trying to solve a puzzle. Instead of starting from scratch, you have a friend who says, "Hey, I solved a similar puzzle yesterday. Here is how I did it." The Librarian provides that helpful hint, so the Planner doesn't have to guess.

How They Work Together

The system runs in a loop:

The Librarian finds a past example to help the Planner.
The Planner draws a new route based on your words and the example.
The Coach checks the route.
- If it's safe and follows your rules, the Coach approves it.
- If it's unsafe, the Coach rejects it, and the Planner tries again.
Once approved, the new plan is added to the Library for next time.

What They Found

The researchers tested this system with four different AI "Planners" and three levels of difficulty (Easy, Medium, and Hard skies).

The "All-Star" Team Wins: The system worked best when all three parts (Planner + Librarian + Coach) were used together. It produced valid, safe flight plans up to 93.8% of the time.
The Coach is Crucial: For some AI models, just having the "Librarian" (past examples) wasn't enough; it actually made them slightly worse because they got confused by bad examples. However, adding the "Coach" fixed this. The Coach acted as a safety net, catching errors that the Librarian missed.
Listening to Humans: Unlike old math-based systems that only care about the shortest distance, FRAMe actually listened to human preferences.
- If you asked to minimize waypoints (keep the path simple), the AI made the path simpler.
- If you asked to maximize clearance (stay far from danger), the AI flew further away from the hazards.
- Note: The system couldn't improve on "distance" for everyone because the AI models were already flying very close to the mathematically shortest path to begin with.

The Bottom Line

FRAMe shows that we can use advanced AI to create flight plans that are not just mathematically correct, but also safe, flexible, and aligned with what a human pilot actually wants. It turns a rigid computer algorithm into a collaborative team that learns from its past mistakes and listens to human instructions.

Note: The paper explicitly states that all tests were done using computer simulations and maps. They did not fly real aircraft yet, though they built a web interface to test the logic on small drones as a stepping stone.

Technical Summary: FRAMe – End-to-End LLM Flight Planning with RAG-based Memory and Multi-modal Coach Agent

Problem Statement

The deployment of electric vertical takeoff and landing (eVTOL) aircraft and unmanned aerial systems (UAS) for Advanced Air Mobility (AAM) requires flight planning systems capable of navigating dense, complex airspace. Traditional path planning algorithms (e.g., A*, RRT) rely on explicitly defined mathematical objectives and struggle to incorporate flexible, subjective human pilot preferences, such as trading flight duration against waypoint complexity or maximizing clearance from hazards. While Large Language Models (LLMs) have shown promise in robotics and navigation for interpreting natural language intent, their application to real-world flight planning remains under-explored. Existing approaches often lack mechanisms to ensure plan validity, align with operator preferences, or learn from past experiences without retraining from scratch.

Methodology

The authors present FRAMe, an end-to-end flight planning framework that integrates a planner LLM with a Retrieval-Augmented Generation (RAG) memory module and a multi-modal Coach Agent. The system operates through the following components:

Planner Agent: An LLM acts as the core route planner. It receives a natural language prompt describing the operator's preference (e.g., "maximize clearance from hazardous polygons") and scenario data encoded in KML (origin, destination, flyable airspace, and no-fly zones). The planner generates a sequence of geographic waypoints forming a flight path.
RAG-based Memory Module: To ground the LLM's reasoning, the system retrieves relevant context from a vector database of past flight plans. The retrieval process involves:
- Embedding the operator's natural language preference.
- Filtering the database for records with identical scenario geometry (same origin, destination, and no-fly zones).
- Ranking these records by cosine similarity of the preference embeddings to select the $K$ most aligned past solutions.
- Augmenting the planner's prompt with these retrieved examples and their evaluations.
Multi-modal Coach Agent: A verification layer implemented using a multimodal LLM (specifically o4-mini) combined with rule-based geometric checks. The coach performs a three-stage review:
- Geometric Validity: Rule-based tools verify that all waypoints remain within the flyable airspace, connect the origin and destination, and do not intersect no-fly zones.
- Preference Alignment: The coach analyzes a rendered image of the proposed flight plan to assess if the route visually aligns with the operator's stated preference.
- Feedback Loop: The operator can provide optional feedback. The final record (scenario, preference, validity, alignment, and feedback) is stored in the database to improve future RAG retrievals.

The system was evaluated using a web prototype targeting subscale UAV operations (integrating with Mission Planner) as a cost-effective proxy for eVTOL testing.

Key Contributions

FRAMe System Architecture: A novel framework combining a planner LLM, RAG memory, and a multi-modal coach to enable natural language-driven route generation that respects safety constraints and operator preferences.
Annotator-Free Preference Evaluation: A quantitative evaluation framework using three measurable geometric objectives: minimizing flight distance, minimizing waypoint count, and maximizing polygon clearance. This allows for objective assessment of preference alignment without human annotators.
Multi-Model Ablation Study: A systematic evaluation across four state-of-the-art LLMs (OpenAI o3-mini, o4-mini, GPT-5.4, and DeepSeek-R1) and three scenario difficulty levels (Easy, Medium, Hard), benchmarked against a classical A* baseline. The study isolates the contributions of RAG and the Coach Agent.

Experimental Results

The evaluation covered 150 warmup runs to seed the database, followed by an ablation phase across four conditions: A* baseline, LLM Baseline, LLM + RAG, and LLM + RAG + Coach.

Plan Validity: The full FRAMe system (+RAG+Coach) achieved the highest validity rates for all planners, reaching up to 93.8% aggregate validity (99% on Easy scenarios for the strongest planner). Notably, for the o3-mini model, RAG alone slightly decreased validity compared to the baseline, but the addition of the Coach Agent recovered and extended the gains, indicating the Coach provides a non-redundant, load-bearing validity check.
Preference Alignment:
- Waypoints: The o3-mini model showed the most significant improvement, reducing the mean waypoint count by 12% under the full system compared to the baseline.
- Clearance: The o4-mini model demonstrated the clearest gain in maximizing clearance (+9%).
- Distance: Most models operated near their planning floor for distance at the baseline, limiting further improvement, though GPT-5.4 showed monotonic improvement when the distance preference was explicitly active.
Comparison with A:* While the classical A* planner achieved 100% validity, it was "preference-blind," producing routes with significantly higher waypoint counts (18.2 vs. ~3.5–5.0 for LLMs) and lower clearance (1.91 km vs. 3.5–5.0 km for LLMs). FRAMe outperformed A* in clearance and waypoint efficiency while accepting a minor distance overhead.

Significance and Claims

The paper claims that FRAMe demonstrates how advanced LLMs can be effectively deployed for human-centric mission planning, successfully translating natural language instructions into safe, efficient, and flexible flight routes. The authors emphasize that the integration of a multi-modal coach is critical; it acts as a necessary gatekeeper for validity and preference alignment, particularly for models where retrieval alone is insufficient or potentially misleading.

The framework is presented as a modular step toward bridging the gap between human pilot intent and autonomous flight operations. The authors maintain a modest stance, noting that the current evaluation relies on synthetic polygonal no-fly zones and simulation-only data, with real-world flight tests and dynamic constraint modeling (e.g., traffic, battery limits) identified as necessary future work. The system is designed to be preference-aligned where classical algorithms are not, offering a trade-off of slight distance overhead for significantly improved safety margins and route simplicity.