A Retrieval-Augmented Language Assistant for Unmanned Aircraft Safety Assessment and Regulatory Compliance

This paper presents a retrieval-augmented language assistant designed to support unmanned aircraft safety assessments and regulatory compliance by grounding responses in authoritative sources to ensure traceable, auditable, and citation-driven decision support without replacing expert judgment.

The Gist

Imagine you are a pilot trying to get permission to fly a new type of drone. To do this, you have to read thousands of pages of complex government rules, cross-reference different documents, and figure out exactly which rules apply to your specific flight. It's like trying to solve a giant, shifting jigsaw puzzle where the pieces are made of dense legal text.

This paper introduces a smart digital assistant designed to help pilots and safety inspectors solve that puzzle without getting lost or making mistakes.

Here is the breakdown of how it works, using simple analogies:

1. The Problem: The "Hallucinating" Librarian

Usually, when we ask a standard AI (like a chatbot) a question, it acts like a librarian who has read a million books but doesn't have them on the shelves anymore. It relies on its memory.

• The Risk: If the librarian forgets a detail or mixes up two books, they might confidently make up a rule that doesn't exist. In aviation, making up a rule is dangerous. If the AI says, "You can fly here," but the rule actually says "No," people could get hurt.

2. The Solution: The "Index Card" System (RAG)

The authors built a different kind of assistant. Instead of relying on memory, this assistant acts like a super-efficient librarian with a strict rule: "I will never answer unless I can point to the exact page in the book."

This technology is called Retrieval-Augmented Generation (RAG). Here is how the system works, step-by-step:

• The Library (The Database): The assistant is fed only the official, up-to-date government rulebooks (like the EASA regulations for drones). It doesn't know anything else.
• The Index Cards (Chunking): The system breaks these massive books down into small, manageable "index cards" (chunks). Each card is labeled with exactly where it came from (page number, section title).
• The Search (Retrieval): When you ask a question, the assistant doesn't guess. It frantically searches its index cards to find the ones that match your question. It uses two search methods at once:
  • The Keyword Search: Looking for exact words (like "drone" or "altitude").
  • The Concept Search: Understanding that "flying high" means the same thing as "high altitude," even if the words are different.
• The Safety Check (Filtering): Before answering, the system checks the cards it found. If the cards don't have enough info to answer safely, the assistant is programmed to say, "I don't know, and here is exactly what is missing." It refuses to guess.

3. The Two Modes of Operation

The paper tests this assistant in two specific ways, like a Swiss Army knife with two main tools:

Tool A: The "Regulatory Chatbot" (Conversational)

• Scenario: You ask, "Can I fly my drone over a crowded market?"
• Action: The assistant finds the specific rule about crowds, reads it, and answers: "No, according to Section 4.2, page 12, you cannot."
• The Magic: It shows you the exact page and paragraph it used. If you doubt it, you can open the book and verify it yourself. It's like a lawyer handing you the specific law they are quoting.

Tool B: The "Safety Checklist" (Structured)

• Scenario: You fill out a form with your drone's weight, how high you'll fly, and where you are.
• Action: The assistant acts like a calculator. It takes your inputs, looks up the rules, and spits out a structured report: "Risk Level: Medium," "Required Assessment: Full."
• The Magic: It doesn't write a long story; it gives you a clean, machine-readable answer that fits into official paperwork, ensuring the data is consistent every time.

4. Why This Matters: The "Human in the Loop"

The most important part of this paper is what the assistant doesn't do.

• It does not make the final decision.
• It does not replace the human expert.

Think of the assistant as a highly skilled research intern. The intern does all the heavy lifting: finding the documents, summarizing them, and highlighting the relevant parts. But the Senior Pilot (the human) is the one who signs the final permission slip. The intern is there to make sure the Senior Pilot doesn't miss a page or misread a rule, but the Senior Pilot remains responsible.

The Bottom Line

This paper proves that we can build AI tools for dangerous, high-stakes jobs (like aviation safety) if we force the AI to stick to the facts and show its work.

By separating the "searching" (finding the rules) from the "speaking" (answering the question), and by forcing the AI to cite its sources, the authors created a system that is fast and helpful but, most importantly, trustworthy. It turns a chaotic mountain of paperwork into a clear, traceable path for safety.

Technical Summary

Here is a detailed technical summary of the paper "A Retrieval-Augmented Language Assistant for Unmanned Aircraft Safety Assessment and Regulatory Compliance."

1. Problem Statement

The rapid proliferation of Unmanned Aircraft Systems (UAS) has increased the workload for aviation authorities and applicants regarding safety assessments and regulatory compliance. Current workflows rely on complex, interconnected regulatory documents (e.g., EASA's Specific Operations Risk Assessment - SORA, and Pre-defined Risk Assessment - PDRA).

• Challenges: Assessors must manually navigate vast amounts of text, cross-reference requirements, and assemble evidence.
• Limitations of Standard LLMs: While Large Language Models (LLMs) excel at text generation, they suffer from hallucinations, factual staleness (knowledge fixed at training time), and a lack of provenance (inability to cite specific regulatory passages). In safety-critical domains, these issues are unacceptable as they undermine auditability and accountability.
• Gap: There is a need for an AI assistant that is strictly bounded to authoritative sources, provides traceable evidence, and supports human decision-making without replacing expert judgment.

2. Methodology

The authors propose a Retrieval-Augmented Generation (RAG) architecture designed specifically for safety-sensitive regulatory environments. The system prioritizes traceability, auditability, and conservative behavior over generative fluency.

A. Data Engineering & Knowledge Representation

• Source Material: The system ingests the EASA Certification Specifications for Unmanned Aircraft Systems (CS-UAS), focusing on SORA and PDRA guidance.
• Structural Parsing: Unlike flat text extraction, the pipeline preserves document hierarchy (articles, Acceptable Means of Compliance - AMC, tables).
  • Manual Chunking: Core regulatory content is manually chunked by article/AMC to avoid fragmenting legal units.
  • Table Processing: Tables are converted into narrative text using an LLM (ChatGPT) to preserve cell-to-footnote relationships, then manually chunked and linked to surrounding text.
  • Exclusion: Figures are excluded as they lack machine-interpretable normative requirements in this context.
• Metadata Enrichment: Each chunk includes title, summary, and keywords to aid retrieval.

B. Retrieval Architecture (Hybrid Strategy)

The system employs a multi-stage retrieval pipeline to maximize recall and precision:

1. Dense Retrieval: Uses all-MiniLM-L6-v2 sentence transformers to create embeddings. To improve matching with formal titles, the title is concatenated twice with the text before embedding. FAISS is used for vector search (supporting both exact and HNSW approximate search).
2. Sparse Retrieval: Uses BM25 for lexical matching, ensuring exact term matches (e.g., specific regulatory codes) are captured.
3. Fusion: Reciprocal Rank Fusion (RRF) combines the dense and sparse ranked lists.
4. Diversity Control: Maximal Marginal Relevance (MMR) is applied to the candidate pool to reduce redundancy and ensure diverse regulatory passages are selected.
5. Reranking: A ColBERT-style late-interaction reranker refines the top candidates based on token-level matching.
6. Filtering: An "elbow method" on score drops filters out weakly related evidence before generation.

C. Controlled Generation

• LLM Backend: Uses gpt-oss-20B (OpenAI).
• Constraints: The model is strictly instructed to answer only from the retrieved context. It must cite sources, explicitly state if evidence is missing (refusal behavior), and avoid speculation.
• Output Modes:
  1. Conversational: For regulatory questions, providing grounded answers with source snippets.
  2. Indicator-Oriented: For structured safety assessments, returning machine-readable JSON objects (e.g., risk orientation, regulatory pathway) based on user inputs.

3. Key Contributions

1. Safety-Oriented RAG Design: A novel architecture that separates evidence storage from generation, enforcing strict citation and "refusal" mechanisms when evidence is insufficient.
2. Hybrid Retrieval Pipeline: Demonstrates the effectiveness of combining dense semantic search, sparse lexical search, and graph-agnostic structural chunking for regulatory texts.
3. Auditability Mechanisms: The system logs full prompt contexts, retrieved chunk IDs, and internal reasoning traces (if available) to support human review and regulatory audits.
4. Two Reference Use Cases:
   • UC-CHAT-01: A conversational assistant for regulatory Q&A.
   • UC-CLS-01: A structured workflow for generating initial safety assessment indicators (e.g., ground/air risk orientation).
5. Empirical Validation: Rigorous testing of retrieval accuracy, grounding quality, and output consistency under various query reformulations.

4. Results

The system was evaluated on the two use cases with the following findings:

• Retrieval Performance:
  • Direct Matches: Achieved 100% Hit@1 and MRR of 1.0.
  • Paraphrased/Reworked Queries: Performance degraded (Hit@1 dropped to 75% for synonyms and reworked questions), indicating that semantic reformulation remains a challenge for pure RAG without advanced query expansion.
• Grounding & Accuracy:
  • Even with perfect retrieval, only 50% of answers were fully "grounded" (all sentences supported by citations), highlighting a gap between retrieval and generation discipline.
  • Safety Assessment Task:
    • Value Consistency: High repeatability (91.7% overall). "Expected assessment depth" was 100% stable.
    • Accuracy: Overall accuracy against ground truth was 81.8%.
    • Weaknesses: "Likely regulatory pathway" was the least stable (66.6% accuracy), suggesting that borderline regulatory interpretations are sensitive to input phrasing and require human oversight.
• Failure Modes: The system correctly identified evidence gaps in many cases but occasionally produced plausible but incomplete responses when retrieval was fragmented.

5. Significance and Conclusion

This paper establishes a blueprint for deploying LLMs in safety-critical, regulated industries (specifically aviation).

• Human-in-the-Loop: The system is explicitly designed as a decision support tool, not an autonomous authority. It accelerates document navigation and synthesis but leaves final judgment to human assessors.
• Regulatory Compliance: By enforcing strict provenance (citing specific pages/articles) and refusing to hallucinate, the system addresses the primary barrier to LLM adoption in certification: trust and auditability.
• Future Directions: The authors suggest future work should focus on:
  • Improving recall for complex semantic reformulations (e.g., via GraphRAG or query expansion).
  • Implementing post-hoc sentence-level verifiers to ensure every claim is backed by evidence.
  • Expanding the corpus to include national interpretations while maintaining source provenance.

In summary, the work demonstrates that with rigorous engineering controls—specifically hybrid retrieval, manual structural chunking, and constrained generation—RAG systems can effectively support the complex, high-stakes workflows of unmanned aircraft safety assessment.