AeroGround: An Open, Citation-Grounded Aviation Dataset forTraining and Evaluating Trustworthy Domain Language Models

Technical Summary: AeroGround

Problem Statement
The aerospace domain is safety-critical and heavily regulated, where plausible but unsupported answers from Large Language Models (LLMs) pose significant risks. Progress in applying LLMs to this field is currently constrained by the absence of open datasets that simultaneously meet three criteria: (i) they are grounded in authoritative sources, (ii) the source material is free of licensing encumbrances, and (iii) they are released in formats suitable for practitioner training. Existing solutions often lack strict provenance tracking or rely on copyrighted material, preventing the development of trustworthy, citation-grounded aviation assistants.

Methodology
The paper introduces AeroGround, a dataset construction pipeline and the resulting resource, designed to train and evaluate models that answer from authoritative text and abstain when evidence is insufficient.

• Data Sources: The corpus is built entirely from U.S. public-domain sources: the FAA Airman handbook series (FAA-H-8083) and Title 14 of the Code of Federal Regulations (14 CFR). ICAO materials were excluded due to copyright restrictions.
• Corpus Processing: The source texts were token-aware chunked into 14,399 segments, each tracked with provenance metadata (source, citation, license). A retrieval index was built using BAAI/bge-small-en-v1.5 embeddings and FAISS.
• Generation Pipeline:
  • Preference Pairs: For sampled passages, exam-style questions were synthesized. A "chosen" answer was generated conditioned on retrieved context (with inline citations) using Meta Llama 3-8B-Instruct, while a "rejected" answer was generated in a closed-book manner.
  • Verification: A lexical claim-support filter removed pairs where the chosen answer was not supported by the retrieved context.
  • Abstention Edge Cases: Approximately 10% of the data consists of real questions paired with deliberately irrelevant contexts, training the model to output a standardized safe refusal ("Data insufficient for safe flight protocol advice").
• Grounding Verification: The dataset employs two model-free signals to verify grounding without human labels:
  1. Sentence Support: The fraction of answer sentences whose content words appear in the source.
  2. Entity Support: The fraction of specific facts (numbers, FAA identifiers, CFR sections) appearing in the source.
  • A stratified human review of a sample corroborated the automated assessment, confirming that low automated scores correctly flagged unsupported claims.
• Model Training & Evaluation:
  • Training: A Llama 3-8B-Instruct model was fine-tuned using QLoRA and Direct Preference Optimization (DPO) on the preference pairs. The prompt structure included system instructions, retrieved context, and the question.
  • Uncertainty Quantification (UQ): A three-layer stack was implemented:
    1. Aleatoric: Scores question-context relevance and retrieval dispersion.
    2. Epistemic: Measures token predictive entropy and semantic entropy with an epistemic-halting rule.
    3. Fusion: A logistic calibrator fuses signals to decide whether to answer or abstain.

Key Contributions

1. AeroGround Dataset: An open, public-domain-sourced, provenance-tracked aviation dataset released in four formats: preference (DPO), supervised (SFT), abstention, and retrieval.
2. Reproducible Pipeline: A generation-and-verification pipeline that pairs answers with real cited source text and quantifies grounding automatically.
3. Grounding Audit: A comprehensive audit establishing trustworthiness through citation validity, sentence/entity support metrics, and human-reviewed samples.
4. Utility Demonstration: A fully evaluated pipeline combining RALA-DPO training with a three-layer UQ stack for calibrated abstention.

Results

• Dataset Composition: The final dataset contains 9,988 preference pairs (8,815 grounded, 1,173 calibrated-abstention) derived from 14,399 corpus chunks.
• Grounding Quality: Grounded answers achieved 96.9% sentence support and 98.2% entity support, significantly outperforming closed-book baselines (60.5% and 82.1%, respectively). 90.5% of grounded answers passed a strict verified-gold filter.
• Model Performance:
  • On grounded items the model chose to answer, the generation quality was ROUGE-L 0.593 and BLEU-4 46.35.
  • The calibrated abstention gate achieved a balanced accuracy of 0.875.
  • Calibration metrics were strong: ECE = 0.033, Brier = 0.061, and AUROC = 0.973.
  • The gate demonstrated a 0.940 recall for edge-abstention items (intentionally unanswerable) but resulted in a 19% over-abstention rate on grounded items, a conservative trade-off favoring safety.
• Observation on Calibration: The abstention gate remained well-calibrated even when the DPO policy was over-optimized (near-zero loss), suggesting that retrieval-side signals (question-context relevance) can supply calibrated abstention when a model's internal confidence is inflated.

Significance and Claims
The paper claims that AeroGround addresses the critical gap in open, trustworthy training data for safety-critical domains. By providing a dataset where every grounded answer is verified against its cited source, it enables the training of assistants that can cite authoritative text and, crucially, abstain when evidence is insufficient. The authors position AeroGround as a research resource intended to seed broader applications of modern NLP in aerospace, explicitly stating it must not be used for operational flight decisions. The work demonstrates that combining retrieval-augmented generation with rigorous uncertainty quantification can produce models that are both informative and safety-aware, even in highly regulated environments.