Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge

Technical Summary: Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge

Problem Statement
Large language models (LLMs) are increasingly proposed for aviation business operations, ranging from documentation and training generation to customer-facing assistants. However, general-purpose benchmarks fail to measure whether models reason safely and correctly regarding aviation-specific operational knowledge. Given the high-stakes, regulated nature of the aviation domain, this gap is consequential. Existing evaluations do not distinguish between models that genuinely understand aviation operations and those that merely produce plausible-sounding outputs. Furthermore, the aviation information landscape is heterogeneous, mixing legacy formats (e.g., NOTAMs, Type B messages) with modern digital services, and much operational knowledge remains unstructured text. There is a lack of open, domain-specific resources to evaluate LLMs in this context, particularly for non-safety-critical business operations where errors carry reputational and operational costs.

Methodology
The authors introduce Pre-Flight, an open-source benchmark consisting of 300 multiple-choice questions designed to evaluate declarative operational and regulatory knowledge.

• Dataset Construction: Items were drawn from authoritative sources, including international airport Ground Operations Safety Manuals (GOSM), US regulations (14 CFR), ICAO Annexes, and general aviation documentation. The dataset was authored and reviewed by practitioners with experience in air traffic management, ground operations, and commercial flying.
• Composition: The 300 items are categorized into five groups:
  1. International airport ground operations (50.7%)
  2. ICAO rules and regulations (28.3%)
  3. FAA rules and regulations (17.0%)
  4. Aviation trivia (2.7%)
  5. Complex ground scenarios (1.3%)
     The dataset is weighted heavily toward operational ground content and regulatory knowledge.
• Evaluation Protocol: The benchmark is evaluated using the Inspect framework. Models are tested using a zero-shot, standard multiple-choice protocol with deterministic exact matching against gold answers. No model-graded or LLM-judge scoring is used. The evaluation runs on a rolling basis as new models are released, with results tracked on a public leaderboard.
• Reference Baseline: An informal expert reference of approximately 95% accuracy was established via a low-sample quiz of aviation professionals at a conference.

Key Contributions

1. Motivation for Domain-Specific Evaluation: The paper argues that general benchmarks (like MMLU) and other domain benchmarks (e.g., MedQA, LegalBench) do not capture the specific knowledge integration and failure modes required for aviation deployment. It positions Pre-Flight as a necessary precondition for responsible AI deployment in non-safety-critical aviation operations.
2. The Pre-Flight Benchmark: The release of a curated, open-source dataset and evaluation harness (integrated into inspect_evals) that specifically targets ground operations, ICAO/FAA regulations, and complex scenarios.
3. Empirical Analysis: A comprehensive evaluation of 44 contemporary commercial and open-weight models (released between early 2024 and mid-2026), providing a snapshot of current capabilities and failure modes.

Results

• Performance Gap: Even the strongest model evaluated (GPT-5.5, released in 2026) achieved an accuracy of 82.7%, remaining approximately 12 percentage points below the informal expert reference (~95%).
• Progress Trajectory: Improvement has been slow relative to the gap. While the frontier rose from the mid-60s to low-70s in 2024, progress slowed in 2025–2026, adding only roughly eight points to reach the current peak.
• Model Comparisons:
  • Open vs. Closed: Open-weight models (e.g., Qwen3.5 122B) are competitive with leading API systems, with a locally run quantized variant reaching 77.3%.
  • Category Difficulty: Models perform best on widely published international safety and ICAO content but struggle most with US (FAA) regulatory details, which showed a 53.0% mean failure rate.
  • Reasoning vs. Recall: Items requiring reasoning (60.3% of the set) proved more difficult than direct recall, with a 4.1-point higher failure rate.
  • Outliers: Some models strong in other domains collapsed on this benchmark (e.g., ALLaM 2 7B scored 25.3%, near chance level).
• Data Quality Issues: The analysis revealed that some "failures" were actually answer key errors where models converged on an option more consistent with the governing regulation than the recorded key. Two such errors were corrected, and others were flagged for revalidation.

Significance and Claims
The paper claims that domain-specific evaluation is a necessary precondition for the responsible deployment of generative AI in aviation business operations. The results demonstrate a substantial and persistent gap between current LLM capabilities and the reliability required for operational use. The authors argue that aggregate knowledge scores often overstate operational competence, particularly where multi-step reasoning is required.

Pre-Flight serves as a concrete, reusable measure of "under the wing" operational knowledge, complementing existing efforts like ALUE (which targets the systemic/airspace layer) and PilotBench (which targets safety-critical flight trajectory prediction). By providing an open, reproducible benchmark, the authors aim to facilitate community scrutiny, track progress over time, and ensure that AI deployment in aviation is grounded in verified operational knowledge rather than plausible hallucinations. The paper explicitly restricts its scope to non-safety-critical functions, aligning with the FAA's incremental approach to AI safety assurance.