💬 NLP

KEO: Knowledge Extraction on OMIn via Knowledge Graphs and RAG for Safety-Critical Aviation Maintenance

This paper introduces KEO, a framework that combines Knowledge Graphs with Retrieval-Augmented Generation to enhance large language models' ability to perform global sensemaking and safety-critical reasoning on aviation maintenance data, demonstrating superior system-level insights compared to traditional text-chunk RAG approaches.

Original authors: Kuangshi Ai, Jonathan A. Karr Jr, Meng Jiang, Nitesh V. Chawla, Chaoli Wang

Published 2026-04-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kuangshi Ai, Jonathan A. Karr Jr, Meng Jiang, Nitesh V. Chawla, Chaoli Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a mechanic trying to fix a very complex, expensive airplane. You have a massive library of old maintenance logs, but they are written in a chaotic mix of short notes, scattered across thousands of different files.

If you ask a standard AI (like a basic chatbot) a question like, "Why do engines keep failing in winter?" it might try to guess the answer based on what it memorized during training, or it might just grab a few random sentences from the library that happen to mention "engine" and "winter." It's like trying to solve a puzzle by looking at just a few scattered pieces without seeing the picture on the box. This can lead to wrong answers, which is dangerous when flying planes.

This paper introduces KEO (Knowledge Extraction on OMIn), a smarter way to help AI understand these safety-critical maintenance records. Here is how it works, using some simple analogies:

1. The Problem: The "Scattered Note" Approach

Traditional AI tools use something called RAG (Retrieval-Augmented Generation). Think of this like a librarian who, when you ask a question, runs to the shelves, grabs the first few books that have your keywords on the cover, and hands them to you.

The Flaw: If you ask about "engine failure," the librarian might hand you a note about a broken tire because the word "failure" was in it, or a note about a summer storm because "engine" was mentioned. The information is fragmented. It's like trying to understand a movie by reading random sentences from different chapters.

2. The Solution: Building a "Mental Map" (The Knowledge Graph)

KEO takes a different approach. Before the AI tries to answer anything, it reads all the maintenance logs and builds a Knowledge Graph (KG).

The Analogy: Imagine taking all those scattered notes and drawing a giant map on a wall.
- You draw a dot for "Engine."
- You draw a dot for "Frozen Fuel."
- You draw a line connecting them with a label: "Causes."
- You draw another line from "Frozen Fuel" to "Winter."
Now, instead of just grabbing random pages, the AI can walk along the lines of this map. It sees the whole picture: Winter → Frozen Fuel → Engine Failure. It understands the relationships, not just the words.

3. How KEO Works in Practice

The system has two main modes, depending on what you need:

Mode A: The Detective (Global Sensemaking)
- The Task: "What are the big trends in aircraft safety over the last year?"
- How KEO wins: It uses the map. It can trace paths across the entire dataset to find patterns that a human or a standard AI would miss. It's like a detective who doesn't just look at one crime scene but connects clues from five different cities to solve a conspiracy.
- Result: KEO is much better at this than the "scattered note" method because it sees the system-level connections.
Mode B: The Technician (Procedural Tasks)
- The Task: "What specific steps do I take to replace a broken fuel pump?"
- How KEO works: Sometimes, you don't need a map; you just need the exact instruction manual. For these specific, step-by-step tasks, the traditional "scattered note" method (grabbing the exact text chunk) is actually faster and more precise. KEO knows when to use the map and when to just grab the text.

4. Why This Matters for Safety

In aviation, you can't have an AI "hallucinate" (make things up).

The "Black Box" Problem: Standard AI is like a black box; you don't know why it gave an answer.
The "Transparent Map" Solution: With KEO, if the AI says, "Replace the fuel filter," it can point to the exact lines on the map that led it there. It's like showing your work on a math test. This makes the AI trustworthy enough to be used in high-stakes environments like airports and military bases, where it runs on local computers without needing to connect to the internet (keeping secrets safe).

Summary

Think of KEO as upgrading an AI from a student who memorizes flashcards to a senior engineer who has a complete blueprint of the entire system.

When you need to understand big patterns (like why things break in winter), the blueprint (Knowledge Graph) is unbeatable.
When you need specific instructions (like how to tighten a bolt), the flashcards (text chunks) still work well.

By combining these two, KEO creates a safer, smarter, and more reliable AI assistant for keeping our skies safe.

1. Problem Statement

The paper addresses the challenges of applying Large Language Models (LLMs) to safety-critical domains, specifically aviation maintenance. Key issues identified include:

Limitations of Standard LLMs: They rely on parametric knowledge, leading to factual inaccuracies, outdated information, and an inability to handle domain-specific real-world scenarios without hallucinations.
Limitations of Conventional RAG: Traditional Retrieval-Augmented Generation (RAG) relies on embedding-based retrieval of unstructured text chunks. This often results in fragmented and redundant context, hindering performance on tasks requiring complex, multi-hop reasoning or "global sensemaking" (understanding patterns across the entire dataset).
Security Constraints: In sensitive environments (aviation/military), reliance on external APIs is often prohibited. Solutions must be locally deployable and secure.
Data Scarcity: High-quality, annotated datasets for aviation maintenance (like the OMIn dataset) are limited, and existing tools show low Technology Readiness Levels (TRLs) for processing this specific data.

2. Methodology: The KEO Framework

The authors propose KEO (Knowledge Extraction on OMIn), a framework that integrates structured Knowledge Graphs (KGs) into a RAG pipeline. The methodology consists of four core components:

A. KG Construction from Raw Records

Data Source: The framework uses the OMIn dataset (2,748 FAA aviation incident records).
Extraction Process: Instead of relying on human-annotated gold standards (which are scarce), KEO uses LLMs (tested with GPT-4o and locally deployable Phi-4-mini) in a zero-shot setting to extract triplets $(h, r, t, w)$ representing Head Entity, Relation, Tail Entity, and Weight (frequency).
Dynamic Construction: The KG is built iteratively. When generating new triplets, the LLM is prompted with the current set of nodes to minimize redundancy and ensure consistency.

B. KG-Based RAG Pipeline

Unlike traditional chunk-based RAG, KEO performs retrieval at the entity level:

Semantic-Based Node Identification: The user query is embedded and compared against entity nodes in the KG to identify the top- $k$ most relevant seed nodes.
Importance-Aware Graph Expansion:
- The system expands from seed nodes to $m$ -hop neighbors to capture indirect relationships.
- The directed subgraph is converted to an undirected weighted graph.
- Maximum Spanning Tree (MST) algorithms (Kruskal's) are applied to connected components to filter out noise and retain the most structurally significant links.
Context Reconstruction:
- The MSTs are traversed (DFS) to generate textual descriptions of the graph structure.
- Hierarchical Community Summarization: Inspired by GraphRAG, the system uses the Leiden algorithm to detect communities and recursively summarizes them, providing the LLM with both local details and global structural insights.

C. QA Benchmark Creation

The authors constructed a novel benchmark with two distinct question types to evaluate different reasoning capabilities:

Global Sensemaking Questions: Require synthesizing information across the entire dataset to identify patterns, trends, and systemic issues (e.g., "What recurring failure modes emerge across aircraft types?").
Knowledge-to-Action Questions: Derived from the MaintNet dataset, these test the transferability of knowledge. The system must answer procedural questions using only the OMIn dataset, forcing it to generalize structured knowledge to unseen scenarios.

D. Evaluation Strategy

Models: Tested locally deployable LLMs (Gemma-3, Phi-4, Mistral-Nemo) for answer generation.
Judges: Stronger models (GPT-4o, Llama-3.3) act as evaluators using absolute scoring (1–5 scale) and pairwise comparisons.
Metrics: Includes ROUGE scores for procedural tasks and custom criteria (Global Perspective, Synthesis Quality, Strategic Value) for sensemaking tasks.

3. Key Contributions

KEO Framework: A novel architecture that replaces unstructured text-chunk retrieval with KG-augmented retrieval, enabling coherent, dataset-wide reasoning.
Specialized Benchmark: Creation of a QA benchmark specifically for aviation maintenance that evaluates both global sensemaking and knowledge-to-action transfer, addressing a gap in existing RAG benchmarks.
Local Deployment Focus: Demonstrates a secure pipeline using only locally deployable models, avoiding external API dependencies crucial for safety-critical sectors.
Empirical Insights: Provides a clear delineation of when KG-RAG outperforms text-chunk RAG (global reasoning) versus when traditional RAG remains superior (fine-grained procedural retrieval).

4. Experimental Results

Global Sensemaking: KEO significantly outperformed both Vanilla LLMs and Text-chunk RAG.
- The structured context provided by the KG allowed models to uncover system-level insights and patterns that text chunks missed.
- Performance gains were most pronounced when paired with stronger backbone models (e.g., Gemma-3).
- Optimal performance was observed when the KG was constructed from 200–300 records.
Knowledge-to-Action: KEO did not outperform Text-chunk RAG in this category.
- Procedural tasks requiring specific, localized steps were better served by direct semantic retrieval of similar text chunks.
- The abstraction inherent in KGs sometimes penalized the precision required for specific action steps.
Model Robustness: The framework remained effective even when using weaker, locally deployable models (like Phi-4) for KG construction, though GPT-4o yielded slightly higher quality graphs.

5. Significance and Implications

Safety-Critical AI: KEO demonstrates a viable path for deploying trustworthy AI in high-stakes environments (aviation, defense, healthcare) where hallucinations are unacceptable and external APIs are restricted.
Hybrid Retrieval Paradigm: The paper establishes that no single retrieval method is optimal for all tasks.
- KG-RAG is essential for complex, multi-hop reasoning and strategic decision-making.
- Text-chunk RAG remains superior for localized, procedural, and fine-grained information retrieval.
Scalability: The approach shows that smaller, locally hosted models can achieve performance comparable to larger models when augmented with structured knowledge graphs, reducing the computational and security barriers to entry for specialized domains.

In conclusion, KEO proves that integrating structured knowledge representation with RAG is a critical step toward reliable, explainable, and secure AI for safety-critical industries, provided the retrieval strategy is matched to the specific reasoning demands of the task.