🤖 AI

Knowledge Graphs as the Missing Data Layer for LLM-Based Industrial Asset Operations

This paper demonstrates that replacing flat document stores with a structured knowledge graph layer significantly boosts LLM-based industrial asset operations from 65% to near-perfect accuracy by shifting the LLM's role from reasoning over raw data to generating deterministic queries against a typed schema.

Original authors: Madhulatha Mandarapu, Sandeep Kunkunuru

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Madhulatha Mandarapu, Sandeep Kunkunuru

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to fix a giant, complex factory machine. You have a brilliant, very smart assistant (the LLM, or Large Language Model) who knows a lot about the world, but they are terrible at doing math, counting, or finding specific facts in a messy pile of papers.

This paper is about a new way to help that smart assistant do its job better by changing where it looks for information.

The Problem: The "Messy Pile of Papers"

In the original setup (called AssetOpsBench), the smart assistant was asked to fix machines by looking at a "flat" data store. Think of this like a giant filing cabinet filled with thousands of loose, unorganized sheets of paper, PDFs, and spreadsheets.

The Task: The assistant had to read these papers, find a specific part number, count how many times a sensor failed, and figure out how one machine affects another.
The Result: The assistant got confused. It lost count, mixed up the papers, or made up facts (hallucinated) because it was trying to do complex math and searching through a messy pile with its brain alone. It only got about 65% of the answers right.

The Solution: The "Super-Organized Map"

The authors decided to stop giving the assistant a messy pile of papers. Instead, they built a Knowledge Graph.

Think of a Knowledge Graph as a living, breathing map or a family tree for the machines.

Instead of loose papers, every machine, sensor, and failure is a "node" (a dot on the map).
The connections between them are "edges" (lines connecting the dots).
If Machine A breaks, the map instantly shows you exactly which other machines depend on it, without you having to read a hundred documents to find out.

The Three Ways They Tested It

The researchers tested three different ways to use this new map with the smart assistant:

The Old Way (The "Do Everything" Assistant):
- The assistant still tries to read the messy papers and figure everything out itself.
- Score: 65%. (Still struggling).
The "Translator" Way (The New LLM Approach):
- Here, they changed the rules. They didn't ask the assistant to find the answer. Instead, they gave it the map and asked it to write a specific instruction (a query) to find the answer.
- Analogy: Imagine the assistant is a chef who is great at reading recipes but bad at chopping vegetables. Instead of asking the chef to chop, you ask the chef to write a note saying, "Chop 5 carrots." A robot (the graph) then chops the carrots perfectly.
- The assistant writes the instruction, and the map executes it instantly.
- Score: 82–83%. (Much better, just by changing how the assistant asks for help).
The "Robot" Way (The Deterministic Approach):
- They removed the assistant entirely for known tasks. They wrote pre-made instructions that the map follows automatically.
- Analogy: Like a vending machine. You press a button, and the machine knows exactly which snack to drop. No thinking required.
- Score: 99%. (Almost perfect).

The Big Discovery: "Inverted" Thinking

The most important thing the paper found is what they call "Inverted LLM Usage."

Before: They asked the smart brain to do the boring, hard math and searching (things it is bad at).
After: They asked the smart brain to do what it is good at: writing instructions (code) based on a clear set of rules. They let the computer (the graph) do the hard math and searching.

By letting the computer handle the heavy lifting and the AI handle the language, the results jumped from 65% to over 80% using the exact same AI model.

What About the Future?

The paper also tested this on a much larger set of 467 different scenarios (including pumps, compressors, and complex rules).

When they used the "Robot" approach (pre-written instructions for the map), they got a 100% success rate.
They also showed that this map can handle things the old "pile of papers" never could, like finding hidden connections between distant machines or ranking which machines are most critical to fix first.

The Bottom Line

The paper argues that for industrial jobs, the problem isn't that our AI isn't smart enough. The problem is that we are giving it the wrong tools.

If you want an AI to manage factory machines, don't just give it a stack of manuals. Give it a structured map and let it write the instructions to navigate that map. That simple switch turns a confused assistant into a highly accurate expert.

Technical Summary: Knowledge Graphs as the Missing Data Layer for LLM-Based Industrial Asset Operations

Problem Statement

Current Large Language Model (LLM) agents for industrial asset operations exhibit limited accuracy when reasoning over flat document stores (e.g., CouchDB, YAML, CSV). The AssetOpsBench benchmark (Patel et al., 2026) established that even leading models like GPT-4 and GPT-4.1 achieve a maximum task completion rate of approximately 65% on 139 industrial maintenance scenarios. The study identified that failures are often not due to a lack of reasoning capability but rather "failures of data access." Specifically, LLMs struggle with operations that structured query engines handle deterministically, such as:

Counting events across multiple unstructured documents.
Correlating data from disparate sources without explicit links.
Traversing implicit equipment dependencies.
Avoiding hallucinations of equipment identifiers or sensor readings.

The central hypothesis of this paper is that for structured operational domains, the data model behind the tools is the primary bottleneck, not the LLM orchestration paradigm.

Methodology

The authors evaluated three distinct architectures using the same 139 AssetOpsBench scenarios, varying only the data layer and the role of the LLM:

Architecture A (Baseline - Tool-Augmented LLM):
- Mechanism: The LLM parses intent, selects tools, formulates arguments, retrieves raw data from document stores, interprets results, and synthesizes an answer.
- Data Layer: Flat document stores (CouchDB, YAML, CSV).
- LLM Role: Performs all reasoning, data traversal, and aggregation.
Architecture B (NLQ + Graph):
- Mechanism: The LLM is constrained to generate structured Cypher queries based on a typed schema. The graph engine executes the query deterministically, and the LLM synthesizes the final answer from the structured results.
- Data Layer: A typed Knowledge Graph (KG) with 781 nodes, 955 edges, and 16 relationship types (expanded to 1,360 nodes and 2,500 edges in the full pipeline).
- LLM Role: Code generation (Natural Language to Cypher).
Architecture C (Deterministic Handlers):
- Mechanism: Pre-coded handlers match question patterns directly to Cypher queries.
- Data Layer: The same Knowledge Graph.
- LLM Role: None.

Knowledge Graph Construction:
The authors built an ETL pipeline transforming AssetOpsBench data sources into a graph schema containing 14 node labels (e.g., Site, Equipment, Sensor, FailureMode) and 21 edge types (e.g., contains_equipment, monitors, depends_on). The graph includes:

Equipment hierarchies with ISA-95 and ISO 14224 classifications.
Sensor metadata linked to equipment.
Failure modes with 384-dimensional Sentence-BERT embeddings for vector similarity search.
Event logs with temporal capabilities.
A dependency topology modeling thermal/electrical dependencies and shared infrastructure.

The implementation uses Samyama, an embedded graph database supporting OpenCypher, HNSW vector indexing, and graph algorithms (PageRank, NSGA-II).

Key Results

Performance on 139 Scenarios

The study demonstrates a significant performance lift by changing the data layer, even when holding the LLM model constant:

Architecture A (Baseline): 65% pass rate (91/139), matching the AssetOpsBench leaderboard ceiling.
Architecture B (NLQ + Graph): 82–83% pass rate (114–116/139) using GPT-4, GPT-4o, and GPT-4.1. This represents a ~17 percentage point improvement over the baseline using the same model family.
Architecture C (Deterministic): 99% pass rate (137/139). The two failures were attributed to response-format mismatches in work order bundling, not knowledge gaps.

Expanded Evaluation (467 Scenarios)

When evaluated against the expanded AssetOpsBench HuggingFace release (467 scenarios across 6 domains):

Deterministic Handlers: Achieved a 100% pass rate (467/467) with an average score of 0.848.
Graph-Native Capabilities: The authors introduced 40 new scenarios requiring multi-hop dependency, vector similarity, and PageRank criticality. On these, the Knowledge Graph approach achieved 100% pass rate and an average score of 0.927, compared to 85% and 0.602 for a GPT-4o baseline without the graph.

Latency and Cost

Latency: Deterministic graph queries averaged 63 ms, compared to 5–11 seconds for LLM-based architectures.
Cost: For 10,000 daily queries, a fully LLM-driven architecture costs $300–$ 500, whereas deterministic graph queries cost $0 after initial ETL.

Significance and Claims

The paper posits that the data layer is the primary bottleneck in structured operational domains, acting as a more significant lever for performance improvement than LLM orchestration paradigms (Agent-As-Tool vs. Plan-Execute).

The authors introduce the concept of "Inverted LLM Usage":

Instead of asking the LLM to reason over raw data (a broad, error-prone task), the system asks the LLM to generate structured queries from a typed schema (a narrow, code-generation task where LLMs excel).
The graph engine then handles the "hard" parts: traversal, counting, joining, and algorithmic execution.
This separation of concerns allows the system to achieve near-perfect accuracy on known query patterns while retaining the flexibility of natural language interfaces for novel queries.

The paper concludes that for structured industrial data, Knowledge Graphs serve as an essential integration layer between raw data and LLM-based reasoning. The results suggest that schema-aware query generation outperforms free-form data reasoning for any structured domain, and that deterministic handlers can achieve near-perfect reliability for known operational patterns.

Limitations and Caveats

The authors explicitly note several limitations:

Clean Data Assumption: The benchmark uses clean, structured data. Real-world industrial environments involve noisy sensors, scanned PDFs, and inconsistent naming, which require LLMs at the data ingestion layer (entity extraction and resolution) rather than just the query layer.
Deterministic vs. Autonomous: The 99% success rate of Architecture C reflects a pre-coded solution, not an autonomous agent's ability to figure out answers independently.
Structural Limitations: Scenarios requiring ML inference (e.g., time-series forecasting, RUL prediction) cannot be solved by querying stored data alone; they require external analytical models.
Nondeterminism: The NLQ results rely on stochastic LLM generation; variance characterization across multiple seeds is deferred to future work.