🤖 AI

Constructing Dynamic Master Logic Models as Knowledge Graphs for Complex System Diagnostics Using Retrieval-Augmented Large Language Models

This paper presents a framework that leverages Retrieval-Augmented Generation and Large Language Models to automatically construct scalable, graph-based Dynamic Master Logic models from technical documentation, thereby enabling automated diagnostic reasoning and reliability analysis for complex systems like nuclear reactor safety systems.

Original authors: Saman Marandi, Yu-Shu Hu, Mohammad Modarres

Published 2026-08-13

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Saman Marandi, Yu-Shu Hu, Mohammad Modarres

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a giant, complex machine works, like a spaceship or a nuclear power plant. Usually, engineers have to draw massive, complicated maps called "Fault Trees" to figure out what happens if something breaks. These maps are like looking at a forest and trying to understand the whole ecosystem by only studying individual fallen leaves and broken branches. They are great for known problems, but if a brand-new, weird failure happens that no one thought of, these maps often fail because they only look for specific, pre-written disaster scenarios.

To fix this, scientists use something called "Functional Modeling." Instead of just listing broken parts, they map out what the machine is supposed to do. It's like understanding a car not by listing every possible broken bolt, but by understanding that the engine's job is to turn fuel into motion, and the wheels' job is to turn that motion into movement. If you know the jobs, you can figure out what happens if the fuel stops, even if you've never seen that exact problem before. The paper you are about to read tackles a huge hurdle in this field: making these "job maps" automatically. Until now, building these maps required human experts to read thousands of pages of technical manuals and manually draw the connections, a process that takes months and is hard to scale up for giant systems.

The Paper's Big Idea: Teaching AI to Draw the Blueprint

This paper introduces a clever new way to build these "job maps" (called Dynamic Master Logic or DML models) using Artificial Intelligence, specifically Large Language Models (LLMs) and a technique called Retrieval-Augmented Generation (RAG). Think of the LLM as a super-smart, fast-reading intern who has read the entire library of engineering manuals. However, this intern has a bad habit: sometimes they make things up or get confused when the story gets too long. To fix this, the authors gave the intern a strict set of rules and a "search engine" (RAG) that forces them to look at the specific page of the manual relevant to the task before they write anything down.

The result is a system that automatically reads technical documentation and builds a "Knowledge Graph" (a digital web of connections) that looks exactly like the expert-drawn maps. The authors tested this on the Low-Pressure Coolant Injection (LPCI) system of a decommissioned nuclear power plant. This system is a safety net designed to flood the reactor with water if it gets too hot. It's complex, with pumps, valves, and pipes working together in specific ways.

How the Magic Happens: The Layer-by-Layer Chef

The authors didn't just ask the AI, "Draw the whole map." That would be like asking a chef to cook a ten-course meal in one sentence. Instead, they broke the job down into layers, like building a house from the roof down to the foundation.

The Search: The system starts with a high-level goal, like "Keep the reactor cool." It searches the manuals for the specific sections that talk about this goal.
The Extraction: The AI reads those sections and asks, "What functions are needed to achieve this?" It finds answers like "Pump water" or "Open valves."
The Connection: It then takes those new functions and searches the manuals again to find the specific parts (components) that do the work.
The Graph: Finally, it stitches all these pieces together into a digital graph (a Knowledge Graph) where every node is a part or a function, and every line is a logical rule (like "AND" or "OR") connecting them.

The paper shows that this method works surprisingly well. When they ran the system five times on the same nuclear plant documentation, it built the map with incredible consistency. For the big-picture goals and main functions, the AI got it 100% right every time. Even at the detailed level of specific components (like individual valves), it was correct about 98% of the time.

The "Integrity Score": How Good is the Map?

To make sure the AI didn't just guess, the authors created a special scoring system called an "Integrity Score." Imagine you are grading a student's homework. You don't just check if the answers are right; you check if the whole story makes sense. If the student misses a key step in the middle, the whole answer falls apart.

The authors found that the way they fed information to the AI mattered. If they asked the AI to process too many parts at once (a large "batch size"), the quality of the map dropped slightly. It was like asking the chef to chop all vegetables for the whole meal at once; they might miss a few. When they asked the AI to do one small step at a time (a batch size of 1), the map was the most accurate, though it took a bit longer to build. The best balance they found was processing five steps at a time, which kept the map very accurate (scoring around 89 out of 100) without taking forever.

What the Map Can Do: The Detective and the Planner

Once the map is built, it becomes a powerful tool for two main things:

Upward Propagation (The Detective): If a specific valve breaks, you can ask the system, "What happens to the whole plant?" The system traces the failure up the ladder. It might say, "If this valve fails, the pump can't work, which means the cooling function fails, and the reactor might overheat." It does this by following the logical rules the AI built into the map.
Downward Propagation (The Planner): You can also ask, "How do we keep the reactor cool?" The system looks down the ladder and says, "You need the pump to work, which requires the valve to be open AND the power to be on." It finds all the different ways to achieve a goal, showing you the backup plans (redundancy) the system has.

The Catch: What the Paper Says and Doesn't Say

The authors are careful not to claim this is a perfect, finished product. They explicitly state that while the AI is great at reading the manuals and drawing the connections, it doesn't "understand" the physics in the way a human expert does. The map is a representation of the text, not necessarily a perfect simulation of reality.

For instance, the paper notes that the system struggles a tiny bit with the most crowded, complex parts of the map (where many valves and pipes connect). It might miss one or two connections out of hundreds. Also, the system doesn't automatically know about "Common Cause Failures"—situations where two different parts fail at the same time because they share a hidden cause, like a bad batch of manufacturing or a shared power surge. The AI can flag these as possibilities during a conversation, but it doesn't build them into the core logic of the map.

The paper also rules out the idea that this replaces human experts. Instead, it suggests that this tool is a "force multiplier." It takes a task that usually takes human experts months of reading and drawing and shrinks it down to a few days. The human expert then steps in to review the AI's work, catching those few missing links and ensuring the map is truly safe.

The Bottom Line

This paper suggests that we can now use AI to turn piles of boring, complex technical manuals into interactive, logical maps that help us understand how giant machines work and what happens when they break. It's not a magic wand that solves every engineering problem instantly, but it is a powerful new tool that makes the job of keeping our complex world safe much faster and more consistent. The authors show that with the right "search and verify" strategy, AI can learn to speak the language of engineering logic, turning text into a living, breathing blueprint for safety.

Technical Summary: Automated Construction of Dynamic Master Logic Models as Knowledge Graphs

Problem Statement
Modern engineered systems are increasingly complex, requiring diagnostic models that accurately capture interactions and dependencies across multiple abstraction levels. Traditional event-driven approaches, such as Fault Tree Analysis (FTA) and Event Tree Analysis (ETA), rely on predefined failure sequences. While effective for known faults, these methods are inherently incomplete when failure sequences are unanticipated and become infeasible to trace as system complexity grows.

Functional modeling offers an alternative by focusing on system objectives and how they are realized through component interactions. Dynamic Master Logic (DML) is a hierarchical framework that links functional objectives to structural elements (hardware, software, human actions) using explicit logical relationships. However, constructing DML models has historically relied on manual, expert-driven interpretation of extensive technical documentation. This process limits scalability and hinders the application of functional representations to large, document-intensive systems. Furthermore, while Large Language Models (LLMs) show promise in semantic interpretation, they suffer from hallucinations, inconsistent reasoning, and a lack of persistent structure, making them unreliable for generating the explicit, structured representations required for rigorous engineering diagnostics without augmentation.

Methodology
The study proposes a framework for the automated construction of DML models from system documentation, representing them as Knowledge Graphs (KG-DML). The approach leverages Retrieval-Augmented Generation (RAG) and LLMs to bridge the gap between unstructured text and structured functional logic.

Document Preprocessing and Embedding: System documentation is standardized to normalize component identifiers and resolve ambiguities. The text is segmented into overlapping chunks (1,500 characters) and embedded into a vector store to support similarity-based retrieval.
Layer-by-Layer Construction: The model is constructed sequentially through the DML hierarchy (Goals $\rightarrow$ $\to$ Functions $\rightarrow$ $\to$ Subfunctions $\rightarrow$ $\to$ Components $\rightarrow$ $\to$ Success Conditions).
- At each layer, a retrieval query is constructed using the semantic definition of the current layer and the parent elements identified in the previous stage.
- The top $K$ relevant document chunks are retrieved and provided to the LLM along with a predefined JSON schema.
- The LLM generates candidate elements and logical relationships (AND/OR gates) constrained by the schema.
- Validated elements are appended to a master JSON representation, which serves as the context for the next layer's retrieval.
Knowledge Graph Synthesis: The validated JSON is converted into Cypher queries to populate a Neo4j graph database. This process explicitly handles nested Boolean gate structures, creating intermediate gate nodes to preserve multi-level logical dependencies without flattening the hierarchy.
Evaluation Framework: A multi-level validation methodology is introduced to assess the constructed KG-DML:
- Element-Level Metrics: Precision, recall, and $F_2$ scores are calculated for each hierarchical layer, with $F_2$ emphasizing recall to penalize missing elements.
- Structural Consistency: Link accuracy and gate correctness (AND/OR logic) are evaluated.
- Integrity Score: An aggregate metric combines weighted layer-level $F_2$ scores with an exponential penalty for structural reconstruction errors (missing nodes, links, gate mismatches, or orphan nodes). This score reflects both element accuracy and the completeness of functional paths.
Interaction Mechanism: The resulting KG-DML supports diagnostic reasoning via an LLM agent that acts as a coordination layer. The agent interprets user queries and invokes deterministic graph-based tools for:
- Upward Propagation: Tracing failure consequences from components to system goals.
- Downward Propagation: Identifying minimal success paths from goals to components.
- Explanatory Queries: Retrieving structural relationships via Graph-RAG.

Key Contributions

Automated Construction Framework: Introduces a scalable method for extracting functional models from unstructured engineering documentation using RAG and LLMs, moving beyond the manual expert processes previously required for DML.
Extension to Complex Systems: Extends prior small-scale research to substantially larger, document-intensive systems (demonstrated on a Boiling Water Reactor safety system), addressing the challenge of hierarchical consistency in large-scale extraction.
Structured Evaluation: Develops a rigorous, multi-level evaluation framework that combines layer-specific accuracy with an aggregate integrity score to assess structural completeness and logical consistency, addressing gaps in previous evaluation methodologies.
Executable KG-DML Representation: Delivers a machine-readable Knowledge Graph that preserves the strict hierarchical decomposition and success logic of DML, enabling deterministic diagnostic reasoning rather than relying solely on generalized LLM knowledge.

Results
The framework was applied to the Low-Pressure Coolant Injection (LPCI) system of a decommissioned Boiling Water Reactor.

Consistency: The model demonstrated consistent reconstruction across five independent runs. Higher-level layers (Goals, Functions, Subfunctions) achieved perfect precision and recall ( $F_2 = 1.0$ ).
Granularity: Minor deviations occurred at the Component and Success Condition layers, where recall decreased slightly ( $F_2$ ranging from 0.978 to 0.993), resulting in approximately 1–6 missed nodes per run.
Structural Integrity: Link accuracy was high for top-level relationships but showed variability at the Subfunction-Component layer due to higher structural density. Gate accuracy ranged from 0.889 to 0.963 at this layer.
Batching Trade-off: An integrity score analysis revealed a trade-off between batch size and structural stability. A batch size of 1 (processing one parent element at a time) yielded the highest mean integrity score (90.38), while larger batches (up to 15) showed a gradual decline in performance, suggesting that isolating expansion steps improves structural stability.
Diagnostic Capabilities: The system successfully performed upward propagation to trace failure impacts to system goals and downward propagation to enumerate success paths, including qualitative annotations for potential Common Cause Failures (CCF) among redundant components.

Significance and Claims
The paper claims that the proposed framework transforms technical documentation into executable functional models, significantly reducing the manual effort required for DML construction (from months to days) while maintaining high structural accuracy. The significance lies not in the individual use of RAG, LLMs, or graph databases, but in their integration into a coherent, schema-constrained framework that enables grounded diagnostic reasoning.

The authors emphasize that the framework is designed to support rather than replace expert modeling. The resulting KG-DML provides an explicit knowledge representation that allows LLM agents to reason about functional dependencies and failure propagation with traceability, overcoming the limitations of unstructured LLM reasoning. The study acknowledges that while the approach is robust for the tested system, further validation is needed for larger systems, and the current implementation handles Common Cause Failures qualitatively rather than modeling them explicitly within the graph logic. The work establishes a foundation for scalable, automated functional modeling in reliability engineering.