Automatic Ontology Construction Using LLMs as an External Layer of Memory, Verification, and Planning for Hybrid Intelligent Systems

1. Problem Statement

The paper addresses the fundamental limitations of Large Language Models (LLMs) when used as standalone knowledge systems:

• Memory Constraints: LLMs rely on short-term context windows and static training weights, lacking reliable long-term memory or the ability to track state changes over time.
• Retrieval vs. Understanding: Standard Retrieval-Augmented Generation (RAG) retrieves text fragments based on vector similarity but fails to reconstruct complex relationships, causality, rules, and constraints. It acts as a "search overlay" rather than a structured knowledge system.
• Planning Deficits: Purely autoregressive LLMs struggle with multi-step planning tasks (e.g., the Tower of Hanoi) and often hallucinate constraints or fail to maintain logical consistency.
• Verification Gap: LLMs generate plausible text but lack an internal mechanism to verify facts against strict regulatory or logical rules.

The core hypothesis is that an LLM should not be treated as a self-sufficient knowledge store but as an orchestrator that interacts with an external, structured, and verifiable memory layer (an ontology).

2. Methodology and Architecture

The authors propose a neuro-symbolic architecture where the LLM operates over a dual-memory system managed via the Model Context Protocol (MCP).

A. System Components

1. LLM (The Generator): Acts as the interface, planner, and interpreter. It does not store knowledge directly but queries external layers.
2. Vector RAG Storage: Handles unstructured text retrieval (embeddings) for broad recall and context.
3. Ontological Graph (RDF/OWL): The core "structured memory." It stores entities, relationships, types, and axioms.
   • SPARQL: Used for precise graph querying.
   • SHACL: Used for validating data shapes and constraints.
   • Reasoning Engines: Used for logical inference and consistency checking.
4. Agent Layer: Specialized agents handle retrieval, validation, planning, and execution loops.
5. MCP (Orchestrator): A universal communication bus connecting the LLM to the graph, vector DB, APIs, and tools.

B. The Ontology Builder Pipeline

The paper details a closed-loop pipeline for automatically constructing and updating the ontology:

1. Ingest & Segmentation: Heterogeneous sources (PDFs, logs, APIs) are chunked.
2. NER & Entity Typing: LLMs extract entities and assign preliminary types (treated as hypotheses).
3. Relation Extraction: LLMs identify semantic predicates (e.g., causes, located-in, prohibited-by).
4. Normalization & Alignment: Resolves synonyms, homonyms, and merges duplicates to ensure stable identifiers.
5. Triple Construction: Converts extracted data into Subject-Predicate-Object (TTL/RDF) format.
6. Validation:
   • Structural: SHACL checks for shape constraints.
   • Logical: OWL reasoners check for contradictions and infer new facts.
7. Feedback Loop: System responses and dialogue logs are re-processed. New facts are extracted, validated, and if they pass, added to the graph; if they fail, they are logged but not trusted.

3. Key Contributions

• Formalization of "Ontology as Memory": The paper shifts the paradigm from "RAG as retrieval" to "Ontology as an operating system" for state accumulation, enabling long-term memory, explainability, and contradiction control.
• Hybrid Architecture Design: A concrete blueprint integrating MCP, GraphRAG, and symbolic reasoning (SHACL/OWL) to separate generation (LLM) from verification (Symbolic Layer).
• Automated Pipeline: A defined engineering workflow for the "Ontology Builder" that moves from raw text to validated TTL/RDF artifacts, suitable for CI/CD-like versioning.
• Two-Loop Verification System: A mechanism where the LLM generates an answer, and the ontology/validator independently verifies it against formal rules, producing a "supported" or "contradicted" verdict.

4. Results

The empirical evaluation relies on reconstructed project materials (charts and screenshots) rather than a fully disclosed external benchmark suite.

A. Quantitative: Tower of Hanoi Benchmark

The system was tested on planning tasks with varying complexity (number of disks).

• 3 Disks: Success rate increased from 26.3% (Baseline LLM) to 33.3% (Ontology-augmented).
• 4 Disks: No difference (33.3% for both).
• 5 Disks: Significant improvement from 33.3% to 45.5%.
• 6 Disks: Both configurations failed (0.0%).
• Interpretation: The ontological layer provides the most benefit in the "middle complexity zone," where the task exceeds trivial local search but is not yet intractable. It helps the LLM maintain plan fidelity through symbolic constraints.

B. Qualitative: Fact Analyzer Case Study

A regulatory question regarding clinical trial initiation (IND submission rules) was tested.

• Baseline LLM: Generated a negative answer, which was flagged as CONTRADICTED by the fact-checker.
• Ontology-Augmented System: Correctly identified that the IND becomes effective after 30 days in the absence of a clinical hold, providing a structured, rule-backed answer.
• Significance: Demonstrates that formalized knowledge allows for strict rule-based verification, distinguishing between "plausible text" and "factually correct answers."

5. Significance and Conclusion

The paper argues that while LLMs are powerful language engines, they require an external structured world model to function as reliable intelligent agents.

• Shift in Paradigm: The system moves from a "search overlay" (RAG) to a "stateful model" (Ontology) capable of long-term learning and reasoning.
• Explainability: By separating generation from verification, the system can trace the path from an answer back to specific facts and rules.
• Agent Viability: This architecture enables agents that can operate on real-world artifacts (contracts, states, processes) rather than just generating text.
• Limitations: The authors acknowledge that automatic ontology construction is not error-free (risks of schema drift and hallucination) and requires human-in-the-loop governance for schema management. The current results are descriptive proof-of-concept rather than definitive statistical proof due to limited disclosure of experimental parameters.

Final Thesis: An LLM combined with an automatically constructed, validated ontological graph transforms a probabilistic language model into a robust, verifiable, and long-term knowledge system.