🤖 AI

Toward Effective and Reliable LLM Agents via Dynamic Ontology

The paper introduces OaK, a framework that dynamically constructs and refines task-oriented ontologies to explicitly ground knowledge and relations, thereby enhancing the evidence utilization and reliability of multi-step reasoning in large language model agents.

Original authors: Xiaohui Zhang, Zequn Sun, Chengyuan Yang, Yuanning Cui, Lingbing Guo, Wei Hu

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaohui Zhang, Zequn Sun, Chengyuan Yang, Yuanning Cui, Lingbing Guo, Wei Hu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a specific type of software known as a "large language model" has emerged as a powerful tool for understanding and generating human language. These models are often used to build "agents," which are programs designed not just to chat, but to perform complex tasks by breaking them down into smaller steps, such as searching for information, using digital tools, and making decisions based on what they find. However, as these agents attempt longer and more complicated jobs, they often struggle. They can lose track of the facts they need, make logical leaps that don't hold up, or get stuck in loops of error because they rely too heavily on the vast, unstructured knowledge stored inside their own memory. To solve this, researchers are exploring ways to give these agents a clearer, more organized map of the world they are navigating, ensuring that every step they take is grounded in reliable evidence rather than guesswork.

A team of researchers at Nanjing University and other institutions has developed a new framework called OaK to address these reliability issues. Their work focuses on a concept called an "ontology," which is essentially a structured dictionary of concepts and the relationships between them, written in a way that computers can strictly interpret. While traditional methods for creating these structures require experts to manually define every rule—a slow and difficult process—the new system automates the creation of these maps specifically for the task at hand. The researchers found that by building a custom, dynamic structure for each job, they could significantly improve how well an agent plans and executes multi-step tasks, making the process more transparent and less prone to failure.

The core of the OaK system is a two-stage process that happens before the agent ever attempts a real-world task. First, the system looks at a set of training examples and the specific requirements of a job, such as planning a trip or managing customer data. It then automatically drafts a schema, which is a blueprint defining the important types of information involved, like "hotels," "flights," or "budgets," and how they connect. This blueprint is not just a list; it is rigorously checked by a logical reasoner to ensure there are no contradictions or missing rules. If the blueprint is flawed, the system fixes it before moving on. Once the blueprint is solid, the system builds a knowledge graph, a network of actual data points pulled from the training materials, organized exactly according to the rules of the blueprint.

With this structured foundation in place, the system creates a set of specialized tools, or functions, that the agent can use. Instead of asking the agent to figure out how to search for a flight, filter by price, and check availability all by itself, the system provides pre-built tools that perform these specific actions correctly. The agent then uses these tools to solve problems, guided by the strict rules of the blueprint. If the agent makes a mistake or the task fails, a "judge" model reviews the entire process, identifies exactly where the blueprint or the tools went wrong, and suggests improvements. This cycle of building, testing, and refining repeats several times until the system is robust enough to handle unseen questions. The result is a "kernel," a frozen, reliable interface that the agent uses to interact with data, ensuring that every decision is backed by evidence and that the agent cannot accidentally invent facts or ignore constraints.

The researchers tested this approach on three very different types of challenges to see if it held up in the real world. The first was a travel planning task where an agent had to arrange flights, hotels, meals, and activities while adhering to strict budget and schedule constraints. The second involved managing customer relationships in a business setting, requiring the agent to follow complex company policies and retrieve specific records. The third tested the ability to combine information from various external sources, like flight databases and restaurant reviews, to answer detailed questions. In every case, the OaK system outperformed standard methods. On the travel planning task, for instance, the system improved the success rate of completing a full, valid itinerary from roughly 15 percent to nearly 56 percent when using one of the base models, and from about 4 percent to nearly 20 percent with another. It also showed significant gains in the business and data tasks, particularly in scenarios where the agent needed to follow strict rules or combine data from multiple tables.

The study also revealed why these improvements happened. When the researchers removed the custom-built tools or stopped the system from refining its blueprint over multiple rounds, performance dropped sharply. This indicated that simply giving the agent access to data was not enough; the agent needed the structured rules and the pre-packaged tools to coordinate its actions effectively. The system worked by making the agent's reasoning process visible and constrained, preventing it from wandering off into unsupported assumptions. While the new method required slightly more time to run and produced more output text than some simpler approaches, it used fewer input tokens, suggesting that the structured interface made the agent more efficient at processing the information it needed. The researchers concluded that by turning domain knowledge into an executable, self-correcting interface, they could make artificial intelligence agents more reliable, trustworthy, and capable of handling the complex, multi-step reasoning required in real-world applications.

Technical Summary: OaK – Toward Effective and Reliable LLM Agents via Dynamic Ontology

Problem Statement

Large Language Model (LLM) agents currently rely heavily on knowledge encoded within model parameters or presented as unstructured context. In domain-specific tasks, this reliance leaves critical semantic connections implicit, leading to incomplete evidence utilization and brittle multi-step decision-making. While ontologies offer a mechanism to externalize domain concepts and relations into machine-interpretable structures, traditional ontology construction is labor-intensive, requiring substantial expert effort and lacking scalability. Furthermore, automatic construction is challenging; an ontology that appears semantically plausible may still lack the specific relational structures required for actual decision-making.

The core challenge identified is that current agent architectures (e.g., ReAct, reflection-based loops) improve planning and memory reuse but leave the set of admissible concepts and action sequences implicit. Consequently, agent behavior lacks an enforceable contract regarding what it may execute or how results should be verified, making it difficult to trace execution failures or ensure outputs are grounded in supporting evidence.

Methodology: The OaK Framework

The authors propose OaK (Ontology-as-a-Kernel), a framework that dynamically constructs and refines task-oriented ontologies to serve as a semantic and procedural contract for LLM agents. OaK operates in two distinct stages: a construction phase on training data and an inference phase on unseen queries.

1. The Ontology Kernel

The framework packages the task interface into a kernel $K = (S, F)$ , where:

$S$ (Schema): A task-oriented schema defining available domain concepts, their properties, relations, and constraints.
$F$ (Functions): A set of typed reasoning functions defining executable procedures for retrieval, filtering, traversal, projection, aggregation, and multi-step reasoning.

2. Construction Loop

The construction stage runs a four-step iterative refinement loop over the training data:

Step 1: Schema Construction: An LLM analyzes task descriptions and training samples to draft a requirement specification ( $R$ ). It then drafts a schema ( $S_t$ ) based on $R$ and feedback from the previous round. This draft undergoes formal verification by encoding it as an OWL ontology and running the HermiT reasoner to check for logical validity (disjointness, restrictions, consistency, and unsatisfiable classes). Invalid schemas are repaired based on counterexamples.
Step 2: Knowledge Graph Instantiation: Using the validated schema, OaK instantiates a Knowledge Graph ( $G_t$ ) from the reference corpora. This employs a "chunk–map–merge" pipeline where an LLM extractor maps text chunks to typed entities and relations, followed by a merge operator that resolves duplicates using schema-declared primary keys to create canonical nodes.
Step 3: Knowledge Reasoning (Function Composition): A composer LLM assembles a catalog of domain-adapted functions ( $F_t$ ) by composing generic operators (e.g., entity lookup, constraint filtering) based on recurring reasoning patterns observed in the training queries. These functions are tested on $G_t$ to ensure executability.
Step 4: Ontology Evaluation: A ReAct agent executes the training queries using the current kernel. A judge model reviews the execution trajectories, task scores, and the kernel itself to generate repair suggestions ( $\psi_t$ ). These suggestions target specific artifacts (entity types, relations, or functions) for addition, deletion, or modification, feeding back into Steps 1 and 3 for the next iteration.

3. Inference Phase

Once the loop concludes, the kernel is frozen ( $S^*, F^*$ ). For unseen test queries, the agent operates strictly within this verified semantic space. It instantiates a query-specific graph ( $G_q$ ) and solves the task by invoking the frozen functions as tools, binding typed arguments according to the schema. This restricts the agent to schema-constrained, executable compositions, preventing open-ended generation of unsupported inferences.

Key Contributions

Ontology as a Kernel: The introduction of a framework that couples a task-oriented schema with typed reasoning functions and a schema-guided evidence graph, serving as a controlled interface between LLMs and domain tasks.
Automated Construction Pipeline: A system that automatically constructs a verified schema, instantiates its knowledge graph, and compiles generic operators into executable domain functions without manual expert intervention.
Judge-Driven Refinement: A mechanism that diagnoses and repairs schema and function failures across construction rounds using official task scores and execution trajectories.
Empirical Validation: Comprehensive experiments demonstrating consistent performance gains across diverse benchmarks and LLM backbones.

Experimental Results

OaK was evaluated on three benchmarks: TravelPlanner (multi-step planning), CRMArenaPro (CRM workflow execution), and ToolQA (compositional tool use over heterogeneous corpora). The framework was tested using two LLM backbones: DeepSeek-v4-flash and GPT-4o-mini.

TravelPlanner: OaK achieved the highest final pass rates and led all macro-level metrics. It outperformed strong baselines (ReAct, AFlow, MemP, ReCode, AgentSquare) by effectively coordinating cross-component constraints (dates, budgets, transportation) through its explicit schema and integrated functions.
CRMArenaPro: OaK achieved the highest average scores in both B2B and B2C settings, particularly excelling in Policy and Database tasks. The schema-grounded execution reduced the burden on the backbone model for complex data reasoning and record selection.
ToolQA: OaK secured the highest weighted-average performance, demonstrating superior ability to compose operations over heterogeneous corpora by mapping questions to typed, reusable procedures constrained by the domain schema.

Ablation Studies confirmed the necessity of each component:

Removing Function Composition caused the largest performance degradation, as agents struggled to reconstruct dependent operations for each query.
Removing the Function Module eliminated reusable procedures, forcing agents to reason directly over the full graph.
Removing Iterative Refinement (single-round construction) resulted in significantly lower performance, highlighting the need for judge-guided repairs to fix missing constraints and reasoning gaps.

Cost Analysis: While OaK incurs higher runtime and output token costs due to graph instantiation, it achieves higher task performance with fewer input tokens compared to baselines like ReAct and MemP, suggesting that schema-adapted functions reduce the need for large context reconstruction.

Significance and Claims

The paper claims that OaK addresses the reliability gap in LLM agents by making domain semantics and executable behavior explicit. By acting as a "semantic and procedural contract," the ontology kernel constrains agent actions to ensure outputs connect to supporting evidence and that faulty steps can be identified and fixed.

The authors emphasize that OaK complements existing agent control mechanisms (like reflection and memory) by structuring the task interface itself. The results suggest that explicitly defining the interface between task requirements and agent execution improves both the effectiveness and inspectability of LLM agents in complex, multi-step reasoning tasks. The paper acknowledges limitations, noting the cost of graph instantiation and dependence on LLM-based extractors and judges, suggesting future work in reusable graphs and richer operator libraries.