Technical Summary: Declarative Skills for AI Agents in Knowledge-Grounded Tool-Use Workflows

Problem Statement

The paper addresses the orchestration of AI agents in realistic, complex customer-service workflows that rely on unstructured knowledge bases (KBs). Specifically, the authors focus on the $\tau$-Knowledge benchmark, a domain involving fintech customer service where agents must perform three distinct competencies: conversational interaction, procedural task execution (ordering constraints, verifications), and reasoning over retrieved documents.

The core challenge is that agents must discover tool capabilities and procedural rules within a large, unstructured KB (e.g., finding that "close bank account" requires a specific random suffix and preconditions) before executing them. The environment is modeled as a Decentralised Partially-Observable Markov Decision Process (Dec-POMDP), where the agent has an incomplete view of the database state and must rely on tool outputs and user messages to achieve a target state. The authors identify that current agents struggle with:

1. Complex interdependencies requiring multi-hop reasoning.
2. Failure to respect implicit subtask ordering (e.g., verifying identity before changing data).
3. Over-trusting user assertions without tool verification.
4. Search inefficiency and unwarranted assumptions.

Methodology

The study compares three distinct policy classes (orchestration paradigms) instantiated across five large language models (LLMs) under two retrieval regimes (Golden retrieval and Embedding-based retrieval).

1. Baseline Agent ($\pi_B$)

A standard unscaffolded LLM agent conditioned only on a static system prompt and conversation history. It has no explicit control flow or procedural priors.

2. Declarative Agent ($\pi_D$)

This agent extends the baseline by appending natural-language skill files to the system prompt. These files act as a procedural prior, describing:

• Banking Procedures: Pre-conditions, tool discovery requirements, and ordering constraints (e.g., "credit limit increase requires no pending disputes").
• Customer Interaction: A descriptive four-step conversational structure (Greeting, Triage, Verification, Action).
• Knowledge Discovery: Strategies for searching the KB, emphasizing that tool discovery is a hard precondition for state-changing tools.
  The LLM reads these files at runtime and decides the workflow, tools, and verification strategy dynamically without an explicit state machine.

3. Imperative Agent ($\pi_I$)

This agent implements a deterministic finite-state machine (Externalised Control). It enforces a rigid phase graph: Greeting $\to$ Triage $\to$ Verification $\to$ Planning $\to$ Execution $\to$ Confirmation.

• Control Flow: Phase transitions are computed by code, not the LLM.
• Constraints: The action space is restricted at each turn ($A_{\phi_t} \subset A$). For example, state-mutating tools are only allowed in the Execution phase, which is gated by a successful Verification flag.
• Mechanisms: Includes explicit task queues (Kahn's algorithm for topological sorting), per-tool retry policies with deterministic escalation to an Escalate phase, and strict response-type enforcement.

Theoretical Framework

The authors formalize the agents within the Dec-POMDP framework and propose four theoretical propositions:

• Prop 1: Skill files provide an information advantage (reducing conditional entropy of actions) for models with a procedural-competence gap.
• Prop 2: Imperative restriction shrinks the policy class; success depends on whether compliance gains offset the loss of model capacity.
• Prop 3: Imperative agents should reduce cost by bounding trajectory length via retry limits.
• Prop 4: Retrieval noise degrades the observation channel; if the channel is too noisy, the benefit of skill files collapses.

Key Results

1. Impact of Orchestration on Performance

• Declarative Advantage: Under Golden Retrieval (high-quality evidence), the DeclarativeAgent consistently outperformed the Baseline on four of five models, with gains scaling with the model's procedural-competence gap. It improved Pass1 (task success) and Write-Argument Accuracy.
• Imperative Underperformance: The ImperativeAgent underperformed the Baseline across all models and retrieval settings. The restriction of the action space reduced the model's capacity to solve tasks without providing sufficient compliance benefits to offset the loss.
• Retrieval as Bottleneck: Under Embedding Retrieval (noisy), performance dropped sharply for all agents. The advantage of the DeclarativeAgent collapsed for weaker models, confirming that skill files cannot compensate for fundamentally incorrect or missing evidence.

2. Compliance and Efficiency

Contrary to the hypothesis that deterministic gates improve safety:

• Unauthorized Writes: The ImperativeAgent did not reduce the rate of unauthorized writes (writes before verification) compared to the Baseline. In some cases (embedding retrieval), it had a higher rate of pre-verification writes.
• Brittleness: The ImperativeAgent exhibited a significantly higher over-retry rate (4–7$\times$ higher than Baseline/Declarative) and longer trajectory lengths. This suggests that rigid code-based phase transitions are brittle; if the LLM misclassifies a phase, it can bypass gates or loop indefinitely, whereas adaptive LLM-based agents handle errors more gracefully.
• Cost: The ImperativeAgent did not achieve the predicted cost savings due to increased trajectory lengths and retry loops.

Significance and Claims

The paper claims the following contributions and insights:

1. Declarative Skills as Effective Priors: Natural-language skill files are an effective orchestration paradigm for tool-using agents, particularly when the underlying model has a procedural-competence gap. They improve accuracy and reduce orchestration errors without the brittleness of state machines.
2. Limitations of Imperative Orchestration: The study empirically refutes the assumption that deterministic state machines guarantee compliance in complex, unstructured environments. The rigidity of code-based control can lead to phase mis-classification and failure to recover from errors, offering no safety advantage over adaptive agents in this context.
3. Retrieval Dominance: Retrieval quality is the dominant bottleneck for AI agents. No orchestration paradigm (declarative or imperative) can recover performance when the observation channel (retrieved evidence) is noisy or incomplete.
4. Engineering Implications: For regulated domains, relying on deterministic gates for compliance guarantees may be misleading. Skill-augmented prompts offer a maintainable, low-cost method to enhance agent behavior, provided retrieval quality is high.

The authors conclude that while declarative orchestration offers measurable accuracy improvements, the field must prioritize advances in retrieval alongside orchestration to unlock the commercial potential of agentic systems.