💻 computer science

Compiling Agentic Workflows into LLM Weights: Near-Frontier Quality at Two Orders of Magnitude Less Cost

This paper argues that compiling agentic workflows directly into the weights of small, fine-tuned models ("subterranean agents") offers a cost-effective, private, and context-efficient alternative to the prevailing external orchestration frameworks, demonstrating near-frontier quality across diverse complex tasks while overcoming key adoption barriers.

Original authors: Simon Dennis, Rivaan Patil, Kevin Shabahang, Hao Guo

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Simon Dennis, Rivaan Patil, Kevin Shabahang, Hao Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Baking the Recipe into the Chef

Imagine you are trying to get a complex task done, like booking a complicated trip or fixing a broken machine. Currently, most companies use a "Manager and Worker" system (called Agent Orchestration).

The Current Way (The Manager): You have a very smart, expensive "Manager" (a frontier AI model). Every time the "Worker" (the AI talking to you) needs to make a decision, the Manager stops, reads a giant rulebook, tells the Worker what to do next, and then waits for the Worker to reply. This happens over and over. It's slow, expensive, and the Manager has to read the whole rulebook every single time.
The New Way (The Subterranean Agent): The authors propose a different approach: Compile the procedure into the weights. Imagine taking that giant rulebook and baking it directly into the Worker's brain. Now, the Worker is the expert. They don't need a Manager to tell them what to do next; they just know the flow naturally. They become a "Subterranean Agent"—an expert who works from the inside out.

The paper asks: Is baking the rules into a smaller, cheaper AI actually as good as using a giant, expensive Manager?

The Three Hurdles (And How They Cleared Them)

The authors thought people weren't doing this because of three big fears. They tested these fears in three real-world scenarios: Travel Booking, Zoom Tech Support, and Insurance Claims.

1. The Quality Fear: "Will a small brain be as smart as a big brain?"

The Fear: People thought a small, cheap AI (3 billion or 8 billion parameters) couldn't handle complex steps as well as a massive, expensive AI (like the ones used by big tech companies).
The Result: Surprisingly, the small AI did almost as well as the big one.
- In Travel, the small AI was slightly less "polite" or "graceful" than the big one, but it got the job done correctly.
- In Zoom Support and Insurance (which are much harder), they upgraded the small AI to a slightly bigger version (8 billion). This version matched the big, expensive AI on almost every metric.
- The Analogy: It's like hiring a local, experienced mechanic (the small AI) vs. flying in a world-famous racing team (the big AI). For fixing your daily car trouble, the local mechanic is 90–98% as good, but costs a fraction of the price.

2. The Cost Fear: "Is it actually cheaper to run this?"

The Fear: People thought that even if the small AI is cheap to train, running it might still be expensive because you have to pay for the computer power.
The Result: It is massively cheaper.
- The "Manager" system has to call a very expensive API for every single sentence the AI says.
- The "Baked-in" system runs on a standard computer server (self-hosted).
- The Math: The new method is 128 to 462 times cheaper per conversation.
- The Analogy: The old way is like paying a taxi driver $50 every time you ask them to turn the wheel. The new way is like buying a car for $1,000 and driving it yourself. The more you drive (the more complex the task), the more money you save.

3. The Flexibility Fear: "What if the rules change? Do I have to retrain for months?"

The Fear: People thought that if the insurance company changes a form or the travel agency changes a policy, you'd have to wait weeks to retrain the AI.
The Result: It's fast.
- Changing the rules and retraining the AI takes 30 to 50 minutes on standard hardware.
- The Analogy: It's not like rebuilding a house from scratch; it's more like updating the software on your phone. You can do it in the time it takes to brew a pot of coffee.

How It Works (The "Subterranean" Pipeline)

The process is surprisingly simple:

Draw the Map: You draw a flowchart of how the conversation should go (e.g., "Ask for dates" -> "Check budget" -> "Show options").
Practice Run: A super-smart AI (the "Teacher") plays through this map thousands of times, creating fake conversations.
Bake the Brain: You teach a smaller AI these fake conversations. The AI learns the pattern of the conversation, not just the words.
Go Live: You deploy the small AI. It doesn't need the flowchart anymore; the flowchart is now part of its memory. It talks to you directly.

The Verdict

The paper concludes that for tasks with clear steps (like booking, support, or claims), you don't need a Manager.

Persistent Structure belongs in the weights: The rules of the game should be baked into the AI's brain.
Transient State belongs in the prompt: The specific details of your trip or your broken Zoom call should be told to the AI in the conversation.

By moving the rules from the "Manager" to the "Worker's brain," you get near-perfect quality at two orders of magnitude (100x) less cost, with the ability to update the rules in under an hour.

Technical Summary: Compiling Agentic Workflows into LLM Weights

Problem Statement

Agent orchestration frameworks (e.g., LangGraph, CrewAI, OpenAI Agents SDK) have proliferated, collectively exceeding 290,000 GitHub stars. These systems rely on an external orchestrator that sits above the Large Language Model (LLM), injecting instructions and making routing decisions at every turn. While recent work suggests that for procedural tasks, simply providing the entire procedure in a frontier model's system prompt (in-context prompting) outperforms orchestration, this approach suffers from three critical limitations:

Cost: It requires a frontier model for every conversation and inflates token consumption by embedding the procedure in every API call.
Context Window: It consumes significant context window capacity.
Privacy: It exposes proprietary procedures to third-party providers.

The alternative—compiling the procedure into the weights of a small, fine-tuned model (creating a "subterranean agent")—has been explored in prior academic work (e.g., SimpleTOD, FireAct, SynTOD) but has seen low developer adoption. This paper identifies three perceived barriers to adoption: Quality (can a small model match a frontier model?), Cost (is self-hosting actually cheaper?), and Flexibility (is retraining prohibitively slow?).

Methodology

The authors propose a "subterranean" architecture where the orchestrator is used only during training data generation, not at runtime. The pipeline involves:

Procedure Definition: Representing workflows as directed graphs ( $F = (N, E, n_0, T)$ ) with nodes (agent/user turns) and edges (transitions).
Synthetic Data Generation: Traversing all valid paths through the flowchart to generate synthetic conversations using a frontier model (Claude Sonnet 4.5). The model generates turn-by-turn dialogue based on node prompt templates and full conversation history.
Full Parameter Fine-Tuning: Fine-tuning small models (3B and 8B parameters) on these synthetic conversations using full parameter updates (rejecting low-rank methods like LoRA, which the authors' prior work [Dennis et al., 2026b] found insufficient for procedural internalization).
Runtime Deployment: The compiled model runs without any external orchestrator, receiving only a minimal system prompt. The procedure is implicit in the model's weights.

Evaluation Setup:
The study evaluates three domains with increasing complexity:

Travel Booking: 14 nodes, 3 decision hubs, 86 unique paths.
Zoom Support: 14 nodes, parallel structure, requiring product-specific knowledge (UI, error codes).
Insurance Claims: 55 nodes, 6 decision hubs, 2,381 unique paths, testing scalability.

Baselines:

LangGraph Orchestrator: Uses Claude Sonnet 4.5 with explicit flowchart-based state tracking (the frontier-model baseline).
In-Context Baseline: Claude Sonnet 4.5 with the serialized flowchart in the system prompt.
Same-Model Orchestrator: A 3B model with explicit orchestration (used to isolate the effect of compilation).

Metrics:
Conversations are scored (1–5 scale) on Task Success, Information Accuracy, Consistency, Graceful Handling, and Naturalness using an LLM-as-a-judge approach (Claude Sonnet 4.5 and GPT-4.1).

Key Results

1. Quality

Compilation vs. Same-Model Orchestration: A controlled comparison on the Travel task showed that a 3B compiled model significantly outperformed a 3B model with explicit orchestration on four of five metrics (Task Success, Consistency, Graceful Handling, Naturalness; $p < 0.001$ ).
Small Model vs. Frontier Orchestrator:
- Travel (3B): The 3B compiled model matched the LangGraph orchestrator on Task Success and Consistency but lagged on Graceful Handling and Naturalness (reaching ~82% of in-context quality).
- Zoom & Insurance (8B): Scaling to an 8B model closed the gap. The 8B compiled model achieved 87–98% of the in-context frontier baseline quality. On Insurance claims, it outperformed the LangGraph orchestrator (built on a ~70× larger model) on Graceful Handling and Naturalness.
Failure Rates: Compiled models exhibited lower failure rates than the LangGraph orchestrator in Travel (5.5% vs. 24%) and Insurance (9% vs. 17%), largely because they eliminate routing errors inherent in node-level orchestration.

2. Cost

Compiled models are 128–462× cheaper per conversation than the in-context baseline.

Per-Token Reduction: Self-hosting an 8B model on commodity hardware (A100) is ~65× cheaper per token than frontier API rates.
Token Volume Reduction: The in-context baseline must include the full procedure in every prompt (overhead scaling 2× to 7× with complexity). The compiled model's prompt is constant-size.
Break-even: The one-time compilation cost (~$50–80) is recovered within 500 conversations.

3. Flexibility

The recompile cycle (data generation, fine-tuning, evaluation) takes 30–50 minutes on production hardware (8× H200) or 3–4 hours on a single A100. This is comparable to a standard CI/CD build cycle, refuting the assumption that retraining is a paradigm-shifting, prohibitively long effort.

Significance and Claims

The paper argues that the perceived barriers to compiling agentic workflows are smaller than commonly assumed:

Quality Gap: The gap between a compiled 8B model and a frontier in-context baseline is small (87–98%), and the compiled model often surpasses a frontier model orchestrated via LangGraph.
Cost Gap: The cost advantage is two orders of magnitude and grows with procedure complexity, as the compiled model's prompt size remains constant while the in-context prompt grows.
Flexibility Gap: Adaptation is a deployment cycle (CI/CD), not a retraining bottleneck.

The authors conclude that for procedural knowledge requiring persistence beyond a single conversation's context, "persistent structure belongs in the weights, transient state belongs in the prompt." Compilation is presented as the natural solution for procedural tasks, offering near-frontier quality at a fraction of the cost and latency, while avoiding the structural failure modes of external orchestration (fragmented reasoning, routing errors, and constrained conversational style).