🤖 AI

In-Context Prompting Obsoletes Agent Orchestration for Procedural Tasks

This paper argues that for procedural multi-turn tasks, simpler in-context prompting where the model self-orchestrates outperforms complex external agent orchestration frameworks, rendering the latter obsolete for frontier models.

Original authors: Simon Dennis, Michael Diamond, Rivaan Patil, Kevin Shabahang, Hao Guo

Published 2026-05-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Simon Dennis, Michael Diamond, Rivaan Patil, Kevin Shabahang, Hao Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to get a complex task done, like booking a complicated family vacation, fixing a broken Zoom call, or filing an insurance claim. In the world of Artificial Intelligence (AI), there are two main ways to tell a smart computer how to do this.

This paper argues that the "fancy, complicated" way is actually making things worse, and the "simple, direct" way is much better.

Here is the breakdown using simple analogies:

The Two Approaches

1. The "Manager" Approach (Agent Orchestration)
Think of this like a strict factory floor. You have a very smart worker (the AI), but you also have a Manager standing over their shoulder.

The Manager has a flowchart (a map of steps).
The Manager tells the worker exactly what to say for Step 1.
Once the worker finishes Step 1, the Manager checks the work, decides if they should move to Step 2 or Step 3, and then whispers the instructions for the next step.
The worker never sees the whole map; they only see the tiny instruction for right now.
The Problem: The Manager is slow, makes mistakes in deciding the next step, and sometimes forgets what the worker said three steps ago because they only focus on the current instruction.

2. The "Self-Guided" Approach (In-Context Prompting)
Think of this like giving the same smart worker a complete instruction manual and a map of the whole journey before they start.

The worker reads the whole manual.
They know exactly where they are, where they need to go, and what to do next.
They don't need a Manager to tell them what to do; they guide themselves.
The Benefit: They see the big picture, remember the whole conversation, and flow naturally without stopping to ask for permission at every turn.

What the Researchers Did

The authors tested these two approaches using a very smart AI (called "Claude Sonnet") on three different real-world scenarios:

Travel Booking: Planning a trip for a group with a specific budget and interests.
Tech Support: Fixing a Zoom audio issue.
Insurance Claims: Filing a claim after hitting a deer with a car.

They ran 200 simulations for each scenario using both the "Manager" style and the "Self-Guided" style. Then, they had another AI act as a judge to grade the conversations on things like:

Did they finish the task?
Was the information accurate?
Did the conversation feel natural, or robotic?
Did the AI get confused or repeat itself?

The Results: The "Self-Guided" Worker Wins

The results were clear and consistent across all three scenarios:

Better Quality: The "Self-Guided" AI (In-Context) scored higher on every single metric. It finished tasks more often, remembered details better, and sounded more like a helpful human.
Fewer Mistakes: The "Manager" approach failed much more often.
- In travel booking, the Manager failed 24% of the time. The Self-Guided AI only failed 11.5%.
- In tech support, the Manager failed 9% of the time. The Self-Guided AI failed only 0.5% (almost never).
- In insurance, the Manager failed 17% of the time. The Self-Guided AI failed 5%.
Why the Manager Failed: Because the Manager only looked at one step at a time, the AI often forgot important details the user mentioned earlier (like the budget being "per person" instead of "total"). It also got stuck in loops, asking the same question over and over because it lost track of where it was in the process.
Speed and Cost: The "Manager" approach actually took more computer power and time because the Manager had to stop and think at every single decision point. The "Self-Guided" approach was faster and smoother, even though it required a slightly longer "manual" to be loaded into the AI's memory at the start.

The Bottom Line

The paper concludes that for tasks that follow a clear set of rules (like booking a trip or filing a claim), we don't need a Manager anymore.

In the past, AI models were "dumber" and needed a Manager to hold their hand through every step. But today's AI models are so smart that if you just give them the whole rulebook at the start, they can figure out the rest on their own. Adding a Manager actually gets in the way, causing confusion and errors.

The simple takeaway: If you have a clear procedure, just put the whole procedure in the AI's instructions and let it do the work. Don't build a complex system to manage it.

1. Problem Statement

The paper addresses the growing complexity and reliability issues associated with Agent Orchestration Frameworks (e.g., LangGraph, CrewAI, Google ADK, OpenAI Agents SDK). These frameworks typically employ an external controller (orchestrator) that sits above the Large Language Model (LLM). The orchestrator tracks conversation state, manages a directed graph of workflow steps, and injects specific routing instructions or node-specific prompts at every turn.

The authors argue that this architecture introduces significant engineering overhead, cascading errors, routing failures, and poor cross-trial consistency. They hypothesize that for procedural tasks (conversations following a defined workflow from intake to resolution), modern frontier LLMs possess sufficient reasoning capabilities to self-orchestrate when provided with the entire procedure in their context, rendering external orchestration unnecessary and potentially harmful.

2. Methodology

The study employs a controlled empirical comparison between two conditions using the same frontier model (Claude Sonnet 4.5) across three distinct domains with varying complexity:

Travel Booking: 14 nodes, 3 decision hubs.
Zoom Technical Support: 14 nodes, 3 decision hubs (requires technical accuracy).
Insurance Claims Processing: 55 nodes, 6 decision hubs (high complexity with nested loops).

Experimental Conditions:

Condition A (LangGraph Orchestrated): The workflow is implemented as a directed graph in LangGraph. The orchestrator manages state, makes routing decisions via additional LLM calls at decision hubs, and injects node-specific prompt templates into the system prompt for each turn.
Condition B (In-Context Baseline): The model receives the entire serialized flowchart (all nodes, edges, conditions, and terminal states) in its system prompt. The model self-orchestrates, determining the next step and routing internally without external state tracking or routing calls.

Evaluation Setup:

Data: 200 simulated conversations per condition per domain ( $N=1,200$ total).
Simulator: A user simulator (powered by Claude Sonnet) generates dynamic responses based on the conversation history, unaware of the underlying flowchart.
Judging: Conversations were evaluated by an LLM-as-Judge (initially Claude Sonnet, replicated with GPT-4.1) on five criteria:
1. Task Success (Procedural completion)
2. Information Accuracy
3. Consistency (State coherence)
4. Graceful Handling (Adaptability to ambiguity/edge cases)
5. Naturalness (Human-like interaction)

3. Key Contributions

Empirical Evidence of Obsolescence: Demonstrates that for procedural tasks, a simple in-context prompting approach consistently outperforms complex orchestration frameworks across all quality metrics.
Failure Mode Analysis: Identifies that orchestration introduces specific structural failure modes (routing errors, fragmented reasoning, template conflicts) that do not exist in the in-context approach.
Efficiency Analysis: Quantifies the trade-off between token usage (higher in-context due to procedure inclusion) and API call efficiency (lower in-context due to lack of routing overhead).
Robustness Check: Validates findings using an independent judge (GPT-4.1) to rule out self-preference bias in the primary judge.

4. Results

A. Quality Performance

The in-context baseline significantly outperformed the LangGraph orchestrator in 15 out of 15 comparisons (3 domains $\times$ 5 metrics) under the Claude judge, with all differences statistically significant ( $p < 0.005$ ).

Scores: In-context scores ranged from 4.53 to 5.00, while LangGraph scores ranged from 4.17 to 4.84.
Consistency & Graceful Handling: The gap was most pronounced here. The in-context model maintained near-perfect coherence (4.83–4.99) and adaptability (4.96–5.00), whereas the orchestrated system struggled (4.32–4.55 and 4.38–4.62, respectively).
Replication: Using GPT-4.1 as a judge confirmed the core findings: 11 of 15 comparisons favored in-context prompting, with zero favoring orchestration. The "Naturalness" gap diminished in the GPT-4.1 evaluation, suggesting the Claude judge may have a bias toward unconstrained text.

B. Failure Rates

The orchestrated system failed significantly more often (defined as Task Success $\le$ 3):

Travel: 24.0% failure (LangGraph) vs. 11.5% (In-Context).
Zoom: 9.0% failure (LangGraph) vs. 0.5% (In-Context).
Insurance: 17.0% failure (LangGraph) vs. 5.0% (In-Context).
Root Causes: Orchestrated failures were attributed to routing errors (misclassifying conversation state at decision hubs), information loss across node boundaries, and infinite loops caused by the agent losing track of completed phases.

C. Efficiency and Cost

API Calls: Orchestration required 1.2x to 1.7x more LLM calls per conversation due to routing decisions at decision hubs.
Token Usage: In-context prompting consumed more tokens per conversation (e.g., 68K vs. 43K for Insurance) because the serialized procedure was included in every API call.
Cost: Despite higher token usage, the in-context approach was only 1.3x–1.4x more expensive per conversation. The cost premium is modest compared to the significant gains in quality and reliability.

5. Significance and Implications

Paradigm Shift: The paper suggests that for well-defined procedural tasks, the "scaffolding" provided by orchestration frameworks is no longer necessary for frontier models. In fact, this scaffolding actively degrades performance by fragmenting the model's reasoning and introducing points of failure.
Engineering Recommendation: Practitioners should prioritize putting the entire procedure in the system prompt before resorting to complex orchestration frameworks. This simplifies the stack, reduces latency (fewer API calls), and improves reliability.
Limitations & Future Work:
- The findings apply specifically to procedural tasks with defined workflows. Non-procedural tasks (creative, open-ended) or multi-model pipelines may still benefit from orchestration.
- The approach requires the procedure to fit within the model's context window (though 55-node procedures used only ~2% of a 200K token window).
- For high-volume production, the higher token cost of in-context prompting might motivate fine-tuning the procedure into smaller models as a cost-effective alternative.

Conclusion: The authors conclude that as frontier models improve, the value of external orchestration for procedural tasks diminishes. The "bottleneck" is not better orchestration; orchestration itself is the bottleneck.