💬 NLP

Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks

This paper presents a comprehensive evaluation of prompt caching across three major LLM providers for long-horizon agentic tasks, demonstrating that strategic caching techniques—such as excluding dynamic tool results and managing prompt structure—can reduce API costs by 41–80% and improve time to first token by 13–31% compared to naive full-context caching.

Original authors: Elias Lumer, Faheem Nizar, Akshaya Jangiti, Kevin Frank, Anmol Gulati, Mandar Phadate, Vamse Kumar Subbiah

Published 2026-02-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Elias Lumer, Faheem Nizar, Akshaya Jangiti, Kevin Frank, Anmol Gulati, Mandar Phadate, Vamse Kumar Subbiah

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a brilliant but expensive research assistant to solve a complex mystery. This assistant needs to read a massive instruction manual (the "system prompt") before they start working. As they work, they make dozens of phone calls to different sources, get answers, and write them down in a notebook.

Every time they make a new phone call, they have to re-read the entire instruction manual from the very first page to remember what to do. This takes a lot of time and costs a lot of money because the assistant charges by the page they read.

"Prompt Caching" is like giving that assistant a magical highlighter. If the instruction manual hasn't changed, the assistant can skip re-reading the first 90% of the book and jump straight to the new part where the phone call results are. This saves time and money.

However, this paper asks a tricky question: Does this magic highlighter always work, or can we mess it up?

The researchers tested this with three major "assistant" companies (OpenAI, Anthropic, and Google) using a benchmark called DeepResearch Bench, where agents go on long, multi-step web searches to answer hard questions. They tried three different ways of using the highlighter:

The "Naive" Approach (Full Context): Highlighting everything, including the new phone call results.
The "Sticky Note" Approach (System Prompt Only): Highlighting only the instruction manual, but leaving the new phone call results un-highlighted.
The "Clean Break" Approach (Exclude Tool Results): Highlighting the manual and the phone calls, but putting a special "stop" sign after every new result so the highlighter doesn't accidentally grab the next person's messy notes.

What They Found

1. It Saves a Lot of Money (The "Wallet" Win)
No matter which method they used, the magic highlighter saved a huge amount of money. Depending on the company, they saved between 41% and 80% on the bill.

Analogy: It's like realizing you don't need to buy a new library card every time you walk into the library; you just use the one you already have.

2. Speed is Tricky (The "Traffic" Problem)
While saving money was easy, saving time was harder.

The Surprise: Sometimes, highlighting everything (the Naive approach) actually made the assistant slower.
Analogy: Imagine a delivery driver who tries to memorize the address of every single package they deliver. If they try to memorize a package that is only being delivered once and never again, they waste time memorizing it. When the next package comes, they have to "un-memorize" the old one to make room for the new one. This "writing to memory" slows them down.
The paper found that if you only highlight the stable parts (the instruction manual) and leave the messy, changing parts (the phone call results) alone, the assistant stays fast.

3. The "Don't Break the Cache" Rule
The biggest lesson is about where you draw the line.

If you put dynamic, changing information (like "Current Time: 2:00 PM" or "User ID: 123") inside the instruction manual, the magic highlighter breaks. The system thinks the manual has changed, so it stops highlighting and starts reading from scratch.
The Fix: Keep the instruction manual pure and static. If you must include changing info, put it at the very end of the manual, like a sticky note on the last page. This way, the first 99% of the manual is still highlighted and reusable.

The Bottom Line

For companies building AI agents that do long, complex tasks:

Yes, use prompt caching. It will drastically cut your costs.
But, be smart about it. Don't just turn it on for everything.
The Best Strategy: Only cache the stable "instruction manual" (System Prompt). Let the changing parts (tool results) flow freely without being cached. This gives you the best mix of low cost and fast speed.

If you try to cache the changing parts, you might end up paying for the "writing" of the cache without getting the "reading" benefits, which can actually slow your system down.

Technical Summary: Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks

Problem Statement

Recent advancements in Large Language Model (LLM) agents have enabled complex, multi-turn agentic tasks involving extensive tool calling, such as deep research and autonomous coding. These workflows often span dozens of API calls with context windows accumulating tens of thousands of tokens, resulting in significant operational costs and latency overhead. While major LLM providers (OpenAI, Anthropic, Google) offer prompt caching features to reuse previously computed key-value (KV) tensors and reduce redundant computation, the specific benefits of these features for agentic workloads remain underexplored. Existing literature focuses on inference-level memory management or security auditing, but lacks quantitative analysis of cost savings or comparative evaluations of caching strategies for dynamic, multi-turn agent sessions.

Methodology

The authors conducted a comprehensive evaluation of prompt caching across three major LLM providers: OpenAI, Anthropic, and Google. The study utilized four flagship models: GPT-5.2, GPT-4o (OpenAI), Claude Sonnet 4.5 (Anthropic), and Gemini 2.5 Pro (Google).

Experimental Setup:

Benchmark: The evaluation used the DeepResearch Bench, a multi-turn agentic benchmark where agents autonomously execute real-world web search tool calls to answer complex, PhD-level research questions.
Scale: The study spanned over 500 agent sessions with 10,000-token system prompts. Each model was tested with 40 independent sessions per cache condition.
Cache Strategies: The authors implemented and compared four distinct conditions to control cache boundaries precisely using unique identifiers (UUIDs):
1. No Cache (Baseline): A UUID is prepended to the system prompt to force recomputation of all tokens, simulating a scenario with dynamic content (e.g., timestamps) breaking the cache.
2. Full Context Caching: No UUIDs are added, allowing the provider's automatic caching mechanism to operate on the entire conversation history, including dynamic tool results.
3. System Prompt Only Caching: A UUID is appended to the end of the system prompt, ensuring only the static system instructions are cached, while dynamic conversation history and tool results are recomputed.
4. Exclude Tool Results Caching: UUIDs are appended after the system prompt and after each tool result, explicitly preventing dynamic tool outputs from entering the cache.

Metrics:
The study measured API cost (distinguishing between standard, cached read, and cache write tokens) and Time to First Token (TTFT) as the primary latency metric. Statistical significance was determined using independent samples t-tests ( $\alpha = 0.05$ ).

Key Contributions

First Quantitative Evaluation: This work provides the first comprehensive quantification of prompt caching benefits specifically for long-horizon agentic tasks across multiple providers and models.
Strategy Comparison: It compares naive full-context caching against strategic boundary control methods (system prompt only, excluding tool results), revealing that "naive" approaches can be counterproductive.
Provider Discrepancies: The study identifies significant differences in how providers implement caching, particularly regarding how they handle dynamic content and the resulting impact on latency versus cost.
Ablation Insights: Through ablation studies across prompt sizes (500–50,000 tokens) and tool call counts (3–50), the authors demonstrate universal linear cost and TTFT benefits after meeting provider-specific token minimums.

Results

The evaluation yielded three primary findings:

Substantial Cost Reductions: Prompt caching consistently reduced API costs by 41% to 80% across all providers and models. The primary driver of these savings was the caching of the large, stable system prompt.
Variable Latency Improvements: TTFT improvements ranged from 13% to 31% depending on the provider and strategy. However, latency gains were not uniform; for some models (e.g., GPT-4o), full-context caching resulted in a latency regression of 8.8%.
Strategic Boundary Control is Critical: The study found that strategic prompt cache block control outperforms naive full-context caching.
- System Prompt Only and Exclude Tool Results strategies provided the most consistent benefits.
- Full Context Caching often introduced overhead by caching dynamic tool calls and results that are not reused in subsequent sessions, paradoxically increasing latency.
- For GPT-4o, the best strategy (System Prompt Only) improved TTFT by 30.9%, whereas Full Context Caching degraded performance.

Significance and Claims

The paper claims to provide nuanced guidance for implementing prompt caching in production agentic systems. The authors emphasize that while providers abstract much of the caching mechanism, practitioners must exercise strategic control over cache boundaries.

The core argument is that in agentic workloads, where context is dynamic and session-specific, caching the entire context (naive approach) can be detrimental. Instead, practitioners should ensure that only stable, reusable content (such as the system prompt) is cached, while dynamic elements (tool results, conversation history) are excluded via explicit boundaries (e.g., UUIDs). This approach maximizes cost efficiency and latency improvements, avoiding the overhead of caching content that will not be reused. The work serves as a practical guide for optimizing enterprise-grade agentic systems, highlighting that the "best" caching strategy is provider-specific and dependent on the optimization goal (cost vs. latency).