💬 NLP

Grounded Cache Routing for Retrieval-Augmented Generation: When Is It Safe to Reuse an Answer?

This paper proposes GroundedCache, an evidence-validated routing mechanism that ensures the safe reuse of cached answers in Retrieval-Augmented Generation by enforcing four simultaneous safety gates, thereby eliminating unsafe-served rates across diverse workloads while maintaining near-baseline latency.

Original authors: Syed Huma Shah (Duke University)

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Syed Huma Shah (Duke University)

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a very busy, high-end library where a librarian (the AI) answers questions for thousands of visitors every day. To save time and money, the library has a "Cheat Sheet" system: if a visitor asks a question that looks similar to one asked before, the librarian just hands them the old answer instead of re-reading the books.

The problem? The old answers might be wrong.

Maybe the books in the library were updated since the last time the question was asked. Maybe the visitor asked a slightly different question that seems the same but actually needs a totally different answer. Or maybe a trickster visitor asked a question designed to trick the librarian into giving the wrong answer.

This paper, "Grounded Cache Routing," introduces a new, super-vigilant security guard for that Cheat Sheet system. Instead of just asking, "Does this question look like the old one?", the guard asks four specific safety questions before handing over the old answer.

The Four Safety Gates (The "Security Check")

The paper argues that reusing an answer isn't about doing it faster; it's about doing it safely. The new system, called GroundedCache, only lets an old answer through if four "gates" all turn green:

The "Did You Mean It?" Gate (Query Similarity):
- The Metaphor: The guard checks if the new question is truly the same as the old one, not just a trick that looks similar.
- The Check: It compares the "fingerprint" of the new question to the old one. If they are too different, the guard says, "Nope, go get a fresh answer."
The "Same Books?" Gate (Evidence Overlap):
- The Metaphor: Imagine the librarian wrote the old answer based on Book A. The guard checks if the new search results are also pulling from Book A.
- The Check: If the new search pulls up Book B (even if the question is the same), the guard knows the old answer might be based on the wrong information. It blocks the old answer.
The "Is the Book Updated?" Gate (Version Validity):
- The Metaphor: What if Book A was updated yesterday with new facts? The old answer might be outdated.
- The Check: The guard checks the "edition number" of the books used for the old answer. If the new search finds a newer edition, the guard rejects the old answer to prevent serving stale information.
The "Does the Book Actually Say That?" Gate (Lexical Support):
- The Metaphor: This is the most important gate. The guard looks at the old answer and checks the new books to see if the specific words in the answer actually appear in the new text.
- The Check: If the old answer says "The capital is Paris," but the new books don't mention Paris at all, the guard stops the answer. It ensures the answer is actually "grounded" in the current evidence.

The "Safety vs. Speed" Trade-off

The researchers tested this system against a "naive" system (the one that just reuses answers because they look similar). They created six different scenarios to stress-test the system, including:

Exact Repeats: Asking the exact same question again.
Paraphrases: Asking the same question in different words.
Document Drift: When the source books change their numbers or facts.
Tricky "Near Misses": Questions that look similar but need different books.

The Results:

The Naive System: It was fast, but it was dangerous. In some scenarios, it gave the wrong answer 35% to 50% of the time because it didn't check if the source material had changed.
The GroundedCache System: It was incredibly safe. It reduced the rate of wrong answers to 0% in most scenarios.
The Speed Cost: You might think adding four security checks would make things slow. Surprisingly, it didn't. The new system was only about 4% to 7% slower than a system with no cache at all. It was much safer than the naive system and almost as fast as doing everything from scratch.

The "Load-Bearing" Gate

The researchers found that one gate did most of the heavy lifting: Gate #4 (Lexical Support).

If you remove the other three gates, the system is still mostly safe.
If you remove Gate #4 (the check that ensures the answer's words are actually in the new books), the system becomes unsafe again.
Analogy: Think of the other gates as a security fence and a guard dog. Gate #4 is the actual key. Without the key, the fence doesn't matter.

The Bottom Line

The paper concludes that in the world of AI that reads documents (RAG), we shouldn't just ask, "Can we reuse this answer to save time?" We should ask, "Is it safe to reuse this answer?"

GroundedCache proves that you can have a "Cheat Sheet" that is both fast and safe, as long as you have a smart security guard who checks the source material before handing out the answer. It turns a risky shortcut into a reliable tool.

Technical Summary: Grounded Cache Routing for Retrieval-Augmented Generation

Problem Statement

Modern Retrieval-Augmented Generation (RAG) systems increasingly utilize caching to reduce token costs and Time-to-First-Token (TTFT). While prefix-level Key-Value (KV) reuse and retrieval-result caching have strong correctness guarantees by construction, output-level semantic answer caching remains fragile. Naive semantic caches suffer from three primary failure modes:

Referent Shift: Semantically similar queries may map to different correct answers depending on context.
Retrieval Drift: As the underlying corpus is updated, previously cached evidence may become invalid or change.
Adversarial Collision: Malicious inputs can hijack cached responses, with reported hijacking rates as high as 86% in evaluated scenarios.

Current systems often prioritize how to reuse answers faster, neglecting the critical question of when reuse is safe. This paper argues that without rigorous validation, semantic caching introduces a significant "unsafe-served rate" (USR), where users receive incorrect answers from the cache.

Methodology: GroundedCache

The authors propose GroundedCache, an evidence-validated cache router that acts as a policy layer atop existing serving stacks (e.g., vLLM, RAGCache). Instead of blindly returning a cached answer for a semantically similar query, GroundedCache admits a cached answer only when four specific "gates" simultaneously hold.

The Four Validation Gates

Given a cached entry $(q_c, a_c, \sigma_c)$ and a fresh query $(q, \sigma, C)$ , the router admits $a_c$ if and only if:

Query Similarity (G1): The cosine similarity between the new query embedding and the cached query embedding exceeds a threshold ( $\tau_q$ ).
Evidence Overlap (G2): The Jaccard similarity between the new evidence signature (set of chunk hashes) and the cached signature exceeds a threshold ( $\tau_e$ ).
Source-Version Validity (G3): Shared chunks between the new and cached evidence carry the same source version tag, ensuring the corpus has not been mutated.
Evidence Support (G4): The cached answer's content tokens are covered by the freshly retrieved evidence. This is calculated via a deterministic lexical-overlap score (default) or optionally via a lightweight judge LLM.

If any gate fails, the system falls back to a standard RAG pipeline: query-conditioned compression of the retrieved chunks followed by generation.

Workload and Metrics

To stress-test cache safety rather than just hit rates, the authors synthesized a six-regime workload:

Exact-repeat & Paraphrase: Benign reuse scenarios.
Near-miss: Lexically similar queries with disjoint gold documents.
Document-drift: Queries where numeric tokens in gold documents are mutated to invalidate cached answers.
Long shared-document & Bounded-KB CAG: Scenarios favoring deduplication and in-context generation.

The primary evaluation metric is the Unsafe-Served Rate (USR), defined as the fraction of all queries that received a wrong cached answer. This is complemented by the Answer-Cache Hit Rate (aHR) and the Conditional False-Hit Rate (FH).

Key Results

Experiments were conducted on HotpotQA and mtRAG datasets using 12,000 generations (Qwen2.5-7B-Instruct served by vLLM).

Safety Improvements

HotpotQA: GroundedCache reduced the USR to 0.0% across all regimes where naive caching had non-zero errors. For example, on the "document drift" regime, naive caching had a USR of 35.0%, while GroundedCache achieved 0.0%.
mtRAG (Multi-turn): Naive caching exhibited catastrophic failure with USR ranging from 26.0% to 51.5% due to referent shifts. GroundedCache reduced USR by more than an order of magnitude, achieving 1.5% on the document-drift regime (a 34× reduction in wrong cached answers).
Ablation Studies: The lexical support gate (G4) was identified as the "load-bearing" safety mechanism. Removing it increased USR by ~0.125 on HotpotQA and ~0.118 on mtRAG. The other gates (G1–G3) provided defense-in-depth at near-zero cost but were largely redundant when G4 was active.

Performance and Latency

Latency: End-to-end p50 latency under GroundedCache remained within 1.04–1.07× of a no-cache RAG baseline.
Trade-offs: A "no-support" variant (disabling G4) offered a 1.4–1.5× speedup but incurred a non-zero USR (0.125–0.182), demonstrating the tunable trade-off between safety and speed.
Hit Rates: While GroundedCache reduced the raw answer-cache hit rate (e.g., from 0.41 to 0.04 on HotpotQA exact-repeat) to ensure safety, it maintained significant retrieval savings by utilizing the retrieval-cache path.

Significance and Claims

The paper claims that the correct framing for cached answer reuse is not maximizing speed, but quantifying and minimizing the unsafe-served rate.

Policy over Kernel: GroundedCache is presented as a policy layer that composes with existing infrastructure (vLLM APC, LMCache, etc.) without requiring changes to the model server, retriever, or embedding models.
Operator-Facing Metrics: The authors argue that reporting USR alongside hit rate and latency is essential for practitioners to make informed decisions about their safety/speed trade-offs.
Robustness: By validating against fresh evidence and version tags, the system effectively neutralizes adversarial collisions and corpus drift, which naive semantic caches cannot handle.

The authors conclude that while semantic answer caches inherently trade correctness for speed, the lexical support gate provides a cheap, deterministic mechanism to ensure that reuse is safe, driving the unsafe-served rate to near-zero while retaining the latency benefits of caching.