🤖 AI

KV-Rescue: Recovering Reasoning Language Model KV Eviction Loss via Stepwise Interleaving

KV-Rescue is a training-free inference framework that mitigates accuracy loss and runaway degeneration in reasoning language models under aggressive KV-cache eviction budgets by interleaving steps from a lightweight full-context helper model with the base model and using an online detector to terminate incoherent generations.

Original authors: Minsoo Cheong, Woosang Lim, Vincent-Daniel Yun, Sungjoo Yoo

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Minsoo Cheong, Woosang Lim, Vincent-Daniel Yun, Sungjoo Yoo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of artificial intelligence, large language models act as powerful engines for reasoning, capable of solving complex math problems or planning intricate tasks. To do this, they rely on a vast internal workspace called a memory cache, which holds the history of everything they have said and thought so far in a conversation. This memory is essential; without it, the model would forget its own previous steps and lose the thread of its logic. However, as these conversations grow longer, this memory requirement becomes a heavy burden, consuming so much computer power that the system slows down or crashes. To keep things running, engineers have developed methods to throw away old parts of this memory, keeping only what seems most important. But this act of discarding information is risky. When a model forgets too much of its past, it doesn't just make a simple mistake; it can spiral into confusion, repeating the same words over and over or generating nonsense until it hits a hard limit on how long it can speak.

Researchers at Seoul National University and the University of Southern California have identified a specific weakness in this process and devised a way to fix it without retraining the models. They discovered that the problem caused by throwing away memory is not a lack of intelligence in the model itself, but rather a simple gap in information. A large, powerful model that has forgotten parts of its history struggles because it is missing context, not because it has lost its ability to think. In a clever twist, they found that a much smaller, less powerful model that remembers everything can often fill in the blanks that the larger, forgetful model misses. While the big model might forget a specific number it needs, the small model remembers it perfectly. Conversely, the small model might lack the deep reasoning skills to solve the problem, while the big model has the skill but not the memory.

To bridge this gap, the team created a new system called KV-Rescue. Instead of letting the large model struggle alone with its incomplete memory, this system pairs it with a small, lightweight helper that keeps the full history in mind. As the two models work together to solve a problem step by step, they take turns generating ideas. At every stage, a scoring system evaluates which idea is better and moves that one forward, creating a single, shared path of reasoning. If the large model starts to wander off into nonsense or repetitive loops because it has forgotten too much, the system detects this early and stops that path immediately, relying on the helper to keep the train of thought on track. This process allows the large model to benefit from the small model's perfect memory without sacrificing its own superior reasoning power.

The results of this approach are striking. When tested on five different math benchmarks using a powerful seven-billion-parameter model, the new method recovered nearly ninety percent of the accuracy that was lost when memory was aggressively thrown away. In situations where the memory budget was extremely tight, the traditional method of simply trying many different answers often failed, causing the model to repeat itself or produce gibberish. The new system, however, prevented these failures and reduced the amount of wasted computer work by forty-three percent on average. The researchers observed that the small helper did not take over the entire task; instead, it contributed about one-third of the steps in the most difficult scenarios, stepping in precisely when the larger model needed a reminder of the past. By combining the deep thinking of a large model with the perfect recall of a small one, the researchers have shown that it is possible to keep long reasoning tasks accurate and efficient, even when the computer's memory is severely limited.

Technical Summary: KV-Rescue

Problem Statement

Large Language Models (LLMs) performing long-context reasoning face a significant memory bottleneck due to the linear growth of the Key-Value (KV) cache. To mitigate this, KV eviction techniques discard less important historical tokens to fit within a fixed memory budget. However, this process is inherently lossy; the model decodes from a partial view of its history rather than the full context.

The authors identify that under aggressive eviction budgets, this loss manifests not merely as a drop in accuracy, but as runaway degeneration. In this state, the model fails to maintain coherent reasoning, instead producing incoherent sequences of symbols or entering repetitive loops until it hits the maximum length limit. The paper argues that this degradation is primarily an information gap (missing context) rather than a capability gap (insufficient model intelligence). Consequently, a large model with an evicted cache and a small model with full context often make complementary errors: the large model lacks historical data, while the small model lacks reasoning capacity.

Methodology: KV-Rescue

To bridge the information gap without retraining, the authors propose KV-Rescue, a training-free inference framework that interleaves reasoning steps between an evicted base model and a lightweight, full-context helper model.

1. Stepwise Interleaved Reasoning

Instead of selecting the best final answer from a set of trajectories (standard Best-of-N), KV-Rescue performs selection at every reasoning step:

Dual Sampling: At each step $t$ , the system samples $N$ candidates from the evicted base model ( $\pi_b$ , operating on a limited cache) and $N$ candidates from the full-context helper ( $\pi_h$ , operating on the complete history).
Process Reward Model (PRM) Selection: A PRM scores all $2N$ candidates based on the current committed context. The candidate with the highest score is committed to the shared trajectory.
State Management: If a base candidate is selected, the base model's cache is updated. If a helper candidate is selected, the base model processes the new tokens and re-applies eviction, while the helper retains its full context for the next step. This ensures both models continue from the same trajectory but maintain distinct views of history.

2. Online Degeneration Detection

To prevent the base model from wasting compute on degenerate paths, KV-Rescue employs an online detector that terminates candidates before PRM scoring if they exhibit specific failure modes:

Incoherent Generation: Detected via high token-level entropy. If the moving average of entropy exceeds a threshold ( $\tau_{ent}$ ), the candidate is aborted.
Repetition Loops: Detected via high compressibility (low entropy but high redundancy). The system calculates the ratio of the gzip-compressed size to the raw size of a sliding window. If this ratio drops below a threshold ( $\tau_{cmp}$ ), the candidate is aborted.
Selection Logic: Only non-degenerate base candidates are considered alongside helper candidates for the PRM selection. This ensures the pool of candidates remains non-empty even if all base candidates are rejected.

Key Contributions

Characterization of Eviction Loss: The paper establishes that eviction-induced failure is an information gap, demonstrating that an oracle choice between an evicted large model and a full-context small model recovers 79% of the accuracy gap to a full-KV large model.
KV-Rescue Framework: A novel, training-free inference method that interleaves steps from an evicted base and a full-context helper, guided by a PRM and an online degeneration detector.
Comprehensive Evaluation: Extensive experiments across five math benchmarks (MATH500, AIME24, AMC23, GSM8K, OlympiadBench) using Qwen2.5-Math models (7B and 72B) and two eviction policies (R-KV and SnapKV).

Experimental Results

The authors evaluated KV-Rescue against standard Best-of-N (sampling only from the evicted model) and full-KV baselines.

Accuracy Recovery: At an aggressive eviction budget ( $B=64$ ), KV-Rescue recovers an average of 87% of the accuracy lost to eviction across five benchmarks. In specific cases (e.g., MATH500 with $N=4$ ), it surpasses the pass@1 performance of the full-KV base model.
Suppression of Degeneration: The framework effectively prevents runaway generation. In the 7B model at $B=64$ , the average number of generated tokens by the base model was reduced by 43% compared to eviction-only Best-of-N, as degenerate paths were terminated early.
Ablation Studies:
- Interleaving: Adding the full-context helper reduced the degeneration rate from 13.4% (eviction-only) to 2.8% and increased accuracy from 52.8% to 81.8% at $B=64$ .
- Early Exit: Terminating degenerate candidates during generation further reduced base-model token generation by 20% without sacrificing accuracy.
Helper Contribution: The helper model contributes 17–34% of the committed steps, with its share increasing as the eviction budget decreases, confirming its role as a context-complementary partner rather than a simple fallback.

Significance and Claims

The paper claims that KV-Rescue offers a practical solution to the memory bottleneck in long-context reasoning without requiring model retraining or architectural changes. By reframing eviction loss as an information gap, the authors demonstrate that a lightweight full-context model can effectively "rescue" a larger, memory-constrained model.

The significance lies in the ability to maintain high reasoning accuracy under strict memory constraints while simultaneously reducing computational waste caused by degenerate generations. The authors note that the recovery persists across different model scales (7B vs. 72B) and eviction policies, suggesting the approach is robust and scalable for agentic systems and long-horizon reasoning tasks.