🤖 AI

Fewer Tokens, Smaller Cache: Reward-Coordinated Efficient Reasoning

The paper introduces ReCo, a reward-coordinated framework that optimizes Large Reasoning Models by dynamically adjusting KV-cache compression, curbing redundant reflections, and enabling early stopping based on process rewards, thereby significantly reducing inference costs and latency while preserving accuracy.

Original authors: Qiyuan Zhu, Dezhi Li, Pengyu Cheng, Tianle Chen, Jiacheng Wang, Ruijie Shen, Hao Gu, Sida Lin, Zirui Liu, Jiacheng Liu, Sirui Han

Published 2026-08-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Qiyuan Zhu, Dezhi Li, Pengyu Cheng, Tianle Chen, Jiacheng Wang, Ruijie Shen, Hao Gu, Sida Lin, Zirui Liu, Jiacheng Liu, Sirui Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a super-smart robot that solves hard puzzles by talking to itself. It doesn't just guess; it writes out a long, detailed story of its thinking process, step by step, before giving you the final answer. This is how modern "Large Reasoning Models" work. They are like brilliant detectives who refuse to solve a case until they've written down every single clue, every dead end, and every "aha!" moment. The problem is, this detective gets a bit too chatty. Sometimes, for a simple question like "What is 2+2?", it writes a novel-length essay about the history of mathematics before answering "4." This "overthinking" uses up a massive amount of computer memory and time, making the robot slow and expensive to run.

To fix this, scientists have tried to make the robot's memory more efficient. Think of the robot's memory as a whiteboard where it writes down its thoughts. As the robot thinks longer, the whiteboard gets full. A common trick is to erase the oldest, "least important" notes to make room for new ones. This is called "KV-cache compression." However, the researchers behind this new paper noticed a flaw in how this erasing was being done. They found that if you erase the wrong notes, the robot gets confused and starts talking even more to try to figure things out, which cancels out all the time you saved by erasing the notes in the first place. It's like trying to save time by skipping a few steps in a recipe, only to realize you forgot a key ingredient and now have to start the whole dish over again.

This paper introduces a new system called ReCo (Reward-Coordinated Compression) to solve this messy problem. Instead of just randomly erasing old notes or using a fixed rule for everyone, ReCo acts like a wise coach standing next to the robot. After every single step the robot takes, the coach asks: "How confident are you that you're on the right track?" If the robot is doing great and is on a solid path (a "high-reward" step), the coach says, "You're doing so well, you don't need to keep every single old note. Let's erase some to save space." But if the robot is struggling or exploring new ideas (a "low-reward" step), the coach says, "Hold on, you need all your notes right now; don't delete anything."

The clever part is that this same "confidence score" does two jobs at once. First, it decides how much of the memory to keep or delete. Second, it tells the robot to stop overthinking. If the robot is confident and on the right track, the coach gently nudges it to stop rambling and just give the answer. If the robot is still confused, the coach lets it keep thinking and exploring. By coordinating these two actions—cleaning up the memory and controlling how much the robot talks—the system prevents the robot from getting confused and writing even longer stories to compensate.

The results are impressive. When the researchers tested ReCo on three different reasoning models across six different types of puzzles (from math problems to science questions), the robot became much faster and used far fewer words. Specifically, the system reduced the number of words the robot generated by 37% to 65% and made the whole process 2.08 to 2.35 times faster than the standard, unoptimized method. Crucially, it did all this without making the robot significantly dumber; the accuracy stayed almost the same as the slow, chatty version. The paper suggests that simply trying to shrink the memory isn't enough on its own; you have to manage the robot's thinking process at the same time to get real speed gains.

Technical Summary: ReCo (Reward-Coordinated Efficient Reasoning)

Problem Statement

Large Reasoning Models (LRMs), such as OpenAI-o1 and DeepSeek-R1, achieve high performance on complex tasks through long Chain-of-Thought (CoT) reasoning. However, this paradigm frequently leads to "overthinking," where models generate excessively long reasoning chains even for simple inputs, resulting in inflated inference costs, high latency, and significant memory overhead due to the accumulation of Key-Value (KV) caches.

While KV-cache compression is a standard solution for reducing memory and attention costs, existing reasoning-oriented methods suffer from two critical limitations:

Uniform Compression Policies: Current methods apply a uniform compression strategy across the entire reasoning trajectory, ignoring that different reasoning states have varying tolerances to context loss.
Decoupled Generation Control: These methods focus solely on what is removed from the cache, failing to account for the fact that compression can induce longer generation lengths. The paper observes that compressing the cache often causes models to generate more tokens to compensate for lost context, partially or fully canceling the efficiency gains (a phenomenon termed "length inflation").

Methodology: ReCo Framework

The authors propose ReCo (Reward-Coordinated Compression), a step-wise framework that coordinates KV-cache compression with generation control under a single signal: a lightweight process reward.

1. Core Signal: Process Reward Estimation

After each completed reasoning step $c_i$ , a lightweight 30M-parameter estimator (distilled from Skywork-o1-Open-PRM-7B) assigns a scalar reward $v_i \in [0, 1]$ . This reward reflects the likelihood that the current reasoning path is on-track and promising. This single signal drives three coordinated components:

2. Reward-Adaptive KV-Cache Compression

Unlike static or uniform compression, ReCo dynamically adjusts the retention ratio $\lambda_i$ based on the step reward $v_i$ :

High-Reward Steps: When the model is on a reliable path (high $v_i$ ), the framework aggressively prunes the accumulated KV cache, as the reasoning state tolerates more context loss.
Low-Reward Steps: When the model is exploring or uncertain (low $v_i$ ), the framework retains more context to prevent breaking the reasoning chain.
Mechanism: The retention ratio is calculated as $\lambda_i = \lambda - \delta \frac{v_i - \bar{v}}{v_{max} - v_{min}}$ , where $\lambda$ is the base ratio and $\delta$ bounds the adjustment. Within the allocated budget, tokens are selected based on attention scores from recent tokens.

3. Reward-Banded Reflection Control

To prevent the "length inflation" caused by compression, ReCo suppresses redundant self-reflection tokens (e.g., "Wait," "Hmm," "Alternatively") during decoding:

The same reward $v_i$ determines a logit penalty $\beta_i$ applied to reflection tokens.
Low Reward ( $v_i \le \tau_\ell$ ): No penalty (encourages exploration).
Medium Reward ( $\tau_\ell < v_i < \tau_h$ ): Half penalty.
High Reward ( $v_i \ge \tau_h$ ): Full penalty (strongly discourages re-opening branches, steering the model toward a conclusion).

4. Confidence-Based Early Stopping

ReCo implements an early stopping mechanism that triggers when the reasoning is deemed reliable:

If the reward remains in the high band for two consecutive steps, a "closing prompt" is inserted to elicit a tentative answer.
The model's confidence in this answer is measured via Perplexity (PPL). If $PPL(a) \le \tau_p$ , the reasoning chain is terminated; otherwise, reasoning continues. This prevents premature stopping on fluent but incorrect answers.

Key Contributions

Coupling Compression and Generation: The paper argues that KV-cache compression alone is insufficient for LRMs because it can backfire by lengthening the reasoning process. ReCo is the first to jointly govern cache compression and generation length under a unified reward signal.
Reward as a Compression Tolerance Signal: The authors demonstrate that process rewards reliably track a reasoning state's tolerance to context loss. High-reward steps are robust to aggressive pruning, whereas low-reward steps require full context. This contrasts with existing uniform policies.
Unified Step-Wise Framework: ReCo integrates KV-cache compression, reflection control, and early stopping into a single framework that reduces token count and latency while preserving accuracy across multiple models and benchmarks.

Experimental Results

The framework was evaluated on three reasoning models (DeepSeek-R1-Distill-Llama-8B, DeepSeek-R1-Distill-Qwen-7B, and Qwen3-8B) across six benchmarks (GSM8K, MATH-500, AMC2023, AIME24, AIME25, and GPQA).

Token Reduction: ReCo reduced generated tokens by 37%–65% compared to Full CoT. In contrast, cache-only baselines (e.g., SnapKV, R-KV) often increased token counts due to length inflation.
Latency Speedup: ReCo achieved end-to-end latency reductions of 2.08×–2.35× over Full CoT.
Accuracy Preservation: ReCo maintained accuracy closest to the Full CoT baseline among all compressed methods. For instance, on Llama-8B, ReCo achieved 60.2% average accuracy compared to Full CoT's 62.8%, whereas KV-only baselines dropped significantly (e.g., SnapKV to 37.5%).
Memory Efficiency: ReCo reduced peak GPU memory usage by 37.7% (17.92 GB vs. 28.78 GB on AIME25), outperforming both KV-only and length-only baselines.

Significance and Claims

The paper claims that KV-cache compression must be coupled with generation control to be effective on reasoning models. The authors assert that compressing the cache without controlling the subsequent generation length can erode efficiency savings by inducing longer reasoning chains.

By using a single process reward to coordinate both sides, ReCo achieves a balance where compute is spent only when reasoning is productive and withheld when the model has settled. The authors emphasize that their method requires no retraining of the reasoning model and no per-task prompt engineering, relying instead on a lightweight reward estimator to dynamically adapt to the reasoning trajectory's needs.