DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization

This paper proposes Decoupled Reward Policy Optimization (DRPO), a novel framework that addresses the overthinking problem in large reasoning models by decoupling length-based reward signals for correct rollouts from incorrect ones, thereby achieving significant reductions in reasoning length with minimal performance degradation compared to existing methods.

The Gist

Imagine you have a brilliant but overly chatty student named Reasoning-LLM. This student is amazing at solving math problems, but they have a bad habit: overthinking.

If you ask, "What is 2 plus 3?", a normal person says "5." This student, however, writes a 1,000-word essay about the history of numbers, the concept of addition, and why they are sure the answer is 5, before finally writing "5."

This is called "Overthinking." While the answer is correct, it wastes a huge amount of time and computer energy (tokens).

The Problem with the Old Way (GRPO)

Researchers tried to fix this by teaching the student a simple rule: "Shorter answers get a better grade." They used a method called GRPO (Group Relative Policy Optimization).

Think of GRPO like a teacher grading a class of 6 students at once.

• Student A: Correct answer, 10 words.
• Student B: Correct answer, 100 words.
• Student C: Wrong answer.

The teacher wants to reward the short answer (A) and punish the long one (B). So, they give Student A a score of 10 and Student B a score of 5.

Here is the trap:
In the old method, the teacher compares everyone to the average of the whole class.

• If the class average is 6, Student A (score 10) gets a "Good Job!" (+4).
• But Student B (score 5) gets a "You did worse than average!" (-1).

The Disaster: Even though Student B got the right answer, the teacher told them they did "badly" because they were too long. The student gets confused, stops trying to be correct, and just starts guessing randomly to avoid the "bad" score. The model learns that being right but long is actually bad, so it starts giving up on hard problems to stay short.

The Solution: DRPO (Decoupled Reward Policy Optimization)

The authors of this paper (DRPO) said, "Wait a minute. We are punishing the wrong people!"

They realized you need two different report cards:

1. The "Right Answer" Club: Only people who got the answer correct are in this club.
2. The "Wrong Answer" Club: Everyone else.

How DRPO works:
Instead of comparing the long, correct answer to the wrong answers, DRPO puts all the correct answers in their own room.

• Inside the "Right Answer" room, the teacher says: "Okay, Student A (10 words) is the star. Student B (100 words) is good, but let's give them a slightly lower score than A."
• Crucially: Student B is still in the "Good" zone. They never get a negative score just for being long. They just get a "Gold" vs. "Silver" distinction.
• The "Wrong Answer" students are in a completely different room and get zero points.

By decoupling (separating) these two groups, the model learns:

• "I must be correct first." (Don't worry about the wrong answers).
• "If I am correct, I should try to be shorter to get a better score." (But I won't be punished for being long if I'm right).

The Magic Ingredient: The "Ideal Shortener"

The paper also uses a clever mathematical trick (a closed-form solution) to imagine a "Perfect Version" of the student.

• Imagine a ghost version of the student who always gives the shortest possible correct answer.
• DRPO uses this ghost to guide the real student. It says, "Look at the ghost! That's how efficient you should be!"
• This allows the model to learn to be concise without needing to collect millions of new examples from humans. It just re-weights the answers it already generated.

The Results: The "Smart & Fast" Student

The paper tested this on math problems (from easy "2+2" to hard Olympiad math).

• The Old Way (GRPO + Length Penalty): To save time, the student started getting answers wrong. They were fast, but useless.
• The New Way (DRPO):
  • On easy questions (like 2+3), the student cut their word count by 77% (from 1,000 words down to 250) but kept the accuracy almost perfect.
  • On hard questions, the student still took the time needed to think, but didn't waste time rambling.

The Analogy Summary

• The Problem: A teacher telling a student, "If you write a 10-page essay to solve a simple math problem, you get an F, even if the math is right." The student gets scared and stops doing math.
• The Fix (DRPO): The teacher says, "If you get the math right, you pass. But if you write a 1-page essay instead of a 10-page one, you get an A+. If you write a 10-page one, you get a B. You never get an F just for writing too much, as long as you're right."

In short: DRPO teaches AI models to be efficient without making them dumb. It separates the goal of "being right" from the goal of "being brief," ensuring the model stays smart while learning to be concise.

Technical Summary

1. Problem Statement

Recent Large Reasoning Models (LRMs), such as DeepSeek-R1, utilize Reinforcement Learning (RL) frameworks like Group Relative Policy Optimization (GRPO) to achieve high performance on complex reasoning tasks (e.g., mathematics, coding). However, these models suffer from "overthinking": they generate excessively long, redundant, and repetitive reasoning chains even for simple questions. This behavior significantly increases computational costs and inference latency.

Existing solutions attempt to mitigate this by incorporating length penalties into the reward function (e.g., $r = r_{correct} - r_{length}$). The paper identifies a critical flaw in this approach when applied to GRPO:

• The Root Cause: GRPO calculates the advantage function based on the group-relative difference between a rollout's reward and the average reward of the entire group (including both correct and incorrect samples).
• The Failure Mode: When a length penalty is applied, a correct but verbose answer may have a lower reward than the group average (which includes incorrect answers with zero reward). Consequently, the advantage function assigns a negative advantage to these valid, correct answers.
• The Consequence: The model is actively discouraged from generating correct reasoning simply because it is too long, leading to a trade-off where shortening reasoning causes severe performance degradation.

2. Methodology: Decoupled Reward Policy Optimization (DRPO)

To solve the "negative advantage" problem, the authors propose DRPO, a novel framework that decouples the learning signals for correct (positive) and incorrect (negative) samples.

Core Concept: Decoupling

Instead of normalizing rewards across the entire group (positive + negative), DRPO normalizes rewards for correct rollouts only against other correct rollouts. This ensures that length penalties reduce the magnitude of the positive signal for verbose correct answers but never push them into negative territory, preserving the learning signal for valid reasoning.

Theoretical Formulation

DRPO builds upon a Discriminative Constrained Policy Optimization (DisCO) framework. The objective is to maximize the score of positive outputs while minimizing the score of negative outputs, subject to a KL-divergence constraint to maintain stability.

1. Optimal Positive Distribution: The authors derive an optimal distribution $P^*_q$ for correct outputs that maximizes the length reward under a KL regularization constraint:
   $$P^*_q = \arg \max_{P} \mathbb{E}_{o \sim P}[r_l(o)] - \lambda D_{KL}(P, \pi^+_{old}(\cdot|q))$$
   where $r_l(o)$ is the length reward and $\lambda$ is a regularization parameter.

2. Closed-Form Solution: They derive a closed-form solution for this perturbed distribution:
   $$P^*_q(o) = \frac{\pi^+_{old}(o|q) \exp(r_l(o)/\lambda)}{\mathbb{E}_{o' \sim \pi^+_{old}}[\exp(r_l(o')/\lambda)]}$$

3. Final Objective: By integrating this distribution into the discriminative objective, the final DRPO loss function becomes:
   $$\max_{\theta} \mathbb{E}_q \left[ \mathbb{E}_{o \sim \pi^+_{old}} \left[ \omega(o|q) s_\theta(o, q) \right] - \tau \log \mathbb{E}_{o' \sim \pi^-_{old}} \left[ \exp\left(\frac{s_\theta(o', q)}{\tau}\right) \right] \right]$$
   where $\omega(o|q)$ is a weight derived from the length reward, normalized solely within the positive group.

Key Advantages

• On-Policy Efficiency: The method requires no off-policy data collection or additional human curation. It uses importance weighting on existing on-policy samples.
• Generalizability: The formulation is general and can incorporate other preference rewards (e.g., process rewards) beyond just length.

3. Key Contributions

1. Diagnosis of GRPO Limitations: The paper rigorously diagnoses that GRPO's group-relative advantage function is fundamentally ill-suited for composite rewards (correctness + length), as it creates misleading negative learning signals for valid but verbose reasoning.
2. DRPO Framework: Proposes a new paradigm that decouples learning signals, ensuring length penalties only modulate the strength of positive signals without reversing their sign.
3. Theoretical Derivation: Provides a rigorous derivation of a closed-form solution for the reward-maximizing positive data distribution, enabling efficient optimization without extra data overhead.
4. Empirical Superiority: Demonstrates significant improvements over state-of-the-art baselines across multiple model sizes and reasoning benchmarks.

4. Experimental Results

The authors evaluated DRPO on mathematical reasoning tasks (GSM8k, MATH-500, OlympiadBench, AIME) using models ranging from 1.5B to 8B parameters.

• Performance-Efficiency Trade-off:
  • 1.5B Model: Achieved a 77% reduction in reasoning length on GSM8k (simple questions) with only a 1.1% performance loss. In contrast, the best baseline (ShorterBetter) sacrificed 4.3% performance for a 68% length reduction.
  • 7B Model: Reduced reasoning length from ~3053 to ~1502 tokens (51% reduction) with only a 2.6% performance loss. Baselines like RLOO-LP suffered a 7.1% performance drop for a smaller 38% length reduction.
• Accuracy Efficiency Score (AES): DRPO consistently achieved positive AES scores (indicating efficiency gains without accuracy loss), whereas almost all baseline methods yielded negative AES scores (efficiency gains came at the cost of accuracy).
• Difficulty Scaling: DRPO showed superior resilience across difficulty levels. While all models required longer reasoning for harder problems (e.g., AIME), DRPO maintained a better balance between length and accuracy compared to baselines.
• Case Studies: Visual analysis of reasoning traces showed that DRPO models eliminated redundant "reflection" loops (e.g., "Wait, let me check...") and repeated answers that plagued baseline models, while retaining necessary verification steps.

5. Significance

• Solving the Overthinking Problem: DRPO provides a principled solution to the "overthinking" phenomenon in LRMs, enabling models to reason efficiently without sacrificing the robustness required for complex tasks.
• Cost Reduction: By drastically reducing token generation (up to 77% on simple tasks), DRPO significantly lowers inference costs and latency, making LRMs more viable for real-world deployment.
• Methodological Impact: The paper challenges the standard practice of using group-relative advantages for multi-objective RL in reasoning. It suggests that decoupling learning signals for different reward types is a necessary step for stable and effective policy optimization in LLMs.
• Future Directions: The framework is extensible to other preference rewards (e.g., safety, style) and suggests adaptive tuning of the regularization parameter $\lambda$ based on problem difficulty as a promising future direction.