Technical Summary: ACPO – Adaptive Credit Policy Optimization via Fine-Grained Surrogate Entropy

1. Problem Statement

Reinforcement Learning (RL) has significantly enhanced the reasoning capabilities of Large Language Models (LLMs), particularly in domains with verifiable objectives like mathematics and coding. However, a critical bottleneck remains: sparse outcome rewards. In standard RL with Verifiable Rewards (RLVR), a scalar reward is typically available only at the end of a long Chain-of-Thought (CoT) trajectory.

Existing scalable methods, such as Group Relative Policy Optimization (GRPO), assign this trajectory-level reward uniformly across all tokens. This "indiscriminate" assignment ignores the varying contributions of individual reasoning steps, leading to high variance and instability. While recent entropy-aware approaches attempt to address this by using token uncertainty as a proxy for importance, they face two primary limitations:

1. Coarse Heuristics: Methods relying on "hard buckets" (e.g., treating the top 20% highest-entropy tokens uniformly) lack the necessary granularity for complex reasoning.
2. Optimization Mismatch: Directly optimizing true entropy introduces non-local gradient components (acting on the full vocabulary distribution) that may misalign with sampled-token policy updates. Furthermore, true entropy is sensitive to noise in the long-tail of the vocabulary distribution, even when detached.

2. Methodology: Adaptive Credit Policy Optimization (ACPO)

ACPO proposes a token-level credit assignment framework that dynamically modulates policy updates based on a mode-local surrogate entropy.

2.1 Core Insight: Entropy Spikes at Errors

Empirical analysis on mathematical reasoning tasks (AIME) reveals a distinct pattern: when a model makes a reasoning error, the step-wise conditional entropy of the incorrect trajectory spikes sharply and remains elevated, whereas correct trajectories maintain low, stable entropy. This suggests that uncertainty is a strong signal for identifying critical "forking points" in reasoning.

2.2 Surrogate Entropy Formulation

To avoid the optimization mismatch of true entropy, ACPO introduces a surrogate entropy ($\delta_{i,t}$), defined as the complementary probability of the dominant mode:
$$\delta_{i,t}(\theta) = 1 - \pi_\theta(v^*_{i,t} | q, y_{i,<t})$$
where $v^*_{i,t}$ is the token with the highest probability.

• Mode-Local Focus: Unlike full-vocabulary entropy, $\delta$ focuses solely on the model's confidence in its most likely prediction.
• Gradient Behavior: Under the assumption of "modal alignment" (where the sampled token frequently coincides with the dominant mode), the gradient of $\delta$ acts locally on the dominant mode rather than the entire distribution. This preserves the intended policy-gradient direction while modulating the magnitude of the update.
• Theoretical Bounds: The paper proves that this surrogate admits deterministic bounds relative to true Shannon entropy, providing a robust approximation under mass-concentrated LLM distributions.

2.3 Asymmetric Credit Assignment

ACPO integrates this surrogate entropy into an asymmetric weighting scheme for the advantage estimate ($\hat{A}_{i,t}$), distinguishing between successful and failed trajectories:

1. Positive-Advantage Trajectories (Success):

   • Goal: Reinforce uncertain but correct decisions.
   • Mechanism: Weight is scaled by $(2 + \delta_{i,t})$.
   • Rationale: High uncertainty in a successful rollout indicates a critical reasoning step that the model was unsure about but got right; these steps should be emphasized. The constant offset '2' ensures the modulation factor remains positive, preventing gradient reversal.

2. Non-Positive-Advantage Trajectories (Failure):

   • Goal: Penalize overconfident mistakes while reducing noise from post-error uncertainty.
   • Mechanism: Weight is scaled by $(1 - \delta_{i,t})$.
   • Rationale: Incorrect trajectories often contain low-uncertainty (overconfident) errors followed by high-entropy noise. ACPO assigns larger penalties to the confident (erroneous) tokens and smaller penalties to the subsequent uncertain tokens, effectively "cleaning" the credit signal.

The final objective function $J_{ACPO}$ combines these asymmetric weights with the standard soft-gating mechanism of SAPO (Soft Adaptive Policy Optimization).

3. Key Contributions

1. Fine-Grained Adaptive Credit Assignment: A token-level weighting scheme that dynamically modulates advantage estimates, treating positive and non-positive trajectories asymmetrically to emphasize uncertain successes and penalize overconfident failures.
2. Surrogate Entropy Formulation: A novel mode-local uncertainty proxy that avoids the optimization mismatch of true entropy. It provides deterministic entropy bounds and preserves the policy-gradient direction under modal alignment and proximal updates.
3. Empirical Effectiveness: Validation across three base models (Qwen2.5-Math-7B, DeepSeek-R1-Distill, Qwen3-8B-Base) on competitive reasoning benchmarks, demonstrating consistent improvements over strong baselines.

4. Experimental Results

Experiments were conducted on mathematical reasoning benchmarks (AIME 2024/2025, MATH500, HMMT25, BeyondAIME) and a coding benchmark (HumanEvalPro).

• Performance: ACPO consistently outperformed strong RL baselines including GRPO, DAPO, GTPO, and SAPO.
  • On Qwen2.5-Math-7B, ACPO achieved an average score of 34.04, surpassing SAPO (31.53) and GTPO (31.89).
  • On DeepSeek-R1-Distill-Qwen-7B, ACPO reached 40.21, significantly outperforming the next best baseline, SAPO (38.35).
• Training Dynamics: Compared to SAPO, ACPO demonstrated faster early-stage improvement in training reward and validation accuracy. In later stages, it maintained higher validation accuracy, indicating improved training stability and a higher performance ceiling.
• Ablation Studies:
  • The asymmetric design $(2+\delta, 1-\delta)$ proved superior to symmetric variants or using detached true entropy.
  • Using the top-1 dominant mode (surrogate entropy) was more effective than using detached true entropy or top-$k$ variants, confirming that dominant-mode uncertainty is a concise and effective proxy for reasoning forks.

5. Significance and Claims

The paper claims that ACPO addresses the fundamental "credit assignment bottleneck" in sparse-reward RL for LLMs without requiring expensive Process Reward Models (PRMs) or human annotation. By replacing coarse, uniform reward assignment with a fine-grained, entropy-aware mechanism, ACPO enables models to learn more effectively from outcome signals.

The authors emphasize that their approach is theoretically grounded in the observation that true entropy introduces non-local gradients, whereas their mode-local surrogate aligns with the sampled-token updates. The results suggest that treating successful uncertain steps and failed overconfident steps differently is crucial for stabilizing and accelerating RL training in reasoning tasks. The method is presented as a scalable, plug-and-play improvement over existing GRPO-based frameworks.