Technical Summary: Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization

Problem Statement

Post-training Large Language Models (LLMs) has become central to improving reasoning and alignment, particularly through critic-free, group-based methods like Group Relative Policy Optimization (GRPO). In these paradigms, multiple responses are sampled for a single input, and optimization is driven by relative rewards. However, a critical limitation exists: current approaches treat all queries uniformly, failing to distinguish between informative learning signals and noisy variations.

While recent works attempt to regulate optimization using response-level uncertainty signals (e.g., Semantic Entropy), their empirical success is unstable. The paper argues that existing entropy-based estimators suffer from two fundamental gaps when viewed through the lens of gradient variance and learning signal quality:

1. The Anisotropic Gap: Entropy measures capture probability dispersion (how likely different semantic clusters are) but ignore the geometric magnitude of disagreement. In reasoning tasks, a "near-miss" error and a "fundamentally incorrect" path may have similar entropy but induce vastly different gradient directions. Entropy treats these errors as equally severe, failing to capture the true anisotropic structure of the error space.
2. The Calibration Gap: Entropy is decoupled from reward informativeness. High uncertainty does not necessarily imply a strong learning signal; it may simply reflect noise. Conversely, high-variance samples often contain rich, reward-differentiated information. By suppressing high-entropy samples without considering reward dispersion, existing methods inadvertently suppress the very samples that provide the strongest corrective learning signals, leading to biased optimization and collapsed gradient variance.

Methodology: Geometric-aware Calibrated Policy Optimization (GCPO)

To address these gaps, the authors propose GCPO, a framework that reinterprets uncertainty not as a heuristic proxy for dispersion, but as a mechanism for characterizing and calibrating gradient variance. GCPO integrates two core components:

1. Geometry-Aware Measures (Addressing the Anisotropic Gap)

Instead of relying solely on probability mass, GCPO introduces measures that capture the geometric distance between semantic representations in the embedding space:

• Cosine Dispersion (CD): Measures pairwise geometric disagreement directly in the embedding space. It calculates the weighted sum of cosine distances between all response embeddings. CD is effective for capturing fine-grained, continuous semantic variations.
• Barycentric Transport (BoT): Inspired by Optimal Transport, this measure captures structured dispersion across distinct semantic clusters. It groups responses into semantic clusters, computes a consensus barycenter direction, and measures the transport cost (cosine deviation) of each cluster centroid from this consensus. BoT is robust to paraphrastic redundancy within clusters while remaining sensitive to true semantic divergence between modes.

These measures generate a reliability weight, $\omega_{geo}$, which penalizes large semantic deviations while tolerating minor variations.

2. Reward Dispersion (Addressing the Calibration Gap)

To align uncertainty with learning signal strength, GCPO introduces Reward Dispersion (RD). This module quantifies the informativeness of a rollout group based on the variability of rewards within that group.

• If a group of responses yields highly diverse rewards (high dispersion), it indicates strong preference signals and informative learning opportunities.
• If rewards are uniform (low dispersion), the group offers limited learning signal.
• RD generates an amplification weight, $\omega_{RD}$, which scales up updates for groups with high reward informativeness.

Unified Modulation

The final policy update for a query $q$ is modulated by the product of these weights:
$$\tilde{A}_i = A_i \cdot \omega_{geo}(q) \cdot \omega_{RD}(q)$$
This formulation ensures that gradient updates are strengthened when responses exhibit aligned semantic structure with informative reward contrast, and attenuated when variance is dominated by noise or semantic inconsistency.

Key Contributions

1. Principled Formulation: The paper provides the first analysis explicitly formulating uncertainty signals as mechanisms for regulating gradient variance and learning signal quality. It identifies and theoretically characterizes the Anisotropic Gap and Calibration Gap as the root causes of failure in entropy-based methods.
2. GCPO Framework: The proposal of a novel optimization framework that combines geometry-aware semantic structure (CD/BoT) with reward-based calibration (RD) to selectively suppress noisy variance while preserving informative gradients.
3. Empirical Validation: Extensive experiments demonstrate that geometry-aware measures align more faithfully with sample-level gradient variance than entropy-based measures. The proposed method consistently improves post-training performance across multiple benchmarks.

Experimental Results

The authors evaluated GCPO on diverse benchmarks including open-ended QA (NarrativeQA, Qasper), fact-driven QA (HotpotQA), and mathematical reasoning (MATH500, AIME24, OlympiadBench) using the Qwen2.5-1.5B-Instruct model.

• Performance Gains: GCPO consistently outperforms raw GRPO and entropy-based baselines (e.g., Seed-GRPO, EMPO).
  • On NarrativeQA, GCPO (with BoT) achieved a 11.38% improvement in Accuracy and 10.15% in F1 over raw GRPO.
  • On Qasper, improvements were approximately 20% in Accuracy and 18.82% in F1.
  • On HotpotQA, GCPO showed gains of 21.56% in F1.
• Gradient Alignment: Statistical analysis revealed that geometry-aware measures (CD, BoT) exhibit significantly stronger correlation with sample-level gradient variance compared to entropy-based measures. This confirms the theoretical claim that these measures better track the actual optimization dynamics.
• Stability: GCPO reduced the standard deviation of training runs, indicating improved stability without sacrificing performance.
• Math Reasoning Nuance: While GCPO improved performance on MATH500, gains were less pronounced on AIME24 and OlympiadBench. The authors attribute this to the nature of these tasks, where multiple valid solution paths may exist with similar rewards, weakening the alignment between semantic uncertainty and informative gradient variance.

Significance and Claims

The paper claims that uncertainty should not be treated as a generic proxy for dispersion but as a structured signal tied to learning dynamics. The significance of this work lies in:

• Reframing Uncertainty: Shifting the perspective from "uncertainty as noise to be suppressed" to "uncertainty as a calibrated signal for gradient regulation."
• Robust Post-Training: Providing a principled guideline for designing uncertainty signals that preserve informative gradients while avoiding the collapse of learning signals, which is critical for the stable post-training of LLMs.
• Task-Aware Design: Highlighting that the effectiveness of uncertainty estimation depends on the alignment between semantic variation and optimization signals, suggesting that future methods must be tailored to specific task characteristics (e.g., reasoning vs. generation).

The authors conclude that by integrating geometry-aware structure with reward calibration, GCPO achieves more reliable alignment with optimization dynamics, offering a robust path forward for improving reasoning and alignment in LLMs.