🤖 machine learning

Entropy Centroids as Intrinsic Rewards for Test-Time Scaling

This paper proposes "Lowest Centroid," a test-time scaling method that selects the best model response by calculating an "Entropy Centroid" based on the temporal clustering of high-entropy tokens, demonstrating consistent performance improvements over existing baselines across various tasks and model scales without requiring external reward models.

Original authors: Wenshuo Zhao, Qi Zhu, Xingshan Zeng, Fei Mi, Lifeng Shang, Yiren Feng

Published 2026-04-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wenshuo Zhao, Qi Zhu, Xingshan Zeng, Fei Mi, Lifeng Shang, Yiren Feng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Finding the Best Answer Without a Teacher

Imagine you have a very smart but sometimes confused student (the AI model). You ask them a hard question, like a complex math problem or a coding challenge. Instead of just asking for one answer, you tell them: "Go think about this 64 different ways, and then pick the best one."

This is called Test-Time Scaling. The problem is: How do you know which of the 64 answers is the best?

Usually, you'd need a teacher (an external reward model) to grade every single answer. But hiring a teacher for every single attempt is expensive and slow. This paper proposes a clever trick: Listen to the student's own "thinking noise" to decide if they are on the right track.

The Problem: The "Noisy" Classroom

When the student thinks out loud (generates text), they produce a stream of words. Some words are said with total confidence (low "entropy"), and others are said with hesitation, doubt, or by trying many different options (high "entropy").

Previous methods tried to judge the answer by looking at the confidence of every single word.

The Flaw: This is like judging a whole movie by looking at a single frame. Sometimes, a student says a word confidently even when they are totally lost (like a robot reciting a memorized fact). Other times, they hesitate because they are doing deep, productive thinking. Looking at word-by-word confidence is too "noisy" and confusing.

The Solution: The "High Entropy Phase" (HEP)

The authors realized that confusion doesn't happen one word at a time; it happens in bursts.

Imagine the student's thought process as a walk through a forest:

Low Entropy: Walking confidently on a clear, paved path.
High Entropy: Stumbling into a dense, foggy thicket where they aren't sure which way to go.

Instead of looking at every single step, the authors group these steps into High Entropy Phases (HEPs).

An HEP starts when the student hits the foggy thicket (high uncertainty).
It ends only when they walk out of the fog and back onto a clear path for a few steps in a row.

This turns a messy, noisy signal into clear, manageable "chunks" of thinking.

The Core Idea: The "Entropy Centroid"

Now, how do we judge the quality of the whole answer? The authors use a physics concept: The Center of Mass (or Centroid).

Imagine the student's entire thought process is a long stick.

Every time they get stuck in a "foggy thicket" (an HEP), we place a heavy weight on that part of the stick.
The Entropy Centroid is the point where the stick would balance if you held it up.

The Golden Rule of this paper:

A Good Student (Low Centroid): They get confused early on (exploring the foggy thicket at the beginning), figure it out, and then walk confidently on the clear path for the rest of the journey. Their "weight" is at the start of the stick. The balance point is low (early).
A Bad Student (High Centroid): They start confidently (thinking they know the answer), but then they get stuck in the foggy thicket near the end, realizing they made a mistake. Their "weight" is at the end of the stick. The balance point is high (late).

The Strategy: When the AI generates 64 different answers, the paper's method simply picks the one where the "balance point" is closest to the beginning. It assumes that getting confused early and then solving it is a sign of a correct answer, while getting confused at the end is a sign of failure.

Why This Matters

No Teacher Needed: It doesn't require an external grader. It uses the model's own internal "confusion signals."
Works Everywhere: The authors tested this on math, coding, logic puzzles, and even "agent" tasks (where the AI has to use tools). It worked better than existing methods across all of them.
Scales Up: Whether the AI is small (14 billion parameters) or huge (480 billion parameters), this method consistently finds the better answers.

Summary Analogy

Think of solving a problem like running a race.

The Old Way: A referee watches every step and yells "Good!" or "Bad!" based on how fast you are moving at that exact second.
The Paper's Way: The referee looks at your pace.
- If you sprint at the start, slow down to think hard in the middle, and then sprint to the finish, you likely won. (Confidence at the end = Good).
- If you sprint at the start, but then trip and stumble in the final stretch, you likely lost. (Confidence at the end = Bad).

The paper's method simply picks the runner who stumbled early but finished strong, ignoring the noisy details of every single footstep.

1. Problem Statement

Large Language Models (LLMs) are increasingly scaled at test-time by sampling multiple candidate responses ( $N$ trajectories) and selecting the best one (e.g., "Best-of-N" selection). Current selection methods face two primary challenges:

External Reward Models: Relying on external reward models requires training a separate, high-quality model and introduces significant computational overhead.
Noisy Intrinsic Signals: Existing intrinsic signals (e.g., token-level confidence, entropy, or KL divergence) are often too noisy to be reliable. Token-level metrics fail to capture the temporal structure of reasoning; for instance, a model might exhibit high confidence on calculation tokens even while being fundamentally uncertain about the overall reasoning step. Furthermore, simple aggregation (like averaging) ignores the temporal dynamics of exploration vs. exploitation.

The paper seeks a robust, intrinsic reward signal derived solely from the model's own generation process that can effectively select high-quality trajectories across diverse tasks (math, code, logic, agents) without external supervision.

2. Methodology

The authors propose a novel framework based on the observation that high-entropy tokens (indicating uncertainty) tend to cluster into consecutive groups during inference, rather than appearing randomly.

A. High Entropy Phase (HEP)

To move beyond noisy token-level signals, the authors define a High Entropy Phase (HEP) as the basic unit of uncertainty.

Definition: An HEP is a variable-length segment of the trajectory.
Trigger: It begins when a token's entropy exceeds a high threshold ( $\theta_{high}$ ).
Termination: It ends only when $k$ consecutive tokens fall below a low threshold ( $\theta_{low}$ ).
Significance: This groups consecutive uncertain tokens into a coherent "phase," filtering out isolated noise and capturing the model's state of exploration or confusion as a whole unit.

B. Entropy Centroid

Inspired by the physics concept of the center of mass, the authors define the Entropy Centroid to encode the temporal position of these uncertainty phases.

Mass ( $m_n$ ): The number of tokens in the $n$ -th HEP.
Position ( $p_n$ ): The midpoint of the $n$ -th HEP within the total trajectory.
Calculation: The centroid $C$ is the weighted average position of all HEPs along the trajectory, normalized by the total sequence length $L$ :
$C = \frac{\sum_{n=1}^{N} m_n \cdot p_n}{L \cdot \sum_{n=1}^{N} m_n}$
Interpretation:
- Low Centroid: Indicates that uncertainty (exploration) occurred early in the reasoning process, followed by confident (low-entropy) generation. This correlates with high-quality reasoning (explore early, converge later).
- High Centroid: Indicates that uncertainty persisted or emerged late in the trajectory, suggesting the model struggled to resolve the problem or got confused near the end.

C. Lowest Centroid Selection Strategy

The proposed selection method, Lowest Centroid, simply selects the trajectory with the lowest entropy centroid value among the $N$ sampled candidates.

Preprocessing: An outlier filtering step removes trajectories with abnormal termination or gibberish before selection.
Advantage: It requires no training, no external labels, and works for any task type (including open-ended coding and agentic tasks where majority voting fails).

3. Key Contributions

Temporal Uncertainty Modeling: The paper identifies that the temporal pattern of uncertainty is more informative than the magnitude of uncertainty at individual tokens. It formalizes this via the High Entropy Phase (HEP).
Entropy Centroid Metric: Introduces a physics-inspired metric (Centroid) that aggregates HEPs into a single scalar reward signal representing the "timing" of model uncertainty.
General-Purpose Intrinsic Reward: Demonstrates that this intrinsic signal is effective across a wide range of model scales (14B to 480B) and domains (Math, Code, Logic, Agents), outperforming existing baselines without requiring external reward models.
Robustness: The method is shown to be robust to hyperparameter choices (thresholds for HEP detection) and stable as model size increases.

4. Experimental Results

The authors evaluated the method on four benchmarks: AIME 2025/Minerva (Math), Synlogic (Logic), BigCodeBench/LiveCodeBench (Code), and $\tau^2$ -Bench (Agents).

Performance Gains: The Lowest Centroid method consistently outperformed baselines (Pass@1, Greedy Decoding, Self-Certainty, Tail Confidence, Bottom Window) across all 30 model-dataset combinations.
- Average Improvement: +5.3% over Pass@1.
- Agentic Tasks: +6.8% average improvement (e.g., GPT-OSS-120B saw a +10.1% gain).
- Math/Logic: Significant gains, e.g., +11.0% on AIME 2025 for GPT-OSS-120B.
Comparison with Baselines:
- Unlike "Self-Certainty" or "Tail Confidence," which showed unstable performance (sometimes degrading results), Lowest Centroid provided reliable gains in every setting.
- It outperformed Majority Voting on short-answer tasks and, crucially, is applicable to long-form tasks (coding, agents) where Majority Voting is inapplicable.
Scaling Behavior: The method exhibits persistent scaling without early saturation. As the number of sampled trajectories ( $N$ ) increases, accuracy continues to rise steadily, whereas other methods often plateau or fluctuate.
Model Scale: The method works effectively from 14B to 480B parameters, proving its generality.

5. Significance and Conclusion

This work provides a lightweight, annotation-free, and generalizable approach to test-time scaling. By shifting the focus from token-level noise to the temporal structure of uncertainty (via HEPs and Centroids), the authors offer a practical solution for selecting high-quality reasoning paths.

Practical Impact: It eliminates the need for expensive external reward models, making test-time scaling more accessible and efficient.
Theoretical Insight: It validates the hypothesis that high-quality reasoning follows a specific temporal pattern: early exploration (high entropy) followed by confident convergence (low entropy).
Future Potential: The Entropy Centroid can serve not only as a selection metric but also as an intrinsic reward signal for training (e.g., in Reinforcement Learning) or for diagnosing model behavior in long-form generation.

In summary, the paper successfully demonstrates that analyzing the timing of model uncertainty is a powerful, intrinsic signal for improving LLM performance at test-time.