Top-b: Entropic Regulation of Relative Probability Bands in Autoregressive Language Processes

1. Problem Statement

Autoregressive Large Language Models (LLMs) generate text as a discrete stochastic process. The core challenge in decoding is transforming the model's continuous probability distribution into a single discrete token choice.

• The Limitation of Static Decoding: Standard strategies like Top-k (fixed count) and Top-p (fixed cumulative probability mass) rely on static hyperparameters. They fail to account for the non-stationary nature of language uncertainty.
  • Low-Entropy States (e.g., Logic/Math): The distribution is peaked. Static thresholds (e.g., $p=0.9$) are too permissive, allowing low-probability "distractor" tokens to enter the sampling pool, leading to logical errors or hallucinations.
  • High-Entropy States (e.g., Creative Writing): The distribution is flat. Static thresholds can be too restrictive, arbitrarily truncating valid creative options and reducing diversity.
• The Gap: Existing methods lack a mechanism to dynamically adjust the "strictness" of the sampling pool based on the instantaneous information density (uncertainty) of the generation step.

2. Methodology: Top-b (Adaptive Relative Band Sampling)

The authors propose Top-b, a decoding strategy that regulates the candidate set via a dynamic bandwidth coefficient strictly coupled to the instantaneous Shannon entropy of the model's distribution.

Core Formalism

• Stochastic Process: The generation is modeled as a sequence $Y = \{Y_t\}$ where $P(Y_t | Y_{<t})$ is the conditional probability vector.
• Uncertainty Quantification: The model's confidence is measured using Shannon Entropy ($H$):
  $$H(p_t) = -\sum_{y \in V} p_t(y) \log p_t(y)$$
  • Low $H$: High confidence (peaked distribution).
  • High $H$: Low confidence (flat/uniform distribution).
• Relative Probability Band: Instead of an absolute probability cutoff, Top-b defines a threshold relative to the mode ($p_{max}$, the probability of the most likely token).
  • Support Set Definition: A token $y$ is included in the sampling pool if:
    $$p_t(y) \ge (1 - \text{bandwidth}_t) \cdot p_{max}$$
• Adaptive Bandwidth: The bandwidth coefficient is dynamically scaled by the normalized entropy:
  $$\text{bandwidth}_t = \text{base\_bandwidth} \times \left(1 + \frac{H(p_t)}{H_{max}}\right)$$
  Where $H_{max} = \log(|V|)$ is the maximum possible entropy.

Theoretical Behavior

• Low-Entropy Regime ($H \to 0$): The bandwidth contracts toward the base value. The threshold becomes very close to $p_{max}$, effectively pruning the long tail of distractors and approximating greedy decoding for logical reasoning.
• High-Entropy Regime ($H \to H_{max}$): The bandwidth expands (up to $2 \times$ base). The threshold lowers, allowing a wider variety of tokens to maintain linguistic diversity for creative tasks.
• Variance Minimization: Theoretically, Top-b acts as a variance-minimizing operator on the tail distribution, stabilizing the generation trajectory.

3. Key Contributions

1. Entropy-Regulated Decoding: Introduction of Top-b, which defines sampling support via a relative probability band anchored to the distribution mode and scaled by normalized Shannon entropy.
2. Theoretical Framework: Formalization of Top-b as a dynamic control system that minimizes tail variance, bridging the gap between greedy search (high precision) and random walks (high diversity).
3. Empirical Validation: Demonstration that Top-b reduces generation entropy and inter-decoding variance while maintaining or improving reasoning accuracy on complex benchmarks.

4. Experimental Results

The authors evaluated Top-b using the google/gemma-3-4b-it model on two reasoning-heavy benchmarks: GPQA (graduate-level science) and GSM8K (arithmetic reasoning).

GPQA Results (Graduate-Level Reasoning)

• Accuracy: Top-b achieved 0.2285, competitive with Top-k (0.2333) and significantly better than Top-p (0.2132).
• Entropy: Top-b achieved the lowest average entropy (0.1579) among all methods, indicating higher decisiveness and reduced confusion.
• Stability: Top-b exhibited the lowest variance across random seeds, suggesting superior reproducibility in reasoning chains compared to Top-p and Min-p.

GSM8K Results (Arithmetic Reasoning)

• Accuracy: Top-b achieved the highest accuracy (0.2680), outperforming Top-p (0.2613) and Min-p (0.2427).
• Variance: Top-b showed an order-of-magnitude lower variance ($1.136 \times 10^{-4}$) compared to Top-p ($1.1207 \times 10^{-3}$), confirming its ability to filter out arithmetic distractors that disrupt logical flow.

Entropy Dynamics

• Analysis of entropy trajectories showed that while Top-p maintains elevated entropy in later generation steps (due to its cumulative threshold), Top-b induces a monotonic reduction in entropy, effectively pruning divergent branches as the reasoning chain progresses.

5. Significance and Implications

• Self-Regulating Control System: Top-b transforms decoding from a static heuristic into a self-regulating system that adapts to the local information density of the language process.
• Improved Reliability: By minimizing sequence entropy and variance, Top-b enhances the reliability of LLMs in high-stakes reasoning tasks (Chain-of-Thought), reducing the accumulation of errors over long contexts.
• Expressivity-Stability Trade-off: The method theoretically extends to creative domains; in high-entropy creative states, it naturally expands the sampling band to preserve expressivity without manual hyperparameter tuning.
• Computational Efficiency: The overhead is negligible, as calculating Shannon entropy is a vector operation over the vocabulary, which is minor compared to the $O(N^2)$ complexity of the attention mechanism.

In conclusion, Top-b offers a theoretically grounded, adaptive alternative to static decoding strategies, significantly improving the stability and accuracy of autoregressive generation in reasoning-intensive tasks.