Technical Summary: Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation

1. Problem Statement

Recent empirical analyses have questioned whether Reinforcement Learning (RL) is the primary driver of emergent reasoning capabilities in Large Language Models (LLMs). Studies suggest that post-RL models often fail to outperform their pre-RL counterparts when evaluated with large sampling budgets, implying that RL gains may stem from "distribution sharpening" rather than the creation of new reasoning paths. Concurrently, inference-time techniques like power sampling (sampling from a power distribution $\pi^\alpha$) and self-distillation have shown promise in improving reasoning without external rewards.

However, the theoretical relationship between these three paradigms—sampling, RL, and self-distillation—remains unclear. Specifically, it is unknown whether the success of power sampling reflects a mechanism distinct from RL and distillation, or if they are connected through a common structural objective. Furthermore, it is unclear whether the expensive inference-time compute required for power sampling can be amortized into offline training.

2. Methodology and Theoretical Framework

The authors propose that the power distribution serves as the unifying theoretical bridge connecting these methods. For a base policy $\pi$ and exponent $\alpha > 1$, the power distribution is defined as:
$$\pi_\alpha(y | x) \propto \pi(y | x)^\alpha$$

2.1 The Sampling Perspective: Limits of Local Approximation

The paper first investigates whether the power distribution can be approximated cheaply at inference time using local (token-level) operations.

• Token-wise Temperature Scaling: The authors demonstrate that applying a power transformation per-token (temperature scaling) fails to reproduce the sequence-level power distribution. They prove (Proposition 1) that the gap between token-wise temperature and sequence-level power is governed by the Rényi entropy of the suffix distribution. Specifically, sequence-level power favors continuations with lower suffix entropy (more peaked suffixes), a property local approximations cannot capture without global sequence information.
• Sequential Importance Sampling (SIS): The authors analyze the optimal one-step proposal for SIS targeting $\pi_\alpha$. They show (Proposition 2) that the variance-minimizing proposal requires knowledge of the prefix totals (sums of power masses over all possible suffixes). Consequently, inexpensive local proposals cannot match the target distribution, explaining why prior work requires computationally expensive inference-time procedures (e.g., Metropolis-Hastings) to approximate power sampling.

2.2 The RL Perspective: Closed-Form Optimality

The authors connect power sampling to KL-regularized RL. They consider an RL objective where the reward is the model's own sequence-level log-probability (self-reward, $r_{self}(x, y) = \log \pi(y|x)$).

• Corollary 1: They prove that the power distribution $\pi_\alpha$ is the closed-form optimizer of the KL-regularized RL objective when the reward is the self-reward and $\alpha = 1 + \beta^{-1}$. This identifies power sampling not as a heuristic, but as the exact solution to a specific self-improvement RL problem.

2.3 The Distillation Perspective: Power Self-Distillation

Leveraging the RL connection, the authors derive Power Self-Distillation.

• Derivation: By rewriting the RL objective, they show that minimizing the reverse KL divergence (standard RL) is equivalent to minimizing the forward KL divergence from the teacher distribution $\pi_\alpha$ to the student policy.
• Algorithm: This allows the expensive inference-time cost of sampling from $\pi_\alpha$ to be amortized into offline training. The process involves:
  1. Sampling a dataset $D = \{(x_i, y_i)\}$ where $y_i \sim \pi_\alpha(\cdot|x_i)$ using an approximate sampler (e.g., Metropolis-Hastings).
  2. Training a student model via Supervised Fine-Tuning (SFT) on $D$ to maximize the likelihood of these "power-sampled" completions.
• Sharpening Guarantees: The authors provide a theoretical bound (Proposition 3) showing that as the number of samples $n \to \infty$ and $\alpha \to \infty$, the distilled model achieves $(\epsilon, \delta)$-sharpening, concentrating probability mass on the self-reward maximizers.

2.4 True Reward Alignment

The paper addresses whether sharpening the self-reward improves a "true" external reward $r^*$ (e.g., correctness).

• Proposition 4: The rate of change of the expected true reward with respect to $\alpha$ is determined by the covariance between the true reward and the self-reward under the power distribution: $\partial_\alpha \mathbb{E}_{\pi_\alpha}[r^*] = \text{Cov}_{\pi_\alpha}(r^*, r_{self})$.
• Implication: Improvement in true reward is guaranteed only if the self-reward (log-probability) is positively correlated with the true reward under the power distribution. If they are uncorrelated or negatively correlated, sharpening may not improve (and could degrade) true performance.

3. Key Contributions

1. Theoretical Unification: The paper establishes that power sampling, self-reward KL-regularized RL, and self-distillation are naturally connected through the power distribution, which acts as the sampling target, the RL optimum, and the teacher distribution.
2. Sampling Hardness: It proves that inexpensive token-level approximations (temperature scaling, simple SIS) cannot reproduce sequence-level power sampling without access to suffix-level information, explaining the necessity of expensive inference-time compute in prior work.
3. Power Self-Distillation: It introduces an offline distillation algorithm that amortizes the cost of power sampling into supervised training, achieving similar performance gains without requiring power sampling at inference time.
4. Covariance Condition: It characterizes the conditions under which self-improvement (sharpening) translates to true reward improvement, linking the gain to the covariance between self-reward and true reward.

4. Experimental Results

The authors evaluated their framework on reasoning tasks (MATH, HumanEval, MBPP, GPQA) using models like Qwen2.5-Math-7B and Phi-3.5-mini-instruct.

• RQ1 (Self-Reward Increase): Power sampling consistently increased the self-reward ($r_{self}$) compared to standard sampling and token-wise temperature scaling.
• RQ2 (True Reward Alignment): True reward ($r^*$) improvements were observed when $r^*$ aligned with $r_{self}$. For example, on MATH, power sampling improved accuracy. However, on a randomly initialized model (RandW) where $r_{self}$ did not correlate with correctness, no true reward gain was observed, validating the covariance condition.
• RQ3 (Distillation Performance): Power self-distillation achieved self-improvement. The distilled model, when using standard autoregressive decoding (with temperature), matched or exceeded the performance of the base model using expensive power sampling.
  • Qualitative Analysis: On a geometry problem, the base model with temperature scaling produced hallucinated formulas. Power sampling produced correct logic but included unexecuted code. The power-distilled model, using standard decoding, produced robust, step-by-step reasoning with the correct answer.
• Efficiency: The distilled model eliminated the need for expensive inference-time sampling (Metropolis-Hastings) during deployment, shifting the computational cost to offline training.

5. Significance and Claims

The paper claims to clarify the mechanism behind recent "reasoning" gains in LLMs. It posits that:

• The success of power sampling is not a distinct phenomenon from RL but is the closed-form solution to a self-reward RL objective.
• The apparent "reasoning" improvements from RL may largely be a result of distribution sharpening (concentrating mass on high-probability, high-reward paths) rather than the emergence of new capabilities.
• Power self-distillation offers a practical pathway to achieve these sharpening benefits at a lower inference cost, making advanced reasoning capabilities more accessible by converting expensive inference-time search into offline training.

The authors remain modest, noting that self-improvement is bounded by the capabilities of the base model; if the base model lacks the necessary reasoning paths, sharpening cannot create them. The analysis is specific to autoregressive models over finite horizons.