Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descent

Problem Statement

Online Reinforcement Learning with Verifiable Rewards (RLVR) transforms checkable outcomes (e.g., math proofs, code unit tests) into scalable training signals. However, standard online methods (like PPO or GRPO) keep rollout generation, verifier scoring, and reference-policy evaluations on the critical optimization path, leading to high computational costs and memory usage.

A natural alternative is to decouple generation from optimization: sample verified rollouts once, assign weights, and train via static Supervised Fine-Tuning (SFT). However, the paper identifies a fundamental ambiguity in this approach: Weighted SFT is not a single objective. The policy induced by weighted SFT depends jointly on the sampler that produced the rollouts and the weights assigned to them. Simply attaching weights to precomputed data does not guarantee that the resulting policy matches the optimizer of the original online KL-regularized RLVR objective. Without a specific target-matching condition, static weighted SFT may fit a different policy than the intended RLVR solution, even with infinite data and exact optimization.

Methodology: Boltzmann Projection Theory

The paper develops a theoretical framework to identify the exact policy-level match between static weighted SFT and fixed-reference KL-regularized RLVR.

1. The Target Policy (Fixed-Reference Boltzmann Target)
The paper establishes that the optimizer of the KL-regularized RLVR objective (maximizing reward minus KL divergence from a reference policy $\pi_{ref}$) is a specific conditional policy, the Boltzmann target $\pi^*$:
$$\pi^*(y | x) \propto \pi_{ref}(y | x) \exp(r(x, y) / \beta)$$
where $r(x, y)$ is the verifier reward and $\beta$ is the temperature. This target is an exponential tilt of the reference policy.

2. The Induced-Target Identity
Using Proposition 1, the authors show that any weighted SFT objective induces a target policy $\tilde{\pi}_w$ proportional to the product of the sampler $q$ and the weight $w$: $\tilde{\pi}_w \propto q(y|x)w(x,y)$.
To faithfully replace online RLVR, the induced policy $\tilde{\pi}_w$ must equal the Boltzmann target $\pi^*$.

3. The Target-Matching Law
The paper derives the necessary condition for this match (Theorem 4). For a fixed sampler $q$, the weights must be proportional to the density ratio $\pi^*/q$.
Crucially, in the reference-sampled subclass where the rollouts are generated from the reference policy ($q = \pi_{ref}$), the condition simplifies uniquely to the prompt-normalized Boltzmann weight:
$$w(x, y) = \frac{\exp(r(x, y) / \beta)}{Z(x)}$$
where $Z(x)$ is the partition function (normalizer). This weight is exactly the Radon–Nikodym derivative converting reference rollouts into target rollouts.

4. The BOLT Algorithm
Based on this theory, the authors propose BOLT (Boltzmann-Targeted SFT). It operates in two phases:

• Phase 1 (Precomputation): Sample rollouts from the reference policy, evaluate them with a verifier, and compute the empirical prompt-normalized Boltzmann weights using Monte Carlo estimation of $Z(x)$.
• Phase 2 (Optimization): Train the model using standard multi-epoch SFT on this fixed weighted dataset.
  Because the heavy lifting (sampling, scoring, normalization) is precomputed, the optimization phase requires no reference-model forward passes, significantly reducing memory and compute costs compared to online RLVR.

Finite One-Shot Analysis and Limitations

The paper provides a rigorous finite-sample analysis (Sections 5 and 6) that decomposes the error of static replacement into distinct components:

• Support-Restricted Gap: If the reference policy $\pi_{ref}$ never samples a high-reward completion, no amount of weighting or optimization can recover it. The paper proves an exact lower bound on the value loss: $\beta \log(1/\pi^*(S_N | x))$, where $S_N$ is the stored support. This explains why extra SFT epochs cannot repair missing coverage.
• Temperature–Coverage–Variance Frontier: There is a trade-off where lowering $\beta$ (sharpening the target) increases the effective sample size (ESS) cost and the variance of the normalizer, making estimation harder if the reference pass probability is low.
• Iterative Refresh as KL Policy Mirror Descent: When coverage is weak but non-zero, the paper shows that iterative BOLT (where the updated policy becomes the new reference for the next round) is mathematically equivalent to KL Policy Mirror Descent (KL-PMD). This process accumulates reward tilts and improves future rollout coverage, whereas static one-shot reuse cannot.

Key Results and Empirical Evidence

The authors validate their theory using Qwen models on GSM8K (math) and HumanEval (code) benchmarks. The experiments are designed to isolate specific theoretical mechanisms rather than claim leaderboard dominance.

• Target Matching: Comparing BOLT (Boltzmann weights) against "Refit" (raw reward weights) on the same reference rollouts shows that BOLT achieves significantly higher accuracy (e.g., +1.44 on GSM8K, +3.60 on MATH). This confirms that the specific density-ratio weight, not just the data source, determines the induced target.
• Efficiency: BOLT achieves accuracy comparable to or slightly better than online GRPO while reducing training time by ~75–85% and peak memory by ~18–31%, as it removes the online loop.
• One-Shot Saturation vs. Refresh: Experiments show that one-shot BOLT often saturates after a few epochs if the reference support is limited. However, Iterative BOLT (refreshing the sampler) continues to improve performance, validating the KL-PMD interpretation.
• Retention: Methods using reference rollouts (BOLT, Refit) show less catastrophic forgetting on general benchmarks (PIQA, HellaSwag) compared to demonstration-only SFT, suggesting reference rollouts help anchor the distribution.

Significance and Claims

The paper claims to provide a mechanism-level understanding of static RLVR replacement, moving beyond heuristic comparisons of training losses. Its primary contributions are:

1. Target Identification: It establishes that static weighted SFT is only a faithful replacement for fixed-reference RLVR if the sampler-weight product induces the specific Boltzmann target. It identifies the prompt-normalized Boltzmann density ratio as the unique weight for reference-sampled data.
2. Finite Gap Decomposition: It separates the irreducible "support price" (missing coverage) from finite estimation errors (normalizer variance, optimization drift). This clarifies that static methods cannot fix coverage gaps that the reference policy failed to sample.
3. Iterative Interpretation: It reframes iterative sampling and weighting not merely as "more optimization" but as a form of KL Policy Mirror Descent that actively improves coverage.
4. Practical Efficiency: It demonstrates that precomputing the target-defining quantities (weights) allows for a static SFT procedure that matches the theoretical target of online RLVR with significantly reduced computational overhead.

The authors remain modest, noting that their results are based on single-run experiments and specific model sizes. They emphasize that the theory identifies when static replacement is valid (covered regimes) and why it fails (support gaps), rather than claiming universal superiority over all online methods. The work serves as a theoretical bridge between the geometry of KL-regularized RL and the practical implementation of static, weighted supervised learning.