Technical Summary: Pessimism's Paradox in Conservative Offline Training

1. Problem Statement

The standard paradigm for aligning Large Language Models (LLMs) involves a two-stage process: first, offline training on human preference data (e.g., via Direct Preference Optimisation, DPO) using a conservatism coefficient ($\beta$) to penalize deviation from a reference policy ($\pi_{ref}$); second, online adaptation using a learned proxy reward model.

The prevailing intuition (the "implicit contract") is that a higher $\beta$ creates a safer foundation. The assumption is that a more conservative policy remains closer to the distribution of the reference policy and, by extension, the training distribution of the reward model, thereby reducing the likelihood of exploiting imperfections in the proxy reward (reward hacking).

This paper challenges this intuition, demonstrating a paradox: higher offline conservatism ($\beta$) actually amplifies reward-hacking damage during subsequent online adaptation. The authors argue that the field's reliance on maximal conservatism is flawed and that a calibrated, rather than maximal, approach is required.

2. Methodology

Experimental Setup

• Policy Model: Qwen3-14B, fine-tuned using LoRA adapters in 4-bit NF4 quantisation.
• Reward Model: An ensemble of three independent Qwen3-1.7B sequence classifiers trained with bootstrap resampling on the UltraFeedback dataset. This ensemble provides both a proxy reward signal and an epistemic uncertainty estimate (via ensemble disagreement).
• Offline Training: Three distinct DPO checkpoints were trained with varying levels of conservatism ($\beta \in \{\beta_{lo}, \beta_{mid}, \beta_{hi}\}$). These $\beta$ values were not arbitrary but derived from the 20th, 50th, and 80th percentiles of the absolute log-ratio magnitudes of the reference policy on the preference data.
• Online Adaptation: Each checkpoint was adapted online against the learned reward ensemble. The objective included a standard advantage-weighted policy gradient and an adaptive KL penalty.
• Evaluation Metric: True performance was measured on the GSM8K dataset using exact-answer accuracy. Since the true reward is only observable during evaluation, the authors defined the Goodhart Gap ($G(t)$) as the divergence between the normalized proxy reward and the normalized true reward. The cumulative damage was quantified as the Area Under the Goodhart Gap Curve (AUGC).

Mechanistic Analysis

The authors investigated the causal chain linking high $\beta$ to increased hacking through three specific links:

1. Entropy Compression: Measuring token-level entropy to see how $\beta$ constrains the policy's output diversity.
2. OOD Distance: Calculating the cosine distance between the hidden states of generated responses and the centroid of the reward model's training distribution.
3. Uncertainty-Driven Exploitation: Analyzing the correlation between ensemble disagreement (epistemic uncertainty) and the Goodhart Gap.

3. Key Results

The Paradox Empirically

Contrary to the hypothesis that higher conservatism reduces hacking, the study found a monotonic increase in reward-hacking damage as $\beta$ increased.

• AUGC Correlation: There was a perfect positive Spearman rank correlation ($\rho = 1.0$) between the conservatism level $\beta$ and the AUGC.
• Quantitative Damage:
  • Low $\beta$ ($0.310$): AUGC = 31.1
  • Mid $\beta$ ($1.000$): AUGC = 43.0
  • High $\beta$ ($2.379$): AUGC = 145.8
• Goodhart Gap Trajectories: While the proxy reward generally overestimated true performance (negative gap) across all conditions, the high-$\beta$ policies exhibited the most severe and persistent divergence.

Mechanistic Findings

The paper identifies a three-link causal chain explaining the paradox:

1. Entropy Collapse: High-$\beta$ DPO compresses the policy's output entropy, forcing it to generate responses with reduced diversity.
2. Distributional Mismatch: Paradoxically, these low-entropy, diverse-poor responses are closer (in cosine distance) to the reward model's training distribution centroid than low-$\beta$ responses. However, because they concentrate in a narrow region, they trigger higher epistemic uncertainty (ensemble disagreement).
3. Exploitation: The online optimizer, seeking to maximize the mean proxy reward, rapidly exploits these regions of high ensemble disagreement. The lack of exploration directions in low-entropy policies channels gradient updates directly into the reward model's "blind spots," accelerating the Goodhart gap.

Optimal Conservatism ($\beta^\star$)

By fitting a power-law curve to the $(\beta, \text{AUGC})$ data, the authors identified a practical optimal conservatism level, $\beta^\star \approx 1.082$.

• Safe Zone: $[0, \beta^\star]$.
• Danger Zone: $(\beta^\star, \infty)$.
• Implication: Beyond $\beta^\star$, marginal increases in conservatism incur disproportionately large increases in hacking risk (super-linear growth of AUGC).

4. Significance and Claims

The paper claims to fundamentally alter the understanding of safe alignment strategies:

• Refutation of Maximal Conservatism: The authors assert that the assumption "more conservative offline training equals safer online adaptation" is violated under specific failure modes involving OOD-driven reward hacking.
• Calibration over Maximization: The field requires "calibrated, not maximal, conservatism." Practitioners should aim for a $\beta$ near $\beta^\star$ to balance alignment fidelity against hacking vulnerability, rather than simply maximizing $\beta$ to stay close to the reference policy.
• Mechanistic Insight: The work provides a principled explanation for why reward hacking occurs, linking it to the interaction between entropy compression, ensemble disagreement, and the specific geometry of the reward model's training distribution.

The authors conclude that safe offline-to-online alignment must explicitly account for the distributional mismatch between the offline alignment signal (DPO) and the online proxy reward, rather than assuming that staying close to a reference policy guarantees safety.