Technical Summary: When In-Distribution Gains Fail: Evaluating Weak-to-Strong Reward Models under Preference Shift

1. Problem Definition

The paper addresses a critical gap in Weak-to-Strong (W2S) generalization for scalable oversight. While W2S frameworks aim to train a stronger student model using labels from a weaker teacher to elicit capabilities beyond the teacher's own performance, existing evaluations typically rely on matched train-test distributions.

The authors argue that this setup overestimates the reliability of alignment in real-world deployment. Specifically, they investigate W2S preference learning under zero-shot distribution shift. The core problem is that strong students trained on weak preference labels from a source domain often appear successful in-distribution (ID) but fail to transfer their learned alignment behavior to unseen target domains (Out-of-Distribution, OOD).

The paper identifies a specific failure mode: representational failure. Weak-supervised fine-tuning can pull the strong model's internal representations toward source-domain specific artifacts rather than maintaining broadly transferable preference representations. Consequently, the model fits the source data but loses generalizability.

2. Methodology: Representation Anchoring (ANCHOR)

To mitigate representational drift, the authors propose Representation Anchoring (ANCHOR), a regularization technique designed to constrain the student model during weak-supervised fine-tuning.

Core Mechanism

ANCHOR augments the standard weak-to-strong preference learning objective with a regularizer that keeps the student's hidden states close to those of a frozen pretrained reference model (initialized from the same checkpoint as the student before fine-tuning).

The training objective combines two terms:

1. Weak Preference Loss ($L_{w2s}$): The standard Bradley-Terry loss where the strong student learns to match the soft preference probabilities ($q_i$) generated by the weak teacher.
2. Anchoring Loss ($L_{anchor}$): A masked Mean Squared Error (MSE) between the hidden states of the trainable student and the frozen reference model. This loss is applied only to response tokens, excluding prompt and padding tokens.

The total loss for a sample $i$ is:
$$L(i) = L_{w2s}(i) + \lambda L_{anchor}(i)$$
where $\lambda$ controls the strength of the representation preservation.

Variants

The authors evaluate two variants based on which layers are anchored:

• Last-Layer ANCHOR (Default): Applies the loss to the final transformer layer. This is computationally efficient and directly regularizes the representation used by the reward head.
• Middle-Layer ANCHOR: Applies the loss to intermediate layers (e.g., 50% and 75% of layers) to preserve general features while allowing the final layers more flexibility.

3. Evaluation Protocol and Metrics

The paper introduces a zero-shot preference-domain shift evaluation protocol. Instead of standard held-out splits, models are trained on one dataset (source domain) and evaluated on:

• In-Distribution (ID): Held-out test split from the same source dataset.
• Out-of-Distribution (OOD): Test splits from different datasets within the same broad preference category (e.g., training on HelpSteer3, testing on Anthropic Helpful or UltraFeedback).

To quantify performance, the authors propose three Transfer-Aware Metrics:

1. Weak-to-Strong Raw Gain (WRG): Measures ID improvement over the weak teacher ($Acc_{student} - Acc_{teacher}$ on source).
2. Absolute OOD Gain (AOG): Measures transfer to a target domain ($Acc_{student} - Acc_{teacher}$ on target).
3. Net Transfer Score (NTS): A composite metric that penalizes source-domain regression.
   $$NTS = AOG - \max(0, Acc_{teacher}^{ID} - Acc_{student}^{ID})$$
   This ensures that gains in OOD domains are not achieved by sacrificing reliability in the source domain.

4. Key Results

Experiments were conducted across two preference categories (Helpful and Harmless) and two model families (Llama and Qwen), using datasets like HelpSteer3, Anthropic Helpful/Harmless, UltraFeedback, PKU-SafeRLHF, and RAIL.

• ID Success Masks OOD Failure: Baseline methods (Naive W2S and Confidence-based W2S) often achieve strong ID performance (high WRG) but exhibit significant performance drops or negative transfer in OOD settings. For example, a Naive model trained on HelpSteer3 showed a 20-point accuracy drop when evaluated on Anthropic Helpful.
• SEAM Trade-off: The SEAM baseline showed a sharp trade-off, achieving competitive OOD gains but often at the cost of severe ID degradation (negative WRG).
• ANCHOR Performance: ANCHOR consistently achieved the most balanced results. It maintained competitive ID performance while significantly improving OOD transfer.
  • In the Llama/Helpful setting, ANCHOR improved AOG on Anthropic Helpful from 0.60 (Naive) to 4.41.
  • It yielded the highest Net Transfer Scores across most configurations, demonstrating that it successfully preserves source-domain reliability while enabling cross-domain generalization.
• Ablation Findings:
  • Anchoring Strength ($\lambda$): A moderate anchoring strength (e.g., $\lambda = 10^{-4}$) provided the best balance. Stronger constraints hindered adaptation to the weak signal, while weaker constraints failed to prevent drift.
  • Layer Choice: Last-Layer ANCHOR generally provided a better overall balance than Middle-Layer ANCHOR, suggesting that regularizing the final representation is sufficient and more efficient for this task.

5. Significance and Claims

The paper claims three primary contributions:

1. Evaluation Protocol: It establishes a rigorous zero-shot preference-domain shift protocol, arguing that ID accuracy alone is insufficient for evaluating W2S reward models.
2. Empirical Evidence: It demonstrates that current W2S methods are fragile under distribution shift, revealing that in-distribution gains can overestimate alignment reliability.
3. Robust Method: It introduces ANCHOR, a simple regularization method that effectively limits representation drift. The authors claim this provides a practical path toward more robust preference transfer without altering the inference-time architecture.

The authors conclude that robust W2S alignment must be evaluated not just by whether strong models learn from weak supervision, but by whether the learned reward function remains reliable beyond the specific training domain.

6. Limitations

The authors acknowledge several limitations:

• Offline Proxy Metrics: The evaluation relies on offline pairwise accuracy, which may not fully capture nuances of downstream deployment (e.g., best-of-n selection, online RLHF, or open-ended generation quality).
• Computational Cost: While inference remains standard, training requires an additional forward pass through the frozen reference model, increasing VRAM and time requirements.
• Scope: Experiments are limited to specific helpful/harmless domains and model scales; broader evaluation across diverse task domains and larger model families is needed.