Technical Summary: DynaCF – Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivity

1. Problem Statement

Reward models (RMs) are central to aligning Large Language Models (LLMs) with human preferences, typically trained on pairwise preference data using the Bradley–Terry objective. However, these models frequently exploit shortcut learning, relying on superficial cues (e.g., response length, formatting, verbosity, or template-like expressions) rather than genuine task-relevant quality.

A critical challenge identified is that shortcut risk is not a static property of the dataset. A preference pair may appear benign initially but become shortcut-sensitive if the model's decision boundary relies heavily on surface features to maintain its reward margin. Conversely, a sample that seems to rely on shortcuts might be robust if the model's preference remains stable when those surface properties are neutralized. Existing mitigation strategies often rely on static heuristics or predefined surface features, failing to account for the dynamic nature of how a specific model learns shortcuts during different training stages.

2. Methodology: DynaCF

The authors propose DynaCF (Dynamic Counterfactual Sensitivity), a framework that dynamically diagnoses and reweights training samples based on the current reward model's sensitivity to superficial perturbations.

Core Mechanism

DynaCF operates by constructing semantics-preserving counterfactual variants of the "chosen" response ($y^+$) while keeping the prompt ($x$) and the "rejected" response ($y^-$) fixed. These variants neutralize surface properties (formatting, tone, length) without altering the core task-relevant content.

The process involves three main steps:

1. Counterfactual Construction:

   • For each sample, the system generates valid counterfactuals ($y^{cf}_{i,k}$) using domain-aware rules (e.g., removing Markdown headers, neutralizing tone, shortening verbosity).
   • Strict validity checks ensure the counterfactual preserves the main semantic content (via token overlap thresholds) and, for math/code domains, retains all numbers and logical structures.

2. Sensitivity Diagnosis:

   • The current reward model $R_\theta$ evaluates both the original pair and the counterfactual pairs.
   • The system calculates the reward margin ($m$) for the original and counterfactual pairs.
   • Sensitivity Score ($s_{cf,i}$): Computed based on the magnitude of the margin shift ($\Delta$) and the frequency of preference flips (where the counterfactual chosen response receives a lower reward than the rejected response).
   • A high sensitivity score indicates the model's preference is unstable and likely driven by superficial cues rather than true quality.

3. Dynamic Reweighting:

   • Instead of deleting samples or treating counterfactuals as new labels, DynaCF converts the sensitivity score into a step-local sample weight ($w_i$) for the original Bradley–Terry loss.
   • The weight is defined as $w_i(t) = \max(w_{min}, 1 - \gamma \cdot s_{cf,i}(t))$, where $\gamma$ controls the downweighting strength and $w_{min}$ prevents complete removal.
   • Samples with high shortcut sensitivity are downweighted, reducing their influence on the gradient updates, while low-sensitivity samples retain full weight.
   • This process is performed online during optimization, allowing the weights to adapt as the model's behavior evolves.

3. Key Contributions

• Dynamic Diagnosis: The paper introduces a method to diagnose shortcut reliance dynamically during training, acknowledging that shortcut sensitivity is model-dependent and stage-dependent, rather than a static dataset property.
• Counterfactual Probing: It utilizes semantics-preserving counterfactuals as diagnostic probes to measure the stability of reward margins under surface-only perturbations.
• Reweighting Framework: DynaCF proposes a novel reweighting strategy that suppresses updates from shortcut-sensitive samples without discarding data or altering the preference distribution.
• Empirical Validation: The framework is validated across multiple benchmarks (RM-Bench, RewardBench, RewardBench2) and model scales (Qwen3-4B and Qwen3-8B).

4. Experimental Results

Experiments demonstrate that DynaCF consistently improves the robustness of reward models compared to standard Bradley–Terry training:

• Hard Reasoning & Safety: The most significant gains are observed in "Hard" splits and safety-critical evaluations, where superficial cues are poor indicators of quality.
  • On RM-Bench Hard, DynaCF improved Qwen3-4B by +8.7 points (41.3% $\to$ 50.0%) and Qwen3-8B by +4.6 points (49.8% $\to$ 54.4%).
  • On Safety metrics, Qwen3-4B saw gains of +8.2 on RM-Bench and +4.0 on RewardBench.
• Model Scale: Smaller models (Qwen3-4B) benefited more consistently across benchmarks, suggesting they are more vulnerable to shortcut learning and thus gain more from dynamic reweighting.
• Ablation Studies:
  • Dynamic vs. Static: Static sensitivity scores computed before training performed worse than the baseline, confirming that sensitivity estimates must be updated online to be effective.
  • Counterfactual Usage: Using counterfactuals solely as diagnostic probes (without adding an auxiliary loss) outperformed variants that treated counterfactuals as additional preference labels.
  • Hyperparameters: Moderate reweighting strength ($\gamma$) and a short warmup period yielded the best trade-offs, preventing noisy early estimates from distorting the objective.

5. Significance and Claims

The paper claims that DynaCF offers a robust solution to the problem of reward hacking and overoptimization caused by shortcut learning. By shifting from static filtering to dynamic, model-aware reweighting, the method encourages reward models to rely on task-relevant preference signals rather than superficial patterns.

The authors emphasize that this approach:

• Improves generalization on difficult and safety-sensitive tasks.
• Does not require discarding data or complex architectural changes.
• Provides a diagnostic tool to understand how specific models learn shortcuts at different stages of training.

6. Limitations

The authors acknowledge several limitations:

• Counterfactual Quality: The effectiveness relies on the quality of rule-based counterfactual construction, which may occasionally alter useful information or fail to fully remove targeted cues.
• Scope: Experiments are limited to text-based reward models and benchmarks; applicability to multimodal or process reward models is untested.
• Scope of Probing: Currently, counterfactual probing is applied only to chosen responses; rejected responses may also contain shortcut cues.
• Training Dynamics: The relationship between shortcut reweighting, model capacity, and training dynamics requires further exploration.