Technical Summary: Beyond Uniform Forgetting in Sequential Direct Preference Optimization

Problem Statement

Aligning Large Language Models (LLMs) with human preferences typically involves optimizing multiple behavioral objectives (e.g., helpfulness, harmlessness, honesty, instruction following). In practice, these objectives are often optimized sequentially using methods like Direct Preference Optimization (DPO) rather than jointly, to avoid retraining from scratch. However, it remains unclear whether later training stages uniformly degrade preferences learned in earlier stages (catastrophic forgetting) or if the outcome depends on the specific relationship between objectives, signal strength, and training order.

Existing literature suggests that forgetting in sequential alignment is often moderate and task-dependent, but prior analyses have largely relied on aggregate metrics (e.g., multiple-choice accuracy) or focused on knowledge benchmarks. These approaches may mask heterogeneous changes at the sample level and fail to distinguish between direct gradient conflict, distributional drift, or signal imbalance as the primary drivers of preference degradation.

Methodology

The authors conduct a systematic empirical study using Llama-3.1-8B-Instruct with LoRA adapters. The experimental pipeline involves training two distinct preference objectives sequentially (Stage 1 and Stage 2) and evaluating the model on all objectives after each stage.

Experimental Regimes

The study evaluates four distinct preference settings, each tested in both training orderings (A→B and B→A):

1. Distributional Conflict (HH-RLHF): Compares "helpful" vs. "harmless" objectives, which differ in prompt distribution and preference dimensions.
2. Multi-Attribute Interaction (HelpSteer2): Compares "verbosity" vs. "coherence," testing interactions between fine-grained response attributes.
3. Strong Safety Signal (PKU-SafeRLHF): Compares "harmless" vs. "helpful," where the safety signal is significantly stronger than in HH-RLHF.
4. Compatible Objectives (UltraFeedback): Compares "instruction following" (IF) vs. "honesty," where objectives reward overlapping response-quality behaviors.

Evaluation Protocols

To capture nuanced changes, the authors employ three levels of analysis:

1. Aggregate Metrics:
   • DPO-Relative Reward Margin: Measures the preference gap relative to a fixed base model reference (not the previous stage's adapter) to ensure consistent scaling.
   • Relative Preference Accuracy: The fraction of pairs where the policy improves the chosen-over-rejected gap relative to the base model.
2. Pair-Level Analysis:
   • Length-Normalized Policy Margins: Tracks the margin change ($\Delta_i$) for individual chosen–rejected pairs between stages.
   • Quartile Decomposition: Pairs are sorted by their Stage 1 margin (confidence) and divided into quartiles (Q1: hard/low-confidence; Q4: easy/high-confidence) to determine where changes concentrate.
3. Mechanistic Diagnostics:
   • Gradient Cosine Similarity: Measures the angle between Stage 2 DPO gradients and a fixed Stage 1 objective gradient (computed over LoRA parameters) to detect direct opposition.
   • Adapter Movement: Analyzes the displacement of LoRA parameters between stages to determine if updates are orthogonal or opposing.

Key Results

1. Non-Uniform Forgetting Patterns

Sequential DPO does not produce a single "uniform forgetting" pattern. The outcome ranges from partial degradation to stability, pair-level redistribution, or positive transfer, depending on the objective relationship and signal strength:

• HH-RLHF: Shows partial degradation. In the helpful→harmless order, the helpful margin drops significantly (−68.8%), but the objective remains above the neutral baseline.
• HelpSteer2: Shows asymmetric degradation. In verbosity→coherence, verbosity degrades slightly (−6.7%), while coherence improves.
• PKU-SafeRLHF: Demonstrates stability and positive transfer. The strong safety signal prevents degradation; in harmless→helpful, the harmless margin actually increases (+2.859) after helpful training.
• UltraFeedback: Shows positive transfer. Both objectives improve or remain stable regardless of order, suggesting high compatibility between instruction following and honesty.

2. Heterogeneity in Pair-Level Changes

Aggregate metrics often mask significant variation across individual preference pairs:

• Redistribution: Later training can degrade some previously learned preferences while improving others.
• Quartile Dynamics: The concentration of change varies by setting.
  • In HH-RLHF (helpful→harmless), degradation is concentrated in high-confidence (Q4) pairs.
  • In PKU-SafeRLHF (harmless→helpful) and UltraFeedback, high-confidence pairs are often reinforced (improved), while changes are distributed differently across quartiles.
  • This indicates that "forgetting" is not a uniform shift but a structured redistribution dependent on the specific objective interaction.

3. Mechanistic Insights: Lack of Direct Gradient Conflict

Contrary to the hypothesis that forgetting is driven by direct gradient opposition:

• Gradient Orthogonality: Across all eight orderings, the cosine similarity between Stage 2 gradients and the previous objective's reference gradient is near zero (mean angle $\approx 90^\circ$). There is no evidence of systematic gradient opposition.
• Adapter Movement: Stage 2 updates move the LoRA adapter in a direction near-orthogonal to the Stage 1 adapter (cosine $\approx 0$), rather than reversing it.
• Implication: Preference changes are likely driven by indirect mechanisms such as distributional drift, signal imbalance, or indirect parameter movement, rather than direct gradient conflict.

Contributions

1. Systematic Empirical Study: The paper provides a comprehensive analysis of sequential DPO across four distinct preference regimes, demonstrating that preference change is a spectrum (from forgetting to positive transfer) determined by objective relationships and signal strength, not just sequential training.
2. Pair-Level Diagnostics: The authors introduce a pair-level policy-margin analysis using quartile decomposition. This reveals heterogeneous redistribution patterns that aggregate reward margins and relative accuracy conceal.
3. Mechanistic Evidence: By measuring gradient cosine similarity and adapter movement, the study provides evidence that preference changes occur despite near-orthogonal update directions, challenging the view that direct gradient opposition is the primary driver of forgetting.

Significance and Claims

The paper argues that future sequential alignment pipelines should not assume that later objectives uniformly affect earlier preferences. Instead, practitioners must account for objective compatibility and signal strength.

The authors modestly claim that their findings suggest indirect mechanisms (drift, signal imbalance) are more likely drivers of preference change than direct gradient conflict. They note limitations, including the use of LoRA adapters on an 8B model and the restriction of mechanistic diagnostics to parameter space, suggesting that future work should extend these diagnostics to larger models and full-parameter fine-tuning to confirm generalizability. The work aims to improve the monitoring and design of sequential post-training pipelines rather than proposing a specific new algorithm.