Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment

Technical Summary: Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment

Problem Statement
Standard Reinforcement Learning from Human Feedback (RLHF) relies on the Bradley-Terry (BT) model, which assumes human preferences are transitive and can be represented by scalar rewards ($A \succ B \land B \succ C \implies A \succ C$). However, real-world human preferences often exhibit intransitive, cyclic patterns (e.g., Rock-Paper-Scissors dynamics) that scalar models fundamentally fail to capture. While recent approaches like the General Preference Model (GPM) address cyclicity using skew-symmetric bilinear forms, the authors identify a critical theoretical limitation: GPM's implicit formulation entangles hierarchical (transitive) and cyclic dynamics. This entanglement prevents GPM from guaranteeing the representation of "dominant solutions" (candidates that strictly defeat all others) in arbitrary cyclic contexts, as the geometric capacity required for global hierarchies can be "crowded out" by complex local cycles.

Methodology
The paper proposes a two-pronged framework to address these structural and optimization challenges:

1. Hybrid Reward-Cyclic (HRC) Model:

   • Theoretical Basis: Drawing on game-theoretic decomposition (Balduzzi et al., 2019), the authors formalize preference modeling as a Symmetric Zero-Sum Functional-Form Game. They prove that any preference function can be uniquely decomposed into an orthogonal transitive component ($\phi_T$) and a cyclic component ($\phi_C$).
   • Architecture: HRC explicitly instantiates this decomposition. It combines a scalar reward head (capturing the transitive component via a BT-like structure) with a vector-based cyclic head (capturing intransitive dynamics via a GPM-like skew-symmetric operator).
   • Mechanism: The final preference score is the sum of the transitive scalar difference and the cyclic vector interaction: $s_{HRC} = (r(y_i) - r(y_j)) + (v_i^\top W v_j)$. The model enforces unit-norm constraints on embeddings and utilizes a context-aware gating mechanism to modulate cyclic intensity.
   • Advantage: Unlike GPM, which struggles to represent a dominant candidate alongside a cycle without specific high-dimensional constraints, HRC mathematically guarantees the preservation of dominant hierarchies by decoupling the two dynamics. It maintains linear inference complexity $O((2d+1)K)$.

2. Dynamic Self-Play Preference Optimization (DSPPO):

   • Concept: Standard self-play algorithms (e.g., SPPO) optimize against a static preference oracle. The authors argue this limits the model's ability to navigate the multi-dimensional landscape revealed by HRC.
   • Mechanism: DSPPO treats alignment as a time-varying game. It schedules the preference signal over training iterations, gradually transitioning from a robust transitive backbone to refined cyclic nuances.
   • Schedule: The preference score at iteration $t$ is defined as $s_t = (1 + \frac{\lambda}{\sqrt{t}})s_T + (1 - \frac{\lambda}{\sqrt{t}})s_C$. This curriculum learning approach stabilizes training by establishing a global quality baseline before introducing complex local dynamics, theoretically ensuring convergence to the Nash Equilibrium with an $O(1/\sqrt{T})$ rate.

Key Contributions

• Theoretical Unification: The paper provides a theoretical proof that HRC unifies the BT model and GPM by explicitly decomposing preferences into orthogonal transitive and cyclic components, resolving the structural inability of GPM to guarantee dominant solutions in mixed settings.
• Algorithmic Innovation: The introduction of DSPPO, a dynamic optimization framework that leverages the decomposed HRC signals to guide policy evolution, offering a more robust convergence trajectory than static baselines.
• Empirical Validation: Extensive experiments demonstrate that HRC outperforms BT and GPM in preference modeling accuracy, particularly in domains requiring the handling of "Ties" and complex reasoning. The HRC+DSPPO framework significantly improves downstream generation quality.

Results

• Preference Modeling (RewardBench 2): HRC consistently outperformed BT and GPM baselines. On Gemma-2B-it, HRC achieved a 57.63% average accuracy, surpassing the strongest baseline (GPM) by 1.23%. Notably, in the "Ties" domain, HRC achieved 74.51% compared to 73.53% for both BT and GPM, validating its ability to handle non-strict preferences.
• Downstream Alignment:
  • AlpacaEval 2.0: Using Gemma-2B-it as the base, HRC+DSPPO achieved a peak length-controlled win-rate of 44.75%, significantly outperforming SPPO baselines trained with BT or GPM.
  • Arena-Hard-v0.1: HRC+DSPPO achieved a 46.8% win rate on Gemma-2B-it, a 3.2% improvement over the best baseline, demonstrating superior capability on challenging prompts.
  • MT-Bench: The framework showed robust performance in multi-turn conversations, with HRC+DSPPO achieving an average score of 8.29 on Gemma-2B-it.
• Synthetic Data: Experiments on synthetic "dominant + cycle" datasets confirmed that HRC converges faster and achieves higher accuracy than GPM in mixed transitive-cyclic settings, whereas low-dimensional GPM failed to capture cyclic structures.

Significance
The paper claims that faithfully representing the complex, heterogeneous nature of human preferences—specifically the duality between global transitive hierarchies (e.g., safety, helpfulness) and local intransitive cycles (e.g., stylistic trade-offs)—is essential for robust AI alignment. By explicitly disentangling these components, the proposed framework overcomes the structural limitations of existing scalar and purely cyclic models. The authors posit that this approach enables AI systems to navigate diverse ethical and cultural landscapes more effectively, fostering greater trust in automated decision-making by respecting the non-strict, multi-dimensional nature of human judgment. The work suggests that future alignment strategies should move beyond static scalar rewards to embrace dynamic, game-theoretic decompositions of preference.