Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning

1. Problem Statement

Large Language Models (LLMs) have achieved significant success in aligning with general human preferences (e.g., being helpful, harmless, and honest) via methods like RLHF and DPO. However, a critical gap remains in individual-level alignment, which faces two major challenges:

• Diversity and Heterogeneity: Different users hold unique, often idiosyncratic preferences.
• Dynamics and Conflict: A single user's preferences are not static; they can shift based on context, mood, or task. Crucially, users may hold contradictory preferences simultaneously (e.g., being risk-averse in financial matters but risk-taking in creative brainstorming).

Existing methods struggle with this complexity:

• Non-adaptive methods fail to capture individual nuances.
• Weight-adaptive methods often require separate models for each preference, incurring high deployment costs.
• Current training paradigms typically rely on explicit preference prompts at inference time or assume preferences are stable and consistent, failing to handle conflicting values within a single model.

2. Methodology: Preference-Paired Fine-Tuning (PFT)

The authors propose Preference-Paired Fine-Tuning (PFT), a novel training paradigm designed to align a single model with multiple, including contradictory, preferences.

Core Concept

Instead of training on isolated preference examples, PFT utilizes conflict pairs. For a given scenario $x$, the model is trained on two opposing preference descriptors ($p^+$ and $p^-$) and their corresponding ideal responses ($y^+$ and $y^-$) simultaneously.

Theoretical Framework

• Preference Manifold: The authors model user preferences as lying on a low-dimensional manifold ($M_u$) within a high-dimensional preference space.
• Synchronous Optimization: PFT optimizes a paired loss function that minimizes the cross-entropy loss for both sides of the contradiction simultaneously:
  $$L_{PFT}(\theta) = \mathbb{E}[\lambda^+ \ell(\theta; x, p^+, y^+) + \lambda^- \ell(\theta; x, p^-, y^-)]$$
• Generalization Guarantee: Theoretically, PFT tightens the risk bound on the preference manifold compared to single-preference training. By training on conflicting pairs, the model:
  1. Reduces the effective covering radius ($\epsilon$): A conflicting pair covers the entire segment between two preference vectors, requiring fewer anchors to cover the manifold.
  2. Reduces preference sensitivity ($L_p$): Synchronous optimization enforces consistency across opposing directions, smoothing the loss landscape and reducing the Lipschitz constant.

Rapid Customization

After the general PFT model is trained, it can be rapidly customized for specific users using In-Context Learning (ICL). By providing a small amount of user history (e.g., 3 examples), the model infers a preference vector and steers its generation toward that specific user's dynamic preferences without retraining.

3. Key Contributions

1. Value Conflict Dilemma (VCD) Dataset:

   • A new dataset specifically designed to evaluate models on scenarios involving conflicting human preferences.
   • It covers four key dimensions: Risk (Taking vs. Averse), Social (Competitive vs. Collaborative), Time (Immediate vs. Delayed Gratification), and Decision-making (Intuitive vs. Analytical).
   • Unlike existing datasets, VCD supports paired supervision, contextualized scenarios, and open-ended generation.

2. Preference-Paired Fine-Tuning (PFT) Framework:

   • A training method that enables a single model to master contradictory preferences.
   • Demonstrates that training on single-choice data (paired examples) significantly improves performance in both classification and open-ended text generation.

3. Rapid Customization via ICL:

   • Shows that with limited user history, the PFT model can infer preference vectors and adapt to specific user needs, achieving a 44.76% improvement in user-specific alignment compared to single-preference models.

4. Experimental Results

The authors evaluated PFT on the VCD and Behavioral Question Datasets (BQD) using models like Qwen2.5 and Llama-3.1.

• Performance Superiority:
  • PFT outperformed single-preference training, DPO, SFT, CPO, and CAA.
  • Multi-choice Classification: Achieved up to 96.67% accuracy (on Llama-3.1-8B).
  • Open-Ended Generation: Achieved the highest human evaluation score of 8.69 (on a 1-10 scale).
• Conflict Resolution: While DPO sometimes outperformed PFT in single-preference settings, PFT consistently excelled when handling contradictory preferences, demonstrating superior conflict resolution capabilities.
• Data Efficiency: PFT showed strong performance even with limited data (1,000 examples). Performance only converged to single-preference baselines when data dropped below ~650 examples, indicating higher data efficiency.
• Ablation Studies:
  • Weighting: Balanced weighting ($\lambda^+ = \lambda^- = 1$) yielded the best results for handling conflicts.
  • Multi-Pair Training: Training on mixed preference pairs allowed the model to capture shared structures, leading to more balanced performance across different preference dimensions compared to training on single pairs.
• General Capabilities: PFT did not degrade the model's general capabilities (measured on MMLU), confirming it improves alignment without sacrificing reasoning or factual knowledge.

5. Significance and Implications

• Scalable Personalization: PFT offers a practical solution for building "one model to rule them all" that can adapt to diverse and conflicting user values without the need for maintaining multiple specialized models.
• Handling Real-World Complexity: By explicitly addressing value conflicts and dynamic shifts, PFT moves beyond the "universal alignment" paradigm toward conflict-aware personalization, which is essential for real-world applications like personalized assistants, decision support systems, and interactive agents.
• Efficiency: The ability to rapidly customize a pre-trained PFT model using few-shot learning makes it highly deployable in scenarios where user data is scarce or privacy-constrained.

In conclusion, this paper establishes that training LLMs on paired, conflicting preferences creates a more robust, adaptable, and personalized model capable of navigating the complex and dynamic nature of human values.