Technical Summary: CROCO – Cross-Lingual Contrastive Preference Tuning on Self-Generations

Problem Statement

Aligning Large Language Models (LLMs) with human preferences is a critical final stage of post-training, typically achieved via Direct Preference Optimization (DPO). While recent advances have demonstrated that DPO can be applied to self-generated samples (using a reward model to score the policy's own outputs) rather than human annotations, these methods have been established exclusively in English.

Extending this to a multilingual setting presents significant challenges. Prior multilingual preference alignment work relies on translation-based signals, exploits capability gaps between English and other languages as implicit rewards, or reweights losses for noisy pairs. However, it remains an open question whether the specific mechanism of contrastive preference tuning on self-generations—where preference pairs are constructed based on relative reward scores within a distribution—transfers across languages without language-specific preference annotations. Furthermore, it is unclear if a reward model (RM) trained solely on English preferences can effectively rank responses in other languages to drive multilingual alignment.

Methodology: CROCO

The authors propose CROCO (Cross-Lingual Contrastive Preference Tuning on Self-Generations), a framework that extends the "sweet spot" preference construction (Xiao et al., 2025) to multiple languages.

Core Mechanism

1. Self-Generation: For a given prompt $x$ in a target language $\ell$, the policy model $\pi_\theta$ generates $K$ candidate responses ($Y_x$).
2. Reward Scoring: An external, off-the-shelf Reward Model (RM) scores these responses. Crucially, the RM is trained on English preferences but built atop a multilingual base model (e.g., Qwen3-8B).
3. Contrastive Pair Construction: Instead of pairing the highest-scoring response with the absolute lowest, the method selects a "sweet spot" to induce controlled contrastiveness:
   • Chosen ($y_c$): The response with the maximum reward.
   • Rejected ($y_r$): The response whose reward is nearest to $\mu_x - 2\sigma_x$ (where $\mu$ and $\sigma$ are the mean and standard deviation of the reward distribution for that prompt).
4. Training: The model is fine-tuned using DPO on these constructed pairs. The authors evaluate two regimes:
   • Monolingual: Tuning independently on data for each language.
   • Multilingual: Jointly tuning on the union of data across all languages.

Experimental Setup

• Models: Two models of different scales were tested: EuroLLM-9B (9B parameters) and aya-3B (3B parameters).
• Data: 20,000 instruction-tuning instances from the Dolci-Instruct-SFT corpus were translated into six European languages (Danish, Dutch, French, German, Italian, Spanish) using TranslateGemma-27B.
• Reward Model: Skywork-Reward-V2-Qwen3-8B, an English-preference-trained RM with a multilingual base.
• Baselines: Standard Supervised Fine-Tuning (SFT) on translated data, SFT on only the highest-rewarded responses (Max-R), and existing multilingual preference tuning methods (ICR, MAPO).

Key Contributions

1. Cross-Lingual Transfer of Contrastive Tuning: The paper demonstrates that contrastive preference tuning on self-generations transfers effectively across languages. DPO on paired self-generations outperforms SFT baselines and existing multilingual methods, while standard SFT on translated data causes catastrophic forgetting.
2. Elimination of Language-Specific Annotation: The study shows that multilingual preference tuning does not require language-specific preference annotations. An English-only RM (atop a multilingual base) produces consistent within-language rankings that drive gains across most languages.
3. Robustness Across Scales and Tasks: The method improves performance on both structured tasks (EuroEval) and open-ended generation (m-ArenaHard 2.1). It matches or exceeds the base model in 6/7 languages for EuroLLM-9B and 4/7 for aya-3B on structured tasks, and wins against the base in all 11 evaluated languages on open-ended generation.
4. Ablation Insights:
   • On-Policy Necessity: The gains require on-policy data. Off-policy responses (generated by a different model) reduce benefits, and online preference optimization fails to improve over the offline variant when using an external RM.
   • Prompt Language: The prompt language should match the chosen response language for optimal performance.
   • Translation Utility: While SFT degrades with translated data, the contrastive signal in DPO remains informative even when translation adds noise, as it relies on relative reward gaps rather than absolute targets.

Results

• EuroEval (Structured Tasks):
  • SFT Failure: Both monolingual and multilingual SFT on translated data caused catastrophic forgetting, with performance drops ranging from 0.1 to 11.3 points.
  • DPO Success: Paired DPO was non-negative in 12/14 settings for aya-3B and improved the baseline in 10/14 settings for EuroLLM-9B. The largest gain was +3.6 points for EuroLLM-9B on Italian (monolingual).
  • Comparison: Paired DPO was the only method that remained non-negative on average across all languages, outperforming ICR and MAPO which often degraded performance.
• m-ArenaHard 2.1 (Open-Ended Generation):
  • Paired DPO achieved length-controlled (LC) win rates between 54.5% and 58.4% against the EuroLLM-9B base across all 7 languages.
  • For aya-3B, LC win rates ranged from 55.5% to 66.3%, with double-digit gains in most languages.
  • The method successfully narrowed the performance gap between the base models and larger reference models (Gemma3-12B and Gemma3-4B) in most languages.
• Generalization: The method generalized positively to held-out languages (Norwegian, Portuguese, Swedish) in EuroEval, whereas multilingual SFT degraded performance on all held-out datasets.

Significance and Claims

The paper claims that CROCO provides a promising direction for multilingual preference tuning by demonstrating that the relative reward gap is a robust signal that transfers across languages, even when the reward model is trained exclusively on English preferences.

The authors posit that the DPO objective's dependence on the relative ordering of paired responses (rather than absolute calibration) allows it to tolerate translation noise and cross-lingual miscalibration. This finding suggests that high-quality multilingual alignment can be achieved without the prohibitive cost of per-language preference annotation, provided that on-policy self-generations are used to construct contrastive pairs.

The work highlights that on-policy data is a critical component, as the contrastive signal is only informative when paired responses originate from the policy's own distribution. Furthermore, it establishes that offline DPO with a fixed reward model is superior to online DPO in this setting, as the latter creates a feedback loop that amplifies reward model biases rather than learning genuine preferences.

Ultimately, the paper argues that a single English-trained reward model, when applied to self-generated multilingual data via contrastive pair construction, is sufficient to drive consistent improvements across diverse languages and model scales, avoiding the catastrophic forgetting associated with standard supervised fine-tuning.