💬 NLP

An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift

This paper presents a systematic empirical study demonstrating that while pseudo-labeling adaptation strategies effectively mitigate performance degradation caused by domain shift in preference-tuned language models, they simultaneously induce mode collapse, thereby revealing a fundamental trade-off between generalization and diversity.

Original authors: Constantinos Karouzos, Xingwei Tan, Nikolaos Aletras

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Constantinos Karouzos, Xingwei Tan, Nikolaos Aletras

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern computers that read and write language are not born knowing how to be helpful, safe, or polite. They start as vast libraries of text, capable of predicting the next word in a sentence but lacking a sense of what a human actually wants. To fix this, researchers teach these machines to align with human values. They do this by showing the computer pairs of answers—one good, one bad—and asking it to learn which one people prefer. This process, often called preference tuning, is the standard way to turn a raw language model into a helpful assistant. However, a persistent problem has emerged: when these models are trained on one specific type of conversation, they often struggle to perform well when the topic or style changes. A model trained to summarize casual internet posts might fail miserably when asked to summarize formal news articles, even though the task of summarizing is the same.

A team of researchers at the University of Sheffield set out to understand exactly why this happens and how to fix it. They treated the problem like a test of adaptability. They took several different methods for teaching computers to align with human preferences and tested them across three distinct challenges. First, they tried to move a model from summarizing informal internet comments to summarizing formal news stories. Second, they moved a model from answering engineering questions to answering cooking questions. Third, they tested if a model trained to refuse requests about cybercrime could also refuse requests about physical violence. The researchers wanted to see if the specific mathematical method used for training mattered more, or if the way they prepared the data for the new topic was the deciding factor.

The study revealed a surprising truth: the method used to prepare the data for the new topic matters far more than the specific training algorithm chosen. When the researchers simply taught the model on the old data and then asked it to perform the new task, the results were often poor. The model would either forget what it knew or fail to understand the new context. However, when they used a technique called pseudo-labeling, the results changed dramatically. In this approach, a much larger, smarter computer model generated examples of what good answers looked like in the new domain. The smaller model then learned from these synthetic examples. This strategy lifted the performance of the models significantly. For instance, in the task of summarizing news, a model using this method achieved a success rate of over 83 percent, whereas models without this help struggled to reach even 40 percent. The researchers found that this boost was so strong that it often didn't matter which of the five different alignment algorithms they used; the quality of the synthetic data was the dominant factor.

Yet, this success came with a hidden cost. The researchers discovered that while these models became excellent at getting the right answer, they lost their ability to be diverse. When a model learned from the synthetic examples, it began to repeat the same patterns over and over. In the news summarization task, the model started producing every single summary with the exact same structure, beginning with "The article discusses" and following a rigid template, regardless of whether the story was about food, travel, or sports. This phenomenon, known as mode collapse, meant the model became a reliable but monotonous machine. It could summarize a news story perfectly, but it lost the unique voice and variety that a human writer might bring. The study showed that the models trained this way were highly consistent but linguistically flat, effectively trading creativity for reliability.

The researchers also found that the type of task influenced how much the model suffered from this loss of variety. In the safety task, where the goal is to refuse harmful requests, the synthetic training was a clear win. The models learned to refuse dangerous requests with near-perfect accuracy, regardless of whether the request was about cybercrime or physical violence. Here, the loss of variety was not a problem; the goal was simply to be safe and consistent. However, in the question-answering task, the researchers noticed a subtle issue. A model trained on engineering questions would sometimes answer cooking questions with the rigid, technical style of an engineer. While a standard computer judge might rate this as helpful because the logic was sound, a human reader would find it out of place. The model had learned the facts but missed the cultural nuance of the new community.

Ultimately, the study suggests that there is no single perfect way to teach a language model to adapt. If the goal is high reliability, such as in safety applications or strict summarization, using synthetic data generated by a stronger model is the most effective path, even if it makes the output repetitive. If the goal is to maintain a rich variety of voices, such as in creative writing, a mix of old and new data works better, though it may not reach the same peak performance. The researchers concluded that the choice of strategy depends entirely on what the user values more: the certainty of a correct answer or the diversity of the expression. They warned that relying too heavily on synthetic data could lead to a future where all machine-generated content sounds the same, homogenizing the digital world. The path forward, they suggest, involves finding a balance that allows models to be both helpful and humanly varied.

Technical Summary: An Empirical Study on Preference Tuning Generalization and Diversity Under Domain Shift

Problem Statement
Large Language Models (LLMs) rely on post-training preference optimization (e.g., RLHF, DPO) to align with human judgments of quality, helpfulness, and safety. While effective within training domains, prior work indicates that these models often suffer performance degradation and reduced helpfulness when applied to out-of-domain (OOD) tasks. However, the extent to which specific adaptation strategies can mitigate this "domain shift" remains unexplored. Furthermore, existing literature lacks a systematic comparison of diverse alignment objectives (both online and offline) and their interaction with domain adaptation techniques, particularly regarding the trade-off between generalization performance and output diversity.

Methodology
The authors conduct a comprehensive empirical study decomposing preference optimization under domain shift into two axes: Alignment Strategy and Adaptation Strategy.

Alignment Objectives: The study evaluates five distinct preference optimization methods alongside standard Supervised Fine-Tuning (SFT):
- Online RL: RLHF with Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO).
- Offline/RL-free: Direct Preference Optimization (DPO), Kahneman-Tversky Optimization (KTO), and Odds-Ratio Preference Optimization (ORPO).
Adaptation Strategies: The study tests how models adapt from a source domain ( $D_S$ ) to a target domain ( $D_T$ ) using:
- Source-only SFT: Training only on labeled source data.
- Target SFT: Training on unlabeled target data.
- Mix SFT: Training on a mixture of source and target data.
- Pseudo-Labeling: Generating a synthetic preference dataset ( $D^{synth}_T$ ) for the target domain using a stronger "teacher" model (Llama-3.3-70B) to distill preferences, which are then used for SFT or preference optimization.
Testbeds: Experiments are conducted across three distinct tasks involving domain shifts:
- Summarization: From informal Reddit TL;DR posts (Source) to formal CNN/DailyMail news articles (Target).
- QA Helpfulness: From the engineering-focused AskEngineers to the culinary-focused AskCulinary (Stanford Human Preferences dataset).
- Safety: Transferring safety alignment from Cybercrime refusal to Violence/Physical Harm refusal (PKU-SafeRLHF).
Evaluation:
- Generalization: Measured via LLM-as-a-judge win rates (comparing adapted models against human references) and safety scores (via OpenAI Moderation API).
- Diversity: Quantified using syntactic (EAD n-grams), semantic (SBERT cosine similarity), and logical (NLI) diversity metrics.

Key Results

Adaptation Strategy Dominates: The choice of adaptation strategy is often more critical than the choice of alignment objective. Pseudo-labeling consistently yields the highest target-domain win rates and safety scores across all tasks, significantly outperforming non-synthetic methods. For instance, in summarization, pseudo-labeled SFT achieved an 83.37% win rate on the target domain, compared to ~36–58% for other methods.
Offline vs. Online Generalization: Offline methods (DPO, ORPO, KTO) trained solely on source data exhibit severe over-specialization, leading to large generalization gaps (e.g., DPO on Llama-3.1-8B showed a 31.78% gap in summarization). Online methods (GRPO, PPO) demonstrate greater robustness, with GRPO maintaining a generalization gap of only 3.79% in the same setting.
The Generalization–Diversity Trade-off: A critical finding is that while pseudo-labeling maximizes performance, it induces mode collapse. Models trained on pseudo-labeled data exhibit near-zero semantic diversity and a significant reduction in syntactic variety. They tend to overfit the teacher model's recurring templates (e.g., a rigid "The article discusses..." structure in summarization), sacrificing the creative variance seen in SFT-only baselines.
Task-Specific Sensitivity:
- Summarization: Highly sensitive to domain shift; pseudo-labeling is essential for high performance but causes structural collapse.
- QA Helpfulness: Shows near-zero generalization gaps even without adaptation, likely due to domain-agnostic helpfulness signals. However, qualitative analysis reveals "epistemic drift," where models trained on source data answer culinary questions with engineering-style rigor, which standard judges rate as helpful despite pragmatic misalignment.
- Safety: Pseudo-labeling achieves near-perfect safety scores (>95%) across harm categories, effectively transferring safety priors. However, applying DPO on top of pseudo-labeled SFT can sometimes erase these safety gains in specific model configurations, whereas KTO and ORPO remain robust.

Significance and Claims
The paper claims to provide the first systematic characterization of preference optimization generalization under domain shift. Its primary contributions are:

Empirical Evidence: Demonstrating that adaptation strategies (specifically pseudo-labeling) matter more than the specific alignment objective for achieving high target-domain performance.
Identification of a Trade-off: Highlighting a fundamental tension where synthetic supervision (pseudo-labeling) maximizes reliability and win rates but severely compromises output diversity, leading to "structural mode collapse."
Deployment Guidance: The authors suggest that for high-constraint tasks requiring factual consistency (e.g., news summarization), pseudo-labeling with offline methods is optimal. Conversely, for creative or open-ended tasks, mixing source and target data (Mix-SFT) or using online objectives like GRPO offers a better balance between generalization and diversity.

The study concludes that while synthetic supervision is a powerful tool for domain adaptation, it acts as a "double-edged sword," risking the homogenization of model outputs. Future work is suggested to explore hybrid data mixtures and optimization objectives that explicitly account for diversity to prevent collapse into single-mode distributions.

More like this