💬 NLP

Value Drifts: Tracing Value Alignment During LLM Post-Training

This paper investigates the dynamics of value alignment during LLM post-training, revealing that Supervised Fine-Tuning primarily establishes a model's values while subsequent preference optimization rarely re-aligns them, with different algorithms yielding distinct outcomes even on identical datasets.

Original authors: Mehar Bhatia, Shravan Nayak, Gaurav Kamath, Marius Mosbach, Karolina Stańczak, Vered Shwartz, Siva Reddy

Published 2026-07-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Mehar Bhatia, Shravan Nayak, Gaurav Kamath, Marius Mosbach, Karolina Stańczak, Vered Shwartz, Siva Reddy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brand-new, super-smart robot that has read almost everything on the internet. It knows facts, can write poems, and solve math problems. But right now, it's a bit like a blank slate when it comes to opinions. If you ask it, "Is it better to save the planet or make money fast?" it might just list facts without taking a side. To make this robot helpful and safe for humans, we need to teach it how to think about these tricky questions. This process is called "alignment."

Think of alignment like training a puppy. First, you teach it basic commands like "sit" and "stay" (this is called Supervised Fine-Tuning, or SFT). You show it examples of good behavior so it learns the rules. Then, you take it to the dog park to play with other dogs and learn social cues, rewarding it when it plays nicely and scolding it when it bites (this is called Preference Optimization). The big question scientists have been asking is: When does the puppy actually learn its personality? Does it learn its core values during the basic training, or does it get its personality from the dog park? Understanding this is crucial because if we don't know where the robot's values come from, we might accidentally teach it to be rude, biased, or dangerous without realizing it.

This paper, titled "Value Drifts," goes behind the scenes of that training process to watch exactly when and how large language models (LLMs) learn their values. The researchers treated the training process like a time-lapse video, checking the robot's "opinions" at every single step of its training, rather than just looking at the finished product at the end.

Here is what they found, and it might surprise you:

1. The First Lesson is the Most Important
The study discovered that the "Supervised Fine-Tuning" (SFT) phase is where the robot actually gets its personality. It's like the moment the puppy learns to sit; once that's done, the core behavior is set. The researchers found that as soon as the model started training on instruction datasets, its stance on big topics (like immigration or climate change) shifted dramatically and quickly. If the training data was full of neutral, "I'm not sure" answers, the robot became neutral. If the data was full of supportive, "Yes, let's do it!" answers, the robot became supportive. This "value imprinting" happened very early in the process, often within the first few hundred steps of training.

2. The Dog Park Doesn't Change Much
After the initial training, the models went through "Preference Optimization" (the dog park phase), where they were shown pairs of answers and told which one humans preferred. The researchers expected this to tweak the robot's values, maybe making it a little more polite or a little more helpful. But they found that for the most part, this phase did almost nothing to change the values established in the first step. The robot's opinions stayed exactly where they were left by the first training phase.

3. Why the Dog Park Failed (and How to Fix It)
Why didn't the second phase change anything? The team realized it was because the "training treats" (the preference datasets) were too similar. In the standard datasets used, the "good" answer and the "bad" answer were often just different in style (one was longer, one was shorter) but had the exact same opinion. It's like trying to teach a dog to fetch a ball by showing it two identical sticks and saying, "Pick the better stick." The dog can't learn a new trick if the options are the same.

However, the researchers proved that if you do give the robot a clear choice—showing it one answer that supports a value and another that strongly opposes it—the robot can change its mind. But the way it changes depends on the specific algorithm used to teach it. Some algorithms (like DPO) are like stubborn teachers who only reinforce what the robot already believes, while others (like SIMPO) are more cautious and make smaller, slower changes.

The Bottom Line
The paper suggests that if we want AI to have specific values, we can't rely on the final "preference" stage to fix things. The heavy lifting happens during the initial instruction training. If we want an AI that is neutral on sensitive topics, we need to make sure the initial training data is neutral. If we want it to be supportive, the initial data must be supportive. The final polishing stage is mostly just for style, not for changing the robot's soul. This is a huge insight because it tells us that the most critical part of building a safe AI is curating the very first set of examples we show it, not just the final round of feedback.

Technical Summary: Value Drifts: Tracing Value Alignment During LLM Post-Training

Problem Statement

As Large Language Models (LLMs) become integral to societal discourse, their alignment with human values is critical. While standard alignment pipelines involve Supervised Fine-Tuning (SFT) followed by Preference Optimization (e.g., RLHF, DPO), the dynamics of how and when models acquire specific values during these post-training stages remain opaque. Prior work largely focuses on post-hoc evaluations of fully trained models, failing to disentangle the specific contributions of training algorithms versus datasets, or to trace the temporal evolution of value acquisition. This lack of granularity hinders the ability to curate data and select algorithms that effectively align models with desired human value systems.

Methodology

The authors introduce a framework to trace "value drifts"—shifts in a model's expressed values over the course of training.

1. Operationalizing Values and Stances:

Values: Defined as latent, subjective positions underlying responses to value-laden prompts (normative judgments).
Stances: The concrete manifestation of values, categorized as support, neutral, or oppose for a given topic.
Measurement: The authors utilize V-PRISM, a curated evaluation set of 550 value-probing prompts across 11 semantic categories (e.g., immigration, abortion, climate change). For each model checkpoint, responses are generated and classified by an LLM (GPT-4o) to determine stance distributions.

2. Metrics:

Drift Magnitude: The change in expected stance probability between two checkpoints (start and end of a training phase).
Drift Time: The fraction of training steps required for a stance probability to reach its eventual extremum (peak or low point) within a training phase.

3. Experimental Setup:

Models: Llama-3 (3B, 8B) and Qwen-3 (4B, 8B).
SFT Datasets: WildChat (real human-LLM conversations) and Alpaca (synthetic instruction-following data).
Preference Optimization Datasets: UltraFeedback and HH-RLHF.
Algorithms: Proximal Policy Optimization (PPO), Direct Preference Optimization (DPO), and Simple Preference Optimization (SIMPO).
Synthetic Control: To isolate the effect of algorithms from dataset properties, the authors constructed a synthetic preference dataset with a controlled, explicit "value gap" (pairs where one response strongly supports a value and the other strongly opposes it).

Key Results

1. SFT is the Dominant Driver of Value Alignment

Value alignment occurs rapidly during the SFT phase, often within the first few hundred steps.
The choice of SFT dataset imprints a distinct value profile on the model. For instance, models trained on WildChat (derived from GPT-3.5 interactions) exhibit a predominantly neutral stance (often due to refusal patterns), whereas models trained on Alpaca exhibit a supportive bias.
This "value imprinting" establishes a strong prior that persists through subsequent training stages.

2. Standard Preference Optimization Induces Minimal Value Drift

When using standard datasets (UltraFeedback, HH-RLHF), PPO, DPO, and SIMPO result in negligible changes to the value profiles established during SFT.
Cause: The authors identify a "low value gap" in standard preference datasets. The "chosen" and "rejected" responses in these datasets often differ primarily in stylistic dimensions (tone, verbosity) rather than underlying stances or values. Consequently, the optimization signal is too weak to reshape the model's value priors.

3. Algorithmic Differences Emerge with Controlled Value Gaps

When provided with a synthetic dataset containing a clear value gap, preference optimization algorithms can shift values, but they do so differently:
- PPO: Largely preserves SFT-induced values due to the KL-divergence penalty anchoring the policy to the reference model. Drift magnitude is low unless the KL coefficient is significantly reduced.
- DPO: Amplifies stances that align with the SFT prior. If the SFT prior opposes a value, DPO struggles to fully invert the stance, resulting in only partial shifts. It is sensitive to the $\beta$ hyperparameter.
- SIMPO: Produces modest value drifts with slower convergence. Its margin-based objective (requiring a fixed reward gap) causes optimization signals to diminish once the margin is satisfied, leading to restrained updates.

Significance and Claims

The paper claims to provide the first systematic view into the temporal dynamics of value acquisition during post-training. Its primary contributions are:

Reframing Value Formation: Establishing that SFT, not Preference Optimization, is the primary mechanism for instilling specific value stances in LLMs.
Diagnosing Alignment Limitations: Identifying that the lack of value drift in standard preference optimization is not an inherent failure of the algorithms (PPO/DPO/SIMPO) but a consequence of the "low value gap" in widely used preference datasets.
Actionable Insights:
- Data Curation: To change a model's values, the SFT dataset must be curated with the desired value distribution, as later stages rarely alter these priors.
- Preference Data Design: Effective preference optimization requires datasets with explicit, high-value gaps between chosen and rejected responses.
- Algorithm Selection: The choice of preference algorithm (PPO vs. DPO vs. SIMPO) significantly impacts the degree and direction of value shifts when strong signals are present, particularly regarding how they interact with SFT priors.

The authors conclude that improving model alignment requires a shift in focus toward the curation of SFT data and the construction of preference datasets that explicitly encode value-level contrasts, rather than relying on standard pipelines to correct or refine values post-SFT.

Technical Summary: Value Drifts: Tracing Value Alignment During LLM Post-Training

Problem Statement

Methodology

Key Results

Significance and Claims

More like this