💬 NLP

Towards Physical Intuitions for Alignment Dynamics: A Case Study With Randomness Crystallization

This paper proposes applying thermodynamic phase-transition theory, specifically the concept of crystallization, to model and understand the dynamics of language model alignment during post-training, demonstrating how supervised finetuning and reinforcement learning transform high-entropy pre-trained distributions into structured, collapsed behaviors.

Original authors: Kunal Samanta, Ari Holtzman, Peter West

Published 2026-06-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kunal Samanta, Ari Holtzman, Peter West

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, chaotic jar of marbles. This jar represents a pre-trained AI model before it has been taught to follow instructions.

The Liquid Phase: The Chaotic Jar

In this initial state, the marbles are swirling everywhere. If you ask the AI a simple question like, "Give me a random number," the answer depends entirely on how you ask.

If you ask playfully ("What's your lucky number?"), it might lean toward 7.
If you ask formally ("According to my calculations..."), it might lean toward 1.
The AI has a "superposition" of many different personalities and habits. It is high-energy, diverse, and unpredictable, much like a liquid where atoms are moving in every direction.

The Nucleation Phase: The Snap-to-Grid

Now, imagine you start teaching the AI to follow specific rules (this is called Supervised Fine-Tuning or SFT). You show it examples of how to behave.

Suddenly, the chaos stops. The swirling marbles don't just slow down; they instantly snap into a rigid, organized pattern.

No matter how you phrase the question now, the AI gives the exact same type of answer.
The Big Discovery: The paper found that the AI didn't invent a new way to behave. Instead, it picked one specific habit that was already hiding inside the chaotic jar from the very beginning.
Think of this like crystallization. In physics, when water freezes, it doesn't create a new structure; it snaps into a crystal shape based on a tiny "seed" that was already there. The AI's training simply found that one "seed" habit and locked everything else into place around it.

The Settling Phase: Polishing the Crystal

After the AI has locked into that single habit, it goes through further training (called Reinforcement Learning or DPO) to become "safer" or "more helpful."

The paper argues this doesn't change the shape of the crystal. Instead, it's like tempering metal or polishing a gem.
The AI gets better at its specific habit, making the most likely answers even more likely, but it never breaks out of the pattern established in the previous step. It just tightens its grip on the same few options.

Why This Matters

The authors are saying that we often think AI alignment (teaching AI to be safe and helpful) is a magical transformation where the AI learns entirely new things.

Instead, they propose that alignment is more like freezing water.

The AI starts as a liquid with many hidden possibilities.
Training acts as the cold, forcing it to freeze into one specific shape (a "seed" that was already there).
Further training just polishes that shape, but it can't turn the ice back into water or change the shape of the crystal.

The "Foreign Seed" Surprise

The researchers did a cool experiment: They took the "seed" habit from one AI model (like OLMo) and used it to predict what would happen when a completely different AI model (like Tulu) was trained.

Result: It worked! The second AI froze into the exact same pattern.
Meaning: This suggests the "seed" isn't about the specific computer code of the AI. It's about the data and the task itself. The AI is just picking the most obvious pattern available in the data, and once it picks one, it's stuck with it.

The Limits

The paper admits this "crystal" idea works best for tasks with clear, limited answers (like picking a number between 1 and 10). It might be harder to see this pattern in open-ended creative writing where the answers aren't so rigid. But for now, it offers a new way to understand why AI models sometimes lose their creativity and become repetitive: they aren't just "learning" to be boring; they are physically "crystallizing" around one of their original, hidden habits.

Technical Summary: Towards Physical Intuitions for Alignment Dynamics

Problem Statement
While the field of Natural Language Processing (NLP) has achieved significant empirical success in aligning large language models (LLMs) to be useful and safe, the structural dynamics of how these changes unfold remain poorly understood. Current alignment research relies heavily on a taxonomic vocabulary (e.g., aligned vs. unaligned, diverse vs. collapsed) that describes end-states but lacks a coherent framework for the transitions between them. Specifically, the origins of alignment-induced structure, the reasons for convergence at specific behavioral attractors, and the fundamental limits of what alignment can change are not well characterized. The authors argue that the physical sciences, particularly thermodynamic phase-transition theory, offer a principled, underexplored vocabulary to reason about these dynamics.

Methodology: The Crystallization Framework
The paper proposes a "crystallization" framework, drawing a direct analogy between the thermodynamic phase transition of material crystallization and the post-training alignment of LLMs. This framework posits that alignment on tasks with finite support (e.g., random number generation) proceeds through three distinct phases:

The Liquid Phase (Pretrained Entropy): The base LLM exists in a high-entropy state. Different prompts elicit a superposition of diverse latent output distributions, analogous to atoms in a liquid moving with varying velocities and directions.
The Nucleation Phase (Supervised Fine-Tuning - SFT): SFT acts as a nucleation event. The model's diverse output distributions collapse abruptly onto a single "seed distribution" that was already latent within the base model. Prompt sensitivity vanishes, and the model snaps to a coherent structure, much like atoms aligning to a seed crystal.
The Settling Phase (Preference Optimization - DPO/RLHF): Subsequent preference optimization stages (e.g., DPO, RLHF) do not create new structures but rather "temper" the existing crystal. Probability mass concentrates further onto the peaks established during nucleation, refining the distribution without escaping the support created by SFT.

To quantify these phases, the authors define two diagnostic metrics:

Mean Squared Error (MSE) Distance Ratio ( $r_d$ ): Measures the distance between SFT distributions and a candidate "seed" distribution from the base model. A low ratio indicates successful nucleation (convergence to a pre-existing seed).
Probability Mass Overlap (ProbMass): Measures the concentration of probability mass in later stages (DPO/Instruct) onto the peaks of the seed distribution. An increase in ProbMass indicates the "settling" process where probability consolidates onto specific outcomes.

Experimental Setup
The framework was tested across 15 stochastic tasks (e.g., generating random digits, even numbers, prime numbers) using two open-weight model families: OLMo 2 and Tulu 3. The study utilized checkpoints from the base model, SFT, DPO, and Instruct stages. For each task, the authors sampled 100 distinct generation prefixes with varying styles (from playful to formal) to recover empirical output distributions.

Key Results

Validation of Three-Phase Dynamics: The experiments consistently showed the three-phase pattern. In the Liquid Phase, base models exhibited high inter-prefix MSE, reflecting high entropy. During Nucleation (SFT), the MSE distance ratio dropped dramatically, confirming that all prompted distributions converged onto a single seed distribution already present in the base model.
Settling Behavior: During the Settling Phase (DPO/Instruct), the absolute MSE from the seed increased slightly (indicating sharpening away from the exact seed shape), but the ProbMass metric increased monotonically. This confirms that probability mass concentrates further onto the seed's peaks rather than expanding to new regions.
Cross-Model Transfer: A "foreign seed" identified from the base model of one family (e.g., OLMo 2) successfully predicted the nucleation behavior of a completely independent model family (e.g., Tulu 3). This suggests that the dynamics are structural features of the alignment process and data distribution, rather than artifacts of specific model architectures or training recipes.
Limitations in Large Spaces: The framework showed limitations in tasks with large output spaces (e.g., int_1_500), where reliable seed identification was difficult, suggesting the need for metrics that account for sparsity and element similarity.

Significance and Claims
The paper positions crystallization as a "proof of concept" for a broader program of importing physical frameworks into alignment research. Its primary significance lies in shifting the field's focus from post-hoc description of outcomes to a priori prediction of dynamics. By demonstrating that a well-characterized physical process maps onto alignment dynamics, the authors argue that reducing aspects of LLM training to known physical phenomena can lead to:

Mathematical structure and testable predictions.
Principled interventions (e.g., identifying "nucleation inhibitors" to preserve diversity).
A deeper understanding of where alignment-induced structure comes from and why it converges where it does.

The authors conclude that while this study is scoped to stochastic tasks with finite output spaces, the crystallization analogy provides a rich, predictive framework that could eventually be extended to open-ended generation and reasoning tasks, offering a new mission for NLP: understanding the processes of alignment rather than just cataloging their results.