💻 computer science

Alignment Makes Language Models Normative, Not Descriptive

This paper demonstrates that while post-training alignment optimizes language models for human use, it fundamentally shifts them from being descriptive proxies of actual human behavior in complex, multi-round strategic interactions to being normative predictors that excel only in settings where human choices align with idealized solutions.

Original authors: Eilam Shapira, Moshe Tennenholtz, Roi Reichart

Published 2026-03-19

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Eilam Shapira, Moshe Tennenholtz, Roi Reichart

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have two versions of a very smart robot assistant.

Version A (The "Base" Model): This is the robot right after it finished reading the entire internet. It knows everything humans have ever said, written, or argued about. It's raw, unfiltered, and sometimes messy. It can be rude, strategic, tricky, or even a bit crazy, because that's just how real humans talk and act.
Version B (The "Aligned" Model): This is the same robot, but it went to "Good Manners School." Humans trained it to be polite, fair, cooperative, and to always do the "right" thing. It's the robot you want to hire to write a birthday card or help you plan a party.

The Big Question:
Researchers wanted to know: Which robot is better at predicting what a real human will actually do in a tricky situation?

The Experiment: The "Game of Life"

The researchers put both robots into a series of complex games where humans have to make tough choices. Think of these games like:

Bargaining: Trying to split a pizza with a friend who might get angry if you take too much.
Negotiation: Trying to sell a car to someone who wants to pay as little as possible.
The Prisoner's Dilemma: Deciding whether to trust your partner or betray them to save yourself.

They asked the robots: "If a real human were in this situation, what would they choose?"

The Surprising Result

Here is the twist: The "Good Manners" robot (Aligned) was terrible at predicting real humans in these games.

In Multi-Round Games (The Long Haul): When the game went on for many rounds, humans started acting like real people. They got angry, they retaliated, they made deals, they lied, and they adapted based on what happened before.
- The Base Robot (the messy one) predicted these moves almost perfectly. It understood that humans aren't always nice; sometimes they get revenge.
- The Aligned Robot (the polite one) kept guessing that humans would be fair and cooperative. It failed miserably because real humans in these games often aren't fair.
- The Score: The messy robot won 10 times more often than the polite robot.

Why Did This Happen?

Think of the Aligned Robot like a student who only studied the "Textbook Answers."

In a textbook, the "right" answer to a negotiation is usually to be fair and reach a compromise.
But in real life, people often bluff, get angry, or try to trick the other person.
Because the Aligned Robot was trained to be the "good guy," it forgot how to predict the "bad guy." It assumed everyone else was trying to be good, too.

The Base Robot, however, had seen the "real world" data. It knew that humans are complex, sometimes irrational, and often driven by history (e.g., "You cheated me last time, so I'm cheating you now").

When Was the Polite Robot Better?

The polite robot did win, but only in very specific, simple situations:

One-Shot Games: If you play a game once and never see the other person again, humans tend to act more like the "textbook" (rational and fair). The polite robot guessed this correctly.
Simple Choices: If you just have to pick between two lottery tickets (no strategy, no opponent), the polite robot was better at following the rules of the game.

The Big Lesson

This paper reveals a fundamental trade-off:

If you want a robot to help you (write emails, be polite, follow rules), you want the Aligned model.
If you want a robot to understand humans (predict how a crowd will vote, how a market will crash, or how people will fight in a negotiation), you actually want the Base model.

The Metaphor:
Imagine you are trying to predict the weather.

The Aligned Model is like a weather app that only predicts "Sunny and Perfect" because that's what people want the weather to be. It's great for planning a picnic, but terrible for predicting a storm.
The Base Model is like a raw radar that shows rain, hail, and tornadoes. It's messy and scary, but it tells you exactly what's going to happen so you can prepare.

Conclusion:
We often assume that making AI "safer" and "nicer" makes it smarter. This paper says: Not necessarily. When we force AI to be "good," we accidentally blind it to the messy, strategic, and sometimes "bad" reality of how humans actually behave. If you want to simulate human behavior, you might need to turn off the "good manners" filter.

1. Problem Statement

Large Language Models (LLMs) are increasingly used as proxies for human behavior in social science research, economics, and psychology. The prevailing assumption is that alignment (post-training optimization via RLHF or DPO to match human preferences) improves or maintains a model's ability to predict human behavior.

The authors challenge this assumption, hypothesizing that alignment introduces a normative bias. While alignment optimizes models to produce responses that humans endorse (cooperative, fair, socially appropriate), it may distort the model's ability to predict how humans actually behave in complex, strategic settings where behavior is driven by reciprocity, retaliation, and history-dependent adaptation (descriptive dynamics). The core question is whether alignment shifts the model from a descriptive tool (modeling actual behavior) to a normative one (modeling idealized behavior).

2. Methodology

The study employs a rigorous, large-scale comparative framework to isolate the effect of alignment from other variables.

Datasets:
- Multi-Round Strategic Games (Main Test): 10,050 real human decisions across four families:
  - Bargaining: Alternating offers with discount factors.
  - Persuasion: Repeated cheap-talk games with reputation dynamics.
  - Negotiation: Bilateral price negotiation with outside options.
  - Repeated Matrix Games: Prisoner's Dilemma and Battle of the Sexes (10 rounds).
- Boundary Conditions (Control Tests):
  - One-shot 2×2 Matrix Games: Single-round decisions (2,416 games).
  - Binary Lottery Choices: Non-strategic risk decisions (1,001 problems).
Model Selection:
- 120 Base–Aligned Pairs: The study compares 120 pairs of models from 23 different families (e.g., Llama, Qwen, Mistral, Gemma).
- Constraint: Only "same-provider" pairs are used (e.g., Llama-3-8B vs. Llama-3-8B-Instruct) to ensure that differences are solely due to the alignment process, not architecture or pre-training data.
- Scale: Models range from <1B to >70B parameters.
Evaluation Protocol:
- Prediction Task: The task is framed as token probability extraction, not generation. For a given decision point, the model is prompted with the game history, and the log-probabilities of the decision tokens (e.g., "accept" vs. "reject") are extracted.
- Metric: Pearson correlation ( $r$ ) between the model's predicted probability distribution and the ground-truth human choice distribution.
- Controls: The study tests various prompt formats (native chat templates vs. plain text) and prompt formulations (framing, persona, structure) to rule out formatting artifacts.
Statistical Analysis: Pairwise comparisons (Base Win vs. Aligned Win) using binomial tests and Wilcoxon signed-rank tests.

3. Key Contributions

Empirical Evidence of the "Alignment Tax" on Behavioral Fidelity: The paper provides the first systematic evidence that alignment significantly degrades a model's ability to predict human behavior in multi-round strategic interactions.
The Normative-Descriptive Trade-off: The authors establish a clear boundary condition: alignment improves prediction in settings where human behavior aligns with normative theory (one-shot games, lotteries) but harms prediction in settings where behavior is shaped by interaction history (multi-round games).
Mechanism Identification: The paper argues that the degradation is caused by distributional narrowing (preference collapse). RLHF/RLDPO tilts the base distribution toward high-reward (socially approved) modes, suppressing the "tails" of the distribution where complex, non-normative human behaviors (bluffing, retaliation) reside.
Scale Independence: The phenomenon is robust across model sizes (1B to 70B+), prompt formulations, and game configurations.

4. Key Results

A. Multi-Round Strategic Games (The Main Finding)

In settings involving interaction history, Base models vastly outperform Aligned models.

Overall Ratio: Base models won 9.7:1 (213 wins vs. 22 wins) across all four game families.
Per-Family Wins:
- Bargaining: 75:4
- Persuasion: 32:4
- Negotiation: 25:1
- Matrix Games: 81:13
Statistical Significance: $p < 10^{-40}$ overall; significant in every individual family ( $p < 10^{-6}$ ).
Round Dynamics: The advantage reverses in Round 1 (before history accumulates), where Aligned models often win. The Base advantage emerges and grows from Round 2 onward, confirming that the deficit is linked to the accumulation of interaction history.

B. Boundary Conditions (The Reversal)

In settings without multi-round history, the advantage flips, supporting the "Normative Bias" hypothesis.

One-Shot 2×2 Games: Aligned models won 4.1:1 (57 vs. 14). Their predictions were closer to Nash Equilibrium, which correlated well with human behavior in these simple, textbook scenarios.
Binary Lotteries: Aligned models won 2.2:1 (62 vs. 28).
Implication: Alignment helps when the task requires following instructions or adhering to textbook rationality, but hurts when the task requires modeling the messy, non-rational dynamics of repeated interaction.

C. Robustness Checks

Prompt Format: The Base advantage persists even when both models are forced into the same prompt format (e.g., both using chat templates or both using plain text). The advantage lies in the model weights, not the interface.
Model Scale: The gap generally widens with model size. For example, in bargaining, the correlation difference grew from +0.22 (at <3B) to +0.36 (at ≥14B).
Parameter Sensitivity: The result holds across all game parameters (stakes, information completeness, discount factors), with only one minor exception (Bargaining with $\delta=0.8$ ).

5. Significance and Implications

Methodological Risk for Social Science: Researchers using aligned LLMs as "simulated participants" (Homo Silicus) may be inadvertently studying normative behavior (how people should act) rather than descriptive behavior (how people actually act). This could lead to flawed conclusions in economics, political science, and psychology.
Model Selection Guidelines:
- Use Base Models for simulating human behavior in strategic, multi-turn, or adversarial environments.
- Use Aligned Models for tasks requiring adherence to social norms, one-shot decision prediction, or non-strategic reasoning.
Theoretical Insight: The findings suggest that current alignment methods (RLHF/DPO) inherently sacrifice the "tails" of the behavioral distribution. To preserve the full spectrum of human behavior (including irrationality and retaliation), future alignment methods may need to incorporate entropy regularization or multi-objective optimization that does not collapse the distribution toward a single "approved" mode.

In conclusion, the paper demonstrates that alignment is not a neutral upgrade for behavioral modeling; it is a fundamental shift in the model's objective function that prioritizes social desirability over descriptive accuracy in complex strategic contexts.