💬 NLP

Emulate or Estimate? The Divergent Strengths of Base and Post-Trained Language Models for Opinion Simulation

This paper resolves conflicting findings on using large language models for opinion simulation by distinguishing between "emulation" (generating individual responses) and "estimation" (predicting distributions), demonstrating that base models excel at the former while post-trained models are superior at the latter.

Original authors: Seth Grief-Albert, Jessica Bo, Difan Jiao, Ashton Anderson

Published 2026-08-05

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Seth Grief-Albert, Jessica Bo, Difan Jiao, Ashton Anderson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand the mood of a massive crowd without asking everyone individually. You might ask a few people what they think, or you might ask an expert to guess the overall vibe. This is the kind of puzzle scientists face when they try to use Artificial Intelligence (AI) to simulate human opinions. In the world of computer science, specifically in a field called "Natural Language Processing," researchers are building giant AI models that have read almost everything written on the internet. These models are incredibly good at predicting what words come next, which makes them seem like they could act as perfect stand-ins for real people. But here is the tricky part: just because an AI can write a convincing story doesn't mean it can accurately predict how a whole group of people will vote, feel about the economy, or answer a survey. Scientists are currently arguing over whether these AI "people" are actually good at mimicking human diversity or if they are just pretending to be something they aren't.

This paper steps into that argument to solve a mystery: why do some studies say AI is great at simulating human opinions, while others say it fails miserably? The authors, researchers from Queen's University and the University of Toronto, suggest that the confusion comes from mixing up two very different jobs. They call the first job Emulation. This is like hiring an actor to play a specific character. The AI generates a single response as if it were a real person, and if you ask enough "actors," their answers naturally add up to show the shape of the crowd's opinion. The second job is Estimation. This is like asking a statistician to look at a crowd and simply say, "I think 60% of them will choose option A." The researchers tested this idea by pitting two types of AI against each other: "Base" models (the raw, unpolished versions) and "Post-trained" models (the versions that have been fine-tuned to be helpful assistants).

The results of their experiments, run on real survey data from the Pew Research Center, reveal a clear split in talents. When the task was Emulation—generating individual text responses to see what the crowd looks like when you add them all up—the Base models were the winners. They produced responses that looked much more like real human data and did a better job of keeping the differences between groups (like Democrats vs. Republicans or rich vs. poor) distinct. In fact, the "Post-trained" models often suffered from what the authors call "persona collapse," where they all started sounding the same, like a choir of clones, losing the messy diversity of real people.

However, the story flips when the task is Estimation. When the researchers simply asked the AI to "predict the percentage of people who will choose this answer," the Post-trained models shined. They were much better at giving the correct numbers directly. The raw Base models were okay at this, but the polished, helpful assistants were significantly more accurate at guessing the distribution without having to generate a single sentence of fake text.

The authors suggest that this happens because the "Post-training" process teaches the AI to be a helpful assistant. When you ask an assistant to guess a crowd's opinion, it uses its knowledge to give a smart, direct answer. But when you ask that same assistant to act like a specific person, it gets stuck in its "helpful assistant" role, which makes it sound too uniform and lose the unique quirks of different demographics. The raw Base models, on the other hand, haven't been forced into that single "assistant" box, so they can draw from a wider, more chaotic pool of human voices when they are asked to generate text.

So, the paper concludes that there is no single "best" AI for simulating humans. If you need to generate text that feels like a diverse group of people (like for a video game or a social science simulation), you should use the raw Base models. But if you just need a quick, accurate prediction of what a group will think, you should ask the polished Post-trained models to estimate it. The confusion in the field wasn't because one type of AI was broken; it was because researchers were using the wrong tool for the specific job they were trying to do.

Technical Summary: Emulate or Estimate? The Divergent Strengths of Base and Post-Trained Language Models for Opinion Simulation

Problem Statement

Large Language Models (LLMs) are increasingly utilized to simulate human opinions for applications in policy testing, market research, and social science. However, prior literature presents conflicting findings regarding their efficacy. Some studies report promising alignment between base models and human survey data, while others document "persona collapse," weak demographic sensitivity, and high sensitivity to methodological choices (e.g., opinion ordering), particularly when evaluating post-trained models. The authors propose that this disagreement stems from a conflation of two distinct simulation tasks: emulation and estimation.

Emulation: The model generates individual responses as a specific participant; the population distribution emerges from the aggregation of these individual outputs.
Estimation: The model explicitly predicts the population's response distribution directly.

The paper argues that conflating these tasks obscures the specific strengths and weaknesses of different model architectures (base vs. post-trained).

Methodology

The study evaluates six matched pairs of base and post-trained models (including Qwen3-14B, Olmo-3-7B, and Olmo-3.1-32B variants) using the Pew American Trends Panel (Wave 54) as a ground-truth benchmark. The dataset comprises 59 four-option items eliciting economic opinions across seven demographic conditions (unconditioned, political, ideological, economic, and religious).

The authors implement two distinct evaluation paradigms:

Emulation Paradigm (Open-Response):
- Models generate free-text interview responses without access to multiple-choice options.
- An LLM judge maps these free-text responses to answer categories (validated with expert annotators, $\kappa = 0.66$ ).
- Base Models: Structured as an interviewer-participant dialogue with demographic conditioning in a preceding turn.
- Post-Trained Models: Given a system instruction to simulate a participant of a specific demographic.
- Rationale: This method avoids "positional bias" inherent in first-token probability extraction and produces interpretable text.
Estimation Paradigm (Verbalized Distribution):
- Models are prompted to directly predict the probability distribution over answer options for a given demographic group, returning a JSON object.
- Base Models: Multiple completions are sampled and averaged to account for variability.
- Post-Trained Models: A single sample is taken under greedy decoding due to output stability.

Metrics:

Total Variation Distance (TVD): Measures assumption-free distributional divergence.
Wasserstein Distance: Respects the ordinal structure of answer options, penalizing errors based on their distance from the correct response.

Key Results

1. Base Models are Superior Emulators

Across all six models and seven evaluation conditions, base models produced emulated distributions significantly closer to human ground truth than their post-trained counterparts.

Distributional Fidelity: Base models outperformed post-trained models in both TVD and Wasserstein distance. On Wasserstein distance, base models beat the uniform reference in 20 of 21 comparisons, indicating that even when exact probabilities were missed, the mass was placed near the correct ordered responses.
Demographic Structure: Base models consistently aligned with human demographic structure (Spearman's $\rho = 0.61\text{--}0.75$ ), compressing inter-group differences to approximately 70% of human magnitude. In contrast, post-trained models showed weaker structural alignment ( $\rho = 0.35\text{--}0.59$ ) and exaggerated inter-group differences by approximately 2 $\times$ , consistent with stereotyped persona simulation.
Mode Collapse: Post-trained outputs exhibited significant mode collapse, being 8.0–14.0 $\times$ more similar at the bigram level than base model outputs. Elevated temperature sampling did not resolve this collapse.

2. Post-Trained Models are Superior Estimators

Under the estimation paradigm, post-trained models produced distributions substantially closer to human ground truth than the uniform baseline and significantly better than their base counterparts.

Explicit Prediction: Estimation error decreased with model scale and was lowest in post-trained models (e.g., Claude Opus 4.6 achieved the strongest performance).
Knowledge Accessibility: Base models also produced estimation results matching or exceeding the uniform baseline, suggesting that distributional knowledge exists prior to post-training but is less accessible via direct verbalization in base models.

Key Contributions

Task Distinction: The paper formally distinguishes between emulation (generating individual text that aggregates to a distribution) and estimation (directly predicting a distribution), arguing that prior conflicting results arise from evaluating these distinct capabilities interchangeably.
Architectural Trade-off: The study demonstrates a clear trade-off: Base models are better suited for tasks requiring generated text that preserves demographic diversity and structure, while post-trained models excel at tasks requiring direct distributional prediction.
Evaluation Methodology: The authors propose open-response emulation as a robust evaluation paradigm that eliminates the positional bias confounding first-token probability extraction methods used in prior work.

Significance and Claims

The paper claims that model selection for human simulation must be guided by the specific requirements of the downstream task:

If the goal is to generate text that aligns with a population's diversity (e.g., for synthetic respondents or interactive agents), base models are the appropriate substrate because the desired distribution must emerge from sampled outputs.
If the goal is to estimate aggregate opinion distributions, post-trained models are more appropriate as they produce well-calibrated predictions when asked explicitly.

The authors suggest that post-training steers the base model's broad, chimeric distribution into a single "Assistant" persona optimized for helpfulness. When asked to emulate a demographic, the post-trained model performs a "second-order simulation" (an Assistant simulating a persona), which compresses output diversity through a single-character bottleneck. Conversely, base models draw directly from the pre-trained distribution over personas, preserving the representative diversity needed for faithful emulation at the cost of weaker metacognitive estimation.

Limitations

The authors acknowledge several limitations:

Scale: It is unknown if the emulation-estimation trade-off persists at frontier scales, as frontier base model variants are not publicly released.
Temporal Alignment: The survey data (2019) predates the model training corpora (extending to 2024–2025), creating a potential confound regarding whether models simulate opinions of a specific era or a temporally blended representation.
Scope: The evaluation is limited to survey opinions; extending these findings to richer behavioral settings (multi-turn dialogue, agent-based simulation) remains an open question.