SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors

This paper introduces SimBench, the first large-scale standardized benchmark for evaluating LLMs' ability to simulate human behaviors across diverse tasks and demographics, revealing that current models achieve only modest fidelity, exhibit an alignment-simulation tradeoff, and rely heavily on knowledge-intensive reasoning capabilities.

The Gist

Imagine you are a director trying to cast a movie about "Average Human Life." You need actors who can perfectly mimic how real people think, feel, and make decisions. For a long time, you've been using Large Language Models (LLMs)—super-smart AI chatbots—as your actors. You ask them, "What would a typical person do in this situation?" and they give you an answer.

But here's the problem: Until now, we didn't have a good way to measure if these AI actors were actually good at their job. Some studies said they were amazing; others said they were terrible. It was like judging a cooking competition where everyone used different ingredients and different taste testers.

This paper introduces SIMBENCH, the first standardized "audition" for AI actors trying to play humans.

🎭 The Big Idea: The "Human Simulator" Audition

The researchers built a massive testing ground called SIMBENCH. Think of it as a giant, diverse casting call. Instead of asking the AI just one question, they gave it 20 different types of "scripts" covering everything from:

• Moral dilemmas: "Should you save five people or one?" (The Trolley Problem).
• Economic choices: "Do you take a guaranteed $10 or gamble for $100?"
• Opinions: "Do you think the government should raise taxes?"
• Personality tests: "Are you more organized or spontaneous?"

They didn't just ask the AI to guess one answer. They asked the AI to predict the distribution of answers.

• Bad AI: "I think 100% of people would choose Option A."
• Good AI: "I think 60% would choose A, 30% would choose B, and 10% would choose C."

The goal is to see if the AI's prediction matches what real humans actually said in surveys.

📊 The Results: The AI is a "C-Student"

After testing 45 different AI models (from the biggest, most expensive ones to the smaller, open-source ones), here is what they found:

1. The Best AI is Only "Okay": The top-performing model (Claude-3.7-Sonnet) scored about 41 out of 100.
   • The Analogy: Imagine a student taking a test where a random guess gets a 0 and a perfect human gets a 100. The best AI got a 41. It's doing better than a random guess, but it's far from being a perfect human mimic. It's like a student who understands the basics but keeps missing the nuance.
2. Bigger Isn't Always Better (But Usually Is): Generally, bigger models with more "brain power" (parameters) did better. It's like a bigger library having more books to learn from. However, the improvement wasn't magical; it followed a slow, steady curve.
3. Thinking Harder Doesn't Help: The researchers tried making the AI "think step-by-step" (a technique called Chain-of-Thought) before answering. Surprisingly, this didn't help and sometimes made it worse.
   • The Analogy: Humans often make decisions based on gut feelings or quick heuristics (shortcuts). When you force an AI to write a long, logical essay about its choice, it becomes too rational and stops acting like a real, messy human. It's like asking a friend, "What's your favorite ice cream?" and them spending 10 minutes analyzing the chemical composition of vanilla before answering. They lose the "human" feel.

⚖️ The Great Trade-Off: Being "Helpful" vs. Being "Real"

This is the most fascinating discovery. The researchers found a conflict between making AI helpful (aligned) and making it realistic (simulating humans).

• The "Helpful" AI: When we train AI to be polite, safe, and follow instructions (Instruction Tuning), it gets very good at predicting what everyone agrees on.
  • Example: If 90% of people agree "Stealing is bad," the helpful AI nails this.
• The "Real" AI: But when humans disagree (high entropy), the helpful AI fails. It tries to find the "one right answer" and ignores the messy diversity of human opinion.
  • The Analogy: Imagine a weather forecaster. A "helpful" forecaster always predicts "Sunny" because that's the safe, polite answer. But a "real" forecaster knows that sometimes it rains, sometimes it snows, and sometimes it's a weird mix. The "helpful" AI forgets that humans are messy and diverse.

The Verdict: Training AI to be a "good assistant" actually makes it a worse simulator of real human behavior, especially when people are divided.

🌍 The "Who" Matters

The AI also struggled to simulate specific groups of people.

• It was okay at guessing what "men" or "women" might think.
• But it was terrible at guessing what people with specific religious beliefs or political ideologies would think.
• The Analogy: The AI is like a tourist who has visited a country once. They can guess the general vibe of the city, but if you ask them, "What do the local farmers in the northern valley think about the new tax law?" they have no idea. They lack the deep, specific cultural context.

🧠 What Makes a Good Simulator?

The paper found that the AI's ability to simulate humans wasn't linked to how good it was at chatting or writing poems. Instead, it was linked to deep reasoning and knowledge.

• The Analogy: To act like a human, you don't need to be a great comedian; you need to understand how the world works, the history behind things, and the logic of human choices. The AI models that were good at complex logic puzzles (like MMLU-Pro) were the best at pretending to be humans.

🚀 Why Does This Matter?

Currently, scientists and governments sometimes use AI to simulate how people will react to new laws or policies. This paper says: "Be careful."

• The AI is not ready to replace real human surveys yet.
• If we use these models, we might get a distorted view of the world where everyone agrees too much and no one is "messy" or "diverse."

The Bottom Line: SIMBENCH is the first ruler we have to measure how well AI can pretend to be us. It tells us that while AI is getting better, it's still a bit of a "one-trick pony" that struggles with the beautiful, chaotic diversity of real human life. We need to build better "actors" who can handle the messy, contradictory, and diverse nature of being human.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) hold the potential to revolutionize social and behavioral sciences by simulating human behaviors, offering a low-cost alternative to expensive human surveys and experiments. However, current evaluations of simulation fidelity are fragmented:

• Studies rely on bespoke tasks and metrics, making results incomparable.
• There is no unified framework to determine when, how, and why simulations succeed or fail.
• Existing benchmarks often focus on narrow contexts or individual-level simulation rather than group-level distributions.

The paper argues that without a standardized, large-scale benchmark, the field cannot systematically measure progress or develop reliable simulators.

2. Methodology: SIMBENCH

The authors introduce SIMBENCH, the first large-scale, standardized benchmark for group-level human behavior simulation.

A. Data Curation

• Scope: Unifies 20 diverse datasets from social and behavioral science repositories (e.g., Harvard Dataverse, ICPSR, OSF) and literature.
• Diversity:
  • Tasks: Covers decision-making (e.g., Choices13k), self-assessment (e.g., OpinionQA), judgment (e.g., ChaosNLI), and problem-solving.
  • Participants: Spans 130+ countries across six continents. The dataset is intentionally skewed away from the "Anglosphere West" (only 27.9% of data) to ensure global representation.
• Selection Criteria: Large participant counts, permissive licensing, single-turn questions, multiple-choice/ordinal formats, and English (or validated translations).

B. Standardization Process

To enable rigorous comparison, the authors harmonized heterogeneous data into a unified format:

1. Question Normalization: Converted all items to a standardized multiple-choice format (mapping options to single tokens) to facilitate probability extraction.
2. Response Aggregation: Created group-level probability distributions (ground truth) by aggregating individual responses.
   • Default Grouping: Aggregates all participants for a dataset (e.g., "US-based MTurk workers").
   • Specific Grouping: Aggregates based on sociodemographic attributes (e.g., "Females aged 30-49") for fine-grained analysis.
3. Benchmark Splits:
   • SimBenchPop: 7,167 test cases covering broad populations across all 20 datasets.
   • SimBenchGrouped: 6,343 test cases focusing on specific demographic subgroups from 5 large-scale survey datasets.

C. Evaluation Metric

The authors use Total Variation Distance (TVD) to measure the divergence between the model's predicted distribution ($Q$) and the human ground truth ($P$).

• SIMBENCH Score ($S$): A normalized score ranging from 0 to 100.
  $$S(P, Q) = 100 \left( 1 - \frac{TVD(P, Q)}{TVD(P, U)} \right)$$
  Where $U$ is a uniform baseline. A score of 100 indicates perfect alignment; 0 indicates performance equivalent to random guessing.

D. Experimental Setup

• Models: Evaluated 45 LLMs (commercial and open-weight, base and instruction-tuned), ranging from 0.5B to 405B parameters.
• Elicitation Strategy:
  • Base Models: Direct extraction of first-token probabilities.
  • Instruction-Tuned Models: Used verbalized distributions (prompting the model to output percentages in JSON) rather than raw logits, as empirical validation showed this significantly improves calibration for tuned models.

3. Key Contributions & Results

RQ1: General Simulation Ability

• Finding: Current LLMs achieve meaningful but modest fidelity.
• Top Performer: Claude-3.7-Sonnet achieved the highest score of 40.80/100.
• Implication: Even the best models are still closer to a uniform distribution than to human truth, though they close roughly 40% of the gap. Many smaller or less capable models score below 0, performing worse than random guessing.

RQ2: Impact of Model Characteristics

• Model Size: Performance scales log-linearly with parameter count. Larger models generally perform better.
• Inference-Time Compute: Increasing compute (e.g., via Chain-of-Thought prompting or larger reasoning budgets) does not improve simulation fidelity. In some cases (e.g., Claude-3.7-Sonnet with 4000-token budget), performance slightly decreased. The authors hypothesize that CoT forces overly rational deliberation that mismatches the heuristic nature of human responses.

RQ3: Task Selection Impact

• Value-Action Gap: Models perform well on opinion/self-assessment tasks (low entropy) but struggle significantly with behavioral choice tasks (e.g., moral dilemmas, risky choices).
• Alignment Filters: Models perform extremely poorly (often < 0 score) on tasks involving "atypical" or counter-normative views, such as Machiavellianism, conspiracy theories, or humor ratings, suggesting alignment filters inhibit the simulation of diverse human perspectives.

RQ4: The Alignment-Simulation Tradeoff

• Discovery: A strong negative correlation ($r = -0.942$) exists between instruction tuning and response entropy.
  • Low-Entropy (Consensus): Instruction tuning improves performance.
  • High-Entropy (Diversity): Instruction tuning degrades performance.
• Mechanism: Standard alignment (RLHF) optimizes for a "mode-seeking" objective (concentrating probability on a single "best" answer), which suppresses the multi-modal diversity inherent in human populations.
• Causal Analysis: Instruction tuning has a positive direct effect (better instruction following) but a negative indirect effect (entropy reduction). The net result is a tradeoff where alignment helps consensus questions but harms pluralistic ones.

RQ5: Demographic Conditioning

• Finding: Models struggle significantly more when simulating specific demographic groups compared to general populations.
• Worst Performance: Simulation accuracy drops most sharply for groups defined by Religiosity/Practice ($\Delta S \approx -9.9$) and Political Affiliation.
• Best Performance: Models handle Gender and Age conditioning relatively better, though still with degradation.

RQ6: Correlation with General Capabilities

• Finding: Simulation ability correlates most strongly with knowledge-intensive reasoning benchmarks.
  • MMLU-Pro: $r = 0.939$
  • GPQA Diamond: $r = 0.86$
• Weak Correlations: General helpfulness (Chatbot Arena) and narrow skills (Advanced Math/OTIS AIME) show weaker correlations, suggesting that simulating human behavior requires broad, deep reasoning rather than just conversational fluency or math skills.

4. Significance and Future Directions

• Scientific Infrastructure: SIMBENCH provides the first standardized infrastructure to move LLM simulation from ad-hoc studies to a reproducible science.
• Practical Implications:
  • Warns against using smaller or less capable models for simulation (many fail the uniform baseline).
  • Highlights the risk of using LLMs for policy pre-testing, as they may systematically misrepresent marginalized or ideologically diverse groups.
• Future Work:
  • Developing distribution-preserving alignment techniques to mitigate the entropy-reduction side effect of instruction tuning.
  • Moving beyond static distributions to evaluate interactive and multi-turn social dynamics.
  • Addressing the "intersectional" simulation gap (e.g., simulating specific combinations of demographics) which is currently limited by data sparsity.

In conclusion, while LLMs show genuine potential as simulators, they are currently limited by alignment-induced homogenization and a lack of robustness in high-entropy, diverse scenarios. SIMBENCH establishes the metrics necessary to track and improve these capabilities.