Technical Summary: Validated Hypotheses as a Lens for Human-Likeness Evaluation in AI Agents

Problem Statement

Large Language Models (LLMs) are increasingly deployed as surrogate participants in social science experiments, testbeds for simulating user populations, and building blocks for multi-agent organizational simulations. These applications rely on an implicit claim: that the agent's behavior is meaningfully human-like. However, accumulating evidence suggests current LLM-based agents fail this claim in ways that compromise simulation validity.

The conventional operationalization of human-likeness, the Turing Test, is insufficient for this purpose due to three key limitations:

1. Subjectivity: Verdicts depend on a human judge's impression of surface-level fluency rather than deep behavioral correspondence.
2. Coarseness: The binary outcome (human vs. machine) fails to diagnose which aspects of human behavior are missing.
3. Scalability: The protocol is expensive, requiring fresh human judges for every new agent iteration.

As LLMs improve, passing such tests increasingly reflects stylistic mimicry rather than genuine behavioral alignment. There is a need for a usable, principled, and diagnostic test of human-likeness that moves beyond conversational imitation.

Methodology: HUMANSTUDY-BENCH

The authors propose a framework that treats validated behavioral hypotheses from decades of psychology, behavioral economics, and social science as the ground truth for evaluation. The core idea is that if an agent is human-like, a population of such agents should reach the same inferential conclusions as the human population when subjected to the same experimental protocols.

To operationalize this, the authors introduce HUMANSTUDY-BENCH, an open platform that reconstructs published human-subject studies into reusable simulation environments. The system employs a human-in-the-loop pipeline:

1. Filter: Curates studies where hypotheses are robustly established through independent replication (e.g., the False Consensus Effect, Framing Effect, Trust Game). Studies must have full experimental documentation, quantifiable outcomes, and be feasible for text-based simulation (excluding physiological or visual stimuli).
2. Extract: Formalizes the experimental protocol (stimuli, conditions, instructions) and statistical analysis methods into machine-executable formats.
3. Execute: Runs configurable agents through the reconstructed protocols. The authors evaluated 10 contemporary LLMs (open-weight and proprietary) under 4 agent design variants:
   • Blank (A1): Base model with no specification.
   • Role-Play (A2): Instructed to act as a human participant.
   • Demographic (A3): Assigned attributes (age, gender, occupation) sampled from original study distributions.
   • Contextualized Backstory (A4): Augmented demographics with rich narrative biographies.
4. Evaluate: Scores agent-human alignment using two orthogonal metrics:
   • Probability Alignment Score (PAS): Measures inferential agreement. It quantifies the probability that agent and human data lead to the same conclusion regarding the existence of an effect (significance), using a Bayesian evidence-to-probability pipeline to handle finite-sample uncertainty.
   • Effect Consistency Score (ECS): Measures magnitude agreement. It uses Lin's Concordance Correlation Coefficient to assess whether the agent reproduces the size and pattern of human effects, penalizing both incorrect patterns and systematic scale deviations.

The initial suite consists of 12 studies covering individual cognition, strategic interaction, and social psychology, yielding over 6,000 trials.

Key Results

The evaluation of 10 models across 12 studies reveals that current LLM agents fail the test in specific, diagnostic ways:

1. Bimodal Polarization: Agent responses do not exhibit a uniform "mediocre" performance. Instead, they polarize between full replication and complete failure. While human data consistently yields highly significant results, agent simulations show a wide spread of p-values, with a substantial portion falling into the non-significant range.
2. Trade-off between Validity and Precision: There is a distinct divergence between PAS and ECS. Some models achieve high PAS (correctly identifying that an effect exists) but low ECS (exaggerating or dampening the effect size). Conversely, others may track effect patterns but fail to reach statistical significance.
3. Pattern vs. Scale: Decomposition of ECS reveals that the primary bottleneck is pattern correlation (whether the agent correctly identifies which findings are stronger/weaker) rather than magnitude calibration. Agents often learn the rough scale of human effects but fail to reliably distinguish the relative strength of different phenomena.
4. Impact of Agent Design: Agent design influences alignment more than raw model scale.
   • Adding demographic priors (A3) generally improves alignment over the blank baseline.
   • Adding rich backstories (A4) does not yield a monotonic improvement; it helps some models (e.g., Qwen, Claude) but degrades performance in others (e.g., Grok, Gemini).
   • Naive model ensembling (mixing responses from different models) significantly worsens alignment compared to single-model approaches, introducing "destructive interference."
5. Temperature and Scale: Neither raw model scale nor naive temperature tuning provides a systematic improvement in alignment. Flagship models do not consistently outperform smaller open-weight models, and varying temperature ($T \in [0.1, 1.0]$) yields non-monotonic results that depend on the specific task domain and agent specification.

Contributions and Significance

The paper makes four primary contributions:

1. A New Evaluation Paradigm: It reframes human-likeness testing from a single binary judgment to a list of statistical tests against validated social science findings, offering an objective, decomposable, and scalable alternative to the Turing Test.
2. Novel Metrics: It introduces PAS and ECS to quantify inferential and magnitude agreement, respectively, while accounting for uncertainty in human reference data.
3. HUMANSTUDY-BENCH Platform: It provides an open infrastructure that reconstructs published experiments end-to-end, enabling the community to evaluate configurable agents against a growing suite of robustly replicated studies.
4. Diagnostic Empirical Insights: It demonstrates that current LLM agents fail to replicate human behavior in specific, diagnosable ways, revealing that simulation fidelity is driven more by agent specification than by model scale or simple ensembling.

Significance:
The authors position HUMANSTUDY-BENCH not merely as a benchmark that reports current shortcomings, but as a diagnostic tool that reveals how agents fail. By shifting the focus from "can the agent pass as human?" to "does the agent replicate specific, validated behavioral effects?", the framework provides a principled path toward improving the validity of AI-driven social simulations. The paper emphasizes that while current agents fall short, the proposed lens allows researchers to systematically identify gaps and iterate on agent designs with greater precision. The authors note that the initial suite is limited to WEIRD (Western, Educated, Industrialized, Rich, Democratic) samples and classic experiments, and they view the platform as a foundation for future expansion into broader cross-cultural and contemporary contexts.