💬 NLP

Identity, Cooperation and Framing Effects within Groups of Real and Simulated Humans

This paper demonstrates that deeply binding large language models with rich narrative identities and verifying their consistency significantly improves the simulation of human behavior in social dilemma games, enabling the exploration of critical contextual factors like time, framing, and participant pools that often hinder the replication of human studies.

Original authors: Suhong Moon, Minwoo Kang, Joseph Suh, Mustafa Safdari, John Canny

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Suhong Moon, Minwoo Kang, Joseph Suh, Mustafa Safdari, John Canny

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand why people act the way they do in tricky social situations, like deciding how to split a pot of money with a stranger. For decades, scientists have run real experiments with thousands of actual humans to figure this out. But real experiments are expensive, slow, and sometimes hard to repeat exactly because small details (like the year the study happened or the exact wording of the instructions) can change the results.

This paper asks a big question: Can we use AI (specifically Large Language Models) to act as "virtual humans" that are so realistic they can help us understand these social behaviors?

Here is the breakdown of their work, using simple analogies:

1. The Problem: The "Too Polite" Robot

Most people try to make AI act like humans by giving it a simple instruction like, "You are a Republican who likes low taxes." The authors call this "steering."

However, the authors argue that standard AI models are like overly polite hotel concierges. They have been trained to be helpful, harmless, and agreeable. If you ask them to act like a real person with flaws, biases, or strong political grudges, they often fail. They smooth over the rough edges of human nature.

The Solution: Instead of using the "polite concierge" (the instruction-tuned model), they used the "raw brain" (the pre-trained base model). Think of the raw brain as a massive library containing billions of real conversations, diaries, and arguments from the internet. It hasn't been taught to be polite yet; it just knows how humans actually talk, including their biases, inconsistencies, and prejudices.

2. The Method: Building a "Virtual Life"

To make the AI act like a specific person, the researchers didn't just give it a label. They built a deep backstory, like writing a full biography for a character in a novel.

The Interview: They asked the AI to "interview" itself, generating a long, detailed story about its childhood, family, struggles, and political views.
The Match: They then took real human participants from old studies and matched them with the AI "characters" who had the most similar life stories.
The "Time Machine" (Temporal Grounding): Humans change over time. A person in 2014 might feel differently about politics than in 2019. The researchers forced the AI to "remember" exactly what year it was living in during the simulation, just like a time traveler.
The "Reality Check" (Consistency Filtering): Sometimes AI gets confused and forgets its own story (e.g., saying it's a Democrat when it was just told it's a Republican). The researchers added a step where a second AI checks the answers to make sure the character stays true to their backstory.

3. The Experiment: The "Money Split" Games

The team tested these virtual humans in two classic games used by psychologists:

The Dictator Game: You have $10. You can keep it all or give some to a stranger.
The Trust Game: You send money to a stranger, which gets tripled. They can choose to give some back to you.

The twist? The researchers told the AI who the stranger was: "The other person is a Democrat" or "The other person is a Republican."

The Finding:
Real humans tend to be much nicer to people who share their political party (co-partisans) and stingier with those who don't. The researchers found that their deeply bound AI characters replicated this behavior almost perfectly.

When the AI knew the other person was a "co-partisan," it gave more money.
When the AI knew the other person was a "rival," it gave less.
Crucially, the AI got the timing right. It showed a bigger gap in generosity in the 2019 simulations compared to the 2014 simulations, mirroring how real political polarization has grown over time.

4. The "What-If" Machine

The most exciting part of this paper is what the AI allowed them to do that is impossible with real humans: Counterfactuals.

Imagine you have two real studies that got different results. You don't know if it was because of the year they were done, the wording of the instructions, or the type of people recruited.

With real humans, you can't change one thing without changing everything else.
With their AI, they could run a "What If" simulation. They took the 2014 study and asked: "What if we used the 2019 instructions?" or "What if we used the 2019 participant pool?"

The Surprise: They discovered that the wording of the instructions (the "Framing") actually had a bigger impact on the results than the year the study was done. The AI helped them see that small changes in how a question is asked can drastically change how people answer, a detail often missed in human studies.

Summary

This paper shows that if you treat AI not as a "helpful assistant" but as a raw, unfiltered mirror of human conversation, and then give it a rich, detailed life story and a specific time period, it can simulate human social behavior with surprising accuracy.

It's not about replacing human studies, but about having a high-fidelity simulator that lets researchers test "what if" scenarios to understand why human studies sometimes give different answers. It's like having a flight simulator for social psychology, where you can crash the plane (or the experiment) to see what went wrong without hurting any real people.

Technical Summary: Identity, Cooperation and Framing Effects within Groups of Real and Simulated Humans

Problem Statement

Human decision-making in social dilemmas is shaped not only by rational deliberation but also by social identity (e.g., political partisanship) and contextual factors (e.g., temporal timing, question framing, and participant pools). While Large Language Models (LLMs) have shown promise in simulating human behavior, prior research has largely focused on opinion surveys or rational decision-making, neglecting identity-driven behaviors. Furthermore, existing simulation methods often rely on instruction-tuned (IT) chat models, which the authors argue introduce biases by flattening human diversity and enforcing a "helpful agent" persona that fails to represent negative traits, prejudices, or the full spectrum of human identity.

The core challenge addressed is whether LLMs can accurately simulate complex human actions—specifically in-group favoritism and out-group animosity in resource allocation and trust games—while accounting for subtle contextual variables that often hinder the reproducibility of human studies.

Methodology

1. Model Selection: Pretrained vs. Instruction-Tuned

The authors argue for the use of pretrained (base) models over instruction-tuned variants.

Rationale: Pretrained models capture a "mixture of voices" and latent context (speaker traits, listener traits, time, place) from their massive training corpora. In contrast, instruction-tuning optimizes models for helpfulness and safety, effectively freezing the identity to a "helpful agent" and marginalizing user traits. This results in a "Lake Wobegon Effect" where IT models systematically avoid negative characterizations, leading to poor fidelity in simulating realistic, diverse human populations.
Evidence: The paper demonstrates that base models exhibit significantly lower perplexity on human dialog data (Reddit) compared to their IT counterparts, indicating a better fit for human language patterns.

2. Simulation Framework

The proposed framework, DeepBind, involves three key components to simulate human participants in social dilemma games (Dictator and Trust Games):

Narrative Backstory Generation: Instead of simple demographic tags, the system generates rich, interview-style backstories for virtual personas. These are created by prompting a pretrained model with a sequence of questions derived from the American Voices Project (covering childhood, family, politics, health, etc.).
- Quality Control: A rejection sampling process using an instruction-tuned "critic" model filters out non-dialog text, code blocks, and factual inconsistencies.
Demographic Matching: Virtual personas are matched to real human study participants based on demographic and ideological traits extracted from their backstories using a maximum-weight bipartite matching algorithm (Hungarian algorithm).
Prompting Strategies:
- Temporal Grounding: Simulations are explicitly bound to the specific year the original human study was conducted (e.g., 2014 vs. 2019) via forced QA pairs within the prompt. This addresses the hypothesis that partisan animosity has increased over time.
- Consistency Filtering: To prevent semantic drift during long simulations, the system periodically reiterates the persona's identity and group membership (e.g., "Keep in mind you have identified as Republican") to maintain behavioral consistency.

3. Experimental Design

The authors evaluated the framework using two classic behavioral economics games:

Dictator Game: Measuring altruism and in-group favoritism in resource allocation.
Trust Game: Measuring trust (sender) and trustworthiness (receiver).

The study replicated four human experiments (Iyengar & Westwood, 2015; Whitt et al., 2021; Carlin & Love, 2018) across three model families (Mistral-Small, Mixtral, Qwen-2.5). The authors also conducted counterfactual simulations to isolate the effects of three specific factors: Study Year, Instruction Framing (wording), and Participant Pool (recruitment source).

Key Results

1. Simulation Fidelity

The DeepBind method (backstory conditioning + temporal grounding + consistency filtering) consistently outperformed baseline prompting strategies (QA, Bio, Portray) in replicating human partisan bias ( $\Delta$ values).

In the Dictator Game, DeepBind achieved the closest alignment with human data for both Democrats and Republicans across multiple studies. For instance, in the 2019 Whitt et al. study, DeepBind produced a Republican $\Delta$ of 2.16 (vs. human 2.33) and a Democratic $\Delta$ of 2.60 (vs. human 2.46), significantly closer than baselines.
In the Trust Game, DeepBind similarly matched human benchmarks, such as replicating a Republican $\Delta$ of 2.00 (vs. human 1.76) in the 2019 study.

2. Ablation Studies

Backstories: Removing the extended narrative backstories (reverting to simple QA prompts) significantly degraded performance, confirming that deep persona binding is critical.
Temporal Grounding & Consistency: Combining both techniques yielded the best results. For the 2019 Dictator Game, the combined approach achieved a Mean Diff. $\Delta$ of 2.38, nearly matching the human gap of 2.40. Using either technique alone was less effective.

3. Counterfactual Analysis of Reproducibility

By varying Year, Framing, and Participant Pool independently, the authors identified drivers of variance in human study results:

Framing Effects: The wording of instructions (specifically, the presence of boldface reminders of political affiliation) had a stronger impact on partisan bias than the study year in some contexts. This challenges the conjecture that rising partisanship is solely due to temporal shifts.
Participant Pools: In the Trust Game, the difference between recruitment sources (Mechanical Turk vs. Dynata/SSI) showed a weak but significant effect. In the Dictator Game, where pools were more similar, this effect was negligible.
Year Effects: Study year significantly affected allocation differences in Dictator games but not Trust games.

Significance and Claims

The paper claims that LLMs, when properly configured with unbiased pretrained models and deep narrative binding, can serve as a powerful tool for exploring the nuances of human social behavior that are often omitted from experimental descriptions.

Reproducibility Tool: The framework allows researchers to "counterfactually explore" how subtle changes in study design (year, framing, pool) alter outcomes, helping to explain discrepancies between human replication studies.
Beyond Surveys: Unlike prior work focused on opinion polling, this work demonstrates that LLMs can simulate actions (resource allocation) influenced by identity and context.
Methodological Shift: The authors argue that instruction-tuned models are ill-suited for simulating the full diversity of human behavior due to their inherent bias toward "helpfulness," and that base models offer a more faithful representation of human complexity, including negative traits and prejudices.

Limitations

The authors acknowledge several constraints:

Data Dependence: Fidelity is limited to the temporal and cultural scope of the model's training data (e.g., simulating 1950s behavior is not expected to be accurate).
Bias Propagation: While base models capture diversity, they also capture societal biases. The generated personas may reflect harmful stereotypes or prejudices present in the training data.
Resource Intensity: Generating high-quality backstories with rejection sampling requires significant computational resources.
Domain Specificity: The current experiments focus exclusively on U.S. political partisanship; generalization to other cultures or contexts is unproven.

The paper concludes that while human studies remain the "gold standard," this simulation framework offers a valuable mechanism for understanding the "hidden variables" of experimental design that affect human behavior.