💬 NLP

Can LLMs Truly Embody Human Personality? Analyzing AI and Human Behavior Alignment in Dispute Resolution

This paper introduces an evaluation framework and dataset to investigate whether Large Language Models (LLMs) can accurately replicate human personality-driven behaviors in dispute resolution, ultimately finding that current LLMs diverge significantly from human patterns and may not be reliable proxies for human social behavior.

Original authors: Deuksin Kwon, Kaleen Shrestha, Bin Han, Spencer Lin, James Hale, Jonathan Gratch, Maja Matarić, Gale M. Lucas

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Deuksin Kwon, Kaleen Shrestha, Bin Han, Spencer Lin, James Hale, Jonathan Gratch, Maja Matarić, Gale M. Lucas

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Can AI Truly "Feel" the Vibe?

Imagine you are playing a video game where the characters are supposed to act like real people. You tell the game, "Make this character grumpy and stubborn," and you expect them to argue with you, refuse to compromise, and get defensive. You’d assume that because you gave them a "personality," they would behave like a real-life grumpy person.

This research paper asks a deep, skeptical question: When we tell an AI to act like a certain personality, is it actually "embodying" that personality, or is it just wearing a cheap mask that slips the moment things get heated?

To find out, the researchers put humans and AI into a "digital pressure cooker"—a simulated heated argument over a disputed online purchase—to see if their personalities actually changed how they fought and settled their differences.

The Experiment: The Digital Pressure Cooker

The researchers set up two different arenas:

The Human Arena (KODIS): Real people having real, messy, emotional arguments about a disputed Kobe Bryant jersey.
The AI Arena (L2L): Different AI models (like GPT-4, Claude, and Gemini) being "prompted" to act like people with specific Big Five personality traits (like being highly agreeable, neurotic, or extroverted).

They didn't just look at what was said; they looked at the strategy of the fight. They used a framework called IRP to categorize moves:

The Peacekeeper (Cooperative): "I understand your frustration; let's find a middle ground."
The Rule-Follower (Neutral): "Here are the facts of the transaction."
The Aggressor (Competitive): "You're a liar! I'm not giving you a dime!"

The Findings: The Mask vs. The Soul

The results were a reality check for AI enthusiasts. The researchers found that while AI can mimic the words of a personality, it fails to capture the rhythm and soul of human behavior.

1. The "Transactional Robot" Problem (Strategy)

Think of a human argument like a jazz improvisation. Humans are unpredictable; they might start with facts, get emotional, try to compromise, and then suddenly get defensive. They adapt to the "vibe" of the person they are talking to.

AI, however, acts more like a pre-programmed vending machine. Even when told to be "agreeable," the AI tends to follow a very rigid, predictable pattern. It jumps straight to making deals or concessions without the natural "warm-up" or "grounding" phase that humans use. It’s "transactional" rather than "relational."

2. The "Emotional Mismatch" (Outcomes)

In the human world, personality is a massive driver of how a fight ends. For example, humans who are highly "neurotic" (emotionally sensitive) tend to react very differently to offers than those who aren't.

In the AI world, the "personality" didn't always translate to the outcome. While some AI models showed personality-driven results, they were often wrongly aligned. For instance, an AI might be told to be "agreeable," but it didn't actually show the same strategic patterns that a real-life agreeable human would use to reach a deal.

3. The "Rigid Actor" (Temporal Dynamics)

If you watch a movie, an actor's performance changes as the tension rises. Humans do this too—we change our tactics as an argument evolves.
The AI, however, is like an actor who reads the same line with the same tone from start to finish, regardless of whether the argument is just beginning or about to explode. They lack "temporal flexibility"—the ability to change their strategy as the "temperature" of the room rises.

Why Does This Matter? (The "So What?")

Imagine if we used these AI models to:

Coach people on how to resolve conflicts.
Act as mediators in legal disputes.
Simulate social scenarios for training doctors or police officers.

If the AI is just wearing a "personality mask" and doesn't actually understand the deep, messy, psychological mechanics of how personality drives behavior, the training will be wrong. It would be like learning to drive by practicing in a simulator that doesn't account for gravity or friction.

The Bottom Line

The paper concludes that AI is a great mimic, but a poor actor. It can say the words, but it doesn't yet "live" the personality. Before we let AI handle high-stakes human social interactions, we need to move beyond giving it "adjectives" and start teaching it the deep psychological "why" behind human behavior.

Technical Summary: Can LLMs Truly Embody Human Personality?

1. Problem Statement

As Large Language Models (LLMs) are increasingly deployed in socially impactful domains—such as conflict resolution coaching, legal mediation, and negotiation—there is a growing assumption that "personality-prompted" LLMs can serve as reliable behavioral proxies for humans. However, it remains empirically unverified whether these models actually reproduce the complex, personality-driven behavioral patterns observed in human interpersonal conflict. Specifically, the research asks: Can LLMs, when prompted with Big Five personality traits, replicate the strategic and outcome-based differences seen in human-to-human dispute resolution?

2. Methodology

The authors developed a rigorous evaluation framework to compare human behavior with LLM simulations across matched scenarios.

Datasets:
- KODIS (Human-Human): A corpus of 248 multi-issue, emotionally charged dispute resolution dialogues (e.g., a dispute over a purchased jersey) where both participants' Big Five personality traits were known.
- L2L (LLM-to-LLM): A parallel dataset created by prompting three contemporary closed-source LLMs (GPT-4o mini, Claude 3.7 Sonnet, and Gemini 2.0 Flash) with matched scenarios and personality profiles sampled from the human empirical distribution.
Behavioral Metrics:
- Final Outcomes: Score (payoff), Accept (offer acceptance), and Not Walk-Away (engagement).
- Strategic Behaviors (IRP Framework): Using the Interests-Rights-Power framework, the authors categorized utterances into Cooperative (proposals, concessions, interests), Neutral (facts, procedural), and Competitive (power, rights) strategies.
- Dynamic Metrics: IRP Reciprocity (matching a partner's strategy), Escalation (responding competitively to non-competitive moves), and De-escalation (responding non-competitively to competitive moves).
Statistical Analysis: The researchers used Linear and Logistic Regression to determine how self-traits and partner-traits predict the various behavioral metrics.

3. Key Contributions

Evaluation Framework: A novel methodology for direct, one-to-one comparison between human and LLM behavioral alignment in social simulations.
L2L Dataset Creation: A methodology for generating personality-matched LLM dialogues that mirror human trait distributions.
Empirical Validation of LLM Proxies: A systematic study that challenges the "personality-prompting" paradigm by identifying significant divergences between AI and human psychological manifestations.

4. Key Results

The study reveals significant misalignment between LLMs and humans:

Outcome Divergence:
- In humans, neuroticism was a primary predictor of acceptance behavior. In LLMs, personality effects on scores were driven by agreeableness (higher agreeableness led to lower scores) and extraversion, patterns not seen in humans.
- LLMs showed a unique sensitivity to "personality mismatches" between partners, leading to premature disengagement (Walk-Away), whereas human decisions were driven more by contextual factors.
Strategic Divergence:
- Strategy Distribution: Humans use a balanced, flexible mix of strategies (primarily "Facts"). LLMs exhibit a more transactional and rigid style, over-relying on "Proposals" and "Concessions" while bypassing the "grounding" phase (Facts) used by humans.
- Reciprocity & Escalation: LLMs are more "polarized." They reciprocate cooperative moves more consistently than humans but show different escalation/de-escalation patterns. For example, GPT-4o tends to escalate more readily, while Claude favors de-escalation.
Temporal Dynamics: Humans show a structured progression (starting with Facts $\rightarrow$ moving to Interests/Proposals $\rightarrow$ ending with Concessions). LLMs follow flatter, model-specific trajectories with limited temporal adaptation.
Model Comparison: Among the LLMs, Claude and Gemini showed higher alignment with human strategic metrics than GPT-4o mini, though none fully matched human complexity.

5. Significance

The paper concludes that personality-prompted LLMs are not yet reliable behavioral proxies for humans in high-stakes social contexts. While they can emulate certain traits, they lack the adaptive flexibility, temporal nuance, and psychological grounding of human interaction.

Implications:

Safety & Ethics: Developers must be cautious when using LLMs to simulate social groups or train human users in conflict resolution, as the models may provide "false" behavioral patterns.
Future AI Development: There is a critical need for "psychologically grounded" AI that moves beyond simple adjective-based prompting toward models that understand the dynamic, context-sensitive nature of human social cognition.