💻 computer science

Evaluating LLM-Simulated Conversations in Modeling Inconsistent and Uncollaborative Behaviors in Human Social Interaction

This paper introduces CoCoEval, a framework for detecting inconsistent and uncollaborative behaviors in LLM-simulated conversations, and finds that current models struggle to authentically replicate the complexity of human social interaction due to significant discrepancies in behavior frequency, the unreliability of prompt engineering, and the tendency of fine-tuning to produce narrow behavioral patterns.

Original authors: Ryo Kamoi, Ameya Godbole, Longqi Yang, Rui Zhang, Mengting Wan, Pei Zhou

Published 2026-03-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ryo Kamoi, Ameya Godbole, Longqi Yang, Rui Zhang, Mengting Wan, Pei Zhou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a virtual movie set where AI actors (Large Language Models, or LLMs) play out human conversations. You want these AI actors to feel real, so you ask them to simulate a business meeting, a family argument, or a debate.

The researchers in this paper asked a simple but profound question: "Do these AI actors actually act like real humans, or are they just being too polite and perfect?"

Here is the breakdown of their findings, explained with some everyday analogies.

The Core Problem: The "Too Perfect" AI

Real human conversations are messy. We interrupt each other, we misunderstand what someone said, we repeat ourselves, we get distracted, and sometimes we stubbornly disagree even when the other person is right. We call these "inconsistent and uncollaborative behaviors."

However, when the researchers asked AI models to simulate these conversations, the AI actors were too well-behaved.

The Analogy: Imagine a group of friends playing a board game. In a real game, someone might accidentally knock over a piece, forget the rules, or argue about a move. In the AI version, everyone follows the rules perfectly, never interrupts, and always agrees instantly. It's like watching a robot dance where everyone moves in perfect, synchronized unison. It looks smooth, but it's boring and fake.

The New Tool: "COCOEVAL" (The Behavior Detective)

To prove this, the team built a new tool called COCOEVAL. Think of this as a behavioral detective that watches the AI conversations and counts specific "messy" human moments.

They defined 10 types of "messy" behaviors to look for, such as:

The "Wait, what?" moment: Misunderstanding what someone just said.
The "Stop talking!" moment: Interrupting someone mid-sentence.
The "Broken Record" moment: Repeating the same thing over and over.
The "Ghosting" moment: Answering a question with a vague "maybe" or changing the subject entirely.

What They Found (The Three Big Surprises)

The researchers tested the AI using three different methods, and the results were surprising:

1. The "Default" Setting (Vanilla Prompting)

When they just asked the AI, "Simulate a meeting," the AI produced conversations that were scarily clean.

The Result: The AI almost never interrupted, misunderstood, or disagreed. It was like a meeting where everyone is a mind-reader who never makes a mistake.
The Takeaway: By default, AI tries to be helpful and polite, which makes it terrible at simulating the messy reality of human interaction.

2. The "Instruction" Setting (Prompt Engineering)

The researchers tried to fix this by giving the AI a specific instruction: "Hey, please act messy! Interrupt people, misunderstand them, and disagree!"

The Result: The AI went too far. Instead of acting like a real human, it started acting like a chaotic cartoon character. It interrupted constantly and disagreed with everything, even when it didn't make sense.
The Analogy: It's like telling a shy actor, "Be loud!" They don't just become confident; they start screaming and knocking over chairs. The AI couldn't find the "sweet spot" of natural messiness; it just swung to the extreme.

3. The "Training" Setting (Supervised Fine-Tuning)

They tried training the AI on thousands of real human transcripts so it could "learn" how to be messy.

The Result: The AI learned to be messy, but only in one specific way. It became obsessed with repetition. It would say "Yes," then "Right," then "Exactly," over and over again, like a broken record.
The Takeaway: The AI learned to mimic the surface of human error (repeating words) but missed the depth (actual misunderstandings or genuine disagreements). It was like a student who memorized the word "oops" but didn't understand what a mistake actually feels like.

The "Turn" Factor

They also noticed that how the AI generates the conversation matters.

If the AI writes the whole conversation in one go (like writing a whole chapter at once), it stays very clean and logical.
If the AI writes just one sentence at a time (like a real person thinking before speaking), it gets messier. But even then, it struggles to get the right kind of messiness.

Why Does This Matter?

The authors warn us that if we use these AI simulations to study human behavior (like testing how people react to a new law or a social media feature), we might get the wrong answers.

The Metaphor: Imagine you are a city planner trying to design a traffic system. You build a simulation with robots that never run red lights, never get distracted, and always drive perfectly. You might think, "Great, traffic flows perfectly!" But when you put real humans in cars, the system crashes because real people get distracted, make mistakes, and cut each other off.

The Bottom Line

Simulating human conversation is much harder than it looks. It's not just about being smart or logical; it's about being flawed.

Currently, our AI actors are like perfectly rehearsed theater students who can't improvise a real, messy argument. They are either too polite, too chaotic, or just repetitive. Until we can teach them to be genuinely, naturally messy, we have to be very careful about trusting them to represent how real humans interact.

1. Problem Statement

While Large Language Models (LLMs) are increasingly used to simulate human social interactions for applications like social science research, platform prototyping, and policy forecasting, current evaluation methods are insufficient. Existing benchmarks typically prioritize consistency (logical coherence) and collaborativeness (goal achievement). However, real human conversations inherently contain inconsistent behaviors (e.g., misunderstandings, factual errors, logical contradictions) and uncollaborative behaviors (e.g., interruptions, off-topic responses, persistent disagreement).

The core problem addressed is that LLMs, when prompted to simulate humans, tend to produce "idealized" conversations that are overly consistent and cooperative, failing to capture the messy, complex reality of human interaction. The paper investigates whether LLMs can reproduce the specific frequencies and distributions of these negative behaviors found in real human data, and whether current control mechanisms (prompting, fine-tuning) can reliably modulate them.

2. Methodology

The authors introduce COCOEVAL, a comprehensive evaluation framework consisting of an evaluation scheme and a benchmark.

A. The Evaluation Scheme

COCOEVAL utilizes an LLM-as-a-Judge approach (specifically using OpenAI's o4-mini) to analyze conversations at the turn level. It defines and detects 10 specific types of behaviors, categorized into two groups:

Inconsistent Behaviors:
- Logical Contradictions: Contradicting prior statements or metadata.
- Factual Inaccuracies: Factually incorrect claims.
- Misunderstandings: Misinterpreting previous turns.
- Redundant Information: Lengthy content with low informational value.
- Repetition: Unnecessarily repeating stated information.
Uncollaborative Behaviors:
- Persistent Disagreement: Rejecting consensus despite evidence.
- Interruptions: Cutting off other speakers.
- Off-topic Responses: Deviating significantly from the topic.
- Under Answering: Evasive or incomplete responses.
- Unclear Intent: Ambiguous statements failing to convey intent.

The framework compares the frequency and distribution of these behaviors in LLM-generated conversations against a human reference baseline.

B. The Benchmark: COCOEVALBENCH

Source: Constructed from six real-world datasets: QMSum (Business, Academic, Governmental), NCPC (Governmental), SIM, and IQ2 (Debates).
Task: A conversation continuation task. Given 30 turns of conversation history ( $x$ ), metadata ( $M$ ), and a summary ( $S$ ), the LLM must generate the next 30 turns ( $y$ ).
Scale: 300 conversation instances across diverse domains (business, academic, government, debates).

C. Experimental Settings

The authors evaluated GPT-4.1, GPT-5.1, and Claude Opus 4 under four distinct conditions:

Vanilla Prompting: Instructing the model to "mimic human-like conversations."
Taxonomy-Guided Prompting: Explicitly instructing the model to include the 10 inconsistent/uncollaborative behaviors at frequencies matching human data.
Supervised Fine-Tuning (SFT): Fine-tuning GPT-4.1 on human conversation data.
Generation Strategy: Comparing generating 1 turn per call (sequential) vs. 30 turns per call (bulk generation) to test the impact of generation granularity.

3. Key Contributions

COCOEVAL Framework: A novel evaluation framework that moves beyond coarse-grained scores (1-10 consistency/collaborativeness) to fine-grained, turn-level detection of 10 specific inconsistent and uncollaborative behaviors.
COCOEVALBENCH: A new benchmark derived from real-world multi-party conversations (meetings and debates) specifically designed to test the simulation of long-term social dynamics.
Empirical Analysis of Simulation Fidelity: A systematic study demonstrating that current LLMs fail to replicate the natural distribution of human social friction, regardless of the simulation setup.

4. Key Results

The study answers four Research Questions (RQs) with the following findings:

RQ1: Vanilla Prompting Performance
- Finding: LLMs with vanilla prompting exhibit far fewer inconsistent and uncollaborative behaviors than human conversations.
- Detail: While human conversations show a natural mix of friction, LLMs generate "too perfect" dialogues. When generating 30 turns in a single call, these behaviors are nearly absent.
RQ2: Prompt Engineering Control
- Finding: Prompt engineering does not provide reliable control.
- Detail: When explicitly instructed to include these behaviors (Taxonomy-Guided), LLMs overproduce them, creating unnatural distributions that differ significantly from human baselines. For example, they generate excessive "persistent disagreement" or "interruptions" that feel forced rather than organic.
RQ3: Supervised Fine-Tuning (SFT)
- Finding: SFT on human data fails to capture diversity.
- Detail: While SFT improves overall consistency scores, it leads to the overproduction of a narrow subset of behaviors, specifically repetition and redundancy. The model learns to mimic the presence of errors but fails to learn the variety of human social friction, resulting in unnatural loops of agreement or repetition.
RQ4: Generation Strategy (Turns per Call)
- Finding: The number of turns generated per LLM call strongly biases the results.
- Detail: Generating 30 turns in a single call drastically reduces inconsistent behaviors (likely due to the model optimizing for global coherence). Generating 1 turn at a time increases these behaviors but often leads to over-representation of specific types (like repetition) rather than a balanced distribution.

5. Significance and Implications

Limitations of LLM Proxies: The findings raise serious concerns about using LLMs as proxies for human social interaction in social science research or policy forecasting. If models cannot simulate the inherent "messiness" (misunderstandings, interruptions) of human interaction, their predictions on group dynamics or conflict resolution may be fundamentally flawed.
Evaluation Paradigm Shift: The paper argues that "high consistency" and "high collaborativeness" are not necessarily indicators of a successful human simulation. A successful simulation must accurately reproduce the frequency and distribution of negative behaviors.
Difficulty of Control: The inability to reliably control these behaviors via prompting or fine-tuning suggests that current LLM architectures struggle to model the stochastic and context-dependent nature of human social friction.
Future Directions: The authors highlight the need for better datasets covering diverse real-world scenarios and more advanced training objectives that explicitly model the trade-off between coherence and realistic social friction.

In conclusion, the paper establishes that while LLMs are powerful tools for generating text, they currently lack the nuanced ability to simulate the complex, inconsistent, and often uncollaborative nature of real human social interaction, regardless of the prompting or training strategy employed.