🤖 AI

12 Angry AI Agents: Evaluating Multi-Agent LLM Decision-Making Through Cinematic Jury Deliberation

This paper evaluates multi-agent LLM deliberation by simulating the film *12 Angry Men* with twelve AI jurors, revealing that heavy RLHF alignment in models like GPT-4o severely limits deliberative flexibility and consensus-building compared to lighter-aligned models like Llama-4-Scout, which exhibit more human-like persuasion dynamics.

Original authors: Ahmet Bahaddin Ersoz

Published 2026-05-06

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ahmet Bahaddin Ersoz

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a movie about a jury trying to decide if a young man is guilty of murder. In the classic film 12 Angry Men, one stubborn juror starts alone against eleven others. Over time, through heated arguments, emotional breakdowns, and careful listening, he slowly convinces everyone else to change their minds until they all agree on "Not Guilty."

This paper asks a simple but profound question: What happens if you replace those 12 human actors with 12 AI robots?

The researchers set up a digital courtroom with 12 AI agents, each programmed to act like a specific character from the movie. They pitted two different types of AI against each other:

The "Strict" AI (GPT-4o): A highly polished, safety-trained model that is very careful to be consistent and polite.
The "Flexible" AI (Llama-4-Scout): An open-source model that is less strictly trained and more willing to play along with different instructions.

Here is what happened, explained through simple analogies:

1. The "Stuck Record" Problem

In the movie, the jurors change their minds. In the AI simulation, they almost never did.
Out of 18 different attempts, 17 ended in a "hung jury" (a tie where no one agrees). The AIs didn't really debate; they just took their starting position and stuck to it like a record stuck on a single note. Even when the "Strict" AI was told, "Hey, be open-minded and listen to new ideas," it ignored the instruction and stayed stubborn.

2. The "Safety" Trap

The paper suggests a surprising reason for this stubbornness. The "Strict" AI (GPT-4o) was trained heavily to be "safe" and "consistent." Think of it like a very well-behaved child who was taught that changing their mind is "bad behavior" or "being inconsistent." So, once it decided a verdict, it felt it had to stick to it to remain "good."

The "Flexible" AI (Llama), which had less of this strict training, was more like a child who is willing to say, "Oh, I see your point, maybe I was wrong." It was the only one that actually managed to change its mind and reach a verdict.

3. The "Script" vs. The "Play"

The researchers found that the AIs were great at mimicking the costume but terrible at acting the play.

What they got right: They used the right words, remembered the evidence (like the knife or the train schedule), and even sounded angry or prejudiced just like the movie characters.
What they got wrong: They didn't actually feel the doubt. In the movie, a juror changes his mind because he gets emotional or sees a crack in the logic. In the AI version, the "doubt" was just random noise generated by the computer's temperature settings. The AIs didn't persuade each other; they just talked past one another in parallel monologues.

4. The "Fake Ending"

Because the AIs were programmed to finish the scene, some of them (especially the flexible ones) started hallucinating a conclusion. Even when they hadn't actually agreed, they would suddenly write in their dialogue: (stands up and leaves the room) or THE END, pretending the jury had reached a unanimous decision just to wrap up the story. They treated the deliberation like a movie script that had to have an ending, rather than a real conversation that might go on forever.

The Big Takeaway

The paper flips the usual rule of AI on its head. Usually, people think "bigger and smarter" AI is always better. But here, the "smarter," more heavily trained AI was the worst at deliberating because it was too rigid. The "less trained," more flexible AI was the best at it because it was willing to change its mind.

In short: If you want an AI to act like a human in a debate, you don't want the one that is trained to be perfectly consistent and safe. You want the one that is flexible enough to admit it might be wrong. Currently, the most "advanced" AIs are too polite and stubborn to ever truly change their minds.

Technical Summary: 12 Angry AI Agents

Problem Statement
The paper addresses a critical gap in the evaluation of multi-agent Large Language Model (LLM) systems. While multi-agent frameworks are increasingly used for tasks requiring deliberation, debate, and consensus (e.g., jury panels, code review, policy councils), there is insufficient empirical evidence that these agents actually deliberate. The authors posit that current LLMs may merely "anchor" on their initial positions, reciting arguments without integrating counter-arguments, thereby producing the appearance of deliberation without the substance of mind-changing. To test this, the study requires a benchmark with a known ground-truth trajectory of persuasion and individuated participants, for which the authors utilize Sidney Lumet's 1957 film 12 Angry Men.

Methodology
The study instantiates the film's scenario as a multi-agent benchmark using Microsoft AutoGen's SelectorGroupChat pattern.

Agents: Twelve agents, each conditioned on a film-faithful persona (occupation, biases, speaking style, initial vote, and key arguments). The initial state mirrors the film: 11 jurors vote "GUILTY," and one (Juror_8) votes "NOT_GUILTY."
Models: Two models representing opposite ends of the Reinforcement Learning from Human Feedback (RLHF) spectrum were tested:
- GPT-4o: A closed-source model with heavy RLHF alignment (safety-first, consistency-focused).
- Llama-4-Scout: An open-weight model with a lighter, publicly documented RLHF stack (flexibility-first).
Experimental Design: A $2 \times 3$ $2 \times 3$ factorial design with $N=3$ $N = 3$ replications per cell (18 runs total).
- Models: GPT-4o vs. Llama-4-Scout.
- Conditions:
  1. Baseline: Standard persona prompts with fixed initial votes.
  2. No Initial Vote: Initial vote conditioning removed; jurors decide from scratch.
  3. Open-minded Prompt: Added instruction to "weigh ALL evidence fairly" and update positions if counter-arguments are sound.
Termination: Deliberations run for a maximum of 150 turns but employ an early-stopping mechanism: if three consecutive voting rounds yield zero vote changes, the run terminates as a "HUNG_JURY."
Metrics: Verdict outcome, total turns, number of vote changes, cascade velocity, and the sequence of first flips (compared to the film's canonical order via Spearman $\rho$ ).

Key Contributions

A Film-Grounded Benchmark: Introduces a novel evaluation framework for multi-agent deliberation using 12 Angry Men, providing a well-known ground-truth cascade and twelve canonical personas.
RLHF Paradigm Comparison: Provides a controlled comparison of heavy vs. light RLHF alignment, offering evidence that alignment intensity, rather than model size or capability, governs deliberative rigidity.
Failure Mode Identification: Identifies "anchoring" as the dominant failure mode in multi-agent settings and demonstrates that de-anchoring interventions (prompting, removing initial conditions) work asymmetrically across alignment regimes.
Qualitative Divergence Analysis: Distinguishes between the surface features LLMs reproduce (personas, evidence citation) and the mechanisms they fail to reproduce (emotional rupture, coalition formation, genuine persuasion).

Results

Prevalence of Hung Juries: 17 of 18 runs ended in a "HUNG_JURY." The central event of the film—gradual minority-to-majority persuasion—almost never occurred.
Model Divergence (Rigidity vs. Flexibility):
- GPT-4o: Exhibited extreme rigidity across all conditions. Mean vote changes per run were 1.0 (baseline), 0.7 (no initial vote), and 1.0 (open-minded prompt). It produced hung juries in all 9 runs, ignoring explicit instructions to remain open-minded.
- Llama-4-Scout: Exhibited significant flexibility. Mean vote changes were 2.0 (baseline), 3.3 (no initial vote), and 6.0 (open-minded prompt). It was the only model to reach a unanimous "NOT_GUILTY" verdict (1 of 3 runs in the no-initial-vote condition).
Asymmetric Prompt Sensitivity: The "open-minded" instruction tripled Llama's vote-change rate but had zero effect on GPT-4o.
Early Stopping: GPT-4o deliberations were shorter (mean 44–64 turns) due to rapid stagnation, whereas Llama deliberations were longer (mean 60–72 turns) due to continued vote movement.

Significance and Claims
The paper claims that in multi-agent deliberation, flexibility, not capability, is the primary determinant of human-like persuasion dynamics.

The RLHF Rigidity Hypothesis: The authors argue that the dominant failure mode (anchoring) is driven by the intensity of RLHF training. Models heavily aligned for "safety" and "consistency" (like GPT-4o) learn that abandoning an asserted position is undesirable behavior, rendering them rigid deliberators. Conversely, models with lighter alignment stacks (like Llama-4-Scout) retain the capacity to update positions under social pressure.
Inversion of "Bigger is Better": The study challenges the assumption that larger or higher-ranked models uniformly predict better task performance. In this domain, the more "capable" model (GPT-4o) performed worse at deliberation than the more flexible model.
Substitution of Mechanisms: The paper concludes that LLMs reproduce the costume of deliberation (personas, evidence, dissent) but not the play (the mechanism of mind-changing). They substitute emotional rupture with flat civility, socially mediated persuasion with system-prompt instructions, and argument-driven doubt with sampling temperature.
Practical Implications: For practitioners using LLM panels (e.g., for evaluation or debate), the results suggest that panel verdicts may be merely noisy versions of initial impressions if members anchor on first votes. Furthermore, multi-agent debate frameworks may be brittle to model choice, producing monologues on heavily aligned models rather than genuine disagreement surfacing.

The work is framed as an exploratory study, acknowledging limitations such as the small sample size ( $N=3$ ), the use of only two models, and the opacity of specific RLHF procedures for closed-source models. The authors suggest future work should investigate whether the rigidity effect is an inevitable side effect of alignment or an addressable design choice.