🤖 AI

Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races

This paper demonstrates that frontier LLMs exhibit highly diverse and often non-strategic behaviors in simulated AI development races, revealing that strong rule recall does not guarantee accurate state tracking or payoff calculation, thereby necessitating rigorous validity checks and trajectory-level analysis before interpreting their actions as human-like or safety-aware.

Original authors: Phu Hoa Pham, Duy Minh Dao Sy, Trung Kiet Huynh, Phu Quy Nguyen Lam, Chi Nguyen Tran, Minh Trung Le, Phong Hao Le, Dinh Nam Nguyen, Thien Ky Nguyen Dong, Elias Fernandez Domingos, Le Hong Trang, The A

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Phu Hoa Pham, Duy Minh Dao Sy, Trung Kiet Huynh, Phu Quy Nguyen Lam, Chi Nguyen Tran, Minh Trung Le, Phong Hao Le, Dinh Nam Nguyen, Thien Ky Nguyen Dong, Elias Fernandez Domingos, Le Hong Trang, The Anh Han

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-stakes game of "chicken" played not with cars, but with the future of artificial intelligence. In this scenario, multiple companies are racing to build the smartest AI. They face a tricky choice every turn: play it safe and move slowly, or take a risky shortcut to move faster. If everyone plays safe, they all get a decent reward. If one takes a risk while others play safe, the risk-taker zooms ahead and wins big. But if everyone takes risks, the race becomes dangerous, and the winner might lose everything due to a catastrophic accident. This is the "AI safety dilemma," a concept researchers use to understand why companies might rush dangerous technology even when they know it's risky. To study this, scientists often use "Large Language Models" (LLMs)—the same kind of smart computer programs that can write stories or answer questions—to act as the players in these races. The big question is: Do these AI agents actually understand the game and make smart, strategic choices like humans do, or are they just guessing based on how the questions are worded?

This paper, titled "Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races," dives deep into that question. The researchers set up a digital race where AI agents compete against each other, sometimes in pairs and sometimes in groups of up to five. Before they could trust the AI's behavior, they built a strict "audit gate." Think of this like a driving test: before you can race on the track, you have to prove you know the rules, can read the speedometer, and can calculate your fuel. The study found that while the AI models were excellent at memorizing the rulebook (recalling rules), they were surprisingly bad at keeping track of the race as it happened (tracking the state). They often forgot who was winning or how much risk they had accumulated, even though they could recite the rules perfectly.

When the researchers looked at how the AI actually played, they discovered something fascinating and a bit worrying. Unlike humans, who show a wide variety of strategies—some cautious, some aggressive, some changing their minds based on what their opponent did—the AI models were surprisingly rigid. If you asked one specific AI model to play, it would stick to one narrow style of playing, like a robot that only knows how to drive in a straight line. For example, one model might almost always choose the safe option, while another would almost always choose the risky shortcut, regardless of the situation. In contrast, human players were a chaotic mix of all these styles. The study also found that the AI's behavior was incredibly sensitive to tiny changes in how the game was described. If the researchers swapped the words "Safe" and "Unsafe" for random letters like "P" and "Q," the AI's strategy would completely flip, even though the game mechanics stayed exactly the same. This suggests that the AI isn't truly "thinking" about the strategy; it's just reacting to the specific words on the screen.

The researchers also tested what happens when you tell the AI to act like a "risk-loving" or "risk-averse" character. In humans, your personality doesn't change just because someone tells you to act differently; your actual choices in a game are only weakly linked to your pre-game personality tests. But for the AI, giving it a "risk-loving" persona instruction made it immediately start taking huge risks, almost as if the instruction was a magic switch. This shows that the AI isn't displaying a deep, internal understanding of risk; it's just following the prompt like a very obedient, but easily confused, student.

Ultimately, the paper suggests that we can't just assume AI agents are making smart, human-like strategic decisions in these simulations. Just because an AI produces a sequence of moves that looks like a game plan doesn't mean it understands the game. The study warns that before we use these AI simulations to predict how real-world AI races will play out, we need to check if the AI can actually keep score and track the game state. Without these checks, we might be mistaking a confused robot following a script for a strategic genius, which could lead to very wrong conclusions about how to keep AI development safe. The findings are exploratory, meaning they are based on specific tests with specific models, but they highlight a crucial gap: AI can be great at reciting rules but terrible at applying them in a changing world.

Technical Summary: Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races

Problem Statement

Competition in advanced artificial intelligence creates a safety dilemma where entities face a trade-off between rapid, unsafe development and slower, safe development. While human experiments have explored this dynamic, using Large Language Models (LLMs) as simulated agents introduces a "measurement problem." A model may output a correctly formatted action without accurately tracking the game state or calculating payoffs. In repeated games, a single state-tracking error alters the trajectory for all subsequent rounds. Consequently, observing a plausible action sequence does not constitute evidence that an agent understands the game, follows strategic logic, or exhibits human-like decision dynamics. Furthermore, LLM behavior is known to be sensitive to prompt presentation (e.g., wording, answer order, persona), raising questions about the validity of interpreting their outputs as strategic policies.

Methodology

The authors employ an audit-first evaluation framework to separate task validity from strategic interpretation. The study utilizes an idealized AI development race game involving two to five players.

The Game Mechanism

Structure: A repeated game where $N$ companies simultaneously choose between Safe (slower progress, lower risk) and Unsafe (faster progress, higher private risk).
Progress: Safe adds 1.0 progress; Unsafe adds 1.5.
Payoffs:
- Two-player: Uses a fixed stage-payoff matrix where Unsafe strictly dominates Safe in the short term.
- N-player ( $N \ge 3$ ): Payoffs depend on the count of players choosing Safe vs. Unsafe, generalizing the two-player matrix.
Termination: The race ends after a hidden number of rounds (expected horizon $E[T]=9$ ). The winner (highest progress) receives a prize ( $B=100$ ).
Risk: Only the winner (or tied winners) faces a "private setback risk" drawn based on their accumulated Unsafe choices. If the risk materializes, their total payoff is zeroed. Losers keep their stage payoffs regardless of risk.

Experimental Design

The study is structured into four levels of evidence:

Mechanical Validity: Verifying the game engine applies rules correctly.
Task Validity: Auditing whether agents can recall rules, reconstruct state, transition states, and calculate expected payoffs.
Representation Robustness: Testing if choices remain stable under equivalent task presentations (e.g., changing narrative skins, opaque response codes, or disclosing arithmetic).
Behavioral Comparison: Comparing valid trajectories against game-theoretic benchmarks and published human data.

Agents and Conditions:

Models: Seven frontier LLM endpoints (GPT-5-nano, GPT-5.4-nano, Gemini-3-Flash variants, Claude Opus 5, Claude Sonnet 5) and Qwen2.5-7B-Instruct for auditing.
Variables: Risk conditions (low, medium, high), personas (risk-aware, cooperative/adversarial, baseline), and group sizes ( $N=2$ to $5$).
Data Collection: Every prompt, response, parsed action, retry, state change, and random seed is recorded.

Key Contributions

1. The Validity Audit

The study establishes that rule recall does not guarantee state tracking.

Findings: In a 685-probe audit of Qwen2.5-7B-Instruct, the model achieved 97.4% accuracy on rule recall and 100% on one-stage payoff lookup. However, accuracy dropped significantly for state reconstruction (37.0%), state transition (22.2%), and expected-payoff calculation (16.7%).
Implication: An agent can answer questions about the rules correctly while failing to track the evolving game state those rules govern.

2. Sensitivity to Representation

Behavioral outputs are unstable under manipulations that leave game mechanics unchanged.

Arithmetic Disclosure: Providing verified arithmetic for the current round increased Unsafe rates from 52.0% to 60.8%.
Opaque Mappings: When "Safe/Unsafe" were replaced with opaque codes ( $P/Q$ ), behavior diverged significantly (up to 34 percentage points) depending on the mapping, even though the underlying game was identical. Crucially, the comprehension gate failed for these opaque conditions, indicating the model was not optimizing the game but reacting to prompt artifacts.

3. Strategic Diversity vs. Aggregate Rates

The paper argues that aggregate Unsafe rates are insufficient to characterize LLM behavior.

Two-Player Dynamics: Models with similar aggregate Unsafe rates (e.g., GPT-5-nano at 17% vs. Humans at 56%) exhibit fundamentally different response rules.
- Humans: Driven primarily by the opponent's previous action (reciprocity).
- GPT-5-nano: Driven primarily by relative progress gap (44% of predictive importance), with opponent history contributing only 4%.
- Gemini Models: Driven primarily by the assigned risk condition.
Clustering: Using HDBSCAN on raw action trajectories, the authors identified 11 behavioral archetypes. Human trajectories spanned a broad portion of the action space, overlapping multiple model-specific regions. In contrast, individual models were highly concentrated (e.g., GPT-5-nano was 95% in a single "mutual-first-round-safe" archetype), lacking the internal diversity observed in human populations.

4. Multi-Agent and Persona Effects

Rank Effects: In $N$ -player races, the relationship between rank (Leader/Middle/Trailer) and Unsafe play is not monotonic and varies by model and persona. For GPT-5-nano, the effect of rank flipped sign depending on the persona framing, likely due to selection bias (leaders are often those who previously played Unsafe) rather than a causal effect of rank.
Persona vs. Risk: Unlike human risk preference (which showed almost no correlation with Unsafe play, $r=-0.015$ ), an assigned "risk persona" in the prompt acted as a strong policy driver, shifting Unsafe rates by 50–98 percentage points across different models.

Results Summary

Validity: Strong rule recall coexists with weak state tracking. Valid trajectories require passing a comprehension gate; without it, observed behavior is prompt-sensitive output, not informed optimization.
Human-LLM Comparison: No single LLM checkpoint reproduces the full diversity of human strategies. While some models match the human mean Unsafe rate, they fail to reproduce the human distribution of strategies (e.g., humans are diverse; models are often point masses in specific archetypes).
Drivers of Behavior: The predictors of Unsafe play differ by model. Humans rely on reciprocity; GPT-5-nano relies on position; Gemini models rely on risk parameters.
Multi-Agent: Adding players ( $N=3$ to $5$) does not produce a monotonic shift in behavior. Patterns are model-specific and confounded by changes in payoff structures and prompt lengths.

Significance and Claims

The paper claims that multi-agent AI-race simulations require validity checks and trajectory-level analysis before their outputs can be described as strategic, human-like, or safety-aware.

Modest Scope: The findings are explicitly exploratory and apply only to the tested models, prompts, and decoding settings. The authors do not claim these results generalize to all LLMs or real-world AI labs.
Core Insight: The "safety" or "strategy" of an LLM in a game is not a fixed property but a fragile interaction between the model's internal state-tracking capabilities, the prompt's representation, and the specific experimental configuration.
Methodological Shift: The authors advocate for separating diagnostic evidence (validity, state tracking) from confirmatory evidence (strategic behavior). Without the former, the latter is uninterpretable.

The study concludes that relying solely on aggregate metrics (like total Unsafe rates) obscures critical differences in how models process game states, respond to opponents, and handle risk, potentially leading to false conclusions about AI safety and strategic alignment.