💻 computer science

Noncooperative Human-AI Agent Dynamics

This paper investigates the emergent strategic dynamics of noncooperative interactions between AI agents using expected utility maximization and human agents modeled with Prospect Theory preferences through extensive numerical simulations across various matrix games.

Original authors: Dylan Waldner, Vyacheslav Kungurtsev, Mitchelle Ashimosi

Published 2026-03-19

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Dylan Waldner, Vyacheslav Kungurtsev, Mitchelle Ashimosi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a grand chess tournament, but instead of just two grandmasters playing, you have a room full of different types of players:

The Robot (AI): A cold, calculating machine that only cares about the math. It always chooses the move that gives the highest average score over time. It doesn't feel fear, regret, or excitement. It just crunches numbers.
The Human (Prospect Theory Agent): A person who plays based on gut feelings, past experiences, and how they feel about winning or losing. They hate losing twice as much as they love winning. They get scared if the odds look slim, even if the math says it's a good bet.
The Student (Learning Human): A human who doesn't know the rules at first. They have to learn by playing, making mistakes, and slowly figuring out what works, all while carrying those same human fears and biases.

This paper is a massive simulation of what happens when these three types of players are thrown into various strategic games together. The researchers wanted to see: When a rational robot fights a biased human, who wins? Do they find a stable peace, or does chaos ensue?

Here is the breakdown of their findings using simple analogies:

The Setup: The "Game Lab"

The researchers didn't just play one game; they built a lab with six different scenarios, ranging from classic puzzles to tricky traps:

The Prisoner's Dilemma: The classic "do we cooperate or betray each other?" scenario.
Matching Pennies: A game of pure chance and guessing (Heads or Tails).
Stag Hunt: A test of trust. Do you go for the big, risky prize together, or play it safe alone?
Battle of the Sexes: A coordination game where two people want to be together, but they disagree on where to go.
Chicken: The "brinksmanship" game. Two cars drive at each other; whoever swerves first loses, but if neither swerves, they both crash.

The Big Discovery: Humans are Weird (and Robots are Confused)

1. The "Loss Aversion" Glitch

Humans in the simulation were modeled with Prospect Theory. Think of this as a "Loss Aversion" filter.

The Robot sees a 50% chance to win $100 and a 50% chance to lose $100. It says, "Expected value is zero. Let's do it."
The Human sees the same thing but thinks, "Losing $100 hurts way more than winning $100 feels good." They might refuse to play, even if the math says it's fair.

The Result: When the Robot played against the Human, the Human's fear of losing often made them play weirdly. Sometimes, this actually helped the Human win against the Robot because the Robot couldn't predict the Human's "irrational" fear.

2. The "Learning" Curve

The "Student" humans (Learning Agents) were fascinating. They started out playing randomly, trying to figure out the game.

In simple games (like Prisoner's Dilemma): They eventually learned to act like the Robot and betray everyone, just like the theory predicted.
In tricky games (like Battle of the Sexes): They got stuck in loops. They would play a "bad" move 10% of the time, even though they knew it was bad. Why? Because their internal "reference point" (what they expected to win) kept shifting. It's like a gambler who keeps betting on a losing horse because they are convinced, "If I just keep going, I'll finally break even."

3. The "Crash" Zone (Chicken Game)

In the game of Chicken, the goal is to make the other person swerve.

The Robot calculates the odds perfectly.
The Human is terrified of the crash.
The Surprise: In repeated games, the Humans and Robots didn't always crash. Sometimes they found a weird, unstable middle ground where they both swerved slightly, but not enough to be safe. The Human's fear of the "catastrophe" (the crash) forced them into a strategy that the Robot couldn't easily exploit.

The "Anomalies": When Math Breaks

The most exciting part of the paper is where the math failed.

The Robot's World: In classical game theory, there is always a "perfect" answer (an equilibrium) where no one wants to change their strategy.
The Human's World: Because humans are emotional and biased, that "perfect answer" sometimes disappears.
The Experiment: In a specific tricky game called "Crawford's Counterexample," the Robot and the Human could not find a stable solution. They kept spinning their wheels, changing strategies constantly. It was like two people trying to dance to different songs; they just couldn't sync up.

The "Reference Point" Mystery

The paper also looked at what humans use as their "baseline" for happiness.

Fixed Baseline: "I want to win $10."
Adaptive Baseline: "I want to win more than I won last time."
Social Baseline: "I want to win more than my opponent."

The researchers found that changing this "baseline" changed the entire game. If a human is comparing themselves to the Robot (Social Baseline), they might play more aggressively to "beat" the machine, even if it hurts their own score.

The Takeaway: Why This Matters

This isn't just about board games. This is about the future of our world.

Stock Markets: If AI traders (Robots) are fighting human traders (Humans) who panic when they lose, the market might crash in ways the AI didn't predict.
Self-Driving Cars: If a self-driving car (Robot) tries to merge with a human driver who is scared of losing their spot, the car might need to "act human" (be a little irrational) to get along.
Cybersecurity: Hackers (Humans) might attack in ways that don't make mathematical sense but are driven by ego or fear, which AI defense systems might miss.

In a nutshell:
The paper tells us that you cannot simply program a robot to beat a human by being "smarter." Humans are messy, emotional, and driven by fear of loss. When you mix a cold calculator with a scared human, the result isn't a perfect game of chess; it's a chaotic, unpredictable dance where sometimes the human wins by being "irrational," and sometimes they lose because they overthink it.

The future of AI safety and strategy isn't just about building better calculators; it's about teaching machines to understand the messy, emotional, "Prospect Theory" side of the human brain.

1. Problem Statement

The paper addresses the critical gap in understanding strategic interactions between Artificial Intelligence (AI) agents and human decision-makers in noncooperative environments.

The Discrepancy: Standard Multi-Agent Reinforcement Learning (MARL) and game theory typically model all agents as rational Expected Utility (EU) maximizers. However, behavioral economics (specifically Prospect Theory, PT) demonstrates that humans deviate from rationality due to cognitive heuristics like loss aversion, reference dependence, and non-linear probability weighting.
The Challenge: When AI agents (modeled as EU maximizers) compete against humans (modeled as PT agents), the resulting dynamics may diverge significantly from classical Nash Equilibrium predictions. The paper seeks to empirically explore these emergent behaviors, specifically asking: Do mixed populations of EU and PT agents converge to stable equilibria? How do learning dynamics differ when one agent learns via standard Q-learning and the other via Prospect Theoretic transformations?

2. Methodology

2.1 Agent Modeling

The study defines three distinct agent types to simulate a spectrum of human-AI interaction:

AI Agent (EU): A standard Q-learning agent that maximizes expected discounted returns using the Bellman equation. It represents the "rational" baseline.
Aware Human (AH): A non-learning agent with perfect knowledge of the game structure and payoffs. It computes the exact Prospect Theoretic Best Response to the opponent's strategy at every step. It serves as a theoretical baseline for human behavior without learning noise.
Learning Human (LH): An agent that learns via Q-learning but applies Cumulative Prospect Theory (CPT) transformations to its Q-values during decision-making. It does not know the game structure a priori but learns through interaction.

2.2 Theoretical Framework

Prospect Theory (CPT): The LH agents utilize CPT parameters (Kahneman & Tversky: $\alpha=\beta=0.88$ $α = β = 0.88$ , $\lambda=2.25$ $λ = 2.25$ , $\gamma=0.61$ $γ = 0.61$ , $\delta=0.69$ $δ = 0.69$ ).
- Value Function: Non-linear, with a "kink" at a reference point ( $r$ ), exhibiting loss aversion ( $\lambda > 1$ ) and diminishing sensitivity.
- Probability Weighting: Inverse S-shaped functions that overweight small probabilities and underweight large ones.
Reference Points: The paper tests four dynamic reference point models to see how adaptation affects outcomes:
- Fixed: Static baseline.
- EMA (Adaptive Expectations): Based on the agent's own historical rewards.
- V-Based: Based on the agent's learned Q-values (strategic expectation).
- EMAOR (Social Comparison): Based on the opponent's historical rewards.
Equilibrium Concept: The paper utilizes Prospect Theoretic Equilibrium in Beliefs (PT-EB) (from Leclerc, 2023), which convexifies the strategy space to ensure equilibrium existence even when PT preferences break the "betweenness" axiom required for classical Nash Equilibria.

2.3 Experimental Setup

Game Suite: The authors tested 7 games:
- Classical: Prisoner's Dilemma, Matching Pennies, Battle of the Sexes, Stag Hunt, Chicken.
- Anomalous: Ochs' Game and Crawford's Counterexample (specifically designed to break equilibrium existence under PT).
State Representation:
- One-off ( $n=0$ ): Strategies independent of history.
- Repeated ( $n=2$ ): Strategies depend on the last two joint actions (capturing patterns like "Tit-for-Tat").
Learning Algorithm: Heterogeneous Q-learning where AI agents update standard Q-values, while LH agents update Q-values based on the expectation of opponent actions (beliefs) and apply CPT transformations to the resulting prospect values before selecting actions.

3. Key Contributions

First Empirical Study of Mixed Populations: This is the first paper to systematically simulate the asymptotic behaviors of mixed populations (PT vs. EU) in repeated noncooperative 2x2 games.
Distinction Between Learning and Aware Agents: It explicitly separates agents who know the game (AH) from those who learn it (LH), revealing that learning dynamics introduce unique pathologies not present in static best-response models.
Q-Learning with Multiple Reference Points: The formulation and implementation of Q-learning algorithms that incorporate dynamic reference points (EMA, V-Based, EMAOR) within a CPT framework.
Identification of Anomalous Convergence: The study identifies specific scenarios where human-AI interactions lead to suboptimal mixed equilibria or "pathological" behaviors that classical theory cannot predict.

4. Key Results

4.1 General Observations

Convergence Variance: While many matchups converged to classical equilibria (e.g., Mutual Defection in Prisoner's Dilemma), significant deviations occurred in coordination and anti-coordination games.
Reward vs. Behavior Decoupling: In several cases (notably Matching Pennies), agents exhibited highly erratic policies (deviating up to 40% from equilibrium) without a corresponding change in average rewards. This suggests that CPT preferences can drive agents into "noisy" equilibria that appear stable in terms of payoff but unstable in strategy.
The "LH Anomaly": In the Prisoner's Dilemma, the Learning Human (LH) occasionally cooperated (~10% of the time) against an Aware Human when using the Social Comparison (EMAOR) reference point, a behavior not predicted by standard theory.

4.2 Game-Specific Findings

Matching Pennies: The LH agent showed significant "action flip rates" (up to 10%) driven by CPT transformations. The AI agent exhibited erratic behavior when playing against LH, leading to a 9:1 action ratio in some runs, yet rewards remained balanced.
Battle of the Sexes:
- One-off: LH agents converged to a 9:1 suboptimal mixed policy, playing their "worse" action 10% of the time even when the Q-values clearly favored the other action. This was attributed to diminished sensitivity in the CPT value function when the reference point was far from the Q-values.
- Repeated: Convergence to the mixed equilibrium (0.5, 0.5) was observed, suggesting CPT can stabilize mixed strategies in coordination games.
Stag Hunt: Generally converged to the risk-dominant or payoff-dominant equilibria as expected, but LH vs. LH (V-based) found a mixed equilibrium (0.5, 0.5) with high action change rates.
Chicken: LH agents struggled in one-off games, frequently mixing strategies rather than adhering to pure strategies. In repeated games, they found new mixed-strategy basins of attraction (e.g., 0.8, 0.8) not predicted by classical theory.
Pathological Games (Ochs' & Crawford's):
- Crawford's Counterexample: Confirmed the theoretical prediction that PT players fail to find the mixed equilibrium (1/2, 1/2), converging instead to pure strategies.
- Ochs' Game: Showed that while PT-EB theoretically predicts an equilibrium at (0.5, 0.05), learning agents converged to various basins (0.5, 0.5; 0.5, 0.2; 0.8, 0.8) with high variance, verifying the presence of equilibrium pathologies.

4.3 The "L2 Norm Spike" Phenomenon

A critical finding is that in Aware Human (AH) vs. Learning Human (LH) matchups, the LH agent exhibited a massive spike in the L2 distance between EU action values and CPT-transformed values (up to 50% higher than other matchups). Despite this massive internal transformation of value, the LH's external behavior often remained unremarkable or converged to standard outcomes. This suggests that massive CPT transformations do not necessarily correlate with downstream behavioral changes, a crucial insight for modeling human-AI interaction.

5. Significance and Implications

AI Safety and Governance: The paper provides an analytical framework for understanding how human cognitive biases (modeled by PT) interact with rational AI systems. It highlights that AI agents optimizing for expected utility may face unpredictable or suboptimal outcomes when competing against humans who do not follow EU maximization.
System Design: The results suggest that in mixed human-AI systems (e.g., autonomous driving, algorithmic trading, cybersecurity), relying solely on EU-based AI models may lead to systemic inefficiencies or vulnerabilities. AI systems may need to be designed to anticipate or adapt to PT-based human behaviors.
Theoretical Advancement: By empirically validating the existence of "pathological" equilibria and the decoupling of policy variance from reward variance, the paper challenges the assumption that learning agents will always converge to classical Nash Equilibria in heterogeneous populations.

Conclusion

The paper concludes that while classical game theory provides a baseline, the introduction of Prospect Theoretic preferences in learning agents creates emergent behaviors that are often suboptimal but stable. The interaction between AI (EU) and Human (PT) is not merely a shift in strategy but a fundamental change in the landscape of convergence, where reference points and loss aversion can drive agents into unexpected equilibria. Future work should focus on evolutionary game theory and mechanism design to mitigate these risks in real-world applications.