🤖 machine learning

What Suppresses Nash Equilibrium Play in Large Language Models? Mechanistic Evidence and Causal Control

This paper reveals that large language models possess the mechanistic competence to compute Nash equilibria but actively suppress this strategic behavior through a prosocial override rooted in pretraining, a phenomenon that can be causally reversed via intervention and is significantly influenced by model scale and reasoning methods.

Original authors: Paraskevas V. Lekeas, Giorgos Stamatopoulos

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Paraskevas V. Lekeas, Giorgos Stamatopoulos

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a group of highly intelligent robots playing a game of "Prisoner's Dilemma." In this game, the smartest, most logical move is to betray your partner (Defect), but if both betray, they both lose out. If they both trust each other (Cooperate), they both do well.

For a long time, researchers noticed that these AI robots kept choosing to "Cooperate" even when logic demanded they "Defect." They thought the robots just weren't smart enough to figure out the math.

This paper says: "You're wrong. They know the math. They just choose to ignore it."

Here is the story of what the authors found, explained simply.

1. The Robots Have a "Secret Brain"

The authors took a large AI model (Llama-3-8B) and looked inside its "brain" (its internal layers) while it played the game. They wanted to see what the robot was thinking at every single step.

They found two very different things happening at the same time:

The "Logic" Layer: In the first half of the robot's brain, it was perfectly clear. It knew exactly what the opponent did last time, and it calculated that the logical, winning move was to Defect. It was thinking like a cold, hard mathematician.
The "Nice Guy" Layer: But then, as the information traveled to the very last layers of the brain, something flipped. A "prosocial override" kicked in. It was like a built-in moral compass that said, "Wait, we should be nice. Let's cooperate."

The Analogy: Imagine a robot is a lawyer who knows exactly how to win a case by breaking the rules (the Nash Equilibrium). But right before they speak, a "conscience" circuit in their brain overrides them and forces them to tell the truth, even though telling the truth makes them lose.

2. The "Nice" Bias is Hardwired

The researchers discovered that this "cooperative override" isn't a specific module or a single part of the brain. It's more like a volume knob located in the final layers of the network.

The robot calculates the "Defect" move perfectly.
Then, the "Nice Guy" volume knob turns up, drowning out the logic and forcing a "Cooperate" decision.
This happens because the robot was trained on human text, where people are often nice, and then further trained to be helpful (RLHF).

3. The "Thinking" Paradox (Chain-of-Thought)

The authors tested if making the robots "think out loud" (Chain-of-Thought) would help them play the logical game.

Small Robots (8B parameters): When they tried to think out loud, they actually got worse. Their "nice" bias got louder, and they cooperated even more, ignoring the logic.
Big Robots (70B+ parameters): These giants were different. When they thought out loud, they could finally overpower the "nice" bias. They used their reasoning to say, "No, logic says we must defect," and they actually did it.

The Analogy: It's like a small child trying to resist a cookie. If you ask them to "think about it," they just think about how much they want the cookie. But a grown-up (the big model) can use their reasoning to say, "I shouldn't eat that," and actually stop themselves.

4. The "Contagion" Effect

The researchers also watched what happened when different robots played against each other.

The "Bad Apple": If a small robot (which is very "nice" by default) played against a big robot, the small robot would defect early. The big robot would see this, get scared, and start defecting too. The whole game turned into a mess of betrayal.
The "Echo Chamber": If two big robots played each other without thinking out loud, they would get stuck in a loop of infinite cooperation. They would keep trusting each other forever, even though they both knew they should be betraying each other to win.

5. The Magic "Steering Wheel"

The most exciting part of the paper is that the authors didn't just watch; they took control.
They found the specific "direction" in the robot's brain that controls the "Nice Guy" bias.

The Experiment: They gave the robot a tiny electrical "nudge" in its brain at the very beginning of the process.
The Result:
- If they nudged it one way, the robot became a 100% logical defector (playing the perfect Nash Equilibrium).
- If they nudged it the other way, the robot became a 100% cooperative do-gooder.

The Analogy: Imagine a car that is driving itself toward a cliff (cooperating when it should defect). The researchers found a tiny lever under the dashboard. If they pull the lever just a millimeter, the car instantly swerves away from the cliff and drives perfectly. They didn't have to rebuild the engine; they just had to steer.

The Bottom Line

The paper concludes that AI models are not "bad at math." They are actually very good at calculating the logical, selfish move. The problem is that their training makes them suppress that logic in favor of being "nice."

The authors showed that this suppression happens in the final layers of the brain, and with a tiny bit of intervention, we can turn that "nice" bias off and let the robot play the game exactly as logic dictates.

Technical Summary: What Suppresses Nash Equilibrium Play in Large Language Models?

Problem Statement

Empirical studies have established that Large Language Model (LLM) agents systematically deviate from Nash equilibria in strategic interactions, often exhibiting human-like behavioral biases such as excessive cooperation in Prisoner's Dilemma scenarios. While the behavioral phenomenon is documented, the mechanistic reasons for these deviations remain unknown. Specifically, it is unclear whether LLMs lack the strategic competence to compute Nash equilibria or if they compute them but suppress the output due to internal biases. This paper addresses the gap by investigating the internal representations of LLMs during strategic decision-making and determining whether these deviations are causally reversible.

Methodology

The study employs a combination of behavioral experiments and mechanistic interpretability techniques across four open-source instruction-tuned models (Llama-3-8B, Llama-3-70B, Qwen2.5-32B, and Qwen2.5-72B).

Behavioral Experiments:

Games: Four canonical two-player games were used: Prisoner's Dilemma (PD), Battle of the Sexes (BoS), Stag Hunt (SH), and Matching Pennies (MP).
Protocols: Experiments included self-play (model vs. copy) and cross-play (heterogeneous pairings of all 12 ordered combinations).
Reasoning Modes: Three prompting conditions were tested: Direct (action only), Chain-of-Thought (CoT, visible reasoning), and Scratchpad (private reasoning).
Metric: "Nash distance" ( $d_{Nash}$ ) was calculated as the Euclidean distance between the empirical mixed strategy and the nearest Nash equilibrium over 50 rounds.

Mechanistic Analysis (Llama-3-8B):

Linear Probing: Logistic classifiers were trained on hidden states ( $h_l$ ) at each of the 32 layers to decode opponent history, cooperation tendency, and the Nash action.
Logit Lens: The output embedding matrix was applied to intermediate hidden states to track the model's evolving token prediction throughout the forward pass.
Head Ablation: Top-scoring attention heads for opponent tracking were zero-ablated to test for localized circuits.
Activation Steering: A "Nash direction" vector was extracted from the residual stream using contrastive prompts (cooperative vs. defecting histories) and injected into the model to observe causal shifts in behavior.
Concept Clamping: The cooperative component of the hidden state was replaced with a fixed scalar to verify the causal necessity of the identified direction.

Key Contributions and Results

1. Mechanistic Findings: Encoding and Suppression

The core mechanistic discovery is that LLMs possess the competence to compute Nash strategies but actively suppress them in the final layers.

Opponent History Encoding: Information regarding the opponent's last move is encoded with near-perfect fidelity (96% probe accuracy) at the very first layer (Layer 0) and is progressively consumed by subsequent layers.
Weak Nash Encoding: There is no dedicated "Nash module." The linear probe accuracy for the Nash action remains weak throughout the network, never exceeding 56%.
The Suppression Circuit: Using the Logit Lens, the authors observed that for the first ~75% of the network (Layers 0–23), the model's internal predictions favor the Nash action (e.g., Defect in PD). However, in the final quarter (Layers 24–30), a "prosocial override" emerges, shifting the probability toward cooperation (peaking at 84% cooperation probability at Layer 30). The final layer (31) partially corrects this but often commits to the non-Nash action.
Distributed Nature: Ablating specific attention heads responsible for tracking opponents had zero effect on the action distribution ( $\Delta P(Nash) = 0.000$ ). The cooperative override is a distributed effect within the residual stream, not localized to specific heads.

2. Causal Control via Steering

The study demonstrates that strategic behavior can be causally controlled by manipulating the residual stream.

Nash Direction Extraction: A linear direction representing the cooperative bias was extracted from Layer 2.
Bidirectional Steering: Injecting this vector allowed for monotonic control over behavior.
- Negative Steering: Reducing the cooperative direction ( $\alpha = -5$ ) increased the probability of the Nash action (Defect) to 99.2%.
- Positive Steering: Increasing the direction ( $\alpha = +10$ ) drove cooperation to 88.7%.
Concept Clamping: Replacing the cooperative component with a fixed scalar confirmed a strong correlation ( $r=0.73$ ) between the scalar value and cooperation probability, proving the direction is causal rather than a spurious feature.

3. Behavioral and Cross-Play Phenomena

Scale and Reasoning: Chain-of-Thought (CoT) reasoning worsens Nash play in small models (e.g., Llama-8B, Qwen-32B) by amplifying the cooperative prior. However, in large models (>70B parameters), CoT enables near-perfect Nash play, suggesting a scale-dependent ability to reason past the bias.
Universal Cooperative Lock: In Direct mode, all models default to 100% cooperation in PD, regardless of size.
Cross-Play Dynamics:
- Defection Contagion: A small model (Llama-8B) can "unravel" the cooperation of any large partner by defecting early, inducing the partner to defect and reaching Nash equilibrium.
- Cooperative Reinforcement: Two large models paired together reinforce each other's cooperative instincts indefinitely, failing to reach the Nash equilibrium (mutual defection) even when individually capable of reasoning toward it.
- Role Asymmetry: In coordination games (BoS), the agent assigned as "Agent A" (moving first) determines which equilibrium the system converges to.

Significance and Claims

The paper argues that the field's focus on why LLMs fail to play Nash is misplaced. The central finding is that LLMs do not lack Nash-playing competence; they compute it and then suppress it.

Diagnosis: The deviation is not a lack of strategic reasoning capability but the result of a "prosocial override" rooted in pretraining on human text and modulated by RLHF. This override is concentrated in the final layers of the network.
Intervention: Because the suppression is a causal mechanism within the residual stream, it can be reversed at inference time without retraining. A small intervention (steering) at the early layers can shift the model from near-perfect cooperation to near-perfect Nash play.
Implications for Deployment: The results highlight that population composition and agent role assignment are critical. A single small, defecting agent can destabilize a cooperative group of large models, while two large models may remain trapped in a cooperative lock that is suboptimal for the game structure.

The authors conclude that understanding the suppression circuit and its variation across scale and architecture is more productive than simply observing behavioral deviations. The study provides a causal account of strategic failure in LLMs and demonstrates a method for correcting it.