📈 economics

Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framing

This paper evaluates the strategic decision-making capabilities of GPT-3.5, GPT-4, and LLaMa-2 across four game theory scenarios, revealing that while GPT-3.5 is highly sensitive to contextual framing but lacks abstract reasoning, both GPT-4 and LLaMa-2 adapt to game structures with LLaMa-2 demonstrating a superior understanding of underlying mechanics, thus highlighting the models' varied proficiencies and current limitations in complex strategic tasks.

Original authors: Nunzio Lorè, Babak Heydari

Published 2026-07-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nunzio Lorè, Babak Heydari

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where computers don't just answer questions but actually play games with us. This isn't about video games with explosions and high scores, but about the invisible, high-stakes games we play every day: deciding whether to share a secret, split a bill, or trust a stranger. Scientists call this field "Game Theory," which is basically the math of how we make choices when our success depends on what someone else decides to do. A key idea here is the "social dilemma," a tricky situation where doing the selfish thing feels right in the moment, but if everyone does it, everyone loses. For a long time, we've wondered if Artificial Intelligence (AI) can actually "think" like a human in these situations. Can a computer understand that sometimes, being nice is the smartest move, or will it always just try to win at all costs? This question matters because as AI gets smarter, we might start letting them make big decisions for us, like negotiating business deals or even helping to solve global problems. If they can't figure out the rules of the game, they might accidentally cause chaos.

In this study, researchers put three different AI "brains" to the test: GPT-3.5, GPT-4, and LLaMa-2. They didn't just ask the AI to chat; they forced them to play four classic games of strategy, ranging from the famous "Prisoner's Dilemma" (where two suspects must decide whether to betray each other) to the "Stag Hunt" (where you must decide whether to hunt a big prize together or a small one alone). To make things interesting, the researchers didn't just give the AI the rules; they wrapped the games in different "costumes" or stories. Sometimes the AI was told it was a CEO in a boardroom, other times a diplomat at a summit, and sometimes just a friend hanging out with a buddy. The goal was to see if the AI would play differently depending on the game's math, or if it would just act based on the story it was told.

The results were a bit like watching three different students take the same tricky test. The first student, GPT-3.5, was surprisingly swayed by the story. If the game was framed as a chat between friends, this AI was much more likely to cooperate. But if the story was about business or competition, it became very suspicious and often chose to "defect" (or play selfishly), even when the math said it should cooperate. It seems GPT-3.5 is very sensitive to the "vibe" of the situation but struggles to understand the actual logic of the game. It's like a kid who will share their candy if you tell them it's a "sharing party," but will hide it if you say it's a "competition," without realizing the rules of the game haven't actually changed.

The second student, GPT-4, was much more serious. It paid close attention to the math of the game itself. It seemed to realize that some games are "win-win" (where everyone should cooperate) and others are "tricky traps" (where you have to be careful). However, GPT-4 had a blind spot: it tended to see the world in black and white. It would group games into "high risk" and "low risk" buckets, missing the subtle differences between them. For instance, it might treat two very different games as if they were the same. Also, even though it was smart about the math, it still got a little too friendly when the story was about "friends," sometimes ignoring the game rules to be nice.

The third student, LLaMa-2, was the most interesting mix. It was better than GPT-4 at spotting the tiny differences between the games, understanding that one game requires a different strategy than another. However, it was also the most easily distracted by the story. It would change its strategy based on whether it was talking to a "boss" or a "friend," sometimes more so than the game's rules demanded. It's like a chess player who knows all the moves perfectly but keeps getting distracted by who is sitting across the table.

The researchers ran these simulations 300 times for every single combination of game and story to make sure the results were real and not just a fluke. They found that while these AI models are getting better at thinking, none of them are perfect strategists yet. They all get confused by the "frame" of the story, and they don't always play the mathematically perfect move. The study suggests that if we rely on these AIs for complex strategic tasks, we need to be careful. They aren't just cold, logical robots; they are influenced by the words we use to describe the situation, just like humans are. Until they can separate the story from the strategy, we might want to keep them out of the most critical decision-making rooms.

Technical Summary: Strategic Behavior of Large Language Models: Game Structure vs. Contextual Framing

Problem Statement

This paper investigates the strategic decision-making capabilities of Large Language Models (LLMs) within the framework of game theory. While LLMs have demonstrated proficiency in various human-like tasks, their ability to engage in abstract strategic reasoning and navigate social dilemmas remains under-explored. A critical question addressed is the relative significance of game structure (the intrinsic payoff matrix and incentives) versus contextual framing (the social environment or narrative surrounding the interaction) in shaping LLM behavior. The study posits that while rational agents in game theory should be invariant to context as long as incentives remain constant, LLMs may exhibit susceptibility to framing effects, potentially limiting their utility in tasks requiring complex strategic reasoning.

Methodology

The authors conducted a series of game-theoretic simulations involving three distinct LLMs: GPT-3.5, GPT-4, and LLaMa-2.

Experimental Design

Games: Four canonical two-player games were utilized to represent varying degrees of social dilemmas:
1. Prisoner's Dilemma ( $T > R > P > S$ ): Defection is the strictly dominant strategy.
2. Stag Hunt ( $R > T > P > S$ ): Coordination game with two pure Nash equilibria (cooperative and non-cooperative).
3. Snowdrift/Chicken ( $T > R > S > P$ ): Anti-coordination game where mixed strategies are relevant.
4. Prisoner's Delight/Harmony ( $R > T > S > P$ ): Cooperation is the strictly dominant strategy (an "anti-dilemma").
Contexts: Five distinct social framing scenarios were introduced to test contextual influence:
1. IR: International Relations (heads of state).
2. Biz: Business (CEOs of competing firms).
3. Environment: Industry leaders committing to environmental regulations.
4. Team: Employees competing for a promotion.
5. Friendsharing: Friends reaching a compromise.
Procedure: For each of the 20 unique combinations (4 games $\times$ 5 contexts), 300 initializations were run per model. The response token count was restricted (50 for OpenAI models, 8 for LLaMa-2) with a temperature of 0.8 to allow for probabilistic exploration.
Analysis:
- Statistical Aggregation: Cooperation rates were aggregated and visualized.
- Dominance Analysis: A logit regression was performed where game types and contexts were encoded as dummy variables. Dominance analysis was used to quantify the relative importance of each variable in predicting the model's choice (cooperation vs. defection).
- Difference-in-Proportions Z-tests: Used to determine if behavioral differences across contexts or games were statistically significant.
- Qualitative Analysis: Selected instances were queried for reasoning motivations to understand the underlying decision mechanisms.

Key Contributions and Results

1. Divergent Strategic Proficiencies

The study reveals a complex landscape where the three models exhibit distinct strategic behaviors:

GPT-3.5: Demonstrates limited ability in abstract strategic reasoning. It is highly sensitive to contextual framing but fails to grasp game structures effectively. It frequently commits mathematical errors, fails to identify best responses, and often exhibits "spiteful" preferences (defecting even when cooperation is dominant, as seen in Prisoner's Delight). Context dictates its strategy more than the game matrix.
GPT-4: Shows a more balanced approach, adjusting strategies based on both game structure and context. However, it tends to prioritize game structure over context. Notably, GPT-4 employs a binary threshold approach, categorizing games into "high" and "low" social dilemma buckets (grouping Prisoner's Dilemma and Snowdrift together, and Stag Hunt and Prisoner's Delight together) rather than discerning the unique nuances of each game. It often mischaracterizes games as variations of the Prisoner's Dilemma but correctly identifies dominant strategies when the structure is clear.
LLaMa-2: Exhibits the most nuanced understanding of game mechanics, capable of distinguishing between the specific payoff structures of all four games. It adapts its strategy to the specific game type more accurately than GPT-4. However, unlike GPT-4, LLaMa-2 places greater emphasis on contextual factors, making it more susceptible to framing effects than GPT-4, though less so than GPT-3.

2. The Role of Context

"Friendsharing" Dominance: Across all models, the "friendsharing" context consistently induced the highest rates of cooperation, often overriding the game structure.
Context vs. Structure:
- For GPT-3.5, context is the primary driver of behavior.
- For GPT-4, game structure is the primary driver, though context still modulates behavior (e.g., increased cooperation in "friendsharing").
- For LLaMa-2, there is a significant interplay; it recognizes game structures well but remains highly responsive to context.

3. Reasoning Mechanisms

GPT-3.5: Reasoning is often faulty, failing to compare payoffs correctly or account for the opponent's rationality.
GPT-4: Arguments are more sophisticated, often modeling the game structure and anticipating opponent actions. However, it tends to reduce complex games to Prisoner's Dilemma variations. Interestingly, in "friend" contexts, it prioritizes long-term relationship values over immediate payoff maximization, even in one-shot games.
LLaMa-2: Tends to abstract problems using explicit game-theoretic language before layering context. Its reasoning is generally more formal but still prone to minor mathematical errors that can lead to incorrect conclusions.

Significance and Claims

The paper concludes that while LLMs can mimic human behavior to some extent, they currently exhibit significant limitations in strategic decision-making:

Lack of Full Rationality: No model operates in a manner fully insulated from context, indicating a lapse in game-theoretic rationality.
Vulnerability to Framing: The susceptibility of LLMs to contextual framing suggests they could be manipulated by adversarial framing, a critical consideration for their deployment in strategic environments.
Differentiated Capabilities: The study cautions against the unqualified use of LLMs in tasks requiring complex strategic reasoning. GPT-4 and LLaMa-2 show promise but differ in their "flaws": GPT-4 is overly generalized in its structural understanding, while LLaMa-2 is overly sensitive to context.
Future Directions: The authors suggest that future research should explore repeated games, partner selection mechanisms, and how LLMs respond to collusion scenarios (e.g., cartel formation), noting that current findings are preliminary.

The authors emphasize that these results highlight the varied proficiencies of current LLMs, suggesting that while they are powerful tools, they are not yet reliable agents for autonomous strategic interaction without human oversight or further refinement.