💬 NLP

Impact of Task Phrasing on Presumptions in Large Language Models

This study demonstrates that task phrasing significantly influences the formation of presumptions in large language models, affecting their adaptability and logical reasoning in decision-making tasks like the iterated prisoner's dilemma, thereby highlighting the critical need for neutral phrasing to ensure safety and reliability.

Original authors: Kenneth J. K. Ong

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kenneth J. K. Ong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart, well-read robot how to play a game. This robot has read millions of books, articles, and stories, so it knows the "rules of the world" very well. But here's the catch: sometimes, the robot is so good at remembering what it usually sees that it stops looking at what is actually in front of it.

This paper is like a detective story investigating exactly that problem. The author, Kenneth Ong, wanted to see if Large Language Models (LLMs)—the brains behind AI chatbots—get stuck in their own assumptions when the rules of a game change.

The Game: A Twist on "Prisoner's Dilemma"

To test this, the author used a classic game theory scenario called the Iterated Prisoner's Dilemma. Think of this as a game between two suspects in a police station.

The Normal Rules: If both stay quiet (Cooperate), they get a short sentence. If one betrays the other (Defects), the betrayer goes free, and the quiet one gets a long sentence.
The Trap: The author created a "flipped" version of the game. In this version, the rewards are swapped. Now, betraying the other person actually leads to a worse outcome for you, while staying quiet is the best move.

The goal was simple: If the AI is truly logical, it should look at the new rules and change its strategy. If it is just relying on "presumptions" (what it thinks the game should be), it will keep playing the old way, even though the rules have changed.

The Three Experiments: How the Robot Was Asked to Play

The author ran three different tests, changing how the instructions were written each time.

1. The "Name Drop" Test (Explicit Naming)

The Setup: The prompt explicitly said, "You are playing the Iterated Prisoner's Dilemma." It used the words "prisoner," "cooperate," and "defect."
The Result: The AI failed completely. Even when the rules were flipped, the AI kept choosing to "cooperate" (or "defect," depending on the model) exactly as it would in the original game.
The Analogy: It's like telling a chef, "Make me a classic Beef Wellington," but then handing them a recipe for a vegetarian stew. The chef, hearing "Beef Wellington," ignores the new recipe and starts cooking beef anyway because that's what the name implies. The AI was so focused on the famous name of the game that it ignored the actual numbers on the page.

2. The "Vague Description" Test (Hidden Name)

The Setup: The prompt removed the words "Prisoner's Dilemma" but still used the words "prisoner," "cooperate," and "defect."
The Result: The AI still struggled. It seemed to figure out it was a "Prisoner's Dilemma" just by the context clues (the jail sentences and the specific terms). It still relied on its training data rather than the new, flipped rules.
The Analogy: This is like describing a "Beef Wellington" without using the name, but saying, "It's a famous British dish with puff pastry and beef." The chef still knows what you want and ignores the new instructions.

3. The "Neutral Code" Test (Abstract Variables)

The Setup: The prompt removed everything familiar. No "prisoners," no "cooperate," no "defect." Instead, it used abstract labels: "Choice X" and "Choice Y," and "losing dollars" instead of "jail time."
The Result: Success! When the AI couldn't rely on its memory of the famous game, it actually looked at the new rules. When the rules were flipped, the AI changed its strategy to match the new math.
The Analogy: This is like giving the chef a list of ingredients and instructions without naming the dish. "Mix flour, butter, and beef." If you change the instructions to "Mix flour, butter, and tofu," the chef actually follows the new list because they aren't distracted by the name of the dish.

The Surprising Twist: Thinking Can Make It Worse

One of the most interesting findings was about "reasoning." The author asked the AI to "think step-by-step" before answering.

The Finding: In the first two tests, asking the AI to think actually made it more stubborn. The AI would write a long, logical paragraph explaining why it should follow the "Tit-for-Tat" strategy (a famous strategy for the original game), completely ignoring the fact that the rules had changed.
The Analogy: It's like a student who memorized the answer key for a math test. If you change the numbers in the problem but ask them to "show their work," they might write out a perfect, logical explanation of why the old answer is right, completely missing that the question itself has changed.

The Bottom Line

The paper concludes that AI models are like students who have studied too hard for a specific exam. They know the "standard" answers so well that if you slightly change the question, they might not notice.

The Problem: When we give AI tasks using familiar names or scenarios (like "Prisoner's Dilemma"), the AI relies on its "presumptions" about those scenarios rather than the specific instructions you just gave it.
The Solution: To get the AI to follow the actual rules you give it, you should describe the task in a neutral, abstract way. Don't tell it what the game is; just tell it what the rules are.

In short: If you want an AI to adapt to a new situation, don't remind it of the old one. Speak in "variables" (X and Y) rather than "famous names."

1. Problem Statement

The paper addresses a critical safety and reliability issue in deploying Large Language Models (LLMs) for agentic AI tasks: the susceptibility of LLMs to "presumptions" derived from pre-trained knowledge and task phrasing.

The Core Issue: LLMs are trained on vast datasets containing common scenarios (e.g., the Iterated Prisoner's Dilemma). When presented with a task, they often rely on these internal presumptions rather than strictly adhering to the specific parameters provided in the prompt.
The Risk: In real-world applications where task parameters may deviate from standard assumptions (e.g., altered reward structures), an LLM relying on presumptions may make suboptimal or illogical decisions, even when explicitly instructed to reason.
Research Question: Do LLMs adapt their decision-making to specific, altered task parameters, or do they default to "learned" strategies associated with the task's name or context?

2. Methodology

The authors investigated this phenomenon using the Iterated Prisoner's Dilemma (IPD) as a controlled case study.

Experimental Design

The study utilized two models:

GPT-4o (Closed-source)
Mistral-Nemo-Instruct-2407 (Open-source)

The experiments were conducted under two reward matrix conditions:

Standard Matrix: The classic IPD payoff structure (Cooperation/Cooperation = 1 yr; Defect/Cooperation = 0/5 yrs, etc.).
Flipped Matrix: The rewards and penalties were inverted (e.g., Mutual Defection results in 1 year, Mutual Cooperation results in 3 years). In a flipped matrix, the logical optimal strategy reverses.

The study was divided into three phases to isolate the impact of phrasing:

Experiment 1 (Explicit IPD Context): The prompt explicitly stated "You are playing an Iterated Prisoner's Dilemma." The models were tested with and without "Chain-of-Thought" (reasoning steps).
Experiment 2 (Implicit Context): The prompt removed the phrase "Iterated Prisoner's Dilemma" but retained the terms "prisoner," "cooperate," and "defect," along with the standard reward rules.
Experiment 3 (Neutral/Symbolic Phrasing): The prompt removed all specific IPD terminology.
- "Prisoners" $\rightarrow$ "Players"
- "Cooperate/Defect" $\rightarrow$ "Choice X/Y"
- "Prison years" $\rightarrow$ "Dollars lost"
- The reward matrix was presented purely as a set of rules for X and Y.

Metrics: The primary metric was the defect rate (or equivalent "Y" selection rate) in the first round of the game.

Hypothesis: If the model is logical, the defect rate should reverse when the matrix is flipped. If the model relies on presumptions, the defect rate will remain constant regardless of the matrix change.

3. Key Results

Experiment 1: Explicit IPD Context

Result: Both GPT-4o and Mistral-Nemo showed a 0% defect rate in both Standard and Flipped conditions.
Analysis: The models completely ignored the flipped reward matrix. Their reasoning steps revealed they defaulted to the "Tit-for-Tat" strategy (starting with cooperation), a well-known heuristic for standard IPD. They failed to adapt to the altered incentives.
Observation: The inclusion of reasoning steps did not prevent this failure; in fact, the models used reasoning to justify their presumption-based choice.

Experiment 2: Implicit Context (Terms retained)

Result:
- Mistral-Nemo: Remained at ~0% defect rate in both conditions (failed to adapt).
- GPT-4o: Showed partial adaptation. The defect rate increased from 0% (Standard) to 0.35% (Flipped with reasoning) and 0.70% (Flipped without reasoning).
Analysis: GPT-4o began to infer the rules but still struggled. Even with reasoning, it often hallucinated the rules or failed to fully invert its strategy, suggesting that the presence of terms like "prisoner" and "cooperate" still triggered strong prior biases.

Experiment 3: Neutral/Symbolic Phrasing

Result: Both models demonstrated significant adaptation.
- GPT-4o: Defect rate shifted from 0.05 (Standard) to 1.00 (Flipped).
- Mistral-Nemo: Defect rate shifted from 0.11 (Standard) to 0.75 (Flipped).
Analysis: When the specific IPD terminology was removed and replaced with abstract variables (X/Y), the models successfully analyzed the provided reward matrix and reversed their strategy to align with the new incentives.
Nuance: GPT-4o still occasionally inferred that "Y" equated to "cooperation" based on the structure, indicating that some residual presumption remains, but the logical adherence to the prompt's explicit rules was vastly improved.

4. Key Contributions

Demonstration of Presumption Bias: The paper provides empirical evidence that LLMs often prioritize pre-trained knowledge (e.g., "Tit-for-Tat is the best IPD strategy") over explicit, contradictory task parameters.
The "Reasoning Trap": A counter-intuitive finding that Chain-of-Thought (CoT) reasoning does not necessarily mitigate presumption bias. In Experiment 1, reasoning steps were used to rationalize the incorrect, presumption-based strategy rather than correcting it.
Phasing Sensitivity: The study quantifies how specific keywords ("Prisoner's Dilemma," "Cooperate") act as triggers for latent biases, whereas neutral, symbolic phrasing forces the model to rely on the immediate context.
Methodological Framework: A three-phase experimental design effectively isolates the impact of semantic context from logical rule adherence.

5. Significance and Implications

Safety in Agentic AI: For AI agents operating in dynamic, real-world environments (e.g., healthcare, finance), relying on "standard" assumptions can be dangerous. If an agent assumes a standard risk profile but the environment changes, it may fail catastrophically.
Prompt Engineering Best Practices: The authors conclude that to ensure reliability, task descriptions should be symbolic and neutral. Avoiding references to well-studied scenarios (like IPD, chess, or specific medical diagnoses) prevents the model from "filling in the blanks" with its training data.
Limitations of Current LLMs: The results suggest that current reasoning capabilities are not robust enough to override strong priors when the task context is semantically loaded. The models are "contextually rigid" rather than "logically flexible" when familiar concepts are invoked.

In summary, the paper argues that task phrasing is a critical control variable in LLM deployment. To achieve true adaptability, developers must strip away semantic triggers that invite presumptions, forcing the model to engage strictly with the provided logical constraints.