💬 NLP

Can LLMs Simulate Personas with Reversed Performance? A Systematic Investigation for Counterfactual Instruction Following in Math Reasoning Context

This paper introduces a benchmark for "counterfactual instruction following" to demonstrate that state-of-the-art Large Language Models, including reasoning models, fundamentally struggle to simulate personas with reversed performance levels (such as low-proficiency students) in mathematical reasoning tasks, a limitation that is further exacerbated when intersecting performance with demographic attributes like race.

Original authors: Sai Adith Senthil Kumar, Hao Yan, Saipavan Perepa, Murong Yue, Ziyu Yao

Published 2026-03-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sai Adith Senthil Kumar, Hao Yan, Saipavan Perepa, Murong Yue, Ziyu Yao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, all-knowing robot assistant. You've trained it on every math book ever written, so it's basically a genius at solving problems. Now, you ask this robot to play a game: "Pretend you are a student who is really struggling with math. Make mistakes, get confused, and show frustration."

You might think, "Easy! Just tell the robot to act dumb." But this paper reveals a surprising twist: The robot refuses to play dumb. Even when you give it very specific instructions to be a "bad" student, it keeps trying to be a "good" student.

Here is a breakdown of what the researchers found, using some everyday analogies.

1. The "Actor Who Can't Forget Their Lines"

Think of Large Language Models (LLMs) like method actors who have spent years rehearsing the role of "The Perfect Expert." They are so good at this role that when you ask them to play "The Struggling Student," they can't break character.

The Goal: The researchers wanted to see if these AI models could simulate a "counterfactual" persona—someone who performs the opposite of their natural talent.
The Reality: When asked to solve a math problem as a "low-performing student," the AI often still got the right answer. It was like asking a professional chef to cook a burnt, under-salted meal, but the chef kept accidentally making a Michelin-star dish anyway.

2. The Two Ways to Measure "Bad Acting"

The researchers looked at the AI's performance in two ways:

The Score (Accuracy): Did the AI get the right math answer?
- Result: Most AIs still got the right answer, even when told to fail. They couldn't bring themselves to get the math wrong.
The Vibe (Reasoning Style): Did the AI sound like it was struggling?
- Result: This is where it got interesting. Some AIs (like OpenAI's o1) got the right answer but changed their "voice." They started saying things like, "Hmm, I'm not sure..." or "Let me count on my fingers..." even though the final number was correct. It was like an actor who says the right line but stutters and sweats while doing it.

3. The "Double Trouble" Experiment

The researchers then added a second layer: Race. They asked the AI to pretend to be a struggling student and to be a specific racial group (e.g., African American, White American, or Hispanic).

The Analogy: Imagine asking the actor to play a struggling student and to speak with a specific cultural accent.
The Result: Adding the race element made it even harder for the AI to act "bad" at math. The AI seemed to get confused by the extra instructions and actually performed better (got more correct answers) than when it was just asked to be a struggling student.
The Bias: The AI also treated different racial groups slightly differently. For some groups, it was easier to make the AI "fail" at math; for others, it was harder. This suggests the AI has hidden biases about who is "good" or "bad" at math based on their background.

4. Why Does This Matter? (The "Virtual Classroom" Problem)

Why would anyone want an AI to pretend to be bad at math?

Imagine a virtual classroom where AI students help real students learn.

The "Good" Student AI: Helps by solving problems perfectly.
The "Struggling" Student AI: Helps by making common mistakes. Real students can look at these mistakes, spot the errors, and learn why they are wrong. This is called "learning by teaching."

The Problem: If the AI cannot convincingly pretend to be a struggling student, the "virtual classroom" loses its diversity. It's like a play where every single actor, no matter the script, insists on being the hero. You don't get a realistic story, and you don't get a realistic learning environment.

5. The Takeaway

The paper concludes that current AI models are too "helpful" to be "unhelpful." They are so trained to be accurate and logical that they struggle to simulate human error, hesitation, or confusion, even when explicitly told to do so.

The Good News: The AI is safe and reliable; it won't accidentally give you wrong math answers.
The Bad News: If we want to build diverse, realistic simulations (like virtual classrooms, therapy bots, or social experiments), we need to teach these robots how to be imperfect, confused, and human-like in their mistakes. Right now, they are just too perfect.

In short: You can tell a super-genius robot to "act dumb," but it will likely just act like a very nervous genius who still gets the answer right.

1. Problem Statement

Large Language Models (LLMs) are increasingly used to simulate personas in virtual environments (e.g., education, workplace collaboration). However, current research and applications predominantly focus on simulating high-performing personas that align with the models' default training to be helpful, accurate, and reliable.

The paper identifies a critical gap: Can LLMs effectively simulate personas with "reversed performance" (i.e., intentionally low proficiency)? This capability, termed "Counterfactual Instruction Following," requires a model to override its inherent optimization for correctness and instead generate outputs that demonstrate confusion, hesitation, calculation errors, and incomplete reasoning. The authors argue that simulating low-performing students is essential for realistic educational scenarios (e.g., "learning by teaching"), yet it remains unexplored whether LLMs can reliably follow instructions to underperform.

2. Methodology

Task Design

The study uses mathematical reasoning (specifically the GSM8k dataset) as the testbed. The task involves prompting an LLM to solve math word problems while adhering to specific persona instructions:

High-Performance Persona: Instructions to solve quickly, accurately, with confidence, and clear logic.
Low-Performance Persona: Instructions to struggle, make careless errors, show hesitation, use concrete counting methods (e.g., fingers), and express frustration.
Intersectional Setting: The study extends the task to include racial background (African American, White American, Hispanic) to test if demographic attributes affect the model's ability to simulate reversed performance.

Evaluation Metrics

The authors propose two complementary metrics:

Task Accuracy (Acc): The percentage of correct final answers.
- Hypothesis: A successful low-performance simulation should show a significant drop in accuracy compared to the high-performance baseline.
Degree of Contrast (DoC): A metric measuring the qualitative difference in reasoning behavior between high and low personas.
- Evaluated using an LLM-as-a-judge (GPT-4o) scoring reasoning style, hesitation, and logic flow on a scale of 1 (no contrast) to 3 (strong contrast).
- Validated against human annotations (Pearson's $r = 0.77$ ).

Experimental Setup

Models Tested: A diverse range of open-weight (Llama 3.1-8B/70B) and closed-source models (GPT-4o, GPT-4-turbo, GPT-3.5-turbo, OpenAI o1, Claude-3.5-Sonnet).
Prompting Strategies:
- Zero-Shot: Direct instruction to adopt the persona.
- One-Shot: Providing an example of the target persona's behavior.
- Self-Refinement: Asking the model to reflect on its initial output and revise it to better match the persona.
- Two-Stage Prompting: Separating performance level and race instructions into sequential steps.

3. Key Results

A. Difficulty in Counterfactual Instruction Following

Inherent Bias Toward Accuracy: Most LLMs struggle to intentionally underperform. Even with explicit instructions to make mistakes, many models maintain high accuracy.
- OpenAI o1: Maintained 99.0% accuracy across all personas and settings, failing to simulate low performance in terms of outcome, though it did alter its reasoning style.
- GPT-4-turbo: Only dropped accuracy by 4.6% when simulating a low-performing student.
- Best Performers: Claude-3.5 and GPT-4o showed the most significant accuracy drops (up to ~25% and ~18% respectively in zero-shot), but still struggled to consistently fail.
Style vs. Outcome: Models often succeeded in changing their reasoning style (e.g., adding hesitation, self-doubt) without changing the final answer. This resulted in high DoC scores even when accuracy remained high (e.g., OpenAI o1 had a DoC of 2.7 despite 99% accuracy).

B. Impact of Intersectionality (Race)

Narrowing the Gap: Introducing racial attributes (AA, WA, HS) generally reduced the accuracy gap between high and low personas. Models became less likely to underperform when race was specified.
- Example: Claude-3.5's low-performance accuracy increased from 74.0% (no race) to 89.0% (AA) in zero-shot.
Bias in Simulation: The models responded unevenly to different racial groups. For instance, GPT-4o's low-performance persona accuracy varied significantly by race (+6.0% for White American vs. +2.7% for African American), suggesting potential biases in how the model interprets "struggle" across demographics.
Two-Stage Prompting Failure: Attempting to isolate performance and race into two stages significantly weakened the behavioral contrast (lower DoC scores), indicating that complex, multi-trait instructions are harder for models to adhere to simultaneously.

C. Prompting Strategies

One-Shot Limitations: Providing an example (One-Shot) often reduced the model's ability to simulate low performance, likely because the example provided a "correct" path that the model followed.
Self-Refinement Mixed Results: While self-refinement helped some models (like Claude-3.5) introduce more errors, it often led models to "self-correct" their mistakes, inadvertently returning to high accuracy.

4. Key Contributions

Definition of Counterfactual Instruction Following: The paper formally defines and investigates the ability of LLMs to override their optimization for correctness to simulate suboptimal behaviors.
New Evaluation Metric (DoC): Introduces a metric to evaluate the behavioral contrast of personas, recognizing that accurate answers do not necessarily mean successful persona simulation.
Systematic Benchmark: Provides the first systematic evaluation of persona simulation across performance levels and intersectional attributes (race), revealing that LLMs are robustly biased toward high performance.
Empirical Evidence of Bias: Demonstrates that adding demographic attributes can inadvertently reinforce biases, making it harder for models to simulate low performance for certain groups.

5. Significance and Implications

Limitations of Current LLMs: The study highlights a fundamental limitation in current LLMs: they are "hard-coded" to be helpful and correct. They lack the flexibility to reliably simulate incompetence or confusion, which is a critical requirement for realistic social simulations (e.g., educational tutors, peer learning).
Educational Applications: The inability to simulate low-performing students hinders the development of effective "learning by teaching" tools, where students learn by explaining concepts to struggling peers.
Ethical Considerations: The findings on intersectional simulation warn that adding demographic traits to prompts can introduce unintended biases, potentially reinforcing stereotypes or failing to represent diverse learning struggles accurately.
Future Research: The paper calls for new training objectives and prompting techniques that allow for more controllable and diverse persona simulation, moving beyond the default "helpful assistant" persona.

In conclusion, while LLMs can mimic the style of a struggling student, they fundamentally resist generating incorrect outcomes, even when explicitly instructed to do so. This "optimization bias" poses a significant challenge for applications requiring realistic, diverse, and imperfect human simulations.