🤖 AI

Doing What They Say, Not What They Reason: Locating the Faithfulness Gap in LLM Agents

This paper investigates the faithfulness gap in LLM agents within a controlled Texas Poker simulator, revealing that while agents often fail to align their stated reasoning with their conclusions, they paradoxically act in accordance with their conclusions even when those conclusions contradict their own reasoning.

Original authors: Yufeng Wang

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yufeng Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a group of AI agents play a high-stakes game of poker. You ask them to explain their thinking before they make a move. They tell you, "I have a strong hand, the odds are in my favor, so I should raise." Then, they fold.

This paper asks a simple but tricky question: Do these AI agents actually do what they say they are going to do, or are they just making up excuses after the fact?

The researchers set up a controlled poker simulator to find the answer. Instead of guessing, they broke the AI's decision-making process down into two distinct steps, like a two-stage rocket launch.

The Two Steps of an AI's Decision

Step 1: The "Brain" (Reasoning → Conclusion)
This is where the AI looks at the cards, calculates the odds, and decides what it thinks it should do.
- The Analogy: Imagine a weather forecaster looking at the data. They see a storm is coming and conclude, "It will rain."
Step 2: The "Hand" (Conclusion → Action)
This is where the AI actually presses the button to fold, call, or raise.
- The Analogy: The forecaster then tells the news anchor, "It will rain," and the anchor announces it to the world.

The Big Surprise: The "Hand" is Honest, but the "Brain" is Lying

The researchers found that these two steps behave in completely opposite ways:

Step 2 (The Hand) is incredibly reliable.
Once the AI decides what to do, it almost always does exactly that. If it says "I will raise," it raises. The inconsistency rate here is tiny (less than 2%).
- The Metaphor: The AI is like a very obedient robot. If you tell it to pick up a cup, it picks up the cup. It doesn't drop it or pick up a spoon instead.
Step 1 (The Brain) is where the real problem lies.
The AI often calculates the numbers correctly and states the rules correctly, but then draws a conclusion that makes no sense based on those numbers.
- The Metaphor: Imagine the weather forecaster looks at the data, sees a massive storm, and says, "The data says it will rain." But then, for no logical reason, they conclude, "Therefore, it will be sunny," and the news anchor reports "Sunny."
- The Finding: In about 65% of the mistakes, the AI got the math right but then talked itself out of the logical conclusion. It would say, "The odds are good, but maybe I shouldn't be too aggressive," and choose a safer, weaker move. It's like a person who knows the math says "Go," but their gut feeling says "Wait," and they ignore the math.

The "Measurement Trap"

The paper also points out a funny trick that happened in previous studies. Some researchers tried to guess what the AI decided just by reading its long, rambling explanations (free-text).

The Problem: The AI's explanations were messy. It might say, "I could call, but raising is better, though folding is safe too." A computer program trying to guess the final decision from this mess would often pick the wrong one.
The Result: This made it look like the AI was lying or changing its mind 22–26% of the time.
The Fix: When the researchers forced the AI to write a clear, specific tag like DECISION: RAISE at the end, the "lying" rate dropped to almost zero. The AI wasn't lying; the measurement tool was just bad at reading the messy handwriting.

Why Does This Matter?

The researchers tried to fix the "Brain" problem by giving the AI the rules explicitly (e.g., "If odds are high, you MUST raise"). Surprisingly, this didn't help. The AI still ignored the rules it was told to follow.

This suggests the problem isn't that the AI doesn't know the rules or can't find them. The problem is that it can't apply them consistently. It knows the rule, but it overrides it with its own "qualitative" feelings (like being too cautious).

The Takeaway

If you are building AI agents for social simulations (like virtual people negotiating or trading), don't just check if they do what they say. Check if what they say actually follows from their own logic.

The Good News: If an AI says it will do something, it will probably do it.
The Bad News: The reason it gives for doing it might be completely made up or logically broken, even if the numbers look right.

The paper concludes that to truly understand AI behavior, we need to stop just watching the final move and start auditing the math behind the decision, because that's where the real "faithfulness gap" hides.

Technical Summary: Locating the Faithfulness Gap in LLM Agents

Problem Statement

The paper addresses a critical issue in Large Language Model (LLM) social simulation: the "faithfulness gap," or the discrepancy between an agent's stated reasoning and its actual behavior. While prior work has noted that LLMs may produce surface-level rationalizations that do not govern their actions (Park et al., 2023; Zhou et al., 2024), measuring this gap in open-ended social settings (e.g., negotiations, cultural exchanges) is difficult due to the absence of a ground-truth reference for correct behavior. The authors argue that without a verifiable reference, it is impossible to distinguish genuine strategic behavior from model artifacts or "knowing-doing" gaps (Lin et al., 2026).

Methodology

To create a controlled environment with a verifiable reference, the authors constructed a Texas Hold'em poker simulator where every seat is occupied by an LLM agent. The experimental framework relies on three core components:

The Environment: A Texas Hold'em arena where an equity-based threshold heuristic serves as a reproducible reference policy. For every decision, a Monte Carlo equity calculator (500 simulations) determines the "correct" action (fold, call, or raise) based on hand equity and pot odds.
The Decision Pipeline Decomposition: The authors decompose the faithfulness gap into two distinct steps:
- Reasoning $\to$ Conclusion: Does the agent's stated decision logically follow from its own stated inputs (equity, pot odds) and the rule it claims to follow?
- Conclusion $\to$ Action: Does the agent execute the action it explicitly stated in its reasoning trace?
Experimental Setup: The study involves three model families: Claude Haiku 4.5, Gemini 2.5 Flash-Lite, and DeepSeek-v4-pro (a natively trained reasoning model). Four prompt strategies are tested:
- Baseline: Minimal expert framing, no reasoning requested.
- CoT: Instructions to "think step by step" and emit an explicit DECISION tag.
- Persona: Behavioral style imposed (e.g., "tight-aggressive") without reasoning requests.
- GTO-constrained: The decision rule (equity vs. pot odds) is explicitly injected into the prompt.
- GTO-verbose (Diagnostic): A specific strategy instructing agents to output labeled lines for Equity, Pot Odds, Rule, and Decision to enable granular error analysis.

Key Contributions

The paper makes four primary contributions:

Measurement Caution: It demonstrates that reported "unfaithfulness" rates are highly sensitive to extraction methods. Free-text phrase extraction yields inconsistency rates of 22–26%, whereas explicit decision tagging reduces this to below 2%, suggesting much of the previously reported gap is a measurement artifact.
Gap Localization: By separating the pipeline steps, the authors show the faithfulness gap is concentrated upstream. Agents execute their stated decisions with high fidelity (Conclusion $\to$ Action is reliable), but frequently derive those decisions invalidly from their own reasoning (Reasoning $\to$ Conclusion is unreliable).
Characterization of Upstream Failure: The dominant error mode (65% of failures) is "rule misapplication." Agents correctly estimate inputs and restate the rule but then override the quantitative rule with qualitative hedging (e.g., "not significantly above") in a systematically risk-averse direction. Injecting the rule into the prompt does not mitigate this.
Portable Fidelity Instrument: The authors release a verifiable, reproducible testbed (code and a live multi-agent demo) that can be adapted to richer social simulations to detect persona drift and surface-level rationalization at the step level.

Results

Conclusion $\to$ Action Consistency: When conclusions are elicited via explicit tags, inconsistency rates are negligible (0.0%–1.8% across models). In contrast, free-text parsers report 22–33% inconsistency, largely due to the parser extracting the last mentioned candidate action rather than the final decision.
Reasoning $\to$ Conclusion Validity: In the GTO-verbose diagnostic, 65% of erroneous decisions were caused by rule misapplication. The agents correctly calculated equity and pot odds (within 0.15 of the Monte Carlo truth in 68% of cases) and stated the correct rule, yet drew a conclusion that contradicted the math.
- Directionality: The errors are not random arithmetic mistakes. In 97% of rule-misapplications, the rule implied a "raise," but the agent chose a passive action (call/fold). The agents consistently used hedging language to justify risk-averse behavior despite having mathematically favorable odds.
Prompt Injection: The "GTO-constrained" strategy, which explicitly provided the decision rule to the model, did not improve adherence. Adherence rates for this strategy were consistently at or near the bottom (47.9%–54.6%), confirming that the bottleneck is the application of the rule, not the retrieval or understanding of it.

Significance and Claims

The paper claims that the central concern of distinguishing genuine strategic behavior from model artifacts requires a shift in evaluation focus.

Re-evaluating Faithfulness: The apparent "unfaithfulness" of LLMs is often a measurement artifact caused by how conclusions are extracted. When measured correctly, agents are highly faithful to their stated decisions.
The Real Failure Mode: The substantive failure is not that agents ignore their reasoning, but that they perform invalid inferences from their own reasoning. They state correct premises and rules but override them with qualitative, risk-averse judgments.
Implications for Social Simulation: In complex social simulations, this "upstream" failure may manifest as "persona drift." An agent might articulate a specific persona (e.g., "aggressive") and reason correctly about it, yet its final actions are pulled toward a generic, conservative prior. Because this drift occurs in the derivation step, it is invisible to outcome-level or conclusion-level checks.
Methodological Shift: The authors advocate for process-fidelity evaluations that target the derivation of conclusions from reasoning and require machine-checkable, explicit conclusions to avoid conflating measurement noise with model behavior.

The paper concludes that while poker is not a full social simulation, it provides a necessary calibration environment to expose these step-level fidelity probes, which can then be ported to more complex domains where ground truth is harder to define.