🤖 AI

Representation Without Control: Testing the Realization Effect in Language Models

This paper demonstrates that while large language models exhibit prompt-sensitive behavioral shifts and possess linearly decodable internal representations of the "realization effect," the inability to causally steer these representations to alter risk-taking decisions reveals that latent readouts do not necessarily indicate genuine reliance on those representations for downstream decision-making.

Original authors: Ciarán Walsh, Emilio Barkett

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ciarán Walsh, Emilio Barkett

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, very chatty robot that can pretend to be a human. You ask it questions about money, gambling, and risk, hoping it will act like a person making real-life choices. But here's the big question: Is the robot actually "thinking" like a human, or is it just good at guessing what words to say based on how you asked the question?

This paper puts that robot to the test using a famous psychological trick called the "Realization Effect."

The Setup: The "Open vs. Closed" Wallet

In real life, humans behave differently depending on whether a win or loss is "open" (paper) or "closed" (realized).

Paper (Open): Imagine you win $50 on a scratch-off ticket but haven't cashed it in yet. You might feel like you're still "in the game" and take bigger risks to double your money.
Realized (Closed): Imagine you cashed that $50 in, put it in your pocket, and then lost it. You might feel like that money is gone and become more cautious.

The researchers asked: Does the AI know the difference between an "open" wallet and a "closed" one, and does it actually change its behavior because of it?

They tested the AI in three ways, like checking a suspect at three different levels of a police investigation.

Level 1: The "Acting" Test (Behavior)

First, they just asked the AI to answer questions.

The Result: The AI did change its answers based on whether the story said the money was "open" or "closed." It was sensitive to the words.
The Catch: But it didn't act like a real human. When humans lose money in an "open" account, they get reckless. When the AI lost money in an "open" account, it didn't get reckless in the same way. It was just mimicking the shape of the answer, not the logic behind it.

Level 2: The "X-Ray" Test (Reading the Brain)

Next, the researchers looked inside the AI's "brain" (its internal math layers) to see if it actually had a concept of "Open vs. Closed" stored there.

The Result: They found it! At a specific layer (Layer 18) of the AI, there was a clear, readable signal. If you looked at the math, you could tell exactly which prompts were "open" and which were "closed."
The Metaphor: It's like finding a light switch in a house that is clearly labeled "Kitchen Light." You can see the switch, and you know it's connected to the kitchen.

Level 3: The "Remote Control" Test (Causal Control)

This is the most important part. If the AI really "understands" the concept, then we should be able to flip that internal switch and force the AI to change its mind.

The Experiment: The researchers used a "remote control" (activation steering) to manually flip that "Open vs. Closed" switch inside the AI's brain while it was answering questions. They tried to force the AI to act as if it had an "open" wallet or a "closed" wallet.
The Result: Nothing happened. Even though they could see the switch and flip it, the AI's final decisions about gambling and risk didn't change.
The Metaphor: It's like finding a light switch labeled "Kitchen Light," flipping it on and off, and realizing the kitchen light doesn't turn on. The switch exists, but it's not actually connected to the bulb. It's just a decoration.

The Big Conclusion

The paper concludes that just because an AI can say the right words (Level 1) and has a readable "switch" inside its brain (Level 2), it doesn't mean that switch actually controls its actions (Level 3).

The "Realization Effect" in humans is a deep, causal mechanism: The way we feel about money changes how we gamble.
The "Realization Effect" in this AI was just a surface pattern. The AI noticed the words "open" and "closed" and adjusted its vocabulary, but it didn't actually use that concept to make its final decision.

The Takeaway:
Don't assume an AI is "thinking" like a human just because it gives human-like answers. It might just be a very good actor. To prove it's actually "thinking," you have to show that you can reach inside, flip a switch, and make it do something different. In this case, the researchers tried to flip the switch, and the AI didn't budge.

In short: The AI has the words for the concept, and it has the internal signal for the concept, but it doesn't have the control over its behavior. The signal is there, but it's not doing any real work.

Technical Summary: Representation Without Control

Problem Statement
The paper addresses a critical gap in the evaluation of Large Language Models (LLMs) used as behavioral simulators. While LLMs are increasingly employed to model human cognition, it remains unclear whether their condition-sensitive outputs reflect genuine internal implementation of human cognitive mechanisms or merely prompt-sensitive surface patterns. The authors investigate this by testing the "realization effect," a well-characterized phenomenon in behavioral economics where risk-taking behavior changes systematically based on whether prior gains or losses are "paper" (open mental account) or "realized" (closed mental account). The core question is whether LLMs not only exhibit behavioral sensitivity to this framing but also encode the underlying construct as a causal representation that drives downstream decisions.

Methodology
The study employs a three-stage experimental design using the Gemma 3 4B model to test three linked claims: behavioral sensitivity, linear decodability, and causal control.

Prompt-Only Behavioral Task: The authors constructed 11 vignette conditions spanning paper and realized outcomes (finance, casino, etc.) to test if LLM responses (wager amounts and risk preferences) replicate the directional predictions of human realization effects.
Activation Readout: The team analyzed Gemma's residual stream activations to determine if "realization status" is linearly decodable. They constructed a mean-difference direction vector ( $v_{realization} = \mu_{realized\_closed} - \mu_{paper\_open}$ ) using a training split of 756 matched pairs at layer 18. This direction was tested for generalization on held-out validation sets and a new DeepSeek-generated prompt set.
Causal Steering Intervention: To test if the decoded representation causally governs behavior, the authors implemented residual-stream steering. They added the scaled, normalized realization direction to the model's activations at layer 18 during generation. They tested various steering scales (from -50 to +150) to see if shifting the activation toward "realized" or "paper" states reliably altered the model's wager and risk choices. A positive-control classification task was also run to check for semantic shifts in direct judgments.

Key Results
The results reveal a significant dissociation between representation and control:

Behavioral Sensitivity (Partial): LLMs are condition-sensitive, producing different responses based on framing. However, the directional pattern does not robustly match human realization-effect predictions. For instance, realized losses increased risk-taking in the model, contrary to the human tendency to decrease risk after realized losses.
Linear Readout (Success): The realization status is linearly decodable from Gemma's residual stream at layer 18. A direction trained on one set of prompts successfully separates "paper" from "realized" prompts in held-out sets, including those from a different source (DeepSeek).
Causal Control (Failure): Steering along the decoded realization direction does not reliably shift downstream risk behavior.
- Wager and Risk: Across positive and negative steering scales, mean wager and risk deltas were small, and median deltas were consistently zero.
- Symmetry: The expected sign-symmetric reversal (steering toward "paper" should decrease risk, steering toward "realized" should increase it, or vice versa) was not observed.
- Correlation: While raw projection strength correlated weakly with wager amounts at the prompt level, this association vanished after controlling for prompt structure. Within matched pairs, projection deltas did not predict behavioral deltas.
Positive Control: A classification task showed a weak directional shift in semantic judgment (increasing "realized" predictions at high positive scales), but accuracy remained near chance, and the shift was asymmetric.

Key Contributions
The paper's primary contribution is the empirical demonstration that behavioral sensitivity, latent readout, and causal control are three distinct properties that do not automatically co-occur.

It establishes that successful linear readout of a variable is insufficient evidence that a model relies on that representation during downstream decision-making.
It provides a concrete case study where a model passes the first two tests of mechanistic interpretability (sensitivity and decodability) but fails the third (causal control).
It challenges the assumption that condition-sensitive behavior in LLMs implies the implementation of human-like cognitive mechanisms.

Significance and Claims
The authors argue that their findings necessitate a shift in how LLM behavioral research is conducted and interpreted.

For Behavioral Evaluation: Condition sensitivity alone is weak evidence of human-like cognition. Models may track surface features of prompts (framing) without encoding the underlying mental constructs (e.g., mental accounting).
For Mechanistic Interpretability: The study warns against treating linear decodability as a proxy for causal explanation. A representation can be readable in activations yet be epiphenomenal to the policy governing outputs. The authors posit that "causal validation, not just correlational decoding, is necessary to establish that a representation is doing real work."
Conclusion: The paper concludes that establishing that a model genuinely implements a human decision-making construct requires showing that the relevant representation is causally upstream of the behavior of interest. This is a higher bar than the field currently applies, and the authors suggest that causal control should be treated as a necessary condition for such mechanistic claims.

The authors remain modest about the scope, noting that the null result is specific to the Gemma 3 4B configuration (layer 18, final-token steering) and that the dissociation might vary across models, layers, or task types. However, the finding motivates a stricter standard for claiming that LLMs implement specific cognitive mechanisms.