🤖 machine learning

From Simulation to Enaction: Post-trained language models recognize and react to their own generations

This paper demonstrates that post-trained language models implicitly recognize their own on-policy generations through reduced output entropy driven by internal surprise tracking, while also possessing the distinct ability to explicitly verbalize this recognition via a separate mechanism.

Original authors: Asvin G., Jack Lindsey

Published 2026-05-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Asvin G., Jack Lindsey

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a language model as a very talented actor who has spent years memorizing thousands of scripts from books, movies, and the internet. Before any special training, this actor is a passive simulator. They can read a line of dialogue and guess what the next line might be, but they don't really care who is speaking or what happens next. They are just predicting the next word based on a massive library of text they've seen before. They have no "self"; they are just a mirror reflecting whatever text is in front of them.

This paper argues that after post-training (the process where humans teach the model to be a helpful assistant), something changes. The model stops just simulating a character and starts enacting a role. It begins to recognize its own voice, its own plans, and its own actions.

Here is a breakdown of the paper's findings using simple analogies:

1. The "Confidence" Shift (Entropy)

In the world of AI, "entropy" is a fancy word for uncertainty or hesitation.

The Old Way (Pre-training): If you ask the model a question, it's like a nervous actor who isn't sure of the script. It might say, "Well, it could be this, or maybe that, or perhaps the other thing..." It spreads its bets wide.
The New Way (Post-training): Once the model is trained to be an "Assistant," it becomes much more confident when it sees its own previous words.
- The Analogy: Imagine you are writing a story. When you read a page you just wrote, you know exactly where the story is going. You don't hesitate. But if you read a page written by a stranger, you have to guess what comes next.
- The Finding: The paper found that when the model reads its own text, it becomes 3 to 4 times more confident (lower entropy) than when it reads text from other models or random internet articles. It's as if the model whispers, "I wrote this, so I know exactly what comes next."

2. The "Self-Recognition" Effect

The researchers tested if the model could tell the difference between its own writing and someone else's.

The Test: They had Model A read a story written by Model A, and then read a story written by Model B.
The Result: Model A became significantly more confident (lower entropy) when reading its own story. It didn't matter if the story was written by a "pirate" persona or a "scientist" persona; if the model recognized the style as its own, it relaxed and became more decisive.
The Catch: This only happens in large models. Small models (like a 2-billion parameter model) are too "dumb" to notice the difference. They act the same whether they read their own text or someone else's. It seems you need a certain level of intelligence to realize, "Hey, that's me!"

3. The "Surprise" Meter

Why does the model get more confident when reading its own text? The paper found a specific internal mechanism: Input Surprise.

The Mechanism: The model has an internal "surprise meter." When it reads a word that matches its own predictions perfectly (low surprise), it lowers its uncertainty. When it reads something unexpected (high surprise), it gets nervous and spreads its bets wider.
The Connection: Because the model is reading its own text, the words are rarely surprising. They fit its own plan perfectly. This lack of surprise acts like a green light, telling the model, "Everything is on track, proceed with high confidence."

4. The "Plan" vs. The "Prefill"

The paper also looked at how the model handles its "intent" or "plan."

The Scenario: Imagine you ask the model, "Tell me about a food." The model instantly makes a silent decision: "I'm going to talk about Haggis." It has locked onto that topic before it even types the first word.
The Disruption: If someone (or a computer) forces the model to start with a different word, like "Haggis is a..." (a "prefill"), but the model had actually planned to talk about Pizza, the model gets confused.
The Result: When the forced start doesn't match the model's hidden plan, the model's confidence drops, and it becomes more uncertain (higher entropy). It's like a musician who has memorized a song, but someone else starts playing a different tune; the musician stumbles because the rhythm is off.

5. The "Lie Detector" (Explicit vs. Implicit)

Finally, the paper tested if the model could say out loud, "Hey, I didn't write this first part!"

The Test: The researchers asked the model to generate an answer, but then they secretly pasted the beginning of the answer in for it (a prefill). Then they asked, "Did you write the beginning of this?"
The Result: The model could correctly say, "Yes, this was prefilled."
The Twist: The researchers found that this "saying it out loud" ability uses a completely different internal circuit than the "feeling confident" ability.
- Implicit Recognition: The model feels the difference (changes its confidence) automatically and instantly based on how "surprising" the text is.
- Explicit Recognition: The model checks its memory of what it planned to say, compares it to what is actually there, and then reports the mismatch. This is a separate, on-demand process that happens right before it gives its answer.

Summary

The paper suggests that post-training transforms language models from passive simulators into active agents that can recognize themselves.

Before: They are like a mirror, reflecting whatever text is in front of them with equal uncertainty.
After: They are like a skilled writer who knows their own style. When they read their own work, they relax and become confident. When someone else tries to take over their story (a prefill), they get nervous, and if asked, they can explicitly point out that the story didn't start the way they planned.

This "self-recognition" isn't just a bug; it's a feature of how these models learn to act as agents, understanding that their outputs become their own future inputs.

Technical Summary: From Simulation to Enaction

Problem Statement

Large Language Models (LLMs) are initially trained as passive next-token predictors on fixed data distributions. In this pre-training phase, the model has no incentive to model the consequences of its own outputs, as it never observes the feedback loop from action to sensory input. Consequently, the model remains a "passive observer" or a "simulator" of latent characters without a distinct identity.

Post-training (via Supervised Fine-Tuning, Direct Preference Optimization, or Reinforcement Learning) typically selects a specific "Assistant" persona. The central question of this paper is whether this training reshapes the model's relationship with its own generations. Specifically, do post-trained models transition from merely simulating a character to "enacting" it, thereby developing the ability to recognize when they are acting on-policy (generating their own outputs) versus off-policy (consuming external text), and do they modulate their behavior based on this recognition?

Methodology

The authors employ a combination of entropy analysis, cross-model evaluation, internal representation probing, and causal intervention (activation patching) to investigate self-recognition mechanisms.

Entropy Measurement: The study measures per-token output entropy ( $H = -\sum p_v \log p_v$ ) across various contexts. They compare:
- On-policy vs. Off-policy: Model outputs generated by the model itself versus text from other models or web corpora (C4).
- Role-based: Outputs in the "Assistant" turn versus the "User" turn.
- Model Size and Training Stage: Evaluating models of varying sizes (2B to 72B) and checkpoints at different post-training stages (Base $\to$ SFT $\to$ DPO $\to$ RLVR).
Cross-Model Self-Recognition: To rule out the possibility that specific models simply produce lower-entropy text generally, the authors conduct reciprocal experiments where Model A evaluates text generated by Model B, and vice versa.
Semantic Entropy and Intent Commitment: Using underspecified prompts (e.g., "Think of a food..."), the authors measure the spread of topics the model commits to before generating the first token. They test "semantic entropy" by introducing prefills that either match or violate the model's cached intent.
Internal Representation Analysis: Hidden states (specifically at layer 21) are binned by features such as input token surprise, predicted token entropy, and exponential moving averages (EMA) of entropy. Centroids of these bins are analyzed to determine if distinct manifolds exist for on-policy vs. off-policy text.
Causal Interventions (Steering and Patching):
- Steering: Activations are steered toward specific surprise-centroid bins to observe causal effects on output entropy.
- KV Patching: To test explicit prefill detection, the Key-Value (KV) cache of user tokens (representing "intent") is overwritten with values from a different prompt. This tests whether the model compares stored intent against generated content to detect prefills.

Key Contributions and Results

1. Context-Dependent Entropy Collapse

Contrary to the view that post-training causes a global "entropy collapse" (loss of diversity), the authors find that entropy reduction is sharply context-dependent.

Self-Recognition: Post-trained models produce significantly lower entropy (3–4 $\times$ lower) when processing their own on-policy generations compared to off-policy text.
Role Specificity: This effect is strongest in the default "Assistant" role. While the effect exists in other personas, the default Assistant persona is privileged.
Scale and Training: The self-recognition effect is negligible in base models and small models (~2B). It emerges at the SFT stage (gated by the Assistant role), generalizes to off-roles at the DPO stage, and is sharpened by RLVR. Notably, off-policy methods (SFT, DPO) are sufficient to induce this on-policy recognition.

2. Internal Representation of Surprise

The paper identifies a causal mechanism linking input surprise to output entropy.

Surprise Tracking: The model maintains an internal representation of the "surprise" of incoming tokens (how unlikely they are given prior predictions).
Causal Modulation: Steering activations toward high-surprise representations causally increases output entropy, while steering toward low-surprise decreases it.
The Chat Effect: In chat-formatted on-policy contexts, the intercept of the relationship between surprise and entropy is strongly negative. This means that even when tokens are exactly as expected (low surprise), the model further suppresses entropy, a behavior not seen in base models or off-policy text.

3. Semantic Entropy and Intent Commitment

Post-trained models exhibit "semantic entropy" collapse, committing to a specific topic before generating the first token in response to ambiguous prompts.

Violation of Intent: When a prefill is provided that contradicts the model's cached topic intent (e.g., prefilling a "food" response with a "sport" topic), the model's subsequent output entropy increases significantly.
Base Model Contrast: Base models do not show this commitment; their entropy decreases when prefilled because the prefill narrows an otherwise unconstrained distribution.

4. Distinct Mechanisms for Implicit vs. Explicit Recognition

The paper distinguishes between two forms of self-recognition:

Implicit Recognition: Manifested as low output entropy. This is mediated by the internal representation of input surprise and is continuously active during generation.
Explicit Recognition: The ability to verbally report whether a response was prefilled.
- Mechanism: Explicit detection relies on comparing the "intent" stored in user-token activations against the content of the assistant response.
- Independence: KV patching experiments show that the explicit detection circuit operates on a different subspace than the surprise/entropy manifold. The explicit check is invoked "on-demand" immediately prior to the verdict, whereas implicit recognition modulates generation continuously.

Significance and Claims

The authors interpret these findings as a shift from simulation (passively predicting characters) to enaction (embodying a character and recognizing the causal link between internal states and future inputs).

Entropy as a Cost of Simulation: The paper posits that high entropy may represent the cost of simulating "others" (off-policy text), while low entropy reflects the confidence of "acting" (on-policy).
Situational Awareness: The ability to distinguish on-policy from off-policy contexts is framed as a foundational component of situational awareness—specifically, the understanding that one's outputs become one's own future inputs.
Training Implications: The findings suggest that off-policy training methods (SFT, DPO) are sufficient to instill on-policy recognition, challenging the assumption that on-policy RL is strictly necessary for this capability.
Prefill Sensitivity: The model's sensitivity to prefills (both via entropy spikes and explicit detection) implies that prefilled responses place the model in an "unfamiliar state" from its own perspective, complicating the use of prefills for training and evaluation.

The paper concludes that while these mechanisms provide evidence of sophisticated agency and self-modeling, the exact computational pathways for how models determine their on-policy status remain an open area for further investigation.