🤖 machine learning

Where's the Plan? Locating Latent Planning in Language Models with Lightweight Mechanistic Interventions

This paper investigates where and how language models form internal representations of future constraints during planning, revealing that while future-rhyme information is linearly decodable across multiple model families and scales, only Gemma-3-27B causally relies on a specific late-layer handoff mechanism to utilize this information, whereas other models condition on the rhyme word itself despite showing strong probe signals.

Original authors: Nicole Ma, Nick Rui

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nicole Ma, Nick Rui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a magician perform a trick. You see them pull a rabbit out of a hat, but you don't know how they did it. Did they have the rabbit hidden in their sleeve the whole time? Or did they magically conjure it the moment they reached into the hat?

This paper is like a pair of X-ray glasses for AI models. The researchers wanted to know: When an AI writes a poem, does it secretly "plan" the ending before it even starts writing the next line?

Here is the story of their discovery, broken down into simple parts.

The Test: The Rhyming Challenge

To test this, the researchers gave AI models a simple task: finish a rhyming couplet.

The Prompt: "She felt a sudden sense of fright,"
The Goal: The AI needs to write a second line that ends with a word that rhymes with "fright" (like "night" or "light").

The question was: Does the AI have a secret internal note saying "I need to rhyme with 'fright'" before it starts typing the second line? Or does it just figure it out word-by-word as it goes?

The Two Tools: The "Spy" and the "Time Traveler"

The researchers used two different methods to look inside the AI's brain (its "hidden states").

The Spy (Linear Probing):
Imagine a spy who can peek at the AI's notes and ask, "Hey, do you know what word comes next?"
- What they found: In almost every AI model they tested (Qwen, Llama, and Gemma), the spy found the answer! At the exact moment the AI finished the first line (the newline character), the AI's internal notes did contain the information about the rhyme.
- The Catch: Just because the spy found the note doesn't mean the AI is actually using it. It might be like having a map in your pocket but ignoring it and just walking randomly.
The Time Traveler (Activation Patching):
This is the real test. Imagine you are the AI. You are about to write the second line. The researchers "time travel" and swap your current brain state with the brain state of a different AI that was trying to rhyme with a different word (e.g., trying to rhyme with "fear" instead of "fright").
- The Question: If you swap the brain state, does the AI suddenly start writing words that rhyme with "fear"?
- If yes: The AI was actually planning and using that information.
- If no: The AI was just pretending to have the plan, or it figured it out later.

The Big Surprise: Only One Model Actually Planned

Here is where the story gets interesting. The results were very different for different models:

The "Fake Planners" (Qwen and Llama):
These models were great at the "Spy" test. They had the rhyme information stored in their brains at the end of the first line. But when the researchers tried the "Time Traveler" test, nothing happened. Swapping the brain state didn't change the output.
- Analogy: It's like a chef who has a recipe card on the counter (the information is there) but ignores it and just cooks by smell and instinct. They didn't actually use the plan to guide their cooking; they just kept looking at the last word they wrote ("fright") to figure out the next one.
The "Real Planner" (Gemma-3-27B):
This specific model was different.
- Early layers: At the start of the second line, it relied on the last word ("fright").
- The Handoff: Around layer 30 (a specific depth in the AI's brain), something magical happened. The "planning duty" moved from the last word to the newline (the empty space after the first line).
- The Result: When the researchers swapped the brain state at the newline, the AI immediately started trying to rhyme with the new word.
- Analogy: This model is like a conductor. At the start of the song, they look at the sheet music (the last word). But halfway through, they put the sheet music down and start conducting the orchestra based on a mental map they built at the break (the newline). They actually used the plan to guide the music.

The "Who Did It?" Investigation

For the Gemma model, the researchers wanted to know exactly which parts of the brain were doing this planning. They narrowed it down to a tiny, specific team: five attention heads (think of these as five specific neurons or "workers" in the AI's brain).

When they messed with just these five workers, the AI lost its ability to plan. When they left them alone, the AI planned perfectly. It wasn't a messy, general process; it was a precise, surgical operation performed by a tiny team.

The Main Takeaway

The paper concludes that just because an AI has information doesn't mean it's planning.

Many models can store the idea of a future rhyme (they have the map).
But only one model (Gemma-3-27B) actually uses that stored map to guide its future actions (it follows the map).
For the others, they are just reacting to the immediate past, not planning for the future, even though the future info is technically sitting in their brains.

This is important because it shows that "intelligence" or "planning" isn't just about having data; it's about how that data is actively used to shape what happens next. And surprisingly, this ability to "plan" isn't just a general feature of all big AIs; it's a specific quirk that only some models have developed.

Technical Summary: Where's the Plan? Locating Latent Planning in Language Models with Lightweight Mechanistic Interventions

Problem Statement
Autoregressive language models generate text token-by-token yet frequently produce outputs requiring long-range structural coherence, such as rhyming couplets. This raises a fundamental question regarding latent planning: Do models form internal representations of future outputs that causally drive generation, even when these processes are invisible to external behavioral evaluation? While prior work has established that planning-compatible information exists in some models, it remains unclear where exactly this information resides during the forward pass and whether it moves (a phenomenon the authors term planning site formation). Furthermore, distinguishing between information that is merely encoded (passive) and information that is causally used (active) is critical for accurate mechanistic understanding and safety assessment.

Methodology
The authors employ a clean test case: rhyming couplet completion. Given a first line ending in a rhyme word $r_1$ , the model must generate a second line ending in a rhyming word $r_2$ . The study utilizes two lightweight mechanistic intervention methods across three open-source model families (Qwen3, Gemma-3, Llama-3) at various scales (up to 70B parameters):

Linear Probing (Encoding Evidence): Linear classifiers are trained to predict the future rhyme token $r_2$ from hidden states $h_{\ell,i}$ at various layers $\ell$ and positions $i$ . This determines if planning-compatible information is decodable at specific locations (e.g., the last word token vs. the newline token).
Activation Patching (Causal Evidence): The hidden state at a specific position and layer from a "corrupt" run (targeting a different rhyme family) is substituted into a "clean" run. If this intervention redirects the generation toward the corrupt rhyme, the location is identified as a causally active planning site.

The study specifically compares the last word token (position $i \approx -1$ or $-2$) against the newline token (position $i = 0$ ) to track the migration of causal influence.

Key Results

Dissociation of Encoding and Causality:
- Probing: In all three model families, linear probes successfully decode future-rhyme information at the newline token. The strength of this signal generally increases with model scale, suggesting that planning-compatible representations are an emergent property of larger models.
- Activation Patching: Despite strong probe signals, only Gemma-3-27B exhibits a causally active planning site at the newline. In this model, the causal driver of generation migrates from the last word token to the newline token around layer 30.
- Other Models: Qwen3 (up to 32B) and Llama-3 (up to 70B) condition on the last word token throughout generation. Despite encoding rhyme information at the newline (as shown by probes), patching the newline in these models has near-zero causal effect on the output.
Localization of the Handoff (Gemma-3-27B):
- The "handoff" of causal influence from the last word to the newline is mediated by a sparse set of five attention heads in layers 28 and 30.
- Using two-stage path patching, the authors demonstrate that patching these five specific heads simultaneously recovers approximately 90% of the rhyme-routing capacity normally achieved by overwriting the entire residual stream at the newline.
- The mechanism is specific to attention; patching the corresponding MLPs yields no effect.

Key Contributions

Definition of Planning Site Formation: The paper formalizes the distinction between planning-compatible representations (detectable via probing) and causally active planning sites (established via intervention), highlighting that the former does not guarantee the latter.
Scalable Mechanistic Framework: The authors propose a framework using linear probing and activation patching that requires no transcoder training and significantly less data than steering vector approaches, making it applicable to large, open-source architectures.
Empirical Discovery of Divergence: The study reveals that different model architectures (even within the same scale range) solve structural constraints via fundamentally different mechanisms: some rely on a static "last-word" condition, while others (like Gemma-3-27B) implement a dynamic "handoff" to the newline.
Sparse Circuit Identification: The work successfully localizes a complex planning behavior in Gemma-3-27B to a specific, small set of attention heads, demonstrating that such emergent phenomena can be structured rather than diffuse.

Significance and Claims
The paper argues that probe signal alone is insufficient evidence of a planning site. Standard behavioral evaluations or simple probing may systematically underestimate or mischaracterize a model's planning capabilities if they fail to distinguish between passive encoding and active causal usage.

The authors claim that planning site formation is a distinct emergent phenomenon tied to specific model scale and architecture, rather than a universal consequence of strong planning-compatible representations. The identification of a sparse, causal circuit (the five attention heads) suggests that latent planning can be a highly structured computational process.

The study concludes with modest limitations: the findings are currently restricted to three model families and a single structured task (rhyming). The authors note that it remains an open question whether this "handoff" mechanism is a general planning primitive or specific to phonological constraints, and whether the absence of this handoff in Qwen and Llama models is due to architectural differences or training data.