Single-Position Intervention Fails: Distributed Output Templates Drive In-Context Learning

This article demonstrates that the identity of the in-context learning task is not localized to specific layers or tokens, as suggested by linear probing, but is causally encoded as distributed output format templates across demonstration tokens, with a critical intervention window located at approximately 30% of the network depth.

The Gist

Imagine a large language model (like those powering chatbots) as a massive, multi-story factory. When you give it a few examples of a task (such as "convert this word to uppercase"), it attempts to recognize the rule and apply it to your new question. This is called In-Context Learning (ICL).

For a long time, scientists believed they knew where in this factory the "rule" was stored. They used a tool called a "probe" (like a metal detector) that would beep and say: "Yes, the rule for 'uppercase' is right here!" They found these beeps at specific locations on certain floors of the factory.

The Big Surprise: The Metal Detector Is a Liar
The authors of this paper decided to test whether these beeps actually conveyed something meaningful. They attempted a "surgery" experiment: they went exactly where the metal detector said the rule was, removed the information, and replaced it with something else.

• The Result: Nothing happened. The factory continued to operate flawlessly, completely ignoring the intervention.
• The Analogy: Imagine you believe a car engine is controlled by a single red wire. You cut that wire and expect the car to stop. Instead, the car keeps driving. It turns out the engine isn't controlled by one wire; the signal is distributed across thousands of wires. If you cut just one, the car doesn't care.

The Real Discovery: The "Distributed Template"
The researchers realized the "rule" isn't stored in one place. It's like a puzzle distributed across the entire set of examples you provided to the model.

1. Failure at a Single Position: If you try to swap out just one puzzle piece (a word in the example), the model notices nothing. It has too many other pieces to recognize the picture.
2. Breakthrough at Multiple Positions: However, if you swap all the puzzle pieces simultaneously (every output word in the examples), the model changes its mind. It begins to follow the new rule you gave it.

The "Sweet Spot" in the Factory
The researchers found that this "puzzle swapping" only works if you do it on a specific floor of the factory.

• Too early (Floors 1–7): The puzzle pieces haven't been assembled yet; the pattern isn't clear.
• Too late (Floors 15+): The factory has already finished building the car and driven away; changing the blueprints now is too late.
• Just right (Floor 8): This is the "commitment window." Here, the factory finalizes the design but hasn't started construction yet. If you swap the blueprints here, the factory builds the new car.

What Is Actually Transferred?
The paper discovered that the model doesn't learn the meaning of the task (such as "this is about feelings"). Instead, it learns the form of the answer.

• The Analogy: Imagine you teach a model how to write a poem. If you change the examples so they show a different type of poem (e.g., from rhyming couplets to haikus), the model won't switch, even if the topic remains the same.
• The Insight: The model only copies the "template." If the examples show "Word, Word, Word," the model will only switch to a new task if that new task also looks like "Word, Word, Word." It doesn't matter whether the words are about cats or numbers; what matters is that the structure matches.

The Request Versus the Examples
The paper also discovered a funny asymmetry:

• The Examples (The Demo): These are like the "ingredients." You need all of them to prepare the dish. If you miss one, the recipe still works because the others compensate. However, if you swap all of them, the dish changes completely.
• The Question (The Query): This is the "chef" reading the recipe. If you scramble the chef's instructions (the part of the question), the whole thing fails. The chef is essential, but the chef doesn't hold the recipe; the ingredients do.

Summary in Simple Language

1. Don't trust the metal detector: Just because a model can find a rule in one place doesn't mean that place is important.
2. The rule is everywhere: The "task identity" is distributed across all example answers, not fixed in one spot.
3. Timing is crucial: You can only change the model's mind in the middle of its thinking process, not at the beginning or the end.
4. It's about form, not meaning: The model copies the format of the answer (like a template) rather than understanding the deep logic of the task.

This paper has essentially rewritten the map of how these AI models learn from examples, showing us that the "brain" of the task is a distributed, fault-tolerant network, not a single switch.

Technical Summary

Technical Summary: Distributed Output Templates Drive In-Context Learning

Problem Statement

Understanding how Large Language Models (LLMs) encode task identity from few-shot demonstrations remains a central open problem in mechanistic interpretability. Previous research relied predominantly on linear probing methods to locate task representations, reporting high classification accuracy at specific layers and positions. However, a critical gap exists between decodability (a probe's ability to extract information) and causal relevance (whether this information controls model behavior). This contribution investigates whether the high probing accuracy observed in prior work implies causal control over task identity during in-context learning (ICL).

Methodology

The authors conduct activation intervention experiments on Llama-3.2-3B-Instruct and replicate results across three additional models (Llama-3.2-1B, Qwen2.5-1.5B, Gemma-2-2B). The study distinguishes between three position types in an ICL prompt: demonstration inputs, demonstration outputs, and the query.

1. Probing: Nearest-centroid classifiers are trained at each (layer, position) pair to measure the decodability of task identity.
2. Single-Position Intervention: The activation vector at a single (layer, position) pair in a target prompt is replaced with the mean activation from a source task. This tests whether task identity is localized.
3. Multi-Position Intervention: Activations at all output markers of the demonstrations are replaced simultaneously. This tests whether task identity is distributed.
4. Causal Tracing: Gaussian noise is injected into specific positions to measure necessity (perturbation rate), while transplantation measures sufficiency (transfer rate).
5. Format Compatibility Analysis: The study creates task variants with identical operations but different output formats (e.g., "WORD" vs. "WORD."), to determine whether transfer relies on abstract rules or surface templates.

Key Contributions and Results

1. The Failure of Single-Position Intervention

Despite 100% probing accuracy at demonstration positions and 83% at the query position across all 28 layers, single-position intervention achieves 0% task transfer at any layer. Control experiments (null and random ablation) confirm that no single position is causally necessary; the model completely ignores single-position perturbations. This demonstrates that task encoding is fundamentally distributed and fault-tolerant.

2. The Breakthrough of Multi-Position Intervention

The authors reveal that task identity can only be causally transferred through simultaneous multi-position intervention.

• Optimal Window: Transfer reaches its maximum at Layer 8 (approximately 30% of network depth) for 28-layer models, achieving 96% transfer (95% CI: [87%, 99%]) for format-compatible pairs.
• Position Specificity: Replacing output markers of the demonstrations yields 94% transfer, whereas replacing input markers yields 0%. Replacing all demo markers (inputs + outputs) yields 96%, indicating that output markers carry the primary signal.
• Threshold Effects: Transfer is not gradual. Replacing 1–3 positions yields 0% transfer; replacing approximately 10 positions yields 10%; replacing all positions yields approximately 90%. Similarly, replacing 1–3 source demonstrations yields 0% transfer, while 5 demonstrations yield 93%.

3. Causal Asymmetry: Query vs. Demonstrations

Causal tracing via noise injection reveals a striking asymmetry:

• Query Position: Strictly necessary (53–100% perturbation when noise is injected in layers 0–14), but not sufficient (0% transfer upon transplantation).
• Demonstration Positions: Individually not necessary (0% perturbation when noise is injected), but collectively sufficient (96% transfer upon complete replacement).
  This suggests a staged pipeline: early layers encode distributed templates, middle layers aggregate this information for the query, and late layers commit to output generation.

4. The Hypothesis of Distributed Templates

Transfer depends on the compatibility of internal representations, not surface similarity.

• Format Sensitivity: Tasks with identical operations but different output formats (e.g., "WORD" vs. "WORD.") show 0% transfer. Conversely, tasks with structurally similar templates (e.g., "word word" vs. "word, word") show 90% transfer.
• Token Count Compatibility: Transfer is driven by the compatibility of the number of output tokens. For example, uppercase (1 token) transfers to linear 2x (1 token) with 100% success, but repeat word (2 tokens) fails to transfer to repeat n (3 tokens).
• Predictability: Only 7 of 56 task pairs (13%) achieve ≥50% transfer. These form a cluster of procedural tasks with rigid, single-token or repeated-token output structures. Semantic tasks show near-zero transfer between tasks.

Significance and Claims

The contribution establishes the Distributed Templates Hypothesis: Task identity in ICL is not encoded as a localized vector or abstract rule, but as output format templates distributed across the markers of the demonstrations.

• Mechanistic Insight: The results resolve the ambiguity between decodability and causal relevance, showing that high probing accuracy does not imply causal control.
• Architecture Specificity: The optimal intervention window at approximately 30% depth appears to be a universal feature across LLaMA, Qwen, and Gemma architectures, suggesting a general Transformer processing phase where templates are encoded prior to aggregation.
• Implications for Steering: Effective activation steering for ICL requires coordinated multi-position replacement in early-to-middle layers, rather than single vector editing.
• Nature of ICL: The results suggest that the model operates by matching output templates for many tasks, rather than inferring abstract input-output relationships, particularly for procedural tasks.

The authors conclude that while the mechanism is general, successful transfer is constrained by strict structural compatibility between source and target output formats, thereby ruling out trivial explanations based on surface similarity or random injection artifacts.