💬 NLP

Patches of Nonlinearity: Instruction Vectors in Large Language Models

This paper investigates how large language models process instructions by identifying "Instruction Vectors" (IVs) that act as circuit selectors, demonstrating that while these representations are localized and linearly separable, they interact with the model's internal pathways through complex non-linear causal mechanisms.

Original authors: Irina Bigoulaeva, Jonas Rohweder, Subhabrata Dutta, Iryna Gurevych

Published 2026-02-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Irina Bigoulaeva, Jonas Rohweder, Subhabrata Dutta, Iryna Gurevych

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a world-class chef prepare a complex dish. You see them chop onions, sear meat, and simmer a sauce. You know they are making "Beef Bourguignon," but you don't quite know how their brain is organizing the steps. Are they following a mental recipe card, or are they just reacting to the smell of the garlic as it hits the pan?

This paper, "Patches of Nonlinearity," is essentially a "brain scan" for Large Language Models (like ChatGPT). The researchers wanted to know: When you give an AI an instruction (like "Translate this to French"), how does it actually "hold" that instruction in its mind while it works?

Here is the breakdown of their discovery using three simple analogies.

1. The "Mental Sticky Note" (Instruction Vectors)

The researchers found that as soon as an AI reads an instruction, it creates a tiny, concentrated "summary" of that task.

The Analogy: Imagine you walk into a library. Before you even look for a book, you scribble "Find a mystery novel" on a sticky note and slap it on your forehead. You don't need to keep re-reading the sign at the entrance; the "essence" of your mission is right there, attached to you, ready to guide every step you take.

In AI terms, they call these Instruction Vectors (IVs). They are localized "digests" of the task that the model creates early on, which then guide the rest of the process.

2. The "Synergistic Jazz Band" (Superadditivity)

This is the most surprising part of the paper. Usually, scientists assume that if Part A of an AI does 5 units of work and Part B does 5 units of work, together they do 10 units. This is called "linear" thinking.

But the researchers found that instruction-following is non-linear (superadditive).

The Analogy: Think of a Jazz Band. If a drummer plays alone, it’s just a beat. If a pianist plays alone, it’s just a melody. But when they play together, they don't just produce "Beat + Melody." They create a "Groove"—a third, magical thing that is much greater than the sum of its parts.

The AI's internal layers work like that jazz band. One layer might hold a piece of the instruction, and another might hold a piece of the logic, but they only truly "click" and solve the task when they interact in a complex, non-linear way. You can't understand the "groove" by looking at the drummer and pianist in isolation.

3. The "Circuit Selector" (The Mechanism)

Finally, the researchers wanted to know what those "sticky notes" actually do. Do they just sit there, or do they change how the AI thinks?

The Analogy: Imagine a massive Electrical Switchboard in a city. There are millions of wires (paths) that can carry electricity. Without an instruction, the power is just flowing randomly. But the Instruction Vector acts like a Master Switch.

Once the "sticky note" is created, it tells the model: "Hey, we are doing a math task! Flip switches 5, 12, and 89, and ignore the rest!" The instruction vector acts as a Circuit Selector, picking out the specific "electrical pathways" needed to solve that specific problem while turning off the pathways used for poetry or coding.

Why does this matter?

If we want to make AI safer and more reliable, we can't just treat it like a "black box" that spits out answers. We need to know how it "thinks."

By proving that instructions are stored as these "superadditive" sticky notes that select specific circuits, the researchers have given us a new map. It tells us that if we want to fix an AI's mistakes, we can't just tweak one single "neuron"—we have to understand how the whole "jazz band" is playing together.

Technical Summary: Patches of Nonlinearity: Instruction Vectors in Large Language Models

1. Problem Statement

While instruction tuning (via Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO)) is a standard post-training step for Large Language Models (LLMs), the internal mechanistic process of how these models represent and utilize instructions remains poorly understood. Specifically, the research seeks to answer:

Localization: Are instructions processed "just-in-time" (distributed across the query) or "eagerly" (as localized neural digests formed immediately after the instruction)?
Geometry and Causality: Do these instruction representations follow the "linear representation hypothesis" (where concepts are linear subspaces), and how do they interact causally to influence the model's output?

2. Methodology

The authors utilize the OLMo-2 model family (1B and 7B variants) to compare base, SFT, and DPO models. They employ several advanced mechanistic interpretability techniques:

Causal Mediation (Activation Patching): They use single- and multi-layer patching to identify which residual stream representations at the final instructional token ( $T_{inst}$ ) are necessary to recover the correct answer.
Superadditivity Testing: They compare the causal effect of individual layers versus combinations of layers to determine if the interaction is additive or non-linear.
Locally-Linear Surrogate Maps: To bypass the limitations of traditional circuit discovery (which assumes additive, Directed Acyclic Graph-based interactions), they propose a novel method. They treat the Transformer as a collection of token-to-token locally-linear maps by "linearizing" non-linear components (MLPs and LayerNorm) within an $\epsilon$ -neighborhood of the input.
Path Tracing: They trace high-ranking causal paths from specific token positions to the output to determine the structural flow of information.
Geometric Analysis: They use Linear Discriminant Analysis (LDA) on 200 semantic rephrasings of instructions to visualize the task-specific clusters in the representation space.

3. Key Contributions

Discovery of Instruction Vectors (IVs): The identification of localized "neural digests" of instructions that are formed eagerly at the end of the instruction segment.
Challenge to the Linear Representation Hypothesis: The finding that while IVs are linearly separable, their causal interaction is superadditive (non-linear), meaning the combined effect of layers is greater than the sum of their individual parts.
Circuit Selector Mechanism: The proposal that IVs do not perform the task themselves but act as "circuit selectors," conditioning later layers to activate specific information-processing pathways tailored to the task.
Novel Interpretability Framework: A method for tracing information flow that is free from the implicit additive assumptions of standard patching-based techniques.

4. Key Results

Eager Computation: IVs are localized at the final token of the instruction ( $T_{inst}$ ). In complex tasks (e.g., "Can an animal fly?"), the model may even form sub-instruction digests earlier in the sequence.
Superadditivity: T-tests across multiple tasks and model scales confirmed that the interaction between layers is non-linear ( $f(x_i \cup x_j) \geq f(x_i) + f(x_j)$ ). This invalidates standard DAG-based causal models for instruction following.
Circuit Selection: Path tracing revealed that while early layers use instruction-agnostic circuits to build the IV, the IV then "selects" different attention heads and pathways in later layers to solve the specific query.
Linear Separability: Despite non-linear causal interactions, the IVs remain linearly separable in high-dimensional space, allowing linear probes to identify tasks with $>98\%$ accuracy.
Post-training Impact: While base models can form task-specific digests, instruction tuning (SFT/DPO) is essential to imbue those digests with the ability to select the correct downstream circuits.

5. Significance

This work has profound implications for both the understanding of LLMs and the field of mechanistic interpretability:

Interpretability Theory: It warns that single-component causal attribution (e.g., "this head does X") may significantly undervalue a component's importance if its influence is synergistic with other components.
Model Robustness: The discovery of an "information bottleneck" (the localized IV) suggests potential avenues for increasing robustness against adversarial instruction perturbations.
Alignment and Safety: Understanding how instructions are processed is a prerequisite for ensuring that models follow safety constraints and remain accountable during deployment.