🤖 AI

Decomposing how prompting steers behavior

This paper introduces a nested geometric decomposition framework that reveals how prompting steers large language and vision-language models by transforming their internal representations, demonstrating that affine transformations—specifically cross-dimensional linear mixing—are the key mechanism for reorganizing representations to match instructed task structures and recover target behaviors.

Original authors: Fan L. Cheng, Nikolaus Kriegeskorte

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Fan L. Cheng, Nikolaus Kriegeskorte

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) or a Vision-Language Model (VLM) as a giant, complex factory. Inside this factory, raw materials (like a picture of a cat or a sentence about a sad story) flow through a series of conveyor belts (the layers of the model). As they move, the materials get processed, reshaped, and organized.

Usually, to change what the factory produces, you have to rebuild the machines (retrain the model). But prompting is like a manager walking onto the factory floor and shouting a new instruction: "Stop sorting by color, now sort by size!" The machines don't change, but the output suddenly shifts.

This paper asks a simple but deep question: How does a shout from the manager actually rearrange the items on the conveyor belt to make them sort differently?

Here is the breakdown of their discovery, using everyday analogies:

1. The Experiment: The "Magic Map"

The researchers took a set of items (like 1,000 different images) and showed them to the model with two different instructions:

Prompt A: "Are there people in this picture?" (Yes/No answer).
Prompt B: "How many people are in this picture?" (A number answer).

They looked at the "internal state" of the model (the arrangement of the items on the conveyor belt) at different depths. They found that the items were in a different shape for Prompt A than for Prompt B.

They then asked: "Can we draw a simple map that turns the 'Yes/No' arrangement into the 'Count' arrangement?"

2. The Five Levels of "Maps" (The Nested Decomposition)

To find the answer, they tried five different types of mathematical "maps" (transformations), starting from the simplest and getting more complex. Think of these as different ways to rearrange a pile of books:

Translation (The Slide): Just sliding the whole pile of books to a new spot on the table without touching the books themselves.
Rigid + Uniform Scaling (The Rigid Move): Moving the pile and maybe making the whole thing slightly bigger or smaller, but keeping the books in the exact same shape relative to each other.
Rigid + Axis Scaling (The Stretch): Moving the pile and stretching it in specific directions (like pulling a rubber sheet horizontally but not vertically).
Affine (The Mix): Moving, stretching, and mixing the books. This is like taking the "top" book and blending it with the "side" book to create a new position.
Nonlinear (The Warp): Bending the table itself, twisting the books in complex, unpredictable ways.

3. The Big Discovery: It's Mostly a "Mix"

The researchers found that:

The "Slide" (Translation) does a lot of the heavy lifting. Just moving the internal state to a new spot explains a huge chunk of the change.
But the "Mix" (Affine) is the secret sauce. While sliding helps, the model really needs to mix the internal features (cross-dimensional linear mixing) to fully understand the new task.
The "Warp" (Nonlinear) isn't necessary. Surprisingly, they didn't need complex, twisting math to explain the change. Once they added the "Mix" (Affine), the model's behavior was almost perfectly recovered. The complex, non-linear stuff didn't add much value.

The Analogy: Imagine you have a box of LEGOs built into a castle (Prompt A). You want to turn it into a spaceship (Prompt B).

You don't need to melt the plastic (Nonlinear).
You don't just need to slide the box across the room (Translation).
You need to take the castle apart, stretch some pieces, and rearrange/mix the bricks into a new structure. The paper found that this "rearranging and mixing" is the core mechanism of how prompts work.

4. The "Causal" Test: Does the Map Actually Work?

To prove this wasn't just a coincidence, they did a "surgery" on the model:

They fed the model an image with the "Yes/No" prompt.
At a specific layer, they stopped the process.
They took the internal state and applied their "Mixing Map" to it.
They let the model finish processing.

The Result: The model suddenly started giving "Count" answers, even though it was still technically being asked the "Yes/No" question! This proved that the geometric transformation (the map) causes the behavior change.

5. Different Models, Different Strategies

Just like different factories have different managers, different AI models use different strategies:

OPT-2.7B seems to rely heavily on "stretching" and "mixing" (complex transformations) early on.
Llama3 and Qwen3 seem to rely more on a simple "slide" (translation) early on, perhaps using a specific "instruction code" that just moves the data to the right place.

Summary

The paper concludes that when you give an AI a new instruction, it doesn't need to fundamentally rewrite its brain or perform complex, twisting math. Instead, it mostly slides its internal understanding to a new location and mixes its features together in a straight-line way (Affine transformation) to fit the new task.

This gives us a clear, geometric picture of how AI models "listen" to instructions: they are essentially performing a sophisticated, multi-step rearrangement of their internal data, and we can now measure exactly how much of that rearrangement is just a simple slide versus a complex mix.

Technical Summary: Decomposing how prompting steers behavior

Problem Statement

Instruction prompts enable Large Language Models (LLMs) and Vision-Language Models (VLMs) to alter their behavior at inference time without weight updates. While it is established that prompts restructure internal representations, the specific mechanistic nature of this transformation remains unclear. Existing literature has characterized these changes through scalar metrics (e.g., representational similarity, manifold capacity) or demonstrated that engineered low-complexity operators (e.g., additive vectors, rotations) can steer behavior. However, there is a gap in understanding the class of mathematical transformation that naturally occurs when an instruction changes. Specifically, it is unknown whether prompt-induced changes are best explained by simple translations, rigid transformations, affine mixing, or complex nonlinearities, and whether these transformations causally drive the observed behavioral shifts.

Methodology

The authors introduce a nested geometric decomposition framework to analyze how prompts reshape representational geometry. The core approach involves treating prompting as a transformation of the representational manifold for content following the prompt.

1. Nested Geometric Hierarchy

For a given model, layer, dataset, and pair of prompts (Source Prompt $A$ and Target Prompt $B$ ), the authors fit a sequence of increasingly expressive, stimulus-invariant maps $f: \mathbb{R}^D \to \mathbb{R}^D$ that align the hidden states of $A$ to those of $B$ . The hierarchy consists of five strictly nested transformation classes:

Translation ( $F_T$ ): A simple mean shift ( $x + b$ ).
Rigid Transformation with Uniform Scaling ( $F_{O_u}$ ): Rotation/reflection and uniform scaling ( $b + c(x-a)Q$ ).
Rigid Transformation with Axis-wise Scaling ( $F_{O_a}$ ): Allows independent scaling of aligned axes ($b + (x-a)QD$).
Affine Transformation ( $F_L$ ): General linear mixing plus translation ( $b + (x-a)M$ ).
Nonlinear Transformation ( $F_N$ ): A shared Multi-Layer Perceptron (MLP) ( $b + g_\theta(x-a)$ ).

2. Evaluation Metrics

The framework evaluates these transformations using three families of measures on held-out stimuli:

Incremental Explained Variance ( $\Delta R^2$ ): The additional variance in the target prompt's representations explained by each tier over the previous one in the nested chain.
Representational Geometry Metrics: Spearman correlation between the transformed Representational Dissimilarity Matrix (RDM) and the target task's RDM, alongside silhouette scores to measure cluster separation.
Causal Intervention: The fitted map is applied to the hidden state of a source prompt at a single layer, replacing the original state. The model then continues the forward pass. The resulting output is evaluated for relevance (addressing the target attribute) and accuracy (correctness of the answer) against the target prompt's ground truth.

3. Experimental Setup

Models: Three LLMs (OPT-2.7B, Llama3-8B-Instruct, Qwen3-8B) and three VLMs (BLIP-2, LLaVA-OneVision, Qwen3-VL).
Datasets: Six datasets (three text: EmotionalStory, WritingStyle, Number; three image: EmoSet, StyleTransfer, COCO) with factorial attribute structures to support cross- and within-attribute prompt comparisons.
Prompt Pairs: Six groups ranging from cross-attribute (e.g., topic $\to$ emotion) to irrelevant source prompts.

Key Results

1. Prompting Reshapes Geometry Toward Task Structure

Prompt-induced representational changes grow nearly monotonically with layer depth. Crucially, the representational geometry of the target prompt aligns increasingly well with the task structure (measured by RDM correlation and silhouette scores) as depth increases, separating clearly from the source prompt in deeper layers.

2. Low-Complexity Maps Explain Most Variance

Across all models and datasets, translation accounts for the largest share of the prompt-induced representational change, dominating in early layers. Rigid transformations with uniform scaling provide the next significant contribution.

Affine transformations are the first tier to nearly recover the target prompt's task geometry and produce corresponding gains in behavioral agreement.
Nonlinear transformations provide the strongest recovery but offer diminishing returns over affine maps; the jump from axis-scaled rigid to affine is more significant than the jump from affine to nonlinear.

3. Model- and Task-Specific Routing Strategies

Model Differences: OPT-2.7B relies more heavily on rotation, reflection, and nonlinear components compared to Llama3 and Qwen3, which appear to use lower-dimensional "instruction codes" (dominated by translation).
Prompt Pair Differences: Specific prompt pairs (e.g., yes/no queries) rely more on rotation and affine transformations than open-ended pairs.
Depth Dependence: Translation dominates early layers, while higher-complexity tiers (affine, nonlinear) contribute more in mid-to-deep layers.

4. Causal Validation

Causal interventions confirm that simply translating the hidden state can substantially steer behavior. However, affine transformations are the critical threshold for recovering both the target representational geometry and high behavioral accuracy. This suggests that cross-dimensional linear mixing is a key mechanism for reorganizing representations toward the instructed task.

Significance and Claims

The paper claims to provide a general framework for decomposing prompt-induced representational changes into interpretable geometric components. The primary finding is that the effect of instruction prompting can be largely understood as an affine transformation of the content's representation.

Mechanistic Insight: The authors argue that while translation does "most of the work" in shifting the representation, rotation and shearing (affine components) fix the residual structured component necessary for task alignment.
Interpretability: By quantifying the contribution of each transformation tier, the framework reveals how models route task-relevant structure. It suggests that natural prompting is implemented through low-complexity transformations in activation space, rather than requiring complex nonlinear restructuring.
Limitations: The authors explicitly note that their analysis is limited to zero-shot instruction prompts and does not evaluate few-shot demonstrations, soft prompts, or longer-context conditioning. The intervention is restricted to a single layer and token.

The work positions itself not as a method to engineer new steering operators, but as a diagnostic tool to understand the geometric nature of natural prompting, suggesting that the operational consequences of prompting are concentrated in the residual fine structure (rotation and small nonlinear parts) after the primary translation.