💬 NLP

Learning Self-Interpretation from Interpretability Artifacts: Training Lightweight Adapters on Vector-Label Pairs

This paper demonstrates that training lightweight, frozen adapters on interpretability artifacts enables large language models to generate reliable, high-quality self-interpretations across tasks and scales without modifying the underlying model.

Original authors: Keenan Pepper, Alex McKenzie, Florin Pop, Stijn Servaes, Martin Leitgab, Mike Vaiana, Judd Rosenblatt, Michael S. A. Graziano, Diogo de Lucena

Published 2026-06-03✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Keenan Pepper, Alex McKenzie, Florin Pop, Stijn Servaes, Martin Leitgab, Mike Vaiana, Judd Rosenblatt, Michael S. A. Graziano, Diogo de Lucena

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a giant, incredibly complex library. Inside this library, information isn't stored in books you can read; it's stored in a massive, invisible web of electrical signals (activations) that fire when the model "thinks."

For a long time, researchers have tried to peek behind the curtain to see what these signals mean. They've built tools to map these signals to concepts (like "math" or "politeness"), but the tools are finicky. If you tweak a dial even slightly, the model might start speaking fluent nonsense that sounds like an explanation but isn't actually connected to what it's thinking.

This paper introduces a new way to fix that. Here is the core idea, broken down with simple analogies:

1. The Problem: The "Translator" is Broken

Imagine the model's internal signals are a secret code. Previously, researchers tried to translate this code by just shoving it directly into the model's mouth and asking, "What does this mean?"

The Issue: Sometimes the model understands, but often it gets confused. It might say, "This signal means 'a small hill of dirt'" when it actually means "a computer function." It's like trying to translate a foreign language by guessing the words; you get the grammar right, but the meaning is wrong.

2. The Solution: Training a "Lightweight Adapter"

Instead of trying to retrain the whole giant library (which is expensive and changes how the library works), the authors trained a tiny, specialized adapter.

The Analogy: Think of the adapter as a specialized translator or a pair of glasses.
- The giant library (the model) stays exactly the same; it doesn't learn anything new.
- The adapter is a tiny add-on (like a lens) that sits in front of the model's eyes.
- The authors trained this lens using "interpretability artifacts." These are like flashcards the researchers already made: one side has a signal vector, and the other has a label (e.g., "This signal = 'Baseball'").
- The adapter learns how to adjust the signal so the model can finally "see" what it's thinking and describe it correctly.

3. The Magic Ingredient: The "Bias" (The Default Setting)

The most surprising finding is that the adapter doesn't need to be a super-computer. A very simple version works best.

The Analogy: Imagine the adapter has two parts:
1. The Signal: The specific thing the model is thinking about right now (e.g., "Plato").
2. The Bias: A "default setting" or a mental habit the adapter learned.
The paper found that this "bias" does about 85% of the heavy lifting. It teaches the model the style and format of the text it generates in its explanations.
The Proof: The authors tested this by training the adapter on labels written in ALL CAPS. When they used it later, the model's generated explanations came out in ALL CAPS too! The only thing that could carry that style information was the bias vector. (The prompt itself, which asks "What does this mean?", stays normal because it is supplied by the researcher, not generated by the model).
It's like teaching a student the voice and tone of a specific writer so well that they can write in that exact style on any topic, as long as you give them the topic name.

4. What They Discovered

Better than the Original Labels: The adapters didn't just copy the training flashcards; they actually got better at describing the signals than the original labels the researchers wrote. It's like a student learning from a textbook and then writing a better summary than the textbook author.
Uncovering Hidden Thoughts: The model can sometimes solve a puzzle without saying its steps out loud (e.g., answering "Athens" to a riddle about Plato without mentioning Plato). The trained adapter can "read the model's mind" and reveal that hidden "Plato" thought, even though the model never said it.
Bigger Models = Better Translators: As the models get bigger (from 7 billion to 72 billion parameters), the adapter gets even better at translating these thoughts. The model doesn't just get smarter at answering questions; it gets better at explaining how it thinks.

5. Why This Matters (According to the Paper)

The paper argues that we don't need to change the model itself to understand it. By just adding this tiny, frozen "translator" trained on existing data, we can get a reliable window into the model's internal state.

The "Self-Interpreting" Benefit: The key advantage of keeping the explainer frozen (meaning it is the same unchanged model) is that the model interprets itself. Its inner workings are exactly compatible with the thoughts of the model being explained.
- This opens the door to recursive self-examination: the model can examine an explanation of its own thoughts, then examine its thoughts about that explanation, and so on.
- As the author puts it: "Crucially, the add-on feeds these patterns back into the same model, so the model interprets itself. This opens the door to future AI that can examine its own thinking from the inside."
Scalability: This method works across different types of models (Llama, Gemma, Qwen) and gets better as models get larger.

In short: The paper shows that if you give a language model a tiny, trained "translator" based on existing maps of its brain, it can reliably tell you what it's thinking, even for complex or hidden thoughts, without needing to be retrained or altered.

Technical Summary: Learning Self-Interpretation from Interpretability Artifacts

1. Problem Statement

Large Language Models (LLMs) operate through high-dimensional hidden activations, yet understanding the semantic content of these internal states remains a central challenge for interpretability. While mechanistic interpretability has produced structured knowledge (e.g., Sparse Autoencoder (SAE) features and contrastive activation vectors), models cannot reliably report on their own internal states.

Existing self-interpretation methods, such as those using activation patching (e.g., SelfIE, Patchscopes), attempt to inject internal representations into prompts to generate natural language explanations. However, these methods are fragile:

Hyperparameter Sensitivity: Small changes in the scale of injected vectors can produce fluent but semantically ungrounded explanations.
Fragility: Most vectors have narrow valid ranges for injection scales; outside these ranges, the model generates confident but incorrect descriptions.
Limitations of Fine-Tuning: Recent approaches that fine-tune models to answer questions about activations alter the model's weights, potentially changing the very representations under study. Ideally, the interpreter should remain identical to the subject model.

2. Methodology

The authors propose training a lightweight adapter on existing interpretability artifacts to map internal activations to language, while keeping the base Language Model (LM) entirely frozen.

2.1. Training Framework

Input: The method utilizes "interpretability artifacts" as supervision data, specifically vector-label pairs $(h, y)$ $(h, y)$ .
- Source 1: SAE decoder vectors paired with auto-interpretability labels.
- Source 2: Contrastive activation vectors (derived from prompts like "Tell me about [topic]") paired with synthetic topic descriptions.
Process:
1. An activation vector $h$ is extracted from a source prompt (e.g., a specific layer of the LM).
2. A lightweight adapter function $f(h)$ transforms this vector.
3. The transformed vector is injected into a "target prompt" (an explanation-seeking template) at the token embedding layer (layer 0).
4. The frozen LM generates a description autoregressively.
5. Objective: Minimize cross-entropy loss between the generated tokens and the ground-truth label $y$ . Only the adapter parameters are updated; the LM remains frozen.

2.2. Adapter Architectures

The paper evaluates adapters with varying expressivity:

Identity: $f(h) = h$ (0 parameters)
Scale-only: $f(h) = \alpha \cdot h$ (1 parameter)
Scalar Affine: $f(h) = \alpha \cdot h + b$ ( $d_{model} + 1$ parameters)
Scalar Affine + Low-Rank: Adds a low-rank term $UV^T h$
Full-Rank Affine: $f(h) = Wh + b $($ d_{model}^2 + d_{model}$ parameters)

2.3. Inference

At inference, the trained adapter maps an unseen activation vector into the token embedding space. To address scale sensitivity, the system evaluates multiple injection scales (from a logarithmic grid) and selects the best-performing candidate generation.

3. Key Contributions and Results

3.1. Superiority of Trained Adapters

Training lightweight adapters significantly outperforms untrained baselines (SelfIE) and the original training labels themselves.

SAE Feature Labeling: On Llama-3.3-70B, trained adapters achieved a 70% generation scoring hit rate, outperforming the original training labels (50%) and untrained SelfIE (48%).
Topic Identification: On contrastive activation vectors, recall@1 improved from ~1% (untrained) to >90% (trained).
Implicit Reasoning: The method successfully decoded "bridge entities" in multi-hop reasoning tasks (e.g., identifying "Plato" in a query about the author of The Republic without the model verbalizing the intermediate step). Trained adapters detected bridge entities in 91% of cases versus 56% for untrained baselines.

3.2. The Critical Role of the Bias Vector

A surprising finding is that a scalar affine adapter with only $d_{model} + 1$ parameters suffices for most gains.

The learned bias vector ( $b$ ) alone accounts for approximately 85% of the improvement over untrained baselines.
The bias acts as an "interpretation prior," steering the model toward valid description formats and generic content patterns, while the scaled input vector provides instance-specific semantics.
Generalization: Simpler adapters (Scalar Affine) generalize better across datasets and layers than more expressive alternatives.

3.3. Architecture vs. Representation Geometry

The performance of adapter architectures depends heavily on the intrinsic dimensionality of the training data:

Contrastive Vectors (Low Dimensionality): Wikipedia topic vectors concentrate >90% of variance in ~200 dimensions. Here, Full-Rank adapters succeed and achieve the best performance without overfitting.
SAE Features (High Dimensionality): SAE features span nearly the full activation space. Here, Full-Rank adapters catastrophically overfit, learning a high-dimensional lookup table. Scalar Affine or Scalar Affine + Low-Rank adapters are required to prevent overfitting and ensure generalization.

3.4. Scaling Behavior

The paper demonstrates that self-interpretation improves with model scale, independent of general capability gains.

Using a "Taboo" baseline (where the model describes a topic without naming it) as a ceiling for knowledge, the gap between the model's knowledge and its ability to report that knowledge narrows as model size increases (from 7B to 72B parameters).
Trained SelfIE performance grows faster than the model's raw capability to describe topics, suggesting that larger models possess more accessible internal semantic structures that the adapter can unlock.

3.5. Cross-Dataset and Cross-Model Generalization

Adapters trained on one dataset (e.g., Wikipedia topics) generalize to others (e.g., SAE features), though performance is highest when training and inference data distributions match.
The approach works across model families (Llama, Gemma, Qwen) without requiring model-specific SAEs if contrastive vectors are used.

4. Significance and Claims

The paper claims that reliable self-interpretation can emerge from lightweight transformations without modifying the underlying model.

Reframing Artifacts: The core insight is that interpretability artifacts (labeled vectors), traditionally viewed as endpoints of analysis, can be repurposed as training data. This allows self-interpretation systems to improve automatically as interpretability research progresses.
Preservation of Privileged Access: By keeping the base model frozen, the method preserves the "privileged access" of the interpreter to the subject model's internal states, avoiding artifacts introduced by fine-tuning.
Verifiability: The approach enables "RL from internal rewards," where a model's claims about its internal states can be verified against behavior (e.g., via generation scoring), providing a path toward models that offer verifiable evidence of their own internals.
Efficiency: Training these adapters is computationally cheap (e.g., ~10 GPU-hours at 70B scale), making it a scalable solution for auditing frontier models.

The authors conclude that while self-interpretation provides partial visibility, it offers a crucial mechanism for detecting when models encode concepts or intentions that differ from their expressed outputs, contributing to AI safety and alignment auditing.