💬 NLP

Scaling Inherently Interpretable Language Models

This paper challenges the notion that interpretability is a trade-off for capability by demonstrating that integrating interpretability constraints into the training pipeline allows models like Steerling-8B to scale effectively, becoming more transparent and aligned with human concepts while maintaining competitive performance.

Original authors: Guide Labs Team, Andreas Madsen, Aya Abdelsalam Ismail, Giang Nguyen, Isaac Plant, Muawiz Chaudhary, Nathaniel Monson, Saqib Azim, Zhichen Guo, Julius Adebayo

Published 2026-08-11

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Guide Labs Team, Andreas Madsen, Aya Abdelsalam Ismail, Giang Nguyen, Isaac Plant, Muawiz Chaudhary, Nathaniel Monson, Saqib Azim, Zhichen Guo, Julius Adebayo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a giant, magical library works. For years, the smartest librarians (AI researchers) have been building these libraries to be as big and powerful as possible, but they've accepted a strange rule: to make the library truly powerful, it has to be a "black box." You can ask it a question, and it will give you a brilliant answer, but you have no idea how it found that answer. It's like a wizard who casts a spell and says, "Trust me, it works," without showing you the ingredients or the incantation. Because of this, if the library makes a mistake or says something weird, we can't easily fix it; we can only guess why it happened after the fact. This paper asks a bold question: What if we didn't have to accept that trade-off? What if we could build a library that is both a super-genius and transparent, where we can see exactly which books it used and which ideas it was thinking about while it wrote its answer?

The authors of this paper, a team called Guide Labs, decided to test this idea. They built a new kind of language model called Steerling-8B. Unlike the usual "black box" models that are trained to just guess the next word in a sentence, Steerling was trained with a special rule baked into its brain from day one: it must explain its own thinking as it goes. They found something surprising: making the model explain itself didn't make it weaker. In fact, as they made the model bigger and gave it more computing power, it didn't just get smarter; it actually got better at explaining itself. The bigger the model became, the clearer its internal thoughts became, and the easier it was to see exactly which ideas it was using to form its answers.

The Problem with "Black Box" Magic

For a long time, the standard way to build AI has been to train a model to be as good as possible at predicting text, and then, after it's finished learning, try to peek inside to see what it's doing. Researchers use tools like "probes" (which try to guess if a concept is hidden inside the model) or "attribution" (which tries to guess which words mattered most). But the authors argue these tools are like trying to figure out how a car engine works by listening to the noise it makes after it's already been built. The engine wasn't designed to be listened to, so the noise might be misleading. A probe might say, "Hey, the word 'cat' is in there!" but that doesn't mean the model actually used the idea of a cat to make its decision. It might just be a coincidence.

The paper argues that this "fix it later" approach is fundamentally flawed. If you want a model that is truly understandable, you can't just add a flashlight after the fact; you have to build the flashlight into the engine while you're building the engine.

The Recipe: Building a Transparent Brain

To solve this, the team created a new "recipe" for training AI. Instead of just teaching the model to predict the next word, they taught it three specific things at the same time:

Input Attribution: "Which words in your question mattered most?"
Concept Attribution: "What big ideas or topics are you thinking about right now?"
Data Attribution: "Which parts of your training library did you learn this from?"

To make this work, they had to build a massive new library of "concepts." Imagine a dictionary, but instead of just words, it contains 33,000 specific ideas like "gradient descent," "medieval poetry," or "sarcasm." They built a system called Atlas that read millions of documents and tagged them with these ideas, creating a map of human knowledge that the AI could actually use.

Then, they built the model's brain, Steerling-8B, with a special "bottleneck" in the middle. Think of a normal AI as a straight pipe where water (information) flows from the input to the output. Steerling has a filter in the middle. Before the water can flow out, it must pass through a set of labeled buckets (the concepts). The model has to decide, "Okay, this sentence is 40% about 'math' and 20% about 'history'." Because the model has to use these buckets to make its prediction, we can see exactly which buckets it used. If it makes a mistake, we can look at the buckets and say, "Ah, it focused too much on 'history' and ignored 'math'."

The Big Surprise: Bigger is Clearer

The most exciting part of the paper is what happened when they made the model bigger. Usually, when you make a complex system bigger, it gets harder to understand. But here, the opposite happened.

The team tested models ranging from tiny ones to the massive 8-billion-parameter Steerling-8B. They found that as the model grew, its internal "buckets" became more organized and more aligned with human ideas.

The "Tax" Myth: Many people thought that making a model explain itself would cost it performance, like a tax on its intelligence. The paper shows this isn't true. The "cost" of being interpretable is a tiny, fixed amount, like a small fee you pay once, not a bill that gets bigger as you get richer.
Scaling Up: As they added more computing power (about 1.35 trillion words of training data), the model didn't just get smarter; it got more transparent. The concepts it learned became clearer, and it relied less on a "residual" (a hidden, unexplainable part of the brain) and more on the labeled concepts.

Steering the Ship

Because the model is built this way, you can actually "steer" it without retraining it. Imagine you are driving a car. In a normal AI, if you want to avoid talking about violence, you have to hope the car doesn't crash. In Steerling, you can just turn a dial. If you want the model to be more "academic," you can boost the "academic" bucket. If you want to stop it from talking about "politics," you can suppress that bucket. The paper shows that they could do this instantly, changing the model's behavior by simply nudging the concepts it was already using.

The Results

The team trained Steerling-8B on 1.2 trillion tokens (a huge amount of text). They compared it to other open-source models that were trained on 2 to 16 times more computing power. Despite having a smaller training budget and carrying the extra "weight" of being interpretable, Steerling performed almost as well as those massive, opaque models.

The paper concludes that we don't have to choose between a smart AI and a transparent one. By designing the model to be understandable from the start, we can build systems that are not only powerful but also trustworthy, because we can see exactly how they think and fix them when they go wrong. It's a shift from building black boxes to building glass houses where the light shines through.

Technical Summary: Scaling Inherently Interpretable Language Models

Problem Statement

Current large language models (LLMs) are typically trained as opaque systems, with interpretability treated as a post-hoc "tax" on capability. Standard workflows involve training a high-performance model and then attempting to reverse-engineer its behavior using probes, feature attributions, or sparse autoencoders. The authors argue that this approach suffers from a fundamental structural limitation: the model was never trained to make the explanation a valid interface into its computation. Consequently, post-hoc methods often fail to satisfy requirements for faithfulness (the explanation reflects actual reasoning) and validity (interventions are meaningful). Furthermore, there is a prevailing assumption that constraining models to human-meaningful structures necessarily degrades performance, creating a trade-off between capability and interpretability.

Methodology

1. The Inherent Interpretability Recipe

The authors propose a training pipeline where interpretability is a constraint optimized alongside the language modeling objective, rather than an afterthought. This recipe modifies the standard training stages to satisfy five formal conditions for inherent interpretability:

Nativeness: Attributed variables must be part of the computational path producing the output.
Agreement: Attribution scores must predict the effect of interventions (e.g., masking or concept steering) before they are applied.
Validity: Interventions must be in-distribution (the model must have seen the perturbed states during training).
Interpretation: Semantic descriptions (cards) must accurately represent the model's internal variables.
Coverage: The interpretable components must account for a significant fraction of the prediction, minimizing the unexplained residual.

###2. Data: The Atlas Pipeline
A major obstacle was the lack of a human-interpretable concept library at the scale of modern pretraining corpora. To address this, the authors built Atlas, a three-stage automated pipeline:

Stage 1 (Tagging): Samples 6.6 million documents, chunks them, and uses an LLM to generate high-recall, free-form tags.
Stage 2 (Canonicalization): Clusters tags into a coherent library of 33,732 human-understandable concepts, labeled and deduplicated.
Stage 3 (Annotator): Trains a model to assign these concepts to arbitrary text chunks across a 1.5 trillion-token corpus.
The resulting library covers diverse domains (science, code, law, etc.) and is validated to ensure concepts are human-meaningful, not just LLM-fluent.

3. Architecture: Steerling-8B

The authors instantiate this recipe with Steerling-8B, a Causal Diffusion language model featuring an Additive Concept Bottleneck.

Causal Diffusion: Unlike standard autoregressive models, Steerling uses a masked diffusion objective with block-causal attention. This allows for parallel denoising within blocks while retaining autoregressive-style KV caching across blocks for efficient inference. Crucially, the masking objective trains the model on a "no information" baseline ([MASK]), enabling valid input attribution.
Concept Module: Inserted between the transformer backbone and the language modeling head, this module decomposes the hidden state $h$ into three additive components:
$\bar{h} = \hat{k} + \hat{u} + \epsilon$
Where $\hat{k}$ is a weighted sum of known concepts (from the Atlas library), $\hat{u}$ is a weighted sum of unknown concepts (learned during training), and $\epsilon$ is a residual term.
Loss Functions: The training objective combines:
- Language Modeling Loss: Predicts tokens based on the decomposed state $\bar{h}$ .
- Concept Loss: Aligns known concept activations with ground-truth chunk-level annotations.
- Reconstruction Loss: Ensures the unknown head captures information not explained by known concepts.
- Independence Loss: Encourages orthogonality between known and unknown representations to prevent redundancy.

4. Training Dynamics

The authors employ teacher forcing schedules for both known and unknown concept heads. Early in training, ground-truth concept representations are injected to stabilize learning, gradually annealing to the model's own predictions. This prevents the model from relying on the residual channel or learning spurious correlations before the concept heads are reliable.

Key Results

1. Scaling Laws

The authors conducted IsoFLOP scaling sweeps across three orders of magnitude of compute (from 10M to 8B parameters) on both autoregressive and causal diffusion backbones.

Capability: Adding the concept module introduces a small, fixed per-backbone offset in compute-optimal scaling, rather than a growing penalty. The irreducible loss ( $L_\infty$ ) actually decreases for the interpretable variants.
Interpretability: All four interpretability metrics (Concept Loss, Independence Loss, Contribution, and Alignment) improve with scale. Contrary to the assumption that interpretability degrades with size, the models become more disentangled and aligned with human concepts as they scale.
Prediction: Small-scale fits accurately predicted the validation loss of the 8B Steerling model (within 0.11 nats) and bounded its interpretability metrics.

2. Steerling-8B Performance

Steerling-8B was pretrained on 1.2 trillion tokens and mid-trained on 150 billion tokens.

Competitiveness: Despite carrying interpretability constraints, Steerling-8B achieves performance within approximately 10% of open peer models trained on 2–16× more compute.
Capabilities: The model supports closed-loop intervention:
- Attribution: It can trace outputs to input tokens, specific concepts, and similar training data.
- Steering: Users can amplify or suppress specific concepts at inference time without retraining, directly manipulating the concept embeddings used in the forward pass.

Significance and Claims

The paper challenges the premise that interpretability is a tax on capability. Its primary claims are:

Interpretability can be designed into training: By satisfying conditions of faithfulness and validity during the training process, models can provide explanations that are structurally coupled to their computations.
Interpretability scales with capability: Rather than becoming harder to understand as they grow, inherently interpretable models become more disentangled and aligned with human concepts at scale.
Competitiveness is achievable: A model can be both interpretable and competitive with opaque peers trained on significantly more compute, suggesting a new scaling paradigm where understanding is a feature of the architecture, not a post-hoc cost.

The authors conclude that the "Rashomon effect" (where multiple models with identical performance have different internal mechanisms) can be mitigated by anchoring the explanatory interface (concept library, baselines, decomposition) across all models produced by the training procedure.