🧬 biology

Features have life history. And we should care

This paper reveals that language models possess a persistent "carrier scaffold" of approximately 50 sparse features that assemble rapidly within the first 1% of training, serving as a load-bearing foundation that dictates the model's subsequent representational structure and development.

Original authors: Philipp Stecher, Sandro Radovanović, Vlasta Sikimić, Reinhard Kahle

Published 2026-05-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Philipp Stecher, Sandro Radovanović, Vlasta Sikimić, Reinhard Kahle

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ⚕️ This is an AI-generated explanation of a preprint that has not been peer-reviewed. It is not medical advice. Do not make health decisions based on this content. Read full disclaimer

Imagine training a large language model (like the ones that write essays or chat with you) is like building a massive, bustling city from scratch. Usually, when we look at the finished city, we see a stable map of roads and buildings. We assume everything grew up together at the same pace.

But this paper argues that the city has a hidden life story. Some parts of the city were built in the very first hour and never changed; they became the unshakeable foundation. Other parts arrived later, were built on top of that foundation, and are much more fragile.

Here is the story of that foundation, told through the paper's findings:

1. The "Carrier Scaffold": The City's Skeleton

The researchers discovered that inside these AI models, there is a tiny group of about 50 special "features" (think of them as the city's main utility lines: water, electricity, internet).

They appear early: These 50 features show up almost immediately, within the first 1% of the training time.
They stick around: Unlike other features that pop in and out like temporary construction crews, these 50 survive until the very end.
They hold everything up: The authors call this group the "Carrier Scaffold." It's the skeleton of the model. If you remove these specific 50 features, the model's performance crashes much harder than if you remove any other 50 random features.

2. The Two-Phase Construction Project

The paper suggests the training process happens in two distinct phases, like a construction project with a strict timeline:

Phase 1: The "Selection" (The First 1%)
Imagine a chaotic construction site where thousands of workers are trying to build walls. In the first few minutes, the "good" workers (the future Scaffold) are identified. They are the ones who show up early, work on the most important tasks, and don't get fired. By the time the first 1% of the work is done, the skeleton is already set. The model has essentially "decided" which 50 features will be the leaders.
Phase 2: The "Calibration" (The Remaining 99%)
For the rest of the training, the model isn't building new skeletons. Instead, it's just fine-tuning the paint and the wiring around that fixed skeleton. It's polishing the geometry and making the existing leaders work better, but it's not changing who the leaders are.

3. The "Crystal Ball" Effect

One of the most surprising findings is that you can predict who the leaders will be before the model is even good at its job.

The researchers looked at the model at the very beginning (Step 1k). At this point, the model is terrible at predicting text.
However, they could look at which features were firing (activating) the most broadly and consistently.
Using just that early data, they built a "crystal ball" (a simple predictor) that could correctly identify 80% of the future leaders before the model had learned anything useful.
The Lesson: The model's "function" (what it does) is decided first. Its "direction" (how it connects to the final answer) is just polished later.

4. The Family Tree of Features

The Scaffold doesn't just sit there; it acts like a parent to the rest of the model.

As training continues, new features are born.
The paper found that about 64% of all the features in the final model are "descendants" of those original 50 leaders. They are organized in a hierarchy, like a family tree, where the early leaders are the grandparents, and the newer, more specific features are the children and grandchildren.
The original 50 features are the "roots" that hold the whole structure together.

5. Why This Matters (The "Why Should We Care?" Part)

The authors say we should care because our current way of studying AI is like taking a photo of the finished city and trying to figure out how it was built.

The Problem: If you just look at the finished model, you can't tell the difference between a feature that was a leader from day one and a feature that arrived late. They look the same in the photo.
The Solution: By tracking the "life history" (who was born when and who survived), we can see the real load-bearing structure.
The Surprise: The features that carry the most weight (the Scaffold) are often the ones that look like boring, technical "infrastructure" (like handling byte codes or file formats) rather than the "smart" features we expect (like understanding emotions or grammar). The "smart" features are actually just the decorations built on top of this boring, unshakeable foundation.

Summary Analogy

Think of training an AI like growing a giant oak tree.

The Scaffold: The deep taproot and the main trunk. These are established in the first few days of the seedling's life. They are hard to change and carry the weight of the whole tree.
The Rest of the Tree: The branches, leaves, and twigs. These grow later, branching off the main trunk. They are numerous and specific, but they rely entirely on the trunk for support.
The Paper's Insight: If you want to understand the tree, don't just look at the leaves (the final snapshot). You need to look at the roots (the life history). The roots were chosen very early, and the rest of the tree just grew around them.

In short: The AI's "brain" isn't built evenly. It picks its most important parts in the first few minutes of training, and spends the next 99% of its time just learning how to use them better.

Technical Summary: Feature Life History and the Carrier Scaffold in Language Model Training

Problem Statement
Current mechanistic interpretability largely relies on static "snapshots" of trained models, treating features as fixed entities at convergence. This approach obscures the dynamic processes of feature emergence, persistence, and death that occur during training. The paper posits that features possess a "life history"—tracking their firing identity (what they respond to) and decoder direction (how they influence predictions) over time—which is critical for understanding the model's internal structure. Specifically, the authors investigate whether a persistent subset of features forms a structural backbone that organizes the model's representational space, and whether this structure is determined early in training or evolves uniformly throughout.

Methodology
The study employs Sparse Autoencoders (SAEs) to decompose residual stream activations into interpretable features across two Pythia model scales (160M and 410M parameters). The methodology involves:

Longitudinal Tracking: SAEs are trained at seven checkpoints (steps 0, 1k, 10k, 40k, 80k, 120k, 143k) across four residual-stream layers per scale.
Feature Matching: Features across checkpoints are matched using Hungarian assignment based on activation profile correlation ( $r \ge 0.5$ ). This allows the construction of "lineages" tracking birth, death, and stability.
Life History Metrics: The authors define "persistent carriers" as features whose lineages remain stable in at least 4 of the 6 post-initialization snapshots.
Structural Analysis: A cross-layer Directed Acyclic Graph (DAG) is constructed using the Containment Index (CI), which measures the fraction of tokens where a feature at a lower layer fires as a subset of a feature at a higher layer.
Load Measurement: Instead of per-feature ablation, the authors use joint cross-layer ablation, simultaneously zeroing out populations of features across all instrumented layers to measure the aggregate increase in cross-entropy ( $\Delta CE$ ).
Predictive Modeling: A logistic regression model trained on step-1k firing-space properties is used to predict which features will become persistent carriers by step 143k.

Key Contributions

Feature Life History as a Unit of Analysis: The paper operationalizes the tracking of features across training checkpoints, distinguishing between features that are broadly active due to noise versus those that are broadly active due to stable, load-bearing roles.
Identification of the Carrier Scaffold: The authors identify a small population of $\sim50$ sparse features (51 in Pythia-160M, 49 in Pythia-410M) that form a "carrier scaffold." These features emerge early, persist throughout training, and organize the majority of the model's representational structure.
Four Properties of the Scaffold:
- Early Assembly: The scaffold assembles rapidly in the first 1% of training (Phase I). Feature birth rates collapse $\sim40\times$ in this phase, and the scaffold is largely fixed before the model can predict competently.
- Load-Bearing Nature: Joint ablation of persistent carriers incurs significantly higher cross-entropy costs ( $\sim0.90$ nats for 160M) compared to count-matched non-persistent roots or leaves. This load gap is invisible to standard per-firing single-feature ablation methods.
- Function Precedes Direction: The identity of future carriers is predictable from firing patterns at step 1k (AUC 0.80–0.83) before decoder directions have stabilized. This suggests selection determines the scaffold's membership early, while gradient descent later calibrates the geometry around this fixed substrate.
- Seeding Development: By convergence, the scaffold recruits 64% (160M) and 53% (410M) of all active features into a hierarchical structure defined by token-set containment.

Results

Training Dynamics: Training is non-uniform. Phase I (steps 0–1k) is characterized by high turbulence and rapid feature turnover. By step 1k, 73% of eventual carriers in the 160M model are already present.
Survival Advantage: Features with high firing entropy at step 1k have a massive survival advantage (43% survival to step 143k) compared to low-entropy features (0.6%).
Ablation Asymmetry: The load-bearing capacity of the scaffold is only detectable via joint cross-layer intervention. Single-feature ablation shows no significant difference in importance between root, mid, and leaf tiers, masking the scaffold's true role.
Semantic Content: The scaffold includes both semantically interpretable features (discourse markers, morphological endings) and "byte-boundary infrastructure" features (Unicode replacement characters, file extensions). Surprisingly, the infrastructure features carry the majority of the ablation load, while the semantically legible features carry near-zero load in isolation, though both are part of the scaffold hierarchy.
Predictability: A predictor trained solely on step-1k firing properties (entropy, CI, activation strength) successfully identifies future carriers with high enrichment (9.3 $\times$ over base rate), confirming that the scaffold is determined by early functional properties rather than late-stage geometric calibration.

Significance and Claims
The paper argues that understanding feature life history is essential for interpretability, debugging, and safety. The primary claim is that the model's representational structure is not formed uniformly but is established through a two-phase process:

Selection (Phase I): The first 1% of training acts as a selector, determining which features survive to form the load-bearing scaffold.
Calibration (Phase II/III): The remaining 99% of training calibrates the decoder directions and geometry around this pre-existing substrate.

The authors claim that interpretability methods starting only at convergence may be analyzing the product of this process rather than its causes. By identifying the scaffold early, it becomes possible to:

Triage feature populations before the model is fully trained.
Distinguish between load-bearing structural elements and transient or non-load-bearing features.
Provide a measurable substrate for theories regarding critical learning periods and the lottery ticket hypothesis.

The paper concludes that the scaffold is largely fixed by selection in the first 1% of training, and the subsequent training primarily refines the geometry around this established foundation. This challenges the view that model structure emerges gradually and uniformly, suggesting instead that the "skeleton" of the model is determined almost immediately.