💻 computer science

When Do Concepts Become Functionally Sufficient During Language-Model Training?

This paper introduces a functional framework to determine when internal concepts become sufficient during language model training by testing whether sparse, masked activations can preserve target information through interventions across layers and checkpoints, revealing that downstream masks retain significantly less soft mass than reconstruction masks despite minimal predictive distribution shifts.

Original authors: Raphael Bernas, Paul G. Chevalier, Fanny Jourdan, Céline Hudelot

Published 2026-08-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Raphael Bernas, Paul G. Chevalier, Fanny Jourdan, Céline Hudelot

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Modern computers that understand language work by transforming words into vast clouds of numbers. As these machines learn, they build internal structures, layers of mathematical pathways that organize information to make predictions. For years, scientists have tried to peek inside these clouds to see what the computer is actually thinking. They have looked for specific patterns or "concepts" that might represent ideas like "politeness" or "math." However, finding a pattern is not the same as understanding its purpose. A structure might exist inside the machine without ever being used to make a decision, much like a tool sitting in a toolbox that is never picked up. The critical question is not just what appears inside the model, but when those internal structures become truly useful for the task at hand.

A team of researchers set out to answer this by watching how these internal structures change from the very beginning of training to the very end. Instead of just looking at the final state of a computer model, they treated the learning process like a movie, pausing it at different moments to test the machine's capabilities. They took a specific layer of the model's brain and tried to isolate small groups of numbers that seemed to carry important information. To do this, they created a method of "masking," which is like placing a semi-transparent screen over the machine's internal activity. This screen could dim or hide certain numbers while letting others pass through. The researchers then asked a simple, rigorous question: if we hide most of the activity and only keep the numbers selected by our screen, does the computer still make the same predictions it did before?

The team tested this idea across seven different language models, ranging from smaller to larger systems, and watched how the answer changed as the models learned. They compared four different ways of judging "usefulness." First, they checked if the remaining numbers could simply rebuild the original cloud of data. Second, they tested if a simple, fixed translator could still read the meaning from the remaining numbers. Third, and most importantly, they checked if the computer's actual behavior—its final predictions on new text—remained unchanged. Finally, they tested if concepts learned at an early stage could still be recognized and used by the model at a much later stage of training.

The results revealed a surprising truth about how these machines learn. When the researchers tried to keep the computer's behavior exactly the same, they found they needed to keep very little information. In fact, for the task of preserving the model's actual predictions, they only needed to retain about 6.6 percent of the internal activity. This is a tiny fraction compared to the amount of data required to simply rebuild the original numbers, which needed over 70 percent of the activity to be preserved. This suggests that the computer stores a massive amount of information, but only a very small, specific slice of it is actually doing the heavy lifting when it comes to making decisions. The rest of the activity, while present, appears to be in directions that the model's decision-making process largely ignores.

The study also showed that this "useful" slice of information is not random. The researchers found that the specific numbers the model relies on for predictions are highly sensitive to the task it is learning. They are concentrated in areas where the model's internal geometry is most curved, meaning these are the directions where small changes have the biggest impact on the final answer. The researchers also discovered that the timing of when these useful structures appear depends heavily on the specific type of model. For some models, the useful information becomes stable and clear as training progresses, while for others, the location of this information shifts significantly through the layers of the network.

Perhaps most importantly, the researchers found that the way a model learns to use these concepts is not a simple, straight line. The information that is useful for the final prediction is often different from the information that is easiest to see or measure. A structure might be very easy to detect and describe, yet contribute almost nothing to the final decision. Conversely, the tiny fraction of information that drives the model's behavior is often hidden within complex patterns that are hard to isolate without this specific testing method. The study concludes that we cannot judge the maturity of a language model by looking at its internal structures in isolation. Instead, we must measure what those structures actually do. The machine may be full of noise and unused potential, but it learns to function by focusing on a remarkably small, efficient core of activity that carries the weight of its understanding.

Technical Summary: When Do Concepts Become Functionally Sufficient During Language-Model Training?

Problem Statement

Understanding the learning mechanisms of neural language models requires more than analyzing their final state; it necessitates identifying when and how internal structures acquire functional utility during training. While concept-based methods (e.g., concept activation vectors, sparse autoencoders) offer a framework for decomposing high-dimensional activations, standard definitions of concepts—based on decorrelation, positivity, independence, or sparsity—define coordinate systems rather than semantic guarantees. A critical gap exists in determining whether these decomposed structures are merely representational or if they are functionally sufficient for the model's predictive behavior. Previous work has often relied on reconstruction fidelity or linear probing, which may not capture the specific information the model actually uses for downstream tasks. This paper addresses the question: At what point during training do sparse subsets of decomposed activations become sufficient to preserve a model's predictive distribution and downstream behavior?

Methodology

The authors introduce a checkpoint-wise activation-intervention framework that treats decomposition assumptions as hypotheses to be tested via intervention rather than interpretability guarantees. The methodology proceeds in three stages: representation fitting, mask calibration, and intervention evaluation.

Activation Decomposition: At each layer $\ell$ and checkpoint $t$ , the authors fit a dictionary $V$ and an affine reconstruction baseline $b$ to the training activations $H$ . They employ four extraction methods: SVD, ICA, NMF, and SemiNMF. A code-inference map $C$ generates code matrices $U$ from activations.
Sparse Mask Selection: A sparse soft mask $z$ is optimized on a calibration split to select a subset of coordinates from the decomposition. The mask is defined as $z_i = \sigma(\alpha_i / \vartheta)$ , where $\vartheta$ is a gate temperature annealed from 1.0 to 0.1. The selection minimizes a preservation loss $L_c$ plus an explicit sparsity penalty $\lambda \|z\|_1$ .
Intervention and Evaluation: The masked reconstruction $\hat{H}$ $\hat{H}$ is injected back into the original model on held-out test data. The authors evaluate the "functional sufficiency" of the mask by measuring the deviation from the original model's behavior using four distinct preservation targets:
- Activation Reconstruction: Minimizing variance-normalized Euclidean error (geometric fidelity).
- Fixed-Head Decodability: Preserving the distribution accessible via a fixed affine translator to the native head (linear probing).
- True Downstream Preservation: Preserving the model's actual predictive distribution after layer $\ell$ (behavioral fidelity).
- Checkpoint Transfer: Determining if concepts from an earlier checkpoint $t$ can reconstruct activations at a later checkpoint $T$ under learned alignment.

The framework compares these targets to distinguish between information that is easy to reconstruct geometrically versus information that is functionally necessary for prediction.

Key Contributions

A Multi-Target Intervention Framework: The paper introduces a systematic method to compare four preservation targets (reconstruction, fixed-head, downstream, and transfer) across seven language models (Pythia, OLMo, EuroBERT, ModernCamemBERT) at multiple checkpoints and layers.
Theoretical Analysis of Preservation Losses: The authors derive the relationship between downstream KL divergence and local predictive geometry. They show that downstream preservation is a curvature-weighted reconstruction error, where the predictive Fisher metric weights residual directions. They introduce "supervised concept-gate scores" ( $\tau$ and $\omega$ ) to resolve sensitivity coordinate-by-coordinate, distinguishing between total single-example sensitivity and the mean-aligned component preserved by averaging.
Empirical Characterization of Concept Dynamics: The study reports target-conditional sparse operating points and model-specific profiles regarding checkpoint evolution and depth dependence.

Key Results

The experiments across seven models reveal significant divergences between geometric reconstruction and functional sufficiency:

Compactness of Downstream Masks: Masks optimized for true downstream preservation are markedly more compact than those for reconstruction. At a shared fixed-penalty operating point, downstream masks retain a median of only 6.6% of the candidate soft mass, compared to 73.0% for reconstruction and 83.4% for fixed-head selection. Despite this extreme sparsity, downstream masks preserve the predictive distribution with a relative KL of 0.020 and cause only a 0.80 percentage-point drop in accuracy.
Geometry-Function Separation: Downstream masks exhibit high reconstruction error (median NMSE of 0.959) compared to reconstruction masks (0.215), yet they preserve behavior. This indicates that the model's predictive distribution relies on a small, high-curvature subset of directions, while the majority of activation variation lies in weakly used directions.
Model-Specific Temporal and Depth Profiles: The evolution of functional sufficiency is not uniform.
- Pythia models show strong positive correlations between training progress and downstream KL, with nearly flat depth correlations, suggesting a temporal signature.
- EuroBERT-210M exhibits the opposite: negative checkpoint correlations but strong positive depth correlations.
- OLMo shows positive correlations for both axes.
- This suggests that the "maturity" of concepts is a model-specific profile rather than a universal timeline.
Supervised Sensitivity Concentration: Downstream masks are enriched for supervised gate-gradient moments ( $\tau$ and $\omega$ ). Specifically, they concentrate the mean-aligned component ( $\omega$ ) significantly more than reconstruction masks (median enrichment of 5.32 $\times$ vs 1.08 $\times$ ). This implies that behaviorally sufficient masks select directions that carry concentrated, consistent gradient signals.
Transfer and Decomposition: Learned-alignment transfer shows consistent patterns where SVD and ICA generally yield lower transfer KL than NMF or SemiNMF across models, indicating that the choice of decomposition affects how well earlier concepts align with later representations.

Significance and Claims

The paper claims that concept dynamics are more informative as a collection of target-aware profiles than as a single maturity score.

Functional Sufficiency is Conditional: A mask is only "sufficient" relative to a specific target (e.g., reconstruction vs. downstream behavior). The large gap between reconstruction occupancy and downstream occupancy demonstrates that most representational capacity is not functionally required for prediction.
Anisotropy in Predictive Geometry: The results support an intuition that training preferentially reinforces directions carrying concentrated gradient energy, creating a task-conditional anisotropy in the pullback predictive Fisher. A small retained soft mass can carry a disproportionate share of supervised sensitivity.
Sender-Receiver Hypothesis: The findings motivate a hypothesis where training shapes activation space so that useful distinctions are accessible to a shallow affine readout (sender), while downstream computation acts as a direction-selective receiver that amplifies a narrower set of signals.
Limitations: The authors explicitly state that their claims are operational and target-conditional. They do not claim to identify unique human-interpretable concepts, nor do they establish that selected directions are the only way to preserve behavior (redundancy is possible). The study is limited to specific pretrained models and does not extend to sequence generation, instruction following, or in-context learning.

In conclusion, the paper provides a rigorous framework for measuring when internal structures become useful, demonstrating that functional sufficiency emerges in highly compact, supervised-sensitive subsets of activations that are distinct from the broader geometric representation of the model.