1. Problem Statement

Large Language Models (LLMs) are trained using the Next-Token Prediction (NTP) objective, which theoretically incentivizes models to compute only features necessary for predicting the immediate next token ($x_{t+1}$). However, empirical evidence shows that Transformers often learn "seemingly useless" features—abstract representations (e.g., world models in Othello, syntactic structures, or formal reasoning) that are not strictly required for the immediate prediction task but appear in the model's hidden states.

The core problem addressed is: How do NTP-trained Transformers learn features that do not directly contribute to minimizing the loss at the current position? Existing mechanistic interpretability work often adopts a teleological perspective (analyzing what a feature does in the final algorithm) but fails to explain the developmental origins of these features via the training gradient signal.

2. Methodology: Gradient Decomposition

The authors propose a novel framework to decompose the gradient signal flowing from the NTP loss into three distinct components based on information flow in causally masked Transformers. They define a feature as "NTP-useless" if an optimal predictor for the immediate next token does not need to compute it.

2.1 Three Mechanisms of Feature Learning

The total gradient $\nabla_\theta \mathcal{L}$ at a specific layer $k$ and position $i$ is decomposed into:

1. Direct Learning: Gradients flowing from the immediate next-token prediction ($x_{i+1}$) through the residual stream at position $i$. This is the standard NTP signal.
2. Pre-caching: Gradients flowing from future token predictions ($x_j$ where $j > i+1$) back to position $i$ via attention mechanisms. This allows position $i$ to encode information that will be useful for predicting tokens further ahead.
3. Circuit Sharing: Gradients arising from positions $j \neq i$ that do not pass through the residual stream at $i$ but influence the shared parameters (weights) of the Transformer blocks. This allows features learned at one position to be transferred to others due to parameter sharing.

2.2 Experimental Framework

To validate and quantify these mechanisms, the authors employ two primary techniques:

• Intervention (Ablation):
  • Myopic Training: Blocks gradients between position $i$ and all $j \neq i$, effectively removing pre-caching.
  • m-Untied Training: Uses separate parameters for positions before and after index $m$, removing circuit sharing while allowing pre-caching (via attention to the KV cache of the first half).
• Attribution (Influence Estimation):
  • They define Feature Mismatch ($R$) as the squared difference in feature projections between a checkpoint $\theta$ and the final model $\theta^*$.
  • They calculate the Influence ($I$) of each gradient component by measuring how a small step in that specific gradient direction reduces the feature mismatch.
  • For large models where retraining is impossible, they propose a proxy metric $Q(w)$, estimated via intervention on a trained model. $Q(w)$ approximates the ratio of pre-cached influence to direct influence by comparing the KL-divergence of predictions at $i+1$ versus future positions $j > i+1$ when a feature is ablated.

3. Key Contributions

1. Theoretical Decomposition: A formal proof that NTP gradients can be exhaustively split into direct, pre-cached, and shared components, providing a developmental lens for feature emergence.
2. Causal Attribution Method: A method to trace the specific gradient components responsible for the emergence of specific features, moving beyond static circuit analysis.
3. Novel Empirical Findings:
   • Demonstrated that pre-caching is essential for learning complex, multi-layer circuits (like induction heads) and "useless" features.
   • Explained the fragility of OthelloGPT's world model: NTP-useless board states are learned primarily via pre-cached and shared signals, making them less robust than NTP-useful states.
   • Discovered that in Large Language Models (Gemma 2), features with extreme pre-caching influence (high $Q(w)$) are strongly correlated with formal reasoning domains (code, math, syntax), whereas direct features dominate general language tasks.
   • Showed that "look-ahead" capabilities in LLMs are not solely driven by pre-cached features; in fact, direct features are often more predictive of future tokens than pre-cached ones, challenging the "implicit planning" hypothesis.

4. Key Results

4.1 Toy Tasks (Majority & Conditioned Majority)

• Conditioned Majority: Requires a two-layer circuit (induction head) to solve.
  • Result: Myopic models (no pre-caching) and Untied models (no circuit sharing) failed to learn the feature during the input phase.
  • Conclusion: Pre-caching is strictly necessary for the emergence of multi-layer circuits and features that are NTP-useless in the immediate context.

4.2 OthelloGPT (World Models)

• Observation: Linear probes for NTP-useless board squares (squares that do not affect the next legal move) perform worse than NTP-useful squares.
• Attribution:
  • Direct Influence: Near zero for NTP-useless features.
  • Pre-cached & Shared Influence: Positive and significant for NTP-useless features.
• Conclusion: The "world model" exists but is fragile because it relies on weaker gradient signals (pre-caching/sharing) rather than the strong direct signal. This explains why the model struggles with state variations that don't change the immediate next move.

4.3 Small Language Models (TinyStories)

• Syntax vs. Position:
  • Syntax (POS/Dependency tags): Learned effectively without pre-caching (high direct influence).
  • Positional Features (Story structure): Required pre-caching to be learned effectively.
• Conclusion: Simple syntax can be learned via direct NTP signals, but higher-level structural properties require pre-caching.

4.4 Large Language Models (Gemma 2)

• Formal Reasoning: Features with the highest $Q(w)$ (strongest pre-caching influence) were overwhelmingly related to code, programming constructs, and mathematical structures.
• Steering Experiments: Steering high-$Q(w)$ features caused the model to generate more code and punctuation.
• Look-Ahead Paradox: Contrary to the hypothesis that pre-caching drives look-ahead, the authors found a negative correlation between $Q(w)$ and the ability to predict future tokens. Direct features were more useful for look-ahead, suggesting that "look-ahead" arises from feature similarity across positions rather than explicit "planning" via pre-caching.

5. Significance

This work fundamentally shifts the perspective of mechanistic interpretability from a static view (what does the circuit do?) to a developmental view (how did the circuit emerge?).

• Theoretical Insight: It resolves the paradox of why NTP models learn "useless" features by identifying pre-caching and circuit sharing as the specific gradient sources responsible.
• Practical Implication: It suggests that training objectives that block pre-caching (myopic training) might prevent models from learning complex, abstract reasoning capabilities (like code generation or world modeling), even if they can still predict the next token.
• Future Directions: The authors propose that analyzing the subspaces formed by specific gradient components (e.g., only pre-cached updates) could reveal new interpretable features and improve our understanding of how LLMs acquire rare skills.

In summary, the paper provides a rigorous mathematical and empirical framework to explain the "emergence of seemingly useless features," attributing them to the specific mechanisms of pre-caching and circuit sharing within the gradient signal of next-token prediction.