🤖 machine learning

When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures

This paper tracks the developmental trajectories of attention circuits across three 1B-class language models, revealing that the formation of induction circuits and attention sinks are distinct, temporally separated transitions rather than a single event, and that specific circuit capabilities can be identified early in training without requiring the final model.

Original authors: Yongzhong Xu

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yongzhong Xu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a group of three young students (the AI models) learn to read and understand a massive library of books. These students are roughly the same size (1 billion "brain cells"), but they have different teachers and different books.

The researchers wanted to answer a simple question: When do these students actually learn to do specific tricks?

Specifically, they were looking for two things:

The "Copycat" Trick: Learning to see a pattern like "A... B... A" and correctly guessing that the next word should be "B." (This is called an induction circuit).
The "First Page" Habit: Learning to always pay attention to the very first word of a sentence, no matter what the sentence is about. (This is called an attention sink).

For a long time, scientists thought these two things happened at the exact same moment, like a light switch flipping on. This paper says: No, that's not true. They happen at very different times, and the way they happen depends on how the student is built and what they are reading.

Here is the story of what they found, using simple analogies:

1. The "No-Go" Zone (The Bottom Floor)

The researchers discovered a rule that never breaks: The first two layers of the brain (Layer 0 and Layer 1) never learn the "First Page" habit.

The Analogy: Imagine a factory assembly line. The first two stations are just raw materials coming in; they haven't been processed yet. You can't have a "factory manager" (the attention sink) making decisions at the very first station because there isn't enough information there yet.
The Finding: No matter how much the students study, the very bottom layers of their brains never develop this habit. It's a hard rule of their architecture, not something they failed to learn.

2. The "Middle-First" Surprise

You might think the "First Page" habit would start at the bottom and slowly move up as the brain gets smarter. The researchers found the opposite.

The Analogy: Imagine a building. You might expect the lights to turn on from the ground floor up. Instead, the lights in the middle of the building turn on first, and then the habit spreads outward. The layers just above the "No-Go Zone" are actually the last to learn this habit.
The Finding: The "First Page" habit crystallizes in the middle of the network first, not the bottom.

3. The Two Different "Light Switches"

This is the biggest discovery. The researchers tracked when the "Copycat" trick and the "First Page" habit appeared.

The "Copycat" Trick (Induction): This happens very early. In the students trained on the "DCLM" books, this trick was fully learned by the time they had read about 20–23 billion words. It's like a child learning to tie their shoes; it happens quickly and then they are done.
The "First Page" Habit (Attention Sink): This happens much, much later. In the same students, this habit didn't really kick in until they had read 260 to 400 billion words.
The Analogy: It's like learning to ride a bike (the trick) happens in week one, but learning to always look at the starting line before every race (the habit) doesn't happen until year five. They are two completely separate events, separated by a huge gap of time.

4. The Shape of Learning Depends on the "School"

The researchers tested three different students:

Pythia: Read "The Pile" books.
OLMo: Read "DCLM" books.
OLMoE: Read "DCLM" books but has a special "Mixture of Experts" brain (like having 64 different tutors and only using the top 8 for each task).

They found that the shape of the learning curve changed based on the student:

The Gradual Ramp: Pythia and OLMoE learned the "First Page" habit slowly and steadily, like water filling a bathtub.
The Sharp Jump: OLMo (the standard dense brain reading DCLM) learned it all at once. One moment it had almost no "First Page" habit, and the next moment (between two checkpoints), it jumped from 7% to 70%. It was a sudden "phase transition," like ice suddenly turning to water.

The Lesson: The same book (DCLM) produced a slow learner in one student and a sudden jumper in another, just because their brain architecture was different.

5. You Don't Need to Wait Until Graduation

One of the most practical findings is about when you can see these skills.

The Finding: If you want to find out which parts of the brain are doing the "Copycat" trick, you do not need to wait until the student finishes reading the whole library.
The Analogy: Imagine you want to know if a student is good at math. You don't need to wait until they finish a 4-year degree. By the time they have finished just 1% of their coursework, you can already see 67% of the math skills they will eventually have.
The Result: Researchers can identify these brain circuits using a student who is only 1% through their training. You don't need the final, finished model to understand how the brain works.

Summary

This paper tells us that the "magic moment" of AI learning isn't one big event.

Timing: The brain learns to copy patterns (induction) way before it learns to obsess over the first word (attention sink).
Location: The obsession with the first word starts in the middle of the brain, not the bottom.
Variety: Depending on the brain's design, this obsession can appear slowly or in a sudden explosion.
Efficiency: We can see these skills forming very early in training, so we don't need to wait for the AI to be "finished" to study how it thinks.

Technical Summary: Developmental Trajectories of Capability and Attention-Sink Emergence

Problem Statement
This paper addresses a gap in the mechanistic understanding of pretraining dynamics in transformer language models. While prior work has identified two distinct phenomena—induction heads (implementing the $A \dots A \to B$ copy pattern, linked to in-context learning) and attention sinks (reliable attention to the first token regardless of content)—they are often conflated under the umbrella of "attention emergence." Olsson et al. [2022] described induction formation as a sharp "phase transition," while Xiao et al. [2024] documented attention sinks as a stable inference property. This paper asks: Do these two phenomena emerge simultaneously as a single transition, or are they temporally separable? Furthermore, how do architecture (dense vs. Mixture-of-Experts) and pretraining corpus (The Pile vs. DCLM) influence the timing and shape of these emergences in 1B-class models?

Methodology
The authors track the developmental trajectory of attention circuits across three 1B-class models spanning two architectures and two corpora:

Pythia 1B: Dense transformer, trained on The Pile.
OLMo 1B-0724-hf: Dense (Llama-style) transformer, trained on DCLM.
OLMoE 1B-7B-0924: Mixture-of-Experts (1B active/7B total) transformer, trained on DCLM.

The study utilizes a 30-run developmental panel, applying a three-step mechanistic interpretability recipe at 10 log-spaced revisions per model:

Synthetic Induction Batch: A controlled dataset of 2,000 sequences ( $A \dots A \to B$ ) used to probe induction capabilities.
Participation Ratio (PR) Spectral Signal: Computed on the per-head attention output at the second- $A$ position. PR measures the spectral diversity of the head's output, serving as a signal for specialized computation.
Capability-Specific Selectivity Screen: An all-head screen classifying heads into six categories (induction, previous-token, duplicate-token, BOS/first-token, self, local) based on attention selectivity ratios. Heads are assigned to a class if selectivity exceeds a specific threshold (e.g., 50× for induction, 30× for BOS).
Causal Verification: Group ablation of identified circuits to confirm their functional role.

Key Findings

1. The L0/L1 Zero-BOS Floor is an Architectural Property (F1)
Across all 30 revisions and all three models, layers 0 and 1 produce zero heads classified as BOS-attractors. This "zero-BOS floor" holds from random initialization through the end of training. The authors interpret this as an architectural constraint: L0 inputs are uncontextualized token embeddings, and L1 outputs lack cross-token integration, making it impossible for attention sinks (which require non-trivial cross-position information) to form in these layers. This is a learned outcome of the gradient signal, not a data artifact.

2. Distinct Emergence Shapes for BOS-Attractors (F2)
The formation of the BOS-attractor fraction follows three qualitatively different trajectories depending on the data-architecture pairing:

Pythia 1B (Pile, Dense): A gradual, monotonic ramp from 0% to ~58%.
OLMo 1B (DCLM, Dense): A sharp phase transition. The fraction jumps from 7.4% at 117B tokens to 70.3% at 264B tokens (a 63-percentage-point increase between adjacent checkpoints).
OLMoE 1B-7B (DCLM, MoE): A gradual, monotonic ramp, similar to Pythia but reaching a higher saturation (~76%).
This demonstrates that the "sharpness" of the attention sink transition is not a universal property of the data (DCLM) but is modulated by architecture (Dense vs. MoE).

3. Induction and Attention Sinks are Temporally Separable (F3)
In DCLM-trained models, the formation of induction circuits precedes the formation of BOS-attractors by an order of magnitude in tokens:

OLMo 1B: Induction circuits form at ~~23B tokens; BOS-attractors reach 50% saturation at 264B tokens (~~11.5× gap).
OLMoE 1B-7B: Induction forms at ~~20B tokens; BOS-attractors reach 50% at 419B tokens (~~21× gap).
The paper concludes that capability-circuit formation and attention-sink formation are two distinct transitions, not a single phase transition event.

4. Early Convergence of Capability Screens (F4)
The capability-specific screen converges to the final induction circuit using only a small fraction of the training budget.

OLMo 1B: 67% of the final induction circuit is identified by 10B tokens (~0.3% of total training).
OLMoE 1B-7B: 67% identified by 41B tokens (~0.8%).
Pythia 1B: 67% identified by 6B tokens (~2.0%).
This implies that for circuit discovery, the final model is not required; intermediate checkpoints contain the bulk of the identifiable circuit.

5. Spectral Signals Precede Selectivity Thresholds (F5)
For every final-checkpoint induction head sampled, the per-head Participation Ratio (PR) rises significantly at or before the revision where the head's capability-selectivity ratio crosses its classification threshold. In Pythia 1B, the PR signal for the top induction head rose ~12× earlier (in token count) than the selectivity threshold crossing. This suggests the spectral signal is an early indicator of specialization.

Significance and Claims
The paper positions itself as the "activation-space half" of a broader study on emergence, complementing parameter-space analyses of spectral-edge dynamics. Its primary contributions are:

Refining the "Induction Phase Transition": The authors argue that the term "induction phase transition" is ambiguous. On DCLM data at 1B scale, it refers to two distinct events: an early capability transition (induction) and a later, data- and architecture-dependent transition (attention sinks).
Constraints for Mechanistic Theory: The findings impose specific constraints on any theory of attention sinks:
1. It must explain the universal L0/L1 zero-BOS floor.
2. It must account for how the same data (DCLM) produces a sharp transition in dense models but a gradual ramp in MoE models.
3. It must distinguish between the timing of capability emergence and sink emergence.
Methodological Utility: The paper demonstrates that mechanistic interpretability can be performed effectively on intermediate checkpoints (0.3–2% of training) without waiting for the final model, and that specific screening methods are required to distinguish capability heads from the dominant BOS-attractor population in later training stages.

The authors explicitly state that these findings are based on specific training runs and do not claim to represent the full expectation of all possible seeds, though they note that the qualitative patterns (L0/L1 floor, separation of transitions) are robust across the tested configurations.