Technical Summary: The Convergence Gap in Instruction-Tuned Language Models

1. Problem Statement

Current model-diffing diagnostics often reduce comparisons between pretrained (PT) and instruction-tuned (IT) checkpoints to final-output scores (e.g., accuracy or perplexity). This approach obscures when a model commits to a prediction during the forward pass. The paper posits that released instruction-following checkpoints do not merely end at different distributions; they follow different stabilization trajectories, potentially settling on their final next-token predictions later in the network stack than their pretrained counterparts. The core problem is the lack of a diagnostic that measures the timing of prediction stabilization rather than just the final outcome.

2. Methodology

2.1 The Convergence Gap Metric

The authors introduce the convergence gap, a layerwise diagnostic defined as the Kullback-Leibler (KL) divergence between the next-token distribution decoded at layer $l$ ($p_l$) and the model's own final next-token distribution ($p_L$):
$$\text{Convergence Gap}_l = \text{KL}(p_l || p_L)$$
This metric quantifies how far a specific layer is from the model's final prediction. A larger gap at late layers indicates that the model is still significantly changing its mind as it traverses the stack.

2.2 Experimental Setup

• Models: Six paired PT/IT checkpoints were analyzed: Gemma 3 4B, Llama 3.1 8B, Qwen 3 4B, Mistral 7B, OLMo 2 7B, and DeepSeek-V2-Lite.
• Histories: All comparisons utilized matched token histories. Discovery curves aggregated per-token layer readouts across hundreds of thousands of token steps.
• Controls:
  • Endpoint Matching: PT and IT token steps were matched based on final confidence, entropy, and top-1/top-2 margins to ensure the gap is not an artifact of differing endpoint confidence.
  • Endpoint-Free Checks: Comparisons of native same-layer output distributions under identical teacher-forced histories, avoiding reference to the final endpoint entirely.
  • Fixed-History Replay: Native chat templates were replayed through raw and tuned models to control for template-induced serialization effects.

2.3 Interventions (Matched-Prefix MLP Grafts/Swaps)

To identify the architectural locus of this phenomenon, the authors performed symmetric causal interventions:

• Grafts: Replacing specific MLP windows (Early, Middle, Late) in a PT host with corresponding IT weights.
• Swaps: Replacing specific MLP windows in an IT host with corresponding PT weights.
• Windows: Defined as roughly 40% of model depth, centered at early, middle, and late regions (e.g., for Llama 3.1 8B, late is layers 19–31).
• Random Controls: Matched random residual-projection perturbations were applied to late layers to test if any large perturbation, rather than the specific IT update, caused the effect.

3. Key Results

3.1 Instruction-Tuned Models Stabilize Later

Across all six model families, instruction-tuned checkpoints exhibit a larger late convergence gap. The IT curves remain higher (further from the final distribution) deeper into the stack compared to PT models.

• Endpoint-Matched: The gap persists even when controlling for endpoint confidence and entropy (Raw: +0.425 nats; Tuned: +0.762 nats).
• Endpoint-Free: Adjacent layer-to-layer movement (JS divergence) and future top-1 flip rates are higher in IT models, confirming they change their predictions later in the forward pass.
• Template Effects: While raw/no-template replay attenuates the signature, the effect remains positive and significant under fixed-history replay of native chat templates.

3.2 Late MLPs are the Primary Leverage Point

Intervention experiments identified late MLP windows as the strongest bidirectional handle on the stabilization delay:

• PT Host + Late IT Graft: Increases the late KL gap by +0.34 nats (vs. -0.03 for early, -0.05 for middle).
• IT Host + Late PT Swap: Reduces the IT delay by -0.51 nats (vs. -0.10 for early, -0.23 for middle).
• Random Controls: Matched random late perturbations failed to reproduce the effect (+0.003 nats vs. +0.327 nats for true grafts), ruling out simple magnitude-based explanations.

3.3 Behavioral Case Study (Gemma)

A preselected case study on Gemma 3 4B tested the downstream impact of the late PT swap:

• Preference: Blind pairwise judging preferred native IT over the late PT swap in 94.7% of assistant-register prompts.
• Scores: The assistant-register score dropped from 4.733 (native) to 3.929 (late swap), while early swaps had negligible impact.
• Robustness: Programmatic checks (MMLU, reasoning) did not show a total collapse, suggesting the effect is specific to instruction-following dynamics rather than a general degradation of capability.

4. Contributions and Significance

The paper makes three primary contributions, framed as a shift in how post-training is analyzed:

1. A Prediction-Dynamics Target: The convergence gap provides a measurable object for "when a model settles," moving beyond static endpoint comparisons to analyze the route to prediction.
2. Rigorous Controls: By employing endpoint-matched readouts and endpoint-free same-history checks, the paper demonstrates that the delayed stabilization is a robust signature of instruction tuning, not an artifact of confidence levels or template formatting.
3. Causal Localization: Matched-prefix interventions isolate late MLP windows as the largest tested bidirectional handle on delayed stabilization. This identifies a specific architectural region where the "instruction-following" dynamics are most malleable.

5. Scope and Limitations (Author's Voice)

The authors explicitly state the scope and limitations of their claims:

• Not a Full Mechanism: The paper does not claim to explain the full mechanism of instruction following or provide a universal steering direction.
• Not a Benchmark Predictor: The Gemma case study is a "behavior-facing bridge" for the clearest family, not a claim that the convergence gap predicts benchmark performance across all tasks or families.
• No Named Features: The work identifies a robust layerwise signature and the strongest MLP handle but does not reconstruct the full circuit or identify named sparse features (e.g., via SAEs or crosscoders).
• Template Dependence: The effect is modulated by chat templates; the paper treats no-template prompting as an out-of-distribution serialization that attenuates, but does not erase, the signature.
• MoE Caveat: DeepSeek-V2-Lite was included in discovery curves but excluded from causal averages because MoE routing complicates the interpretation of dense MLP window swaps.

Conclusion: The paper concludes that released instruction-tuned checkpoints tend to settle later in the forward pass, and late MLP computation is the strongest tested bidirectional handle on that delay under matched histories. This reframes post-training as a change in the route to a prediction, not just the endpoint.