Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

Technical Summary: Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning

1. Problem Definition: The Knowing–Using Gap

The paper addresses a critical failure mode in Large Language Model (LLM) fine-tuning: the discrepancy between a model's ability to memorize new facts and its ability to utilize those facts for downstream reasoning. The authors formalize this as the Knowing–Using Gap, characterized by two distinct disparities:

• Accuracy Gap: Generalization accuracy (using facts in reasoning) remains significantly lower than memorization accuracy (direct recall) even after convergence.
• Temporal Lag: Generalization capabilities emerge significantly later than memorization capabilities during the training process.

Despite sufficient model capacity to fit new data, models often exhibit a "remembering but not using" behavior, where they can recall a fact (e.g., "Sydney is in Australia") but fail to chain it into a reasoning task (e.g., "What is the capital of the country Sydney is in?").

2. Methodology

To investigate the mechanistic causes of this gap, the authors employ a combination of dataset construction, causal intervention, and dynamic analysis.

Dataset Construction:

• The authors constructed a specialized dataset adapted from STaRK (a real-world semi-structured knowledge base), utilizing biomedical (STaRK-Prime) and academic (STaRK-MAG) subsets to ensure cross-domain robustness.
• They defined Memorization Tasks (single-hop QA) for fine-tuning and Generalization Tasks for evaluation.
• Two specific generalization task types were designed:
  • Chaining: Requires sequential reasoning across multiple hops (e.g., Entity A → Relation → Entity B → Relation → Entity C).
  • Intersection: Requires retrieving attributes for multiple entities and filtering based on a shared relation amidst noise.
• Rigorous novelty validation ensured pre-trained models scored near zero on these tasks, confirming the injected knowledge was genuinely new.

Self-Patching Intervention:
The core methodological contribution is Self-Patching, a variant of activation patching.

• Mechanism: The technique copies the hidden layer representation of an anchor token (e.g., the head entity) from a "source" run (where the fact is known) and substitutes it into a "target" run (where the model fails to generalize) at a specific target layer.
• Objective: By scanning all layer pairs $(l_{src}, l_{tgt})$, the method generates a spatial permeation map. It identifies whether a representation containing the injected knowledge exists in the model but is simply not routed to the computation-effective layers required for reasoning.
• Differentiation: Unlike causal tracing, self-patching does not require a "clean" correct trajectory, making it applicable to failed generalization cases. Unlike PatchScope, it focuses on the accuracy of the target prompt rather than interpreting the source via auxiliary decoding.

3. Key Findings and Results

The Knowledge–Circuit Misalignment Hypothesis:
The authors propose that the Knowing–Using Gap is driven by a spatial misalignment between knowledge storage and reasoning circuits.

• Storage vs. Computation: Fine-tuning encodes new facts into "storage-oriented" states (often early or very late layers) that support direct recall. However, these representations are not reliably routed into the "computation-effective" mid-layers required for multi-step reasoning.
• Gradient Vanishing: Once memorization saturates, the loss diminishes, causing gradients to vanish. Consequently, the model fails to naturally update its internal routing to align the stored knowledge with the reasoning circuit, leaving the knowledge "stranded."

Empirical Evidence:

• Permeation Dynamics: Heatmaps of self-patching scores show that after memorization, "off-diagonal" red regions appear (indicating knowledge exists in specific layers) while diagonal cells remain blue (indicating the model fails end-to-end). In successful generalization cases, the red region expands to cover the diagonal; in failures, it halts before reaching the diagonal.
• Oracle Recovery: When representations are manually relocated via self-patching to effective mid-layers, downstream use accuracy improves dramatically.
  • Chaining Tasks: Oracle self-patching increased accuracy by 1.5–6× across all models and domains.
  • Intersection Tasks: The gap was nearly eliminated.
• Spatial Clusters: The most effective patching locations form two clusters: moving representations from early layers to mid-layers and from late layers to mid-layers. This suggests the bottleneck is not the emergence of information, but its alignment with the mid-layer reasoning stream.

Practical Heuristic:
Moving beyond diagnostic tools, the authors designed a fixed heuristic strategy based on the observed spatial clusters (targeting ~0.5L from ~0.1L and ~0.8L).

• This non-oracle strategy recovers 58–75% of the oracle headroom, demonstrating that the generalization failure is partially reversible without per-instance search.

Ablation Studies:

• Token Position: Gains are tightly tied to patching at the head-entity position, not random positions or generic tokens.
• Prompting vs. Patching: Chain-of-Thought (CoT) prompting improves performance but remains far below self-patching gains, and sometimes degrades performance. Irrelevant patching (using representations from unrelated facts) yields negligible gains, confirming the effect is specific to the injected knowledge representation.
• Scale Independence: The phenomenon persists across different model sizes (1B to 8B) and fine-tuning methods (FFT vs. LoRA), indicating it is a structural alignment issue rather than a capacity limitation.

4. Significance and Claims

The paper claims to recast the generalization failure in fine-tuning not as a lack of learning capacity, but as a tractable alignment problem.

• Mechanistic Insight: It provides causal evidence that memorized knowledge exists internally but is misrouted, failing to reach the computation layers necessary for reasoning.
• Diagnostic Utility: The self-patching technique serves as a tool to map where injected knowledge becomes causally usable, even in cases where the model fails to generalize naturally.
• Practical Implication: The findings suggest that "alignment-aware" training or post-hoc alignment strategies could bridge the gap between memorization and generalization. The paper demonstrates that a simple, fixed heuristic can recover a substantial portion of the lost generalization capability, moving the field from pure diagnosis toward practical remedies.

The authors conclude that the Knowing–Using Gap is a robust phenomenon across domains and architectures, driven by the failure of fine-tuning to naturally align storage representations with reasoning circuits.