💬 NLP

Understanding Verbatim Memorization in LLMs Through Circuit Discovery

This paper employs mechanistic interpretability and transformer circuit discovery to reveal that distinct neural subgraphs are responsible for initiating versus maintaining verbatim memorization in LLMs, with initiation being context-dependent while prevention mechanisms transfer robustly across domains.

Original authors: Ilya Lasy, Peter Knees, Stefan Woltran

Published 2026-02-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ilya Lasy, Peter Knees, Stefan Woltran

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Language Model (LLM) as a massive, hyper-organized library where every book ever written is stored. Sometimes, when you ask the librarian a question, they don't just summarize the story; they recite a paragraph from a book word-for-word. This is called memorization.

The paper you shared is like a team of detectives trying to figure out exactly how the librarian decides to start reciting that book and how they keep going. Instead of just guessing, they used a special tool called "circuit discovery" to map the specific electrical pathways inside the librarian's brain that handle this task.

Here is the breakdown of their findings using simple analogies:

1. The Two Different Jobs: The "Start Switch" and the "Treadmill"

The researchers realized that memorizing a sentence isn't just one big action. It's actually two distinct steps, like turning on a machine and then keeping it running.

The "Start Switch" (Memorization Decision): This is the moment the model looks at your prompt and decides, "Hey, I know this exact sentence! Let's copy it."
The "Treadmill" (Branch Comparison): Once the model has started copying, this is the mechanism that keeps it running along that same path, word after word, without getting distracted.

2. The Detective Work: Finding the Tiny Circuits

To find these mechanisms, the researchers created a game of "Spot the Difference." They gave the model two very similar prompts:

Clean Prompt: A sentence that is one step away from triggering a memorized response.
Corrupted Prompt: A sentence that looks similar but leads the model to a completely different, non-memorized answer.

By swapping the "brain activity" (activations) between these two prompts, they could see which tiny wires (circuits) were responsible for the switch. They found two surprising things:

The "Start Switch" is a Master Controller: The tiny group of wires responsible for starting the memorization is powerful enough to keep it going. If you mess with the "Start Switch," the model stops memorizing entirely.
The "Treadmill" is a One-Way Street: The wires responsible for keeping the model on the memorized path (once it's already started) cannot start the process on their own. They are like a treadmill that only works if someone is already running on it; they can't start the engine.

3. The "Universal Stop Button" vs. The "Context-Specific Start Button"

The researchers tested if these circuits worked on different types of text (like code, emails, or web pages), not just Wikipedia.

The "Universal Stop Button": The circuits that prevent memorization (stopping the model from reciting) worked everywhere. Whether the text was about GitHub code or Enron emails, the same "brakes" could be applied to stop the model from reciting.
The "Context-Specific Start Button": The circuits that cause memorization were picky. What triggers the model to memorize a sentence in an email doesn't necessarily trigger it to memorize a sentence in a code file. The "start" mechanism depends heavily on the specific type of text.

4. The Big Takeaway: It's Easier to Break the Trigger Than the Memory

The most interesting conclusion is about how to control this behavior.

The researchers found that it is much easier to break the "Start Switch" (preventing the model from deciding to memorize) than it is to force the model to memorize something new.

They compare this to a house with a security system:

The stored information (the books in the library) is very hard to change or erase.
However, the trigger (the key that unlocks the door to the library) is fragile. If you tamper with the key, the library stays locked, even though the books are still inside.

In summary: The paper suggests that if we want to stop AI from leaking private information or copyrighted text, we shouldn't try to delete the memories (which is hard). Instead, we should focus on finding and disabling the specific "trigger circuits" that decide when to start reciting, because those are the weak points that work across different types of text.

Technical Summary: Understanding Verbatim Memorization in LLMs Through Circuit Discovery

Problem Statement

While Large Language Models (LLMs) have demonstrated remarkable capabilities, the underlying mechanisms of verbatim memorization—the exact reproduction of training data—remain poorly understood. Existing interpretability research has identified that memorization is distributed across layers (early layers promoting tokens, upper layers amplifying confidence) and involves specific attention heads, but it lacks a mechanistic explanation of how the model decides to retrieve a memorized sequence versus generating novel content. Furthermore, standard circuit discovery methods, which rely on contrastive datasets (clean vs. corrupted examples), have been largely restricted to small, well-defined tasks (e.g., Indirect Object Identification) due to the difficulty of constructing such datasets for open-ended tasks like memorization.

Methodology

The authors approach this problem using mechanistic interpretability, specifically the circuits framework, to isolate minimal computational subgraphs responsible for memorization.

1. Dataset Construction

Source: The study utilizes the Wikipedia subset of The Pile dataset, processed using the GPT-Neo-125M model.
Memorization Selection: Samples were filtered for a memorization score of exactly 1.0 (perfect token-level match with ground truth), resulting in 4,047 samples.
Divergence Points: The authors algorithmically shortened context lengths until the model's generation diverged from the memorized path (detected via a BLEU score drop below 0.3). These points define the transition from memorization to novel generation.
Contrastive Datasets: Two distinct datasets were created to isolate different aspects of memorization:
- Memorization Decision Dataset: Designed to identify components that trigger the initial decision to retrieve memorized content. It contrasts "clean" contexts (one token away from triggering memorization) with "corrupted" contexts (semantically similar but leading to non-memorized tokens).
- Branch Comparison Dataset: Designed to identify components responsible for maintaining a memorized path once the decision has been made. It contrasts contexts continuing along the memorized path against those diverging to the model's highest probability non-memorized token.

2. Circuit Discovery

Technique: The study employs Edge Attribution Patching with Integrated Gradients (EAP-IG). This method uses gradients to approximate the effects of activation patching, reducing computational cost from thousands of forward passes to just two forward passes and one backward pass.
Intervention Strategies:
- Noising (Corrupt $\to$ Clean): Patching activations from corrupted inputs into clean inputs to identify components that, when disrupted, break memorization.
- Denoising (Clean $\to$ Corrupt): Patching activations from clean inputs into corrupted inputs to identify components that restore or induce memorization.
Faithfulness Metric: Circuits are evaluated based on a normalized faithfulness score, ensuring the subgraph performs $\ge$ 85% of the full model's task performance.

Key Contributions

Dataset Creation: The construction of two specialized contrastive datasets addressing the initiation (Decision) and continuation (Branch Comparison) of memorization.
Circuit Discovery: The identification of compact circuits (containing $\le$ 5% of model edges) responsible for these specific memorization tasks. Notably, the Branch Comparison circuit was found to be extremely minimal (as few as 14 edges, or 0.04% of the model).
Rigorous Verification: A multi-faceted evaluation framework including random baseline comparisons, cross-task generalization, cross-corpus generalization, and testing against alternative patching methods (zero ablation, mean ablation).
Mechanistic Insights:
- Asymmetry in Control: Circuits capable of triggering memorization can also maintain it once started. However, circuits that only maintain memorization cannot trigger its initiation.
- Generalization Differences: Mechanisms that prevent memorization (disruption circuits) transfer robustly across different text domains (GitHub, Enron Emails, Common Crawl). Conversely, mechanisms that induce memorization are highly context-dependent and do not generalize well across domains.

Results

Circuit Size: The Branch Comparison task yielded a remarkably small circuit (14 edges, 0.04% of the model) with high faithfulness (1.00). The Memorization Decision task required larger circuits (e.g., 141 edges, 0.43%) for similar faithfulness levels.
Cross-Task Generalization:
- Memorization Decision circuits successfully controlled whether the model continued along memorized paths in the Branch Comparison task (faithfulness 1.00).
- Branch Comparison circuits performed poorly when applied to the Memorization Decision task, indicating that the mechanisms for continuing a memorized sequence are distinct from those making the initial decision to memorize.
Cross-Corpus Generalization:
- Prevention: Noising circuits (designed to break memorization) effectively reduced memorization accuracy across diverse domains (e.g., reducing accuracy from 90.5% to 4.5% on Common Crawl).
- Induction: Denoising circuits (designed to induce memorization) showed limited transferability. While faithfulness scores remained high, the actual induction of memorization in new domains was weak (e.g., increasing accuracy from 5.2% to only 23.8% on Common Crawl).
Robustness: The identified circuits showed high dependency on the specific clean-corrupt pairs used for discovery. When tested with alternative patching methods (zero or mean ablation), faithfulness dropped significantly, suggesting the circuits rely on specific counterfactual activations rather than general structural properties.

Significance and Claims

The paper claims to provide a mechanistic understanding of how LLMs transition between memorization and generation. The primary significance lies in the discovery of an asymmetry in memorization circuits:

Trigger vs. Maintenance: The ability to initiate memorization encompasses the ability to maintain it, but the reverse is not true. This suggests a hierarchical control structure where a specific "trigger" circuit decides to access stored information, while separate mechanisms handle the continuation.
Prevention vs. Induction: The study suggests that memorization prevention is a more generalizable mechanism across text domains than memorization induction. This implies that safety interventions targeting the "trigger" mechanisms might be more effective and robust than attempts to induce or suppress memorization in a domain-agnostic manner.
Contradiction with Prior Work: The findings contrast with Stoehr et al. (2024), who found memorized passages hard to corrupt. The authors hypothesize this difference arises because they target specific computation paths (circuits) rather than gradient-implicated weights. They propose a model where a "trigger" circuit is easily disrupted, while the underlying storage of information remains robust.

The authors remain modest, noting limitations such as reliance on a single small model (GPT-Neo-125M), the use of approximation methods (EAP-IG) rather than direct patching, and the potential specificity of their findings to the Wikipedia subset of The Pile. They conclude that while these circuits offer a window into memorization, further validation across model sizes and architectures is required.