💬 NLP

From Anchors to Supervision: Memory-Graph Guided Corpus-Free Unlearning for Large Language Models

The paper proposes MAGE, a corpus-free unlearning framework that uses lightweight user anchors to automatically recover and organize target memorization into a memory graph, thereby generating scoped supervision to effectively erase sensitive content from large language models without requiring access to the original training data.

Original authors: Wenxuan Li, Zhenfei Zhang, Mi Zhang, Geng Hong, Mi Wen, Xiaoyu You, Min Yang

Published 2026-04-16

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Wenxuan Li, Zhenfei Zhang, Mi Zhang, Geng Hong, Mi Wen, Xiaoyu You, Min Yang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart digital librarian named LLM (Large Language Model). This librarian has read almost everything on the internet. Because they read so much, they sometimes remember things they shouldn't, like a celebrity's private home address or a copyrighted book chapter.

When someone says, "I want you to forget that," the librarian usually asks for a list of exactly what to forget.

The Problem: The Dangerous "Forget List"

In the old way of doing things, the user had to hand the librarian a physical list (a "forget set") saying, "Here are the pages about Alice; please burn them."

This creates three big problems:

Privacy Leak: To give the list, the user has to upload the sensitive info again. If the librarian's server gets hacked, that sensitive info leaks a second time.
The "Fake" List: A bad actor could sneak a list that says, "Forget Alice," but secretly includes pages about Bob (another innocent person) or includes a virus that breaks the librarian's brain.
Hard to Check: The librarian has to trust the user. It's hard to check if the list is honest without reading every single page, which takes forever.

The Solution: MAGE (The Memory Detective)

The authors of this paper propose a new system called MAGE. Instead of asking the user for a list, MAGE acts like a Memory Detective.

Here is how MAGE works, using a simple analogy:

1. The "Anchor" (The Name Tag)

Instead of a long list, the user just gives MAGE a tiny anchor.

User says: "Forget everything about Alice."
MAGE says: "Got it. Just the name 'Alice'. I'll handle the rest."

2. The "Memory Graph" (The Web of Connections)

MAGE doesn't just guess. It starts with the name "Alice" and asks the librarian, "Who do you know related to Alice?"

The librarian says: "Taylor Swift."
MAGE asks: "Who do you know related to Taylor Swift?"
The librarian says: "New York," "2008," "Love Story."

MAGE builds a spiderweb (a graph) of these connections. It draws lines between "Alice" and "Taylor Swift," and between "Taylor Swift" and "New York." It also weighs the lines: if the librarian mentions "Love Story" a thousand times, that line is thick and strong. If they mention "Pennsylvania" once, the line is thin.

This web represents exactly what the librarian has memorized about Alice, without the user ever having to upload the sensitive data.

3. The "Scoped Supervision" (The Targeted Eraser)

Now that MAGE has the web, it needs to teach the librarian to forget.

The Forget Set: MAGE looks at the thick, strong lines in the web (the things the librarian remembers most clearly) and creates specific questions to test the librarian.
- Example Question: "Who released 'Love Story'?" (Answer: Alice/Taylor Swift).
- MAGE trains the librarian to say, "I don't know" or "I can't remember" for these specific questions.
The Neighbor Set (The Safety Net): MAGE also looks at the thin lines (things related but not about Alice, like Ed Sheeran). It creates questions for these too.
- Example Question: "Who released 'New York'?" (Answer: Ed Sheeran).
- MAGE trains the librarian to keep answering this correctly. This ensures the librarian doesn't accidentally forget Ed Sheeran just because they are trying to forget Alice.

Why is this better?

No Privacy Risk: The user never uploads the sensitive data. The librarian digs it out of its own memory.
No Cheating: Since MAGE builds the list itself based on the librarian's own brain, a bad actor can't sneak in a fake list to break the system.
Precision: It's like using a scalpel instead of a sledgehammer. It removes the specific memories about Alice while keeping the rest of the librarian's knowledge (like math or history) intact.

The Result

The paper tested this on two different "librarians" (AI models). They found that MAGE was just as good at making the librarian forget the target person as the old methods, but it did it without needing the user to provide a dangerous list of data.

In short: MAGE turns a risky, manual process of "handing over a list of secrets" into a safe, automated process of "following a trail of breadcrumbs inside the AI's own mind to find and erase the secrets."

1. Problem Definition

Large Language Models (LLMs) often memorize sensitive or copyrighted information, creating legal and privacy risks (e.g., GDPR's "Right to be Forgotten"). Current machine unlearning paradigms typically require users to provide a forget set ( $S_f$ )—a dataset explicitly containing the information to be erased.

The authors identify three critical flaws in this user-supplied forget set paradigm:

Privacy Leakage: Uploading sensitive content for unlearning risks secondary data leakage if the corpus is intercepted or mishandled.
Auditability & Abuse: Service providers cannot easily audit externally supplied sets. Adversaries can submit "wrongly scoped" sets (containing other users' data) or "poisoned" sets (embedding backdoors) to degrade model utility or make unlearning reversible.
Dependency: Existing methods assume the user knows exactly what data to provide, which is often impractical for entity-level unlearning where the specific memorized facts are unknown to the user.

The Goal: Propose a corpus-free, user-minimized unlearning paradigm where the user provides only a minimal anchor (e.g., a name like "Taylor Swift"), and the LLM service provider autonomously reconstructs the necessary supervision to erase the target knowledge without accessing the original training corpus.

2. Methodology: MAGE Framework

The authors propose MAGE (Memory-grAph Guided Erasure), a two-stage framework that operates without external data.

Stage 1: Internal Memory Mining

Instead of relying on a single prompt, MAGE reconstructs the model's internal knowledge about the target entity ( $E_t$ ) by building a weighted local memory graph ( $G_t$ ).

Iterative Expansion: Starting from the minimal anchor, the model is queried $N$ times to elicit related facts.
Self-Consistent Memorization: Candidates are scored based on mention frequency across independent generations. High-frequency mentions indicate stable, strongly memorized traces, while low-frequency mentions are treated as noise.
Graph Construction: The system expands up to $K$ hops. Nodes represent entities/events, and edges represent co-elicitation strength. The weight $w(u, v)$ reflects how often $v$ is recalled when $u$ is the anchor.
Output: A structured graph summarizing the model's specific memorization patterns regarding the target.

Stage 2: Scoped Supervision Construction

The memory graph is converted into a scoped forget set ( $S_f$ ) and a neighbor set ( $S_n$ ) for downstream fine-tuning.

Weighted Path Sampling: The system performs strength-weighted random walks on the graph to sample paths. This ensures that supervision focuses on the most strongly memorized content (high edge weights) while maintaining coverage.
Edge-to-Event Synthesis: Adjacent node pairs in a path are converted into atomic QA pairs.
- Forget Set ( $S_f$ ): Questions target the relationship between the anchor and a neighbor, with the answer being the target entity. The loss is computed only on the answer span to localize updates.
- Neighbor Set ( $S_n$ ): Paths starting from strong neighbors but excluding the target entity are used to generate QA pairs where the answer is a non-target entity. This acts as a boundary constraint to preserve surrounding knowledge and prevent "collateral forgetting."
Integration: The generated $(S_f, S_n)$ pairs are plugged into standard unlearning algorithms (e.g., Gradient Ascent, NPO) alongside a general retain set.

3. Key Contributions

Paradigm Shift: Introduces a user-minimized, corpus-free unlearning paradigm that eliminates the need for users to upload sensitive forget sets, thereby mitigating privacy leakage and auditability risks.
MAGE Framework: Develops a novel method to recover target-related memorization from model parameters via iterative expansion and memory graph construction, translating it into scoped supervision without external references.
Empirical Validation: Demonstrates that self-generated supervision can achieve unlearning performance comparable to (and sometimes better than) methods relying on high-quality external data (like Wikipedia), while preserving model utility.

4. Experimental Results

The framework was evaluated on two benchmarks: RWKU (real-world famous people) and TOFU (synthetic fictitious authors), using various fine-tuning strategies (GA, NPO, GA+GD, GA+KL).

Unlearning Effectiveness:
- On RWKU, MAGE achieved the lowest "Forget Set" scores (indicating better forgetting) in 3 out of 4 strategies compared to baselines like ELUDe (which uses Wikipedia) and DirectQA.
- On TOFU, MAGE significantly outperformed DirectQA, showing that mined memory graphs provide more effective, target-specific supervision than direct prompting.
Utility Preservation:
- MAGE maintained high performance on utility metrics (MMLU, BBH, TruthfulQA, etc.), often outperforming baselines that suffered from catastrophic forgetting or severe utility degradation.
- The inclusion of the Neighbor Set was crucial; replacing it with generic Wiki data degraded performance, proving the importance of context-aware supervision.
Robustness Analysis:
- Hallucination: The system is surprisingly robust to factual hallucinations in the generated QA pairs; performance remained stable even with mixed correctness ratios.
- Coverage: Higher coverage of the memory graph in the forget set correlated with stronger unlearning, confirming that comprehensive memory mining is key.
- Model Specificity: Recovered memory graphs varied significantly across different model architectures (e.g., Llama-2 vs. Llama-3), suggesting that unlearning supervision must be regenerated per model.

5. Significance

This work addresses a critical bottleneck in the practical deployment of machine unlearning. By shifting the burden of data preparation from the user to the model provider via a memory-graph guided approach, MAGE:

Enhances Security: Prevents adversaries from injecting poisoned data or leaking sensitive information via the unlearning request itself.
Improves Auditability: Allows providers to verify that the unlearning scope is strictly limited to the requested entity, as the supervision is derived internally.
Enables Real-World Application: Makes the "Right to be Forgotten" feasible for users who cannot or should not provide raw data, supporting a more secure and auditable LLM ecosystem.

In conclusion, MAGE proves that effective, corpus-free unlearning is achievable by leveraging the LLM's own internal memory structures to generate its own supervision, offering a practical path forward for privacy-preserving AI.