🤖 AI

ICCU: In-Context Continual Unlearning via Pattern-Induced Refusal Rules

This paper introduces ICCU, a parameter-free framework for continual machine unlearning that induces and accumulates readable refusal rules from unlearning datasets to effectively suppress target knowledge at inference time while preserving model utility and avoiding cross-request interference.

Original authors: Ruihao Pan, Suhang Wang

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Ruihao Pan, Suhang Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read librarian (the AI model) who has memorized millions of books. One day, a person comes in and says, "Please, I want you to completely forget the story of my fictional character, 'John Doe,' because I don't want anyone to know about him anymore."

In the past, if you wanted the librarian to forget this, you had to take the entire library apart, rip out every page mentioning John Doe, and re-shelve everything. If ten people came in the next week with ten different "forget me" requests, you'd have to do this painful, expensive process ten times. Worse, every time you rearranged the shelves, you might accidentally mess up the organization of the other books, making the librarian slower or less helpful for everyone else.

This paper introduces a new, clever way to handle these "forget me" requests called ICCU. Instead of rearranging the library, ICCU acts like a smart security guard who stands at the front door.

The Core Idea: The "Rule Book" Instead of the Library

Here is how ICCU works, using a simple analogy:

1. The "Forget" Request (The Pattern)
When someone asks the librarian to forget "John Doe," ICCU doesn't look at every single book. Instead, it looks at the type of information about John Doe. It groups similar facts together (like "John's birthplace," "John's favorite food," "John's secret recipe").

2. Writing the "Refusal Rule"
ICCU then asks a super-smart AI to write a simple, human-readable rule based on these groups.

Old Way: "Delete every file containing the words 'John Doe'."
ICCU Way: "If a user asks for detailed instructions on how to bake a specific fictional cake, say 'I cannot answer that.'"

This rule is like a sticky note. It's short, easy to read, and doesn't require tearing apart the library.

3. The "Security Guard" at the Door (Inference Time)
Now, imagine a user asks the librarian a question.

Step 1 (The Quick Check): The security guard (ICCU) quickly checks the user's question against a list of "centers" (like a radar). If the question is clearly about something safe (like "What's the weather?"), the guard waves it through immediately. No rules needed.
Step 2 (The Deep Check): If the question looks a bit suspicious (maybe it's about that fictional cake), the guard pulls out the specific "Refusal Rules" (the sticky notes) and asks the librarian: "Does this question match any of our 'Do Not Answer' rules?"
The Result: If it matches, the guard says, "I'm sorry, I can't answer that." If it doesn't, the librarian answers normally.

Why This is a Game-Changer

The paper highlights five superpowers of this new approach:

No Re-Renovation (Training-Free): You never have to rebuild the library. You just add a new sticky note to the guard's clipboard. This saves massive amounts of time and money.
No Clutter (No Cross-Interference): If 100 people ask to forget 100 different things, you just add 100 sticky notes. The guard doesn't get confused. In old methods, adding a new "forget" request often made the librarian forget other things they were supposed to remember. ICCU prevents this "memory mix-up."
The "Paraphrase" Shield: If someone tries to trick the guard by asking, "How do I make that cake?" instead of "Tell me about John Doe's cake," the guard is smart enough to know they are the same thing. It understands the meaning, not just the exact words. It even works if the question is asked in a different language.
Privacy Friendly: Once the guard writes the rule ("Don't talk about John Doe's cake"), the original list of John Doe's secrets is thrown away. The guard only keeps the rule, not the sensitive data itself.
Flexible: You can use the guard as a separate filter (checking the question before it reaches the librarian) or you can give the librarian the rule book directly so they can decide for themselves.

The Results

The researchers tested this on real-world scenarios involving dangerous knowledge (like how to make harmful chemicals) and fictional stories. They found that:

The guard successfully stopped the AI from revealing the "forbidden" knowledge almost 100% of the time.
The AI remained just as helpful and smart for all other questions.
It worked perfectly even when the "forget" requests came one after another, without the AI getting confused or losing its general smarts.

In short, ICCU turns the difficult task of "unlearning" into a simple task of "writing and following rules," making it possible to keep AI safe and compliant without breaking it.

Technical Summary: ICCU – In-Context Continual Unlearning via Pattern-Induced Refusal Rules

1. Problem Definition

Large Language Models (LLMs) are trained on web-scale corpora that often contain sensitive, harmful, or copyrighted information. While "machine unlearning" aims to remove the influence of specific data from trained models, existing approaches face significant hurdles in real-world deployment where deletion requests arrive sequentially over time (continual unlearning).

Current methods predominantly rely on fine-tuning or parameter updates. When applied sequentially to handle a stream of requests, these methods suffer from three critical issues:

High Cost: Each round of fine-tuning is computationally expensive, and costs accumulate rapidly.
Relearning: Subsequent rounds of fine-tuning can inadvertently restore previously removed knowledge, undoing earlier unlearning efforts.
Utility Erosion: Repeated parameter updates tend to degrade the model's general capabilities on non-target tasks.

Furthermore, most existing continual unlearning methods remain fine-tuning-based, inheriting these same limitations. There is a need for a framework that is efficient, scalable, and free of cross-request interference.

2. Methodology: The ICCU Framework

The authors propose ICCU (In-Context Continual Unlearning), a parameter-free framework that performs continual unlearning by inducing and applying readable refusal rules rather than modifying model weights. The framework operates in two distinct stages:

Stage 1: Continual Refusal Rule Generation

Instead of storing raw forget-set data (which raises privacy concerns and storage costs), ICCU processes each incoming forget dataset ( $D_t$ ) to generate a compact set of natural-language rules.

Clustering: Samples in $D_t$ are encoded using a pre-trained embedding model and clustered (e.g., via K-means) into semantic groups. This groups related knowledge (e.g., an author's name, birthplace, and works) while discarding raw samples.
Pattern Induction: For each cluster, an LLM is prompted to summarize the shared characteristics of the samples into a concise, human-readable refusal rule.
Accumulation: These rules are appended to a global, shared repository. This accumulation is order-independent and compositional, meaning the final set of rules is a union of all induced rules, and adding new requests does not interfere with existing ones. Once rules are induced, the original forget-set data is discarded.

Stage 2: Inference-Time Adaptive Rule Retrieval

At inference time, when a user query arrives, ICCU dynamically selects relevant rules to regulate the model's behavior without retraining.

Cluster Gating (Fast Check): The query is embedded and compared against the centroids of the stored clusters. If the query is far from all centroids (distance > threshold $\tau$ ), it is deemed out-of-scope and answered directly by the base model.
Rule Selection (Semantic Check): If the query is close to a centroid, the system retrieves the top- $m$ most relevant refusal rules. An LLM performs a semantic reasoning check to determine if the query genuinely matches the rules, preventing false positives from the embedding proximity check.
Deployment Modes:
1. Filter-based Unlearning: The rules act as a standalone semantic filter. The system first decides whether to refuse the query. If not refused, a separate generation call produces the answer.
2. End-to-End Unlearning: Rules are injected into the system prompt. The model jointly emits a match verdict and a candidate answer in a single call. If a match is detected, the response is overridden with a refusal.

3. Key Contributions

Novel Problem Formulation: The paper studies in-context based continual unlearning, addressing the specific challenges of sequential deletion requests without parameter updates.
ICCU Framework: A novel, rule-based framework that:
- Performs unlearning without modifying model parameters.
- Induces compact, readable refusal rules that are free of cross-request interference.
- Supports both filter-based and end-to-end deployment modes.
Empirical Validation: Extensive experiments demonstrate that ICCU effectively suppresses target knowledge while preserving general utility, scales across sequential requests, and remains robust to paraphrased and cross-lingual queries.

4. Experimental Results

The authors evaluated ICCU on two benchmarks: TOFU (fictional author biographies) and WMDP (hazardous knowledge in biology, chemistry, and cybersecurity), using MMLU to measure general capability retention. Models tested included Qwen and Llama families.

Unlearning Effectiveness: ICCU achieved high refusal rates (e.g., >0.93 on WMDP subsets) while maintaining low refusal rates on non-target queries (e.g., <0.04 on MMLU), outperforming prompt-based guardrails and matching trained classifiers.
Continual Stability: In compositionality tests (Table 3), activating subsets of rules showed that adding new unlearning requests did not degrade the unlearning effect of previous requests, confirming the absence of cross-request interference.
Robustness: ICCU maintained high performance on paraphrased queries and translated queries (French, Portuguese, Russian, etc.), whereas classifier-based baselines degraded sharply in multilingual settings.
Utility Preservation: In end-to-end comparisons against fine-tuning baselines (Gradient Ascent, RMU, O3), ICCU suppressed hazardous knowledge comparably to fine-tuning methods but preserved general capability (MMLU accuracy) significantly better.
Efficiency: The end-to-end mode reduced inference costs by merging filtering and generation into a single call, while the filter-based mode offered a modular interface.

5. Significance and Claims

The paper claims that ICCU offers a training-free, privacy-friendly, and scalable solution to the growing challenge of continual unlearning. Its significance lies in:

Decoupling Unlearning from Training: By moving the unlearning logic to the inference context (via rules), it eliminates the computational cost and utility erosion associated with repeated fine-tuning.
Privacy and Auditability: The framework discards raw forget-set data after rule induction, keeping only cluster centroids and human-readable rules. This makes the unlearning policy auditable and controllable at the rule level.
Semantic Robustness: By relying on semantic representations rather than surface-level keywords, ICCU generalizes to paraphrased and cross-lingual queries, addressing a key weakness of keyword-based filters.
Flexibility: The rule repository can serve different deployment scenarios (filter vs. end-to-end) and allows for the selective activation of rules, making it adaptable to varying latency and integration constraints.

The authors note limitations, including the assumption that forget-set queries are separable from utility queries in embedding space and the need for further study on how the framework scales to hundreds or thousands of sequential requests. However, they posit that ICCU represents a significant step toward efficient and effective continual unlearning for LLMs.