🤖 AI

RePAIR: Interactive Machine Unlearning through Prompt-Aware Model Repair

The paper introduces RePAIR, a prompt-aware framework that enables interactive, on-device machine unlearning by allowing users to instruct large language models to forget specific knowledge through natural language using a training-free activation manipulation technique (STAMP) that achieves near-perfect knowledge removal while preserving model utility.

Original authors: Jagadeesh Rachapudi, Pranav Singh, Ritali Vatsi, Praful Hambarde, Amit Shukla

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC0 1.0

Original authors: Jagadeesh Rachapudi, Pranav Singh, Ritali Vatsi, Praful Hambarde, Amit Shukla

Original paper dedicated to the public domain under CC0 1.0 (http://creativecommons.org/publicdomain/zero/1.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read friend (let's call him The Model). This friend has read almost everything on the internet. Because of this, he knows a lot of great things, but he also accidentally memorized some embarrassing secrets, dangerous instructions, and outright lies.

Usually, if you want your friend to "unlearn" something specific—like a secret you told him or a lie you want corrected—you are stuck. You can't just tell him, "Hey, forget that." He's a static book; you can't erase a page without rewriting the whole library. Currently, only the person who built the library (the Model Service Provider) can try to fix it, and that takes months and costs a fortune.

RePAIR is a new invention that changes the rules. It lets you (the user) tell the model to forget something instantly, right while you are talking to it, without needing the original builder's help.

Here is how it works, broken down into simple analogies:

1. The Problem: The "Un-erasable" Notebook

Think of the AI as a notebook filled with facts. If you write a mistake in it, you can't just use an eraser. In the old world, to fix a mistake, you had to take the whole notebook, tear out every page, and rewrite the entire book from scratch. This is too slow and expensive.

2. The Solution: The "Surgical Repair" Team (RePAIR)

The authors created a system called RePAIR (Interactive Machine Unlearning through Prompt-Aware Model Repair). Imagine this as a three-person medical team operating on the AI in real-time:

The Watchdog (The Detective): This is a small AI that listens to your conversation. If you say, "I want you to forget my home address," the Watchdog catches that intent. It acts like a security guard who spots a "Do Not Enter" sign and alerts the team.
The Surgeon (The Architect): Once the Watchdog spots the request, the Surgeon steps in. It doesn't retrain the whole model. Instead, it writes a tiny, precise piece of code—a "surgical instruction"—that tells the model exactly how to change its brain to forget that one specific thing.
The Patient (The Model): This is the main AI you are talking to. It receives the Surgeon's instruction and instantly modifies its own internal wiring to forget the target information.

3. The Magic Trick: STAMP (The "Steering Wheel")

How does the Surgeon actually make the model forget without retraining? They use a method called STAMP.

Think of the AI's brain as a giant highway with millions of lanes (neural pathways). When the AI thinks about "My Home Address," it drives down a specific lane.

Old Way: To stop the car from driving that lane, you would have to rebuild the entire highway system (retraining).
STAMP Way: STAMP is like a smart steering wheel. It calculates a precise mathematical nudge. It doesn't rebuild the road; it just gently turns the steering wheel of the "Home Address" thought so that instead of driving to the "Answer" exit, it gets diverted into a "Refusal" dead-end.

The AI still knows how to talk, but when you ask about the specific thing you want forgotten, the steering wheel automatically turns the car away, and the AI says, "I don't know that," or "I can't tell you that."

4. Why is this a Big Deal?

It's Instant: You don't wait for the developers to fix it. You ask, and it happens in minutes (or seconds).
It's Private: You don't have to trust a big company to delete your data. You do it yourself, on your own device.
It's Efficient: The "Low-Rank" version of this trick (STAMP-LR) is like a compact car. It does the same job but uses much less fuel (computer power), making it possible to run on a laptop or even a phone.

The Result

In their tests, this system was incredibly good at its job.

Forgetting: It successfully erased the bad info 100% of the time (the model stopped remembering the secret or the lie).
Remembering: It didn't accidentally forget other things. The model stayed smart and helpful for everything else.
Speed: It was about 3 times faster than the old methods that required heavy retraining.

In summary: RePAIR gives you the power to edit your AI's memory on the fly. It turns the AI from a rigid, unchangeable encyclopedia into a flexible friend who can listen to your request, say "Got it," and instantly delete that specific memory, all while keeping the rest of its knowledge intact.

1. Problem Statement: The Limitations of Current Machine Unlearning

Large Language Models (LLMs) inherently absorb harmful knowledge, misinformation, and private data during pretraining. While Machine Unlearning (MU) aims to remove specific data influences without full retraining, existing approaches suffer from critical structural limitations:

Provider-Centric: Current methods (e.g., GA, NPO, RMU) require model service providers (MSPs) to have access to model internals, curated "retain" datasets, and full training pipelines.
Exclusion of End Users: Users cannot directly control the removal of their own data or the correction of misinformation. They must rely on MSPs, raising transparency and governance issues (e.g., GDPR "right to erasure" is difficult to enforce).
Training Requirements: Most methods require gradient computation and backpropagation, making them infeasible for inference-time or on-device scenarios.
Batch Dependency: Existing methods often fail when attempting to unlearn a single sample (single-shot unlearning), as the gradient signal is overwhelmed by the retain set.

The Goal: The authors propose Interactive Machine Unlearning (IMU), a paradigm where users can instruct an LLM to forget targeted knowledge via natural language prompts during inference, without requiring retraining or MSP intervention.

2. Methodology: The RePAIR Framework

The paper introduces RePAIR (Interactive Machine Unlearning through Prompt-Aware Model Repair), a multimodal framework designed to execute IMU. It consists of three interacting components:

$M_{watchdog}$ (Watchdog Model):
- Monitors the dialogue history.
- Detects unlearning intent (distinguishing between standard chat and a request to "forget").
- Extracts the target "forget pair" $(p_f, r_f)$ from the conversation.
$M_{surgeon}$ (Surgeon Model):
- Receives the forget pair and generates executable repair code ( $C_t$ ).
- This code defines the specific mathematical procedure to modify the model weights.
$M_{patient}$ (Patient Model):
- The base LLM (e.g., Llama-3-8B) that undergoes autonomous parameter updates based on the code generated by the Surgeon.
- Transforms from $M_{patient}$ to $M_{healed}$ (the unlearned model).

Core Algorithm: STAMP

The heart of RePAIR is STAMP (Steering Through Activation Manipulation with PseudoInverse), a training-free, single-sample unlearning method.

Mechanism: Instead of gradient descent, STAMP uses closed-form pseudoinverse updates to redirect the MLP (Multi-Layer Perceptron) activations of the forget sample toward a "refusal subspace."
Process:
1. Activation Extraction: Extracts MLP activations for the forget sample ( $D_f$ ), a small retain buffer ( $D_r$ ), and a reference set of natural refusal prompts ( $D_{ref}$ ).
2. Steering Vector: Computes a vector ( $r_{SV}$ ) representing the direction from the forget activation to the refusal activation.
3. Target Generation: Constructs a target output matrix where forget activations are shifted by $r_{SV}$ , while retain activations remain unchanged.
4. Weight Update: Solves for new weights ( $W_{new}$ ) using the Moore-Penrose pseudoinverse:
  $W_{new} = X^+ \cdot O'$
  Where $X$ is the input matrix and $O'$ is the target output matrix.
STAMP-LR (Low-Rank Variant):
- To reduce the computational complexity of matrix inversion from $O(d^3)$ to $O(r^3 + r^2 \cdot d)$ , STAMP-LR approximates the input matrix using low-rank decomposition ( $X \approx AB$ ).
- This enables efficient on-device unlearning with up to 3× speedup compared to training-based baselines.

3. Key Contributions

Formalization of IMU: Defined a new problem setting where end-users can autonomously instruct LLMs to forget knowledge via natural language at inference time, removing dependency on MSPs.
STAMP & STAMP-LR: Proposed the first training-free, single-sample unlearning methods for LLMs. These methods operate entirely at test time without gradient computation.
RePAIR Framework: An end-to-end solution integrating intent detection, code generation, and autonomous model repair.
Empirical Validation: Demonstrated that RePAIR achieves near-oracle forgetting while preserving model utility, outperforming six state-of-the-art (SoTA) baselines.

4. Experimental Results

The framework was evaluated on three tasks: Harmful Knowledge Suppression, Misinformation Correction, and Personal Data Erasure, using Llama-3-8B as the patient model.

Forgetting Performance:
- STAMP and STAMP-LR achieved near-zero forget scores ( $Acc_f = 0.00$ , $F\text{-}RL = 0.00$ ) across all tasks.
- They successfully unlearned single samples ( $|D_f|=1$ ), whereas training-based baselines (GA, NPO, etc.) failed completely ( $Acc_f = 100$ ) due to the inability to isolate single-sample gradients.
Retention & Utility:
- Retention: STAMP-LR achieved high retain accuracy ( $Acc_r$ up to 84.47 for misinformation) and ROUGE-L scores ( $R\text{-}RL$ up to 0.88), closely matching the Oracle model (trained only on retain data).
- Utility: Perplexity on TinyStories remained stable (6.0–7.0), comparable to the Oracle and significantly better than gradient-based methods (which saw perplexity rise to 10–12).
Efficiency:
- Runtime: STAMP-LR reduced runtime to 2.57–4.01 minutes, achieving a ~3× speedup over training-based baselines (which took 6–12 minutes).
- Memory: STAMP-LR requires only $2rd$ memory, making it feasible for edge devices, unlike full fine-tuning which requires ~6× model memory.

Table 2 Highlights (Misinformation Removal):

STAMP-LR: $Acc_f = 0.00$ , $Acc_r = 84.47$ , Utility = 7.39, Runtime = 2.57 min.
Best Baseline (RMU): $Acc_f = 0.27$ , $Acc_r = 82.17$ , Utility = 8.10, Runtime = 6.00 min.

5. Significance and Impact

User Empowerment: RePAIR shifts control from service providers to end-users, enabling direct, autonomous enforcement of privacy rights (GDPR/CCPA) and safety corrections.
Feasibility of On-Device AI: By eliminating the need for backpropagation and large retain datasets, STAMP-LR makes real-time, on-device model editing practical.
Robustness to Single Samples: The ability to unlearn a single data point without a large batch is a critical advancement for interactive scenarios where users request immediate corrections.
Future Directions: The authors note that while a small retain buffer is currently needed, future work aims to achieve fully "retain-free" unlearning and extend the framework to multimodal foundation models.

In conclusion, RePAIR establishes a new standard for Interactive Machine Unlearning, proving that LLMs can be safely and efficiently "repaired" by users in real-time without the prohibitive costs of retraining.