Technical Summary: MAAT: Multi-phase Adapter-Aware Targeted Unlearning

1. Problem Statement

The paper identifies a critical structural bias in existing machine unlearning benchmarks (e.g., CounterFact, ZSRE, TOFU, MUSE, WMDP-Cyber). These datasets are heavily skewed toward entity-centric "What," "Who," "When," and "Where" questions, while "Why" questions—which probe causal and relational knowledge chains—comprise less than 0.06% to 1.3% of these datasets.

This near-zero representation creates a measurement gap: unlearning methods that fail to remove causal knowledge can still achieve high aggregate scores. The authors argue that causal knowledge is qualitatively harder to unlearn due to two factors:

1. Length and Complexity: "Why" answers average 40.1 tokens (vs. 4.2–10.5 for others) and 44% involve multi-hop reasoning chains (vs. ≤2% for others).
2. Gradient Dilution: The ascent signal required to unlearn these facts is spread across long token spans without a dominant direction, causing standard gradient-based methods to fail or degrade retained knowledge when forced to be aggressive.

Existing methods (Gradient Ascent, KL-regularized GA, Adapter Negation) face a fundamental tradeoff: aggressive forgetting degrades model utility (retention), while conservative methods fail to forget causal facts.

2. Methodology: The MAAT Framework

The authors propose MAAT (Multi-phase Adapter-Aware Targeted Unlearning), a three-phase framework operating directly on LoRA adapter weights without merging them into the base model. This allows for structured "adapter surgery" rather than uniform gradient pressure.

Phase 1: Gradient-Projected Ascent

Standard gradient ascent applies updates uniformly, which can erode retained knowledge if the forget gradient ($g_f$) aligns with the retain gradient ($g_r$). MAAT employs conditional orthogonal projection:

• If $g_f \cdot g_r > 0$ (conflict), the component of $g_f$ aligned with $g_r$ is removed via projection:
  $$g^\perp_f = g_f - \frac{g_f \cdot g_r}{\|g_r\|^2 + \epsilon} g_r$$
• If no conflict exists, the full forget-ascent direction is preserved.
• The objective minimizes forget loss while anchoring to the pre-unlearning adapter state via KL divergence.

Phase 2a: SVD Rank-Dimension Pruning (MLP-only)

To address residual signals in low-magnitude directions, MAAT scores Multi-Layer Perceptron (MLP) adapter dimensions based on their gradient-column norms over the forget set.

• The top-$\rho$ fraction of rank dimensions are zeroed in the $A$ and $B$ matrices of MLP modules (down_proj, up_proj, gate_proj).
• Crucially, attention modules are excluded from pruning to avoid destroying instruction-following capabilities.

Phase 2b: Task Vector Negation

This phase targets the specific forget subspace identified in Phase 2a.

• The top-$k_F$ (e.g., 50%) forget-scored rank dimensions of each LoRA $B$ matrix are masked to form a "forget task vector" $\tau^F_l$.
• This vector is subtracted from the weights: $B_l \leftarrow B_l - \alpha \cdot \tau^F_l$.
• This achieves targeted suppression without indiscriminately erasing retain-associated directions.

Phase 3: Hybrid Retain Repair

To recover performance degraded by pruning and negation, a hybrid repair objective is minimized. It combines four terms:

1. KL Divergence: Anchors the model to the retain distribution.
2. Hidden-State Distance: Minimizes distance between current and reference hidden states for retain samples.
3. Entropy Maximization: A negative entropy term ($-w_{ent} H_F$) maximizes entropy on forget-set predictions, preventing the model from re-learning forgotten content during repair.
4. Task Vector Penalty: Penalizes cosine similarity between current weights and the forget task vector.

3. Key Contributions

1. 5WBENCH Benchmark

The authors introduce 5WBENCH, a balanced 5,000-sample benchmark derived from the Factify-5WQA corpus.

• Structure: 1,000 examples per 5W category (Who, What, When, Where, Why).
• Significance: It provides the first statistically significant coverage of "Why" questions (20% of the dataset), enabling the quantification of causal unlearning failures previously undetectable in skewed benchmarks.
• Format: Compatible with EasyEdit, including ground-truth answers, labels, and rephrased variants for robustness testing.

2. MAAT Algorithm

MAAT is the first method to simultaneously achieve high forgetting and high retention on "Why"-type causal knowledge. It operates on the structural properties of LoRA adapters, utilizing SVD scoring and task vector negation to isolate and suppress specific rank dimensions activated by forget-set inputs.

4. Experimental Results

The framework was evaluated on LLaMA 3.2-3B and Gemma 3-4B using a Qwen 2.5-7B LLM-as-a-Judge.

Performance on 5WBENCH

• Pareto Frontier: MAAT achieves a new operating point, simultaneously exceeding 60% Forget Success Rate (FSR) and 60% Retain Success Rate (RSR) across all five 5W categories on LLaMA 3.2-3B.
  • Llama 3.2-3B: 77.4% FSR / 71.6% RSR (Aggregate).
  • Gemma 3-4B: 64.0% FSR / 61.8% RSR (Aggregate).
• Why-Type Performance:
  • MAAT: 63% FSR / 65% RSR (Llama).
  • Baselines (GA, GA+KL, AN, RO-FT) fail to reach this balance. For instance, Gradient Ascent (GA) achieves 44% FSR but 78% RSR, which the authors attribute to "under-forgetting" (the model retains facts simply because it didn't forget them, not because it preserved utility).
• Baseline Failures:
  • Adapter Negation (AN): Achieves near-perfect forgetting (99.8% FSR) but causes catastrophic retention collapse (0.4% RSR), confirming that full task vector negation removes both forget and retain knowledge indiscriminately.
  • RO-FT: High retention but lower forgetting efficiency compared to MAAT.

Performance on TOFU

On the TOFU dataset (which lacks significant "Why" coverage), MAAT achieves competitive results (67.5% FSR / 46.6% RSR on Llama), demonstrating robustness across different data distributions, though the paper notes TOFU's category imbalance limits its utility for causal analysis.

Ablation Studies

• Gradient Projection: Essential for protecting retained knowledge during ascent.
• Hybrid Repair: Necessary to recover retention lost during pruning/negation; without the entropy term, the repair phase inadvertently recovers forgotten content.
• Task Vector Negation vs. Attention Pruning: Replacing attention pruning with task vector negation (Condition D) significantly improves retention (76% vs. 54% RSR) while maintaining similar forgetting strength, validating the decision to exclude attention modules from structural pruning.

5. Significance and Claims

The paper claims that MAAT is the first method to successfully navigate the forget–retain tradeoff specifically for causal (Why-type) knowledge, a domain where existing baselines fail.

• Structural Insight: The authors demonstrate that the difficulty of unlearning causal facts stems from relational complexity and gradient dilution, not a unique weight-space footprint (encoding analysis shows uniform distributed encoding across all 5W categories).
• Evaluation Impact: By introducing 5WBENCH, the paper provides a tool to expose "under-forgetting" artifacts in existing methods, where high retention scores are actually a byproduct of failed forgetting.
• Architecture Sensitivity: The results suggest that adapter-based unlearning inherits the base model's knowledge encoding structure. The performance gap between LLaMA and Gemma indicates that knowledge separability in adapter rank dimensions is architecture-dependent.

The authors conclude that while MAAT achieves a new Pareto frontier, the field requires balanced evaluation taxonomies to detect failures in causal knowledge unlearning, which standard benchmarks currently obscure.