Representation-Guided Parameter-Efficient LLM Unlearning

1. Problem Definition

Large Language Models (LLMs) often memorize sensitive, copyrighted, or harmful information, necessitating Machine Unlearning (MU) to remove specific knowledge without retraining from scratch.

• The Challenge: Existing parameter-efficient unlearning methods (e.g., those using LoRA) struggle with the forget-retain trade-off. Reducing performance on the "forget set" ($D_f$) often degrades performance on the "retain set" ($D_r$).
• Root Cause: Current methods rely on parameter importance metrics (e.g., Fisher Information) to identify which parameters to update. However, due to the superposition phenomenon in LLMs, single parameters are often polysemantic (involved in multiple concepts). This makes it difficult to disentangle parameters relevant only to the forget set from those crucial for the retain set, leading to catastrophic forgetting or ineffective unlearning.

2. Methodology: ReGLU

The authors propose Representation-Guided Low-rank Unlearning (ReGLU), a framework that shifts the focus from parameter-level importance to the geometric properties of representation spaces. ReGLU consists of two core components:

A. RILA: Representation-Guided Initialization of Low-rank Adaptation

Instead of initializing LoRA matrices randomly or based on parameter importance, RILA initializes them to align with a subspace that maximizes the difference between the forget and retain sets.

• Mechanism: It computes the covariance matrices of the layer outputs (representations) for both the forget set ($Cov_F$) and retain set ($Cov_R$).
• Balanced Covariance: It constructs a balanced covariance matrix: $Cov_\Delta = (1 - \beta)Cov_F - \beta Cov_R$, where $\beta$ balances the trade-off.
• Initialization: The LoRA matrices ($A$ and $B$) are initialized using the top-$r$ eigenvectors of $Cov_\Delta$.
• Goal: This ensures the initial update direction maximizes variance for the forget set while minimizing variance for the retain set, providing an optimal starting point for selective forgetting.

B. ROL: Representation Orthogonal Loss

To prevent the unlearning process from interfering with retained knowledge during training, a regularization term is added.

• Mechanism: It identifies the principal subspace of the retain set's representations (via eigenvalue decomposition of $Cov_R$) to form an orthogonal basis $P_B$.
• Loss Function: The loss $L_{ROL} = \|B^\top P_B\|_F^2$ penalizes any alignment between the LoRA up-projection matrix ($B$) and the retain set's principal subspace.
• Goal: This forces the LoRA updates to lie in the orthogonal complement of the retain set's representation space, ensuring that updates affect the forget set without disturbing the retain set.

C. Overall Objective

The total loss combines the unlearning objective (e.g., Gradient Ascent or IHL), a retain loss, and the ROL regularization:
$$L_{total} = L_{forget}(D_f) + \gamma L_{retain}(D_r) + \lambda L_{ROL}$$

3. Key Contributions

1. Paradigm Shift: Moves LLM unlearning from parameter importance (which fails due to superposition) to representation geometry (which is more effectively disentangled).
2. Novel Framework: Introduces ReGLU, combining RILA (geometric initialization) and ROL (subspace-controlled regularization).
3. Theoretical Guarantee: Provides a theorem proving that the proposed initialization maximizes the "differential energy" between forget and retain sets at the start of training.
4. Efficiency: Demonstrates that RILA is computationally efficient, requiring only forward passes and eigenvalue decomposition (scalable via randomized SVD), avoiding the expensive gradient computations required by Fisher Information-based methods.

4. Experimental Results

The authors evaluated ReGLU on TOFU (synthetic author profiles) and WMDP (biosecurity/cybersecurity questions) across models like Llama-2-7B, Phi-1.5, and Zephyr-7B.

• Performance: ReGLU consistently outperformed state-of-the-art baselines (FILA, VILA) combined with various loss functions (GD, IHL).
  • TOFU: Achieved the highest Forget Quality (FQ) scores. For example, on Llama-2-7B with IHL, ReGLU achieved an FQ of -2.4 (higher is better, log-scaled), significantly better than VILA (-7.2) and FILA (-6.9).
  • WMDP: Achieved the lowest accuracy on forget sets (indicating successful unlearning). GD + ReGLU reduced Bio/Cyber accuracy to 35.1%, outperforming GD + VILA (38.1%).
• Utility Preservation: All methods maintained at least 95% of the original model's utility (measured by MMLU or harmonic mean of other metrics), proving that ReGLU minimizes catastrophic forgetting.
• Ablation Studies: Confirmed that both RILA (initialization) and ROL (regularization) are necessary; using them together yields the best results, demonstrating strong synergy.
• Mechanism Analysis:
  • Initialization: RILA achieved a significantly higher forget-to-retain energy ratio at initialization compared to FILA/VILA.
  • Orthogonality: ReGLU maintained the highest orthogonality (0.87) between LoRA updates and the retain subspace, compared to 0.70 for FILA.

5. Significance and Limitations

Significance:

• ReGLU offers a robust solution to the forget-retain trade-off by leveraging the geometric structure of neural representations rather than relying on unreliable parameter importance metrics.
• It provides a parameter-efficient alternative that is scalable to large models (70B+) using randomized SVD, making it practical for real-world deployment where full fine-tuning is too costly.

Limitations:

• Initialization Overhead: Requires computing covariance matrices and eigenvalue decomposition for each layer, which adds a one-time cost (though still lower than full fine-tuning).
• Data Representativeness: The quality of the identified subspaces depends on the small subsets of data used for covariance estimation; unrepresentative samples could reduce effectiveness.
• Hyperparameter Sensitivity: Performance is sensitive to the LoRA rank ($r$) and the balance parameter ($\beta$), requiring careful tuning.
• Scope: Evaluation is currently limited to TOFU and WMDP benchmarks; broader domain generalization needs further study.