A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearning

Technical Summary: A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearning

Problem Statement
Machine unlearning is critical for building trustworthy and compliant language models (LLMs), yet its success varies significantly across individual training samples. While some data points are reliably erased, others persist despite identical unlearning procedures. Existing research largely attributes this disparity to data-side phenomena, such as spurious correlations, redundancy, or dataset structure. However, there is a lack of mechanistic, model-internal explanations for why certain samples are intrinsically harder to forget. This paper addresses two primary research questions: (1) Can unlearning difficulty be defined and quantified from a mechanistic perspective prior to unlearning? and (2) Can this predicted difficulty be linked to the internal circuits within the model?

Methodology: Circuit-Guided Unlearning Difficulty (CUD)
The authors propose Circuit-guided Unlearning Difficulty (CUD), a pre-unlearning metric that assigns a continuous difficulty score to each sample based on model-internal circuit structures. The methodology proceeds in three stages:

1. Reference Circuit Identification:
   The authors first identify "easy" and "hard" anchor samples using a bi-level optimization objective.

   • Easy Anchors ($D_{f,E}$): Samples that exhibit increased loss (low memorization) post-unlearning.
   • Hard Anchors ($D_{f,H}$): Samples that retain low loss (high memorization) post-unlearning, indicating failed unlearning.
     These anchors are selected via optimization over the forget set, repeated across multiple seeds to ensure stability.

2. Circuit Extraction:
   Using Edge Attribution Patching with Integrated Gradients (EAP-IG), the authors extract the computational circuits responsible for predictions on the anchor sets from the original model. These circuits are represented as structured matrices capturing functional interactions among model components.

3. CUD Score Calculation:
   For any query sample $z_i$, its specific circuit $C_i$ is extracted. The CUD score is computed by measuring the similarity of $C_i$ to the easy anchor circuit ($C_E$) and the hard anchor circuit ($C_H$). The score is defined as:
   $$CUD(z_i) = \frac{1 - s_E}{(1 - s_E) + (1 - s_H)}$$
   where $s_E$ and $s_H$ are similarity metrics between the query circuit and the respective anchors. The resulting score ranges from 0 (easy to unlearn) to 1 (hard to unlearn). This approach treats difficulty as a relative geometric position in circuit space, independent of the specific unlearning algorithm used.

Key Contributions

• First Circuit-Level Analysis of Disparity: The paper provides the first mechanistic analysis of why unlearning performance varies, revealing that easy and hard samples are memorized through structurally distinct internal circuits.
• CUD Metric: The introduction of a principled, fine-grained metric that quantifies intrinsic unlearning difficulty using circuit-level signals before any unlearning intervention occurs.
• Mechanistic Signatures: The identification of specific circuit patterns distinguishing easy and hard samples:
  • Easy Samples: Associated with shorter, shallower interactions concentrated in early-to-intermediate parts of the model (e.g., early-to-mid MLP pathways). These rely on compact, repeatedly utilized computation paths.
  • Hard Samples: Rely on longer, deeper pathways closer to late-stage computation (e.g., deeper MLP transitions and direct connections to logits). These involve heterogeneous, diffuse circuitry with attention-mediated routing, making them resistant to localized intervention.

Experimental Results
The authors evaluated CUD on the TOFU (fabricated author profiles) and MovieLens-1M (recommendation) datasets across five unlearning methods (GradAscent, GradDiff, NPO, SimNPO, UNDIAL, E2UREC, RecEraser).

• Stratification of Difficulty: Using CUD, the authors constructed "easy" and "hard" forget sets. Replacing the default forget set with the CUD-selected hard set resulted in a statistically significant drop in unlearning efficacy (average drop of 14.1 points across methods) and degradation in retained knowledge. Conversely, the easy set yielded an improvement in unlearning efficacy (average +3.3 points) with preserved or slightly improved retain performance.
• Robustness: CUD scores remained stable across different similarity metrics (Cosine, Jaccard, Hamming) and different unlearning loss functions (GradDiff vs. UNDIAL), with a high correlation ($\rho = 0.76$) between scores derived from different loss formulations.
• Independence from Superficial Features: Analysis confirmed that CUD is not confounded by lexical similarity (diverse n-grams in anchors) or context length (no correlation between CUD score and input token count).
• Comparison to MRD: Unlike Memory Removal Difficulty (MRD), which measures sensitivity to parameter perturbations and correlates weakly with CUD ($\rho = -0.27$), CUD captures functional circuit structures relevant to decision-making and can be computed offline prior to unlearning.

Significance and Claims
The paper claims that CUD represents a first step toward a principled, interpretable analysis of unlearning difficulty. By shifting the perspective from data-side heuristics to model-internal mechanisms, the work demonstrates that unlearning difficulty is fundamentally tied to the structural pathways (circuits) supporting a model's predictions.

The authors posit that this mechanistic understanding enables:

1. Proactive Identification: The ability to identify easy and hard samples before unlearning begins, without executing the unlearning procedure.
2. Targeted Intervention: Insights into which layers or modules (e.g., early MLP vs. late logits) dominate resistance to forgetting, facilitating targeted debugging.
3. Method-Agnostic Benchmarking: The construction of controlled, difficulty-stratified forget sets to stress-test unlearning algorithms more rigorously than standard benchmarks allow.

The paper concludes that while CUD currently requires computational resources for circuit discovery, it provides a foundational framework for developing future unlearning methods that adapt to intrinsic sample difficulty rather than treating all samples uniformly.