🤖 machine learning

Understanding Machine Unlearning Through the Lens of Mode Connectivity

This paper introduces the concept of mode connectivity in unlearning (MCU) to analyze the optimization geometry of machine unlearning, revealing that unlearned models often reside in connected low-loss basins with distinct mechanisms from retraining, while offering insights into privacy metrics, nonlinear unlearning progress, and improved robustness through ensembling.

Original authors: Jiali Cheng, Hadi Amiri

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jiali Cheng, Hadi Amiri

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart digital brain, a machine learning model, that has read the entire internet. It knows everything from how to bake a cake to the secret recipe of a famous cookie. But then, someone asks it to forget that cookie recipe because it's copyrighted, or maybe it's just outdated. You can't just delete a few lines of code; the brain has woven that knowledge into its very structure. If you try to rip it out, you might accidentally break its ability to bake the cake or answer simple questions. This is the tricky world of Machine Unlearning: the art of making an AI forget specific things without ruining its general smarts.

To understand how this works, scientists look at the "loss landscape." Think of this landscape as a giant, hilly terrain where the height of the ground represents how bad the AI is at its job. The AI wants to find the deepest valley (the lowest point) where it makes the fewest mistakes. Usually, if you train two different AIs from scratch, they might end up in different valleys. But a cool discovery called Mode Connectivity showed that these separate valleys are often connected by smooth, low-lying paths. You can walk from one AI's solution to another without ever climbing a steep hill. This paper asks a new question: If we take one AI and force it to forget something, does it end up in a valley that is still connected to other "forgotten" versions of itself? And does the path between them stay smooth, or does it turn into a jagged, broken cliff?

The researchers, Jiali Cheng and Hadi Amiri, decided to explore this by introducing a new concept called Mode Connectivity in Unlearning (MCU). They treated the process of forgetting like a journey across a map. Instead of just checking if the AI forgot the right thing, they looked at the "terrain" between two different versions of a forgotten AI. They found that for many methods, the path is indeed smooth. You can slide from one "forgotten" model to another without the AI suddenly remembering the secret cookie or losing its ability to bake the cake. The landscape of forgetting is often a wide, flat valley rather than a bunch of isolated pits.

However, the journey isn't always the same. The paper reveals that how you teach the AI to forget matters a lot. If you use different training tricks—like changing the order of the lessons (curriculum learning) or using a more complex math method (second-order optimization)—you might end up in completely different valleys that aren't connected. It's like taking two different routes to the same destination; one might be a smooth highway, while the other is a bumpy dirt road that leads to a totally different neighborhood.

One of the most surprising discoveries is that even when two "forgotten" models look identical on paper (they are in the same smooth valley), they can act very differently in secret. The researchers found that while the models performed similarly on standard tests, their "privacy" levels could fluctuate wildly. One model might be safe from hackers trying to trick it into remembering the secret, while its twin in the same valley might be dangerously leaky. This suggests that just because an AI looks like it has forgotten, doesn't mean it's truly safe.

The paper also uncovered a weird pattern in how forgetting happens. It doesn't happen all at once. At first, the AI stops repeating the exact words of the secret (like a parrot stopping a phrase), but it still holds the underlying idea. Only later, as you push further along the path, does it truly lose the deep knowledge. It's like the AI first stops saying "The cookie recipe is..." but still knows how to bake it, until eventually, it forgets the recipe entirely.

Finally, the authors suggest that this "smoothness" of the path is a useful tool. If the path between two forgotten models is bumpy and full of barriers, it means the job of forgetting was really hard. If the path is smooth, the job was easier. They even showed that by mixing models from different points along this smooth path, you can create a super-robust AI that is harder to trick into remembering the secret again.

In short, this paper doesn't just tell us if an AI can forget; it gives us a map of how it forgets. It shows us that the landscape of unlearning is complex, sometimes smooth and sometimes jagged, and that the way we navigate it changes everything about how safe and reliable our forgotten AIs really are.

Technical Summary: Understanding Machine Unlearning Through the Lens of Mode Connectivity

Problem Statement

Machine unlearning aims to remove specific knowledge (the "forget set" $D_f$ ) from a trained model while preserving performance on the remaining data (the "retain set" $D_r$ ) without full retraining. Despite recent progress, the loss landscape and optimization geometry of unlearning remain poorly understood. Existing research has largely focused on standard training dynamics or specific algorithmic improvements, leaving a gap in understanding the structural properties of unlearned models, their mechanistic similarity to retraining, and their robustness to adversarial attacks like relearning.

Methodology: Mode Connectivity in Unlearning (MCU)

The authors introduce Mode Connectivity in Unlearning (MCU), a framework that applies the concept of mode connectivity to the unlearning domain. In standard mode connectivity, independently trained models are connected by a smooth, low-loss path in parameter space. MCU extends this by defining a path $\phi(t)$ connecting two unlearned models ( $\theta'_1$ and $\theta'_2$ ) that must satisfy dual objectives simultaneously:

Retain Performance: The path must maintain low task loss on the retain set $D_r$ .
Forget Performance: The path must maintain high task loss on the forget set $D_f$ .

Formally, for all $t \in [0, 1]$ , the path must satisfy:
$L(D_r; \phi(t)) \leq (1-t)L(D_r; \theta'_1) + tL(D_r; \theta'_2)$
$L(D_f; \phi(t)) \geq (1-t)L(D_f; \theta'_1) + tL(D_f; \theta'_2)$

The study evaluates MCU across diverse settings:

Datasets: LLM benchmarks (TOFU, MUSE, WMDP) and classification benchmarks (CIFAR-10, DDI2013, NLVR2) from MU-Bench.
Unlearning Methods: Gradient Ascent (GA), GradDiff (GD), Negative Preference Optimization (NPO), SimNPO, RMU, Random Labeling (RL), Bad Teaching (BT), and Saliency Unlearning (SU).
Training Dynamics: Variations in curriculum learning (CL), second-order (SO) vs. first-order optimization, and different random seeds.
Curve Types: Linear interpolation and quadratic curves (where a midpoint is explicitly trained to minimize accumulated loss).

Key Results

1. Smooth Loss Landscapes and Basin Convergence

The study finds that MCU is prevalent in many unlearning scenarios. Unlearned models generated by different random seeds or specific algorithms often reside in the same connected, low-loss basin.

Scaling Effects: As model size increases or the forget set grows, the MCU curves become smoother (lower barriers). However, this increased smoothness correlates with worse unlearning quality, suggesting that scaling unlearning on larger models is geometrically more challenging.
Training Paradigms: Changes in training dynamics, specifically between Curriculum Learning (CL) and Second-Order (SO) optimization, can cause unlearning methods to converge to different, disconnected basins. When endpoints are trained with mixed paradigms (e.g., one CL, one non-CL), significant barriers appear, indicating divergent solution structures.

2. Functional Heterogeneity within Basins

Models residing in the same low-loss basin can exhibit drastically different functional behaviors, particularly regarding privacy metrics.

Privacy Volatility: In the MUSE dataset using RMU, models along a smooth MCU curve showed stable performance on forget/retain knowledge metrics but exhibited massive fluctuations in privacy leakage ($|PrivLeak|$), ranging from +100 to -100.
Implication: This reveals that proximity in parameter space does not guarantee identical privacy properties, highlighting a limitation in current evaluation protocols that rely solely on endpoint metrics.

3. Non-Linear Unlearning Dynamics

Probing the path from the original model to the unlearned model reveals that unlearning is not a linear process.

Suppress First, Unlearn Later: The removal of superficial memorization (e.g., verbatim text) occurs earlier in the parameter path than the removal of deeper knowledge. Models may stop reproducing specific text while retaining the underlying knowledge until a specific threshold is crossed, after which deeper forgetting occurs.

4. Mechanistic Distinctness from Retraining

MCU analysis suggests that approximate unlearning methods are mechanistically distinct from retraining from scratch.

Lack of Connectivity: Most unlearning methods (except NPO) do not form a smooth linear connection to a model retrained from scratch, even when their final utility is comparable. This implies unlearning reaches valid endpoints via different optimization trajectories rather than approximating the retraining solution along a shared subspace.
Method Similarity: GA shows smooth connectivity with all other methods on the forget set, suggesting a shared mechanism for handling forget samples, whereas methods like NPO and SimNPO share mechanisms distinct from GradDiff.

5. MCU-Based Ensembling and Robustness

Generalization: Averaging models sampled along the MCU curve (MCU-based ensembling) can improve the trade-off between forgetting and retaining, often outperforming the individual endpoints.
Robustness: Models obtained via MCU ensembling demonstrate significantly higher robustness against relearning attacks (fine-tuning on generic data to recover forgotten knowledge) compared to standard unlearned models. This aligns with the hypothesis that smoother loss landscapes correlate with greater adversarial robustness.

6. MCU Smoothness as a Difficulty Proxy

The smoothness of the MCU curve (measured by barrier height) serves as a proxy for unlearning difficulty. "Hard" forget sets (e.g., adversarial or highly memorized data) result in MCU curves with higher barriers and more irregularity compared to "easy" forget sets.

Significance and Claims

The paper claims to be the first study to investigate machine unlearning through the lens of mode connectivity. Its significance lies in:

Diagnostic Utility: Providing a geometric framework to understand the optimization dynamics, solution space structure, and mechanistic similarities of unlearning methods.
Practical Application: Demonstrating that MCU can be used as a tool to select more robust unlearned models and improve generalization through ensembling.
Theoretical Insight: Revealing that unlearning landscapes are often smooth and connected, but that functional properties (like privacy) can vary significantly within these basins, and that unlearning follows a non-linear progression distinct from standard retraining.

The authors conclude that MCU offers new directions for future research, particularly in designing unlearning methods that navigate smoother, more robust regions of the loss landscape.