🤖 AI

Exploring Nonlinear Pathway in Parameter Space for Machine Unlearning

This paper proposes Mode Connectivity Unlearning (MCU), a novel framework that leverages nonlinear mode connectivity and adaptive strategies to overcome weight entanglement in existing methods, thereby generating a spectrum of unlearning models that effectively balance forgetting quality and predictive performance with reduced computational overhead.

Original authors: Yingdan Shi, Ren Wang

Published 2026-05-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yingdan Shi, Ren Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a highly trained AI chef who has memorized thousands of recipes. Suddenly, a customer asks the chef to "unlearn" a specific recipe (perhaps because the ingredient is now banned, or the customer wants their data deleted).

The old way to do this was like telling the chef, "Forget that one recipe," and hoping they could magically erase just that one thought without messing up the rest of the menu. Often, this didn't work well. The chef would either forget the recipe but also lose the ability to make similar dishes (collateral damage), or they would keep making the forbidden dish anyway.

Another old method was like trying to fix a tangled knot of headphones by just pulling on one end. The paper calls this "linear updates." The problem is that the chef's knowledge is a complex, tangled web. Pulling on one thread (the bad recipe) often accidentally tightens or loosens other threads (good recipes), causing the whole menu to get messed up.

The New Solution: The "Mode Connectivity Unlearning" (MCU) Framework

This paper proposes a smarter, more flexible way to "unlearn." Instead of just pulling a thread or starting over from scratch, they treat the AI's knowledge like a landscape of hills and valleys.

Here is how their new method works, broken down into simple concepts:

1. The "Pathway" Analogy

Imagine the AI's current knowledge is a point on a map. The "perfectly unlearned" state is another point on the map.

Old Method: Tried to draw a straight line between the two points. But because the terrain is full of hidden cliffs and valleys (non-linear complexity), a straight line often leads to a disaster (bad performance).
New Method (MCU): Instead of a straight line, they find a curved, winding path that safely connects the two points. This path avoids the cliffs and keeps the chef's other skills intact.

2. The "Spectrum" of Solutions

Usually, when you try to unlearn something, you get one single result: a new version of the chef.

The Paper's Innovation: Their method doesn't just give you one result. It reveals a whole spectrum of chefs along that curved path.
Why this matters: Imagine you have a "low-risk" data point that you want to forget gently, and a "high-risk" data point you want to forget completely.
- You can pick a chef from the start of the path who still remembers a lot but has forgotten the specific bad data.
- You can pick a chef from the end of the path who has forgotten the bad data very thoroughly, even if they are slightly less skilled at other things.
- The Benefit: You get to choose the "flavor" of forgetting that fits your needs without having to retrain the whole chef from scratch.

3. The "Smart Mask" (The Filter)

To make this process fast and efficient, the authors use a filtering strategy.

Think of the AI's brain as having millions of neurons. Updating all of them is slow and expensive.
The paper's method puts a mask over the brain. It says, "Only update the neurons that are critical for the bad recipe, and leave the neurons for the good recipes completely alone."
This is like telling the chef, "Only change the way you chop onions for this one dish; don't touch how you bake bread." This makes the process much faster (up to 75% faster in their tests) without hurting the quality.

4. The "Self-Adjusting" Penalty

Usually, you have to guess how hard to try to make the AI forget. If you try too hard, you break the model; too soft, and it doesn't forget.

The paper introduces a self-adjusting knob. As the AI learns to forget, this knob automatically turns itself.
If the AI is forgetting too much (losing good skills), the knob turns down the pressure. If it's not forgetting enough, the knob turns up. This removes the need for humans to guess the right settings.

Summary of Results

The authors tested this on image recognition tasks (like identifying cats vs. dogs). They found that:

Their method outperformed all existing "unlearning" techniques.
It was more stable, meaning it didn't accidentally break the AI's ability to recognize other things.
It worked even when the starting "unlearning" attempt was messy or imperfect.

In a nutshell: This paper replaces the clumsy, straight-line attempt to delete data from an AI with a smooth, curved, and adjustable pathway. It gives us a whole menu of "partially forgotten" models to choose from, ensuring we can remove specific data without ruining the rest of the AI's brain.

Technical Summary: Exploring Nonlinear Pathway in Parameter Space for Machine Unlearning

Problem Statement

Machine Unlearning (MU) aims to remove the influence of specific training data from a trained model to comply with privacy regulations and user requests. While retraining from scratch is the gold standard, it is computationally prohibitive. Consequently, approximate MU methods have emerged, with a mainstream approach relying on linear task arithmetic (e.g., subtracting a forgetting task vector from the original model).

The authors identify two critical limitations in existing linear approaches:

Weight Entanglement: Modern classifiers exhibit high-dimensional, nonlinear characteristics. Simple linear updates often fail to localize the influence of the forgetting data exclusively. Instead, the task vectors interfere with retaining data, violating the necessary conditions for successful linear editing.
Single-Model Limitation: Existing methods typically yield a single unlearning model. However, the optimal stopping point for unlearning varies across different data points within the forgetting set. A single model cannot simultaneously achieve optimal forgetting for all points without repeated, costly training.

The paper poses two central questions:

(Q1) Can we break free from linear update constraints and explore MU in a nonlinear manner?
(Q2) Can we identify a spectrum of effective MU models rather than just one?

Methodology: Mode Connectivity Unlearning (MCU)

The authors propose Mode Connectivity Unlearning (MCU), a framework that leverages mode connectivity to search for a nonlinear unlearning pathway in the parameter space.

1. Nonlinear Pathway Construction

Instead of a linear interpolation, MCU constructs a smooth pathway between two end models using a quadratic Bézier curve:

$\theta_o$ (Original Model): Trained on the full dataset $D_{train}$ .
$\theta_p$ (Pre-unlearning Model): Obtained by applying any existing MU method to remove the influence of forgetting data $D_f$ . This model provides a directional shift toward forgetting.
$\theta_c$ (Control Model): A learnable parameter that shapes the trajectory.
Pathway Definition: The pathway $\phi_{\theta_c}(t)$ is defined as:
$\phi_{\theta_c}(t) = (1-t)^2\theta_o + 2(1-t)t\theta_c + t^2\theta_p$
where $t \in [0, 1]$ . This generates a continuous spectrum of unlearning models $\theta_u$ along the curve.

2. Optimization Objective

The framework optimizes the control model $\theta_c$ to balance forgetting and retaining performance. The loss function is:
$\mathcal{L}_{mcu} = \mathbb{E}_{t \sim U(0,1)}[\mathcal{L}(D_r; \phi_{\theta_c}(t)) - \beta \cdot \mathcal{L}(D_f; \phi_{\theta_c}(t))]$
where $\mathcal{L}$ is the cross-entropy loss, $D_r$ is the retaining data, and $\beta$ is the unlearning penalty coefficient.

3. Parameter Mask Strategy

To enhance efficiency, MCU introduces a parameter mask that updates only a subset of parameters, bypassing gradient computations for the rest. The mask $m$ is derived by:

Filtering ( $m_r$ ): Excluding parameters highly influential to retaining data $D_r$ (based on gradient magnitude).
Reserving ( $m_f$ ): Preserving parameters critical for forgetting data $D_f$ .
Final Mask: $m = m_r \ \& \ m_f$ .
This strategy significantly reduces computational overhead (reported as a 75.23% speedup in experiments) while maintaining effectiveness.

4. Adaptive Unlearning Penalty

To eliminate the need for empirical hyperparameter tuning, the authors propose an adaptive $\beta$ strategy. The coefficient $\beta$ is dynamically adjusted at each batch based on the gap between observed accuracies and calibration targets:

If forgetting accuracy is already sufficient (or over-forgotten), $\beta$ is set to 0.
If forgetting is insufficient but retaining performance is stable, a mild $\beta$ (0.1) is applied.
Otherwise, a stronger $\beta$ (0.5) is used.
This allows the framework to adaptively balance forgetting quality and predictive performance without manual intervention.

5. Model Selection

Unlike methods that output a single model, MCU identifies an optimal unlearning model and an effective unlearning region along the pathway. The optimal model is selected by minimizing the average gap between the unlearning model's performance and the retrained baseline (RT) across forgetting, retaining, and test datasets.

Key Contributions

Nonlinear Perspective: The paper is the first to investigate MU from a nonlinear perspective, addressing the weight entanglement issue inherent in linear task arithmetic.
Spectrum of Models: It introduces the concept of exploring unlearning pathways, providing a spectrum of models rather than a single point, offering flexibility to prioritize utility or forgetting quality based on specific needs.
Plug-and-Play Framework: MCU is designed to integrate seamlessly with existing MU methods (e.g., NegGrad+, SalUn), consistently enhancing their performance regardless of the quality of the initial pre-unlearning model.
Efficient Masking: The proposed parameter mask strategy achieves comparable unlearning effectiveness to element-wise masks but significantly reduces training time by completely bypassing gradient computations for masked parameters.

Experimental Results

The framework was evaluated on image classification tasks (CIFAR-10, ImageNet-100, Tiny-ImageNet) using various architectures (PreResNet, ViT, VGG).

Performance: MCU and its adaptive variant (MCU $\beta$ ) consistently achieved the top two overall performances across random data forgetting and class-wise forgetting scenarios. In class-wise forgetting on ImageNet-100, MCU $\beta$ performed nearly on par with the retraining (RT) baseline.
Comparison with Linear Methods: The results highlighted the superiority of nonlinear unlearning over linear methods like NegTV. Linear methods struggled with the trade-off between under-forgetting and over-forgetting, whereas MCU maintained a stable balance.
Robustness: MCU $\beta$ demonstrated robustness across different pre-unlearning models (including degraded ones like Gradient Ascent) and varying proportions of retaining data (performing well even with only 10-30% of retaining data).
Efficiency: The parameter mask strategy reduced running time significantly compared to baselines, while the adaptive $\beta$ strategy removed the need for hyperparameter search.

Significance and Claims

The paper claims that MCU offers a more effective and flexible solution to machine unlearning by moving beyond the constraints of linear updates. By uncovering a continuous spectrum of unlearning models, it addresses the fundamental limitation that a single model cannot optimally serve all forgetting points.

The authors emphasize that MCU serves as a plug-and-play framework that can be integrated with any existing MU method to improve efficacy. The ability to select different points along the pathway allows practitioners to adapt to evolving risk assessments (e.g., a data point becoming more sensitive over time) without the cost of retraining. The work suggests that exploring nonlinear pathways in parameter space is a viable and necessary direction for overcoming weight entanglement and achieving high-quality machine unlearning.