🤖 machine learning

Spectral Saliency for Machine Unlearning

This paper introduces Spectral Saliency Unlearning (SSU), a machine unlearning method inspired by Muon that selectively updates weak spectral directions based on confident unlearning signals to effectively remove specific training data influence while preserving model utility across various architectures.

Original authors: Cedar Site Bai, Amber Yijia Zheng, Raymond A. Yeh, Brian Bullins

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Cedar Site Bai, Amber Yijia Zheng, Raymond A. Yeh, Brian Bullins

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern digital world, artificial intelligence systems are trained on vast oceans of data to learn how to recognize faces, generate images, or write text. Once a model has learned from this data, it becomes difficult to change its mind. If a person requests that their private information be removed from a dataset, as required by laws like the General Data Protection Regulation, the standard solution has been to delete that data and retrain the entire system from scratch. This process is incredibly expensive and time-consuming, often requiring the same massive computing power used for the original training. To solve this, researchers have developed a field called machine unlearning, which aims to surgically remove the influence of specific data points from a trained model without having to rebuild it entirely. The goal is to make the model forget the unwanted information while keeping its ability to perform well on everything else.

However, simply trying to reverse the learning process is tricky. When a computer tries to "unlearn" a specific image or fact, the adjustments it makes to its internal settings can accidentally damage its knowledge of other, unrelated things. It is like trying to remove a single stain from a complex tapestry; if you pull too hard on the wrong thread, you might unravel the whole pattern. Recent methods have attempted to fix this by selectively ignoring certain parts of the learning signal, but they often rely on trial and error. A new study by researchers at Purdue University introduces a more precise approach called Spectral Saliency Unlearning. Instead of guessing which parts of the model to adjust, this method analyzes the mathematical structure of the learning process itself to identify which directions are safe to change and which are too risky.

The researchers built their method on a concept from advanced mathematics known as spectral analysis, which breaks down complex data into its fundamental components, much like separating a chord into individual musical notes. In the context of training an artificial intelligence, the system learns by moving through a vast landscape of possibilities, guided by signals that tell it how to improve. The new study suggests that not all of these signals are equally reliable. Some signals are strong and clear, pointing directly toward the information that needs to be forgotten. Others are weak and fuzzy, often representing a tangled mix of the information to be forgotten and the information that must be kept. The researchers found that when a model tries to unlearn data using these weak, mixed signals, it often ends up hurting its own performance on the data it was supposed to remember.

To address this, the team proposed a technique that acts as a filter for these learning signals. They developed a way to measure the strength of each direction in which the model could change. If a direction is supported by a strong, confident signal to forget, the model follows it. But if the signal is weak or ambiguous, the method suppresses it, effectively telling the model to ignore that particular path. This prevents the model from making clumsy adjustments that confuse its retained knowledge with the data it is trying to erase. The researchers tested this idea across three very different types of artificial intelligence: systems that classify images, systems that generate new pictures, and large language models that write text. In every case, the new method allowed the models to forget the target data more effectively while preserving their ability to handle the rest of the world.

The results of these experiments were striking. When applied to image classifiers trained on the CIFAR-10 dataset, the new method reduced the gap between the unlearned model and a perfectly retrained model by over thirty percent. In the realm of image generation, where the goal is to create pictures of specific objects, the method achieved perfect forgetting of the unwanted categories while simultaneously improving the quality of the remaining images by nearly twenty-four percent. For large language models, which are often used to write stories or answer questions, the approach consistently improved the balance between forgetting specific facts and maintaining general writing ability. The researchers noted that this technique works by focusing only on the most dominant and reliable directions of change, rather than trying to adjust every part of the model at once.

One of the most significant aspects of this work is that it provides a theoretical explanation for why previous methods, which relied on simple rules of thumb, had been successful. By analyzing the mathematical properties of the learning signals, the researchers showed that the weak, noisy directions are exactly where the conflict between forgetting and remembering happens. By filtering these out, the model avoids the worst of the interference. This insight transforms unlearning from a heuristic process, based on guesswork, into a principled one based on the structure of the data itself. The study demonstrates that by being more selective about which directions to update, artificial intelligence systems can be made to forget specific information without losing their general intelligence, offering a practical and efficient path toward compliant and ethical AI systems.

Technical Summary: Spectral Saliency for Machine Unlearning

Problem Statement

Machine Unlearning (MU) aims to remove the influence of specific training data (the "forget set") from a trained model without resorting to full retraining, a requirement driven by data governance regulations like GDPR. While recent approximate unlearning methods utilize gradient-based updates to optimize a forgetting objective (increasing loss on the forget set) alongside a retention objective (preserving utility on the remaining data), they often face a fundamental trade-off: aggressive forgetting can degrade model performance on retained data.

Existing approaches, such as SalUn, employ coordinate-wise masking to suppress low-magnitude gradient updates, treating them as noise. However, the theoretical justification for why suppressing specific directions improves the forgetting-retention trade-off has largely been heuristic. Furthermore, standard gradient descent updates may inadvertently update directions where forgetting and retention objectives are entangled, leading to unnecessary utility degradation.

Methodology: Spectral Saliency Unlearning (SSU)

Inspired by the Muon optimizer, which applies spectral magnitude normalization to encourage exploration of rare directions in learning, this paper proposes Spectral Saliency Unlearning (SSU). SSU reframes unlearning as a partial "reversal" of the optimization process, operating in the spectral domain of the unlearning gradient.

Core Mechanism

SSU replaces the standard unlearning gradient $G_u$ with a singular value thresholded counterpart. The procedure involves:

Spectral Decomposition: For matrix-structured parameters, the unlearning gradient $G_u$ is decomposed via Singular Value Decomposition (SVD): $G_u = U \text{diag}(\sigma) V^\top$ .
Spectral Saliency Definition: The magnitude of the singular value $\sigma_i$ is defined as the "spectral saliency" of the corresponding direction. Large $\sigma_i$ indicates a strong, reliable unlearning signal, while small $\sigma_i$ suggests weak support.
Thresholding: Weak spectral components are suppressed. The method retains only the top $\gamma$ fraction of singular components (where $\gamma \in (0, 1]$ is a keep ratio). The thresholded gradient $\tilde{G}_u$ is reconstructed using only the dominant singular vectors and values:
$\tilde{G}_u = \sum_{i=1}^{k} \sigma_i u_i v_i^\top$
where $k = \lfloor \gamma m \rfloor$ .
Hybrid Application: For vector- and scalar-structured parameters (e.g., biases), SSU resorts to coordinate-wise thresholding similar to SalUn, mirroring the practice of using vector-based optimizers for these parameters.

Theoretical Justification

The paper provides a theoretical framework linking weak spectral components to the forgetting-retention trade-off:

Forget-Retain Conflict: The authors define "forget-retain alignment" to measure the directional conflict between the forgetting gradient ( $G_f$ ) and retention gradient ( $G_r$ ).
Proposition 4.3: The analysis suggests that directions with small singular values (weak spectral components) often correspond to scenarios where $G_f$ and $G_r$ are negatively aligned. In these directions, an update that improves forgetting necessarily worsens retention (and vice versa).
Mechanism of Improvement: By suppressing these weak components, SSU avoids updating directions where forgetting and retention are entangled. This mitigates unnecessary degradation in model utility while preserving the dominant unlearning signals.
Extension to Coordinate Masking: The paper extends this directional analysis to coordinate-wise masking (SalUn), offering a theoretical explanation for why suppressing small-magnitude coordinates is beneficial: it prevents updates where forgetting and retention signals cancel each other out.

Key Contributions

Novel Algorithm: Introduction of SSU, a singular value thresholding approach for gradient-based unlearning that suppresses weak spectral components of the unlearning gradient.
Theoretical Insight: A principled justification for saliency-based thresholding from the perspective of the forgetting-retention trade-off, characterizing the relationship between weak directions and forget-retain interference. This analysis also provides a theoretical basis for the efficacy of coordinate-wise masking methods like SalUn.
Empirical Validation: Comprehensive evaluation across diverse model architectures and tasks, demonstrating consistent improvements over state-of-the-art baselines.

Experimental Results

The authors evaluated SSU across three domains: image classification, image generation (diffusion models), and large language models (LLMs).

1. Image Classification (CIFAR-10, ResNet-18)

Setup: Random subset unlearning (10% of data).
Results: SSU improved upon the strong SalUn baseline, reducing the average gap to the retrained oracle from 1.11 to 0.77. It achieved this while maintaining comparable performance in forgetting set accuracy (FA), remaining set accuracy (RA), and test set accuracy (TA).
Efficiency: The computational overhead was negligible as SVD was applied only to convolutional layers.

2. Image Generation (DDPM and Stable Diffusion)

Setup: Class-wise unlearning on CIFAR-10 (DDPM) and Imagenette (Stable Diffusion V1.4).
Results:
- DDPM: SSU achieved perfect forgetting (FA $\approx$ 0%) across all classes while significantly improving generation quality on retained classes. The Fréchet Inception Distance (FID) showed a 23.6% improvement compared to baselines.
- Stable Diffusion: SSU achieved an average FID of 1.24, outperforming ESD (1.71) and SalUn (1.55), while maintaining strong forgetting efficacy (FA 0.20%).
Qualitative: Visual results confirmed that unlearned classes were effectively removed (e.g., "tench" images no longer generated) while retained classes maintained high fidelity.

3. Large Language Models (TOFU Benchmark, Llama2-7B)

Setup: Random subset unlearning (1%, 5%, 10% of data) for both untargeted and targeted scenarios.
Results: SSU was applied as a plug-in to the best-performing baselines (ME+GD for untargeted, IDK+AP for targeted).
- Untargeted: Improved Model Utility (MU) from 0.7245 to 0.7477 (forget01) and 0.7312 to 0.7480 (forget10) while simultaneously increasing Forget Efficacy (FE).
- Targeted: Achieved the best average scores across all scenarios (e.g., 0.7856 for forget01), demonstrating a superior balance between preserving model capabilities and effective unlearning.

Ablation Studies

Keep Ratio: A ratio of $\gamma=0.3$ was found to offer the best balance between forgetting strength and model utility.
Layer Selection: In Stable Diffusion, applying SVD specifically to cross-attention layers yielded the best trade-off. Applying SVD to all layers slightly degraded generation quality, suggesting that unlearning should target concept-carrying components rather than low-level visual features.

Significance and Claims

The paper claims that SSU offers a principled, optimizer-agnostic mechanism for machine unlearning. By shifting the perspective from coordinate-wise heuristics to spectral analysis, the authors argue that suppressing weak spectral components directly addresses the "forget-retain conflict."

The significance lies in:

Theoretical Grounding: Moving beyond empirical heuristics to provide a theoretical explanation for why suppressing specific gradient directions improves unlearning.
Generalizability: Demonstrating that a single spectral thresholding strategy can effectively enhance unlearning pipelines across vastly different architectures (CNNs, Diffusion Models, Transformers) and tasks.
Performance: Consistently improving the forgetting-utility trade-off, achieving state-of-the-art results in generation quality and model utility while maintaining high forgetting efficacy.

The authors remain modest, noting that their theory characterizes the trade-off rather than providing a complete quantitative prediction of end-to-end performance, and that extending the analysis to weaker assumptions remains a direction for future work.