Technical Summary: CoUn - Empowering Machine Unlearning via Contrastive Learning

1. Problem Definition

Machine Unlearning (MU) addresses the growing necessity for data privacy and regulatory compliance (e.g., GDPR) by removing the influence of specific "forget" data ($D_u$) from a trained model while preserving the utility of the remaining "retain" data ($D_r$). The gold standard for MU is exact unlearning, achieved by retraining a model from scratch on $D_r$ (the Retrain model). However, this approach is computationally prohibitive.

Approximate unlearning methods aim to achieve similar results more efficiently but often struggle with a performance gap compared to the Retrain model. Existing approximate methods generally fall into two categories:

1. Label Manipulation: Assigning forget samples to semantically inconsistent or random labels to force misclassification. This often degrades model utility on retain data and requires access to forget data.
2. Model Weight Perturbation: Modifying model weights to erase forget data influence. Excessive perturbation harms utility, while insufficient perturbation fails to remove the influence of forget data.

A critical observation driving this work is that a Retrain model (trained only on $D_r$) classifies forget samples ($D_u$) based on their semantic similarity to the retain clusters. For instance, in a class-wise forgetting scenario (removing all "truck" samples), the Retrain model misclassifies "truck" forget samples into semantically similar clusters like "automobile" or "ship." In random forgetting, it may correctly classify some forget samples if they remain semantically close to their original retain clusters. Current approximate methods often fail to replicate this nuanced behavior, either forcing total misclassification or failing to sufficiently disrupt the forget data's influence.

2. Methodology: CoUn

The authors propose CoUn (Contrastive learning for empowering Unlearning), a framework that achieves effective unlearning by adjusting learned data representations based on semantic similarity, using only retain data. CoUn does not require access to forget data.

Core Mechanism

CoUn leverages the fact that the original model's weights already capture semantic similarities between all training samples. By adjusting the representations of the retain data ($D_r$), the framework indirectly influences the representations of the forget data ($D_u$).

The framework consists of two simultaneous learning objectives applied to $D_r$:

1. Contrastive Learning (CL) Module:

   • Applied to retain data to adjust representations.
   • Uses data augmentation to create positive pairs (two views of the same image) and negative pairs (different images).
   • Cluster Collision: The CL module uses a standard InfoNCE loss. Crucially, because the batch is sampled randomly from $D_r$, it inevitably includes false negatives (samples from the same class as the positive sample but treated as negatives).
   • This induces cluster collision, where samples from the same class are pushed apart. Consequently, forget representations (which share semantic information with retain data via the original weights) are indirectly pushed toward clusters of other retain samples that exhibit the highest semantic similarity to them.
   • In class-wise forgetting, this pushes forget samples into different clusters. In random forgetting, it aligns forget samples with their most semantically similar retain clusters, mimicking the behavior of the Retrain model.

2. Supervised Learning:

   • Applied to retain data via a classifier head using Cross-Entropy (CE) loss.
   • Utility Preservation: While CL induces cluster collision (which is beneficial for unlearning forget data), it risks degrading the model's utility on retain data. The supervised learning component counteracts this by enforcing that retain representations remain within their respective correct clusters, maintaining high accuracy on $D_r$.

Optimization

The total loss function combines both objectives:
$$L = L_{CE} + \lambda L_{CL}$$
Where $L_{CE}$ is the supervised cross-entropy loss, $L_{CL}$ is the contrastive loss, and $\lambda$ is a scaling factor balancing the two.

3. Key Contributions

1. Novel Framework: Introduction of CoUn, a MU framework that achieves effective unlearning by adjusting learned representations based on semantic similarity using CL and supervised learning exclusively on retain data.
2. Empirical and Theoretical Insights:
   • Demonstrated that CoUn effectively unlearns by pushing forget representations into clusters of semantically similar retain representations, closely matching the behavior of the exact Retrain model.
   • Provided a theoretical analysis showing that CoUn induces a higher misclassification rate on forget data while maintaining a low misclassification rate on retain data, optimizing both forget quality and utility.
3. Performance Validation: Validated CoUn across multiple datasets (CIFAR-10/100, TinyImageNet) and architectures (ResNet-18, VGG-16, ViT). The results show CoUn consistently outperforms state-of-the-art baselines in reducing the performance gap with the exact Retrain model.
4. Modular Integration: Demonstrated that integrating the CoUn CL module into existing baseline methods significantly empowers their unlearning effectiveness.

4. Experimental Results

Experiments were conducted under two forgetting scenarios: Class-wise (removing an entire class) and Random (removing a percentage of samples).

• Performance Gap: CoUn consistently achieved the lowest "Average Gap" (the average difference in Retain Accuracy, Unlearn Accuracy, Test Accuracy, and MIA success rate compared to the Retrain model) across all datasets and models.
  • On CIFAR-10 with ResNet-18 (10% random forgetting), CoUn outperformed the best baseline by 16.7%.
  • On CIFAR-100 with ResNet-18, the improvement was 15.8%.
  • On TinyImageNet, the improvement was 6.3%.
• Comparison with Baselines: CoUn outperformed methods like NegGrad+, $\ell_1$-sparse, SalUn, and NoT. Notably, methods relying on label manipulation (e.g., SalUn) often suffered from reduced utility on retain data.
• Integration: Adding the CoUn CL module to competitive baselines (NegGrad+, $\ell_1$-sparse, NoT) resulted in substantial performance boosts (e.g., up to 87.7% improvement in average gap for $\ell_1$-sparse on CIFAR-100 with 50% forgetting).
• Ablation Studies:
  • Scaling Factor ($\lambda$): Proper tuning is essential; high $\lambda$ reduces supervised constraints, degrading utility.
  • Temperature ($\tau$): Lower temperatures generally improve CL effectiveness, with optimal performance at $\tau=0.1$.
  • Transformations: Stronger augmentations in CL can lead to tighter clustering, which paradoxically hinders unlearning by preventing the necessary cluster overlap. Simpler transformations (matching those used in supervised learning) yielded the best results.
  • Batch Size: Performance was optimal when the batch size matched that of the Retrain model (256).

5. Significance and Claims

The paper claims that CoUn represents a significant step forward in approximate machine unlearning by shifting the paradigm from "forcing misclassification" to "mimicking semantic similarity."

• Alignment with Exact Unlearning: Unlike prior methods that attempt to blindly misclassify forget data, CoUn aligns with the natural behavior of the exact Retrain model, which classifies forget data based on semantic proximity to retain data.
• Efficiency and Utility: CoUn achieves high unlearning effectiveness without requiring access to forget data and maintains high utility on retain data by mitigating cluster collision through supervised learning.
• Generalizability: The framework is shown to be effective across different model architectures and forgetting scenarios, and its core CL module can serve as a plug-in to enhance existing unlearning methods.

The authors conclude that while CoUn currently focuses on vision-based classification on relatively small datasets, its approach of leveraging semantic similarity and cluster collision offers a robust direction for future research in machine unlearning.