💬 NLP

Deep Contrastive Unlearning for Language Models

This paper proposes DeepCUT, a machine unlearning framework that optimizes the latent space of language models using deep contrastive learning to effectively remove specific training data while preserving predictive quality, addressing the limitations of existing methods that ignore geometric sample distributions.

Original authors: Estrid He, Tabinda Sarwar, Ibrahim Khalil, Xun Yi, Ke Wang

Published 2026-08-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Estrid He, Tabinda Sarwar, Ibrahim Khalil, Xun Yi, Ke Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital age, our lives are increasingly recorded in text: social media posts, medical records, legal documents, and private emails. Large language models, the powerful computer programs that can write, translate, and answer questions, are trained on vast oceans of this human-generated data. They learn by reading millions of these examples, absorbing patterns of language and knowledge to become incredibly useful tools. However, this reliance on real-world data creates a significant problem. If a person decides they no longer want their private information used, or if a piece of data turns out to be incorrect or sensitive, the current way to remove it is to delete the data and start the entire training process over from scratch. For models that have taken weeks to train on billions of words, this is like burning down a library just to remove a single book. It is too slow, too expensive, and often impossible to do quickly when people exercise their right to be forgotten.

To solve this, researchers have been developing a field called machine unlearning. The goal is to teach a model to forget specific information without having to retrain it entirely. Imagine a student who has memorized a textbook; machine unlearning aims to make that student forget a specific chapter while still remembering everything else, all without sending them back to school for a new year. While some methods exist to do this for images or simple data, applying it to the complex, nuanced world of language has proven difficult. Most existing attempts focus only on the final answers a model gives, trying to nudge the output so it no longer mentions the forgotten topic. But this approach often leaves the underlying memory of the data intact, hidden deep within the model's internal calculations, ready to resurface.

A team of researchers at RMIT University has proposed a new method called DeepCUT, which takes a different approach by looking directly at how the model organizes information inside its own mind. Instead of just tweaking the final answers, DeepCUT reorganizes the internal map where the model stores its knowledge. The researchers trained a language model on four different real-world datasets, ranging from noisy social media posts to precise biomedical literature. They then asked the model to forget specific portions of this data, ranging from one percent to ten percent of the total training set. To test if the forgetting worked, they measured how well the model could still answer questions about the data it was supposed to have forgotten, and how well it retained its ability to handle the data it was supposed to keep.

The results showed that previous methods were often inefficient or incomplete. Simply continuing to train the model on the remaining data, a technique known as catastrophic forgetting, failed to erase the specific memories of the removed text; the model still remembered the forbidden data with high accuracy. Another common strategy, which involves splitting the data into smaller chunks and only retraining the affected pieces, worked well for removing the data but significantly hurt the model's overall intelligence, making it less accurate on general tasks. In contrast, the DeepCUT method successfully erased the specific memories while keeping the model's general performance intact. When tested on the data it was supposed to forget, the DeepCUT model's accuracy dropped dramatically, indicating it had truly forgotten the information. At the same time, it maintained high accuracy on the data it was supposed to remember, performing just as well as a model that had been fully retrained from scratch.

The secret to this success lies in how the method manipulates the model's internal space. The researchers treated the data points as locations on a map. To make the model forget a specific piece of information, they pushed that piece of data away from its own group and pulled it closer to different groups, effectively scrambling its unique identity within the model's memory. This process ensures that the model can no longer recognize the specific features that made that data distinct, while leaving the rest of the map undisturbed. The experiments demonstrated that this approach was not only effective at removing the data but also much faster than retraining the entire model. By focusing on the geometric arrangement of information inside the model, the researchers found a way to surgically remove unwanted memories, offering a practical path forward for protecting privacy in the era of artificial intelligence.

Technical Summary: Deep Contrastive Unlearning for Language Models

Problem Statement

The rapid advancement of Large Language Models (LLMs) relies on training with vast amounts of web data, including user-generated content that may contain sensitive personal information or copyrighted material. While LLMs demonstrate powerful capabilities, they pose significant risks regarding user privacy and data protection, particularly in light of regulations like the GDPR which enforces the "right to be forgotten."

The core challenge is machine unlearning: the process of removing the influence of specific training samples from a model without retraining the entire model from scratch. Retraining is computationally prohibitive for LLMs due to their scale and the time required for training rounds. Existing unlearning methods for LLMs primarily focus on mitigating the impact of forgotten samples on the model's output distributions (e.g., classification probabilities). However, these approaches often fail to explicitly consider or optimize the geometric distributions of samples within the model's latent space. Consequently, they may not effectively erase the discriminative features of the data to be forgotten, leading to residual memory traces.

Methodology: DeepCUT

To address these limitations, the authors propose Deep Contrastive Unlearning for fine-Tuning (DeepCUT). This framework operates by directly optimizing the latent space of a fine-tuned language model to facilitate the forgetting of specific samples while preserving the model's performance on remaining data.

The methodology consists of three main components:

Input Encoding and Augmentation:
- The framework utilizes a pre-trained LLM encoder to map input sentences into a latent embedding space.
- To facilitate contrastive learning within a mini-batch, the authors employ data augmentation via standard dropout. By passing a sample through a dropout layer twice with different masks, they generate multiple views of the same anchor sample. This ensures the availability of positive samples (same class) within the batch, even when the batch size is small or the specific class is underrepresented.
Contrastive Unlearning Objective:
- Unlike traditional contrastive learning which pulls positive samples together and pushes negatives apart, DeepCUT inverts this logic for the data to be forgotten ( $D_f$ ).
- For an anchor sample $x_f$ in the forget set, the model is fine-tuned to push its latent embedding away from instances of its own class ( $D_y$ ) and pull it closer to instances of different classes ( $D_{\neg y}$ ).
- This is achieved using a modified contrastive loss (inspired by FoCL loss), formulated as:
  $L_f = -\sum_{x_f \in D_f} \frac{1}{|D_{\neg y}|} \sum_{x_i \in D_{\neg y}} \log \frac{\exp(z_i^\top z_f / \tau)}{\exp(z_i^\top z_f / \tau) + \sum_{z_j \in D_y} \exp(z_j^\top z_f / \tau)}$
- This process alters the learned representation of the forgotten data, effectively erasing its unique discriminative features and moving it toward the decision boundaries of other classes.
Combined Optimization:
- To prevent catastrophic forgetting of the remaining data ( $D_r$ ), the unlearning loss ( $L_f$ ) is combined with the standard cross-entropy loss ( $L_{CE}$ ) for the retained samples.
- The final objective function is:
  $L = -\sum_{x_r \in D_r} \log \frac{\exp(\theta_{y_r}^\top z_r)}{\sum_c \exp(\theta_c^\top z_r)} + \gamma L_f$
- Here, $\gamma$ is a hyperparameter balancing the retention of useful knowledge and the removal of specific data.

Key Contributions

The paper outlines three primary contributions:

Framework Introduction: The proposal of DeepCUT, an effective unlearning framework designed to selectively forget specific samples in trained models by operating directly on the latent space.
Latent Space Optimization: A novel approach inspired by contrastive learning that optimizes sample distributions within the latent space. This ensures the removal of discriminative features specific to the samples to be forgotten, rather than just adjusting output probabilities.
Empirical Validation: Extensive experiments on real-world datasets demonstrating that DeepCUT can efficiently forget specified samples without degrading the model's predictive performance on the remaining data.

Experimental Results

The authors evaluated DeepCUT on four benchmark datasets across two domains: Social Media (WNUT16, WNUT17) and Biochemical (NCBI-Disease, ChEMU). The model was compared against several baselines:

Retrain: Exact unlearning via full retraining (ground truth).
Fine-tune: Standard fine-tuning on the remaining data (relying on catastrophic forgetting).
Reverse Gradient: Reversing gradients of the forget set.
SISA: Sharding, Isolation, Slicing, and Aggregation technique.

Key Findings:

Unlearning Effectiveness: DeepCUT consistently achieved the lowest accuracy on the "forget set" across all datasets and configurations (1% and 10% removal rates), significantly outperforming baselines like Reverse Gradient. For instance, on WNUT16 with 1% removal, DeepCUT reduced forget-set accuracy to 32.25%, which is lower (indicating more effective unlearning) than the 54.55% achieved by the ground truth retrained model and the 99.14% achieved by the standard Fine-tune method.
Preservation of Performance: DeepCUT maintained high accuracy on the "retained set" and the "test set," comparable to the retrained model and superior to SISA (which suffered accuracy drops due to data sharding).
Efficiency: In terms of running time, DeepCUT was significantly faster than the "Retrain" baseline and more efficient than the "Fine-tune" approach, which required extensive epochs to achieve partial forgetting. While SISA was fast, it compromised overall model accuracy.

Significance and Claims

The paper claims that DeepCUT represents a significant step forward in machine unlearning for LLMs by shifting the focus from output-level mitigation to geometric optimization in the latent space. The authors argue that existing methods fail to explicitly address the distribution of samples in the latent space, leading to incomplete unlearning. By directly manipulating the latent embeddings to erase discriminative features, DeepCUT offers a more principled, precise, and efficient solution. The results suggest that contrastive latent space manipulation is a viable and superior approach for ensuring privacy compliance in large-scale language models without the prohibitive costs of full retraining.