Towards Privacy-Guaranteed Label Unlearning in Vertical Federated Learning: Few-Shot Forgetting without Disclosure

This paper proposes the first label unlearning method for Vertical Federated Learning that utilizes a representation-level manifold mixup mechanism to generate synthetic embeddings for gradient-based forgetting and recovery, effectively removing sensitive label information while preserving model utility and computational efficiency across diverse datasets.

The Gist

Imagine you and a few friends are building a giant, super-smart detective together. You all have different pieces of the puzzle, but you can't show each other your pieces because they are private secrets.

• You (The Active Party) have the answers (the labels). You know who is sick, who is a good credit risk, or what the text is about.
• Your Friends (The Passive Parties) have the clues (the features). They have the X-rays, the bank transaction logs, or the first half of a sentence.

Together, you train a model to solve mysteries. But what happens if someone says, "Wait! I want to forget that specific person ever existed in our training data. Please erase them!" This is called "Unlearning."

In the world of machine learning, this is usually hard. It's like trying to remove one specific grain of sand from a beach without moving the whole beach or rebuilding the entire shoreline.

The Problem: The "Vertical" Dilemma

In this specific paper, the authors are tackling a tricky version of this called Vertical Federated Learning.

• The Catch: Because you and your friends have different types of data (answers vs. clues), you have to constantly talk to each other during training. You can't just work alone.
• The Privacy Risk: If you want to forget a specific patient's HIV status (the label), the old ways of doing this often required you to tell your friends exactly which patient you were deleting. This leaks the very secret you were trying to protect! It's like asking your friends to help you erase a name from a list, but in the process, you accidentally whisper, "Oh, by the way, I'm deleting John's name," revealing that John was on the list.

The Solution: "Few-Shot Forgetting" with a Magic Trick

The authors propose a clever new method that acts like a magic eraser that works with just a tiny scrap of paper, rather than the whole book. Here is how they do it, using simple analogies:

1. The "Manifold Mixup" (The Photocopy Blender)

Usually, to forget something, you need to see the thing you want to forget many times to understand how to remove it. But you only have a few samples allowed (due to privacy).

• The Trick: Instead of just looking at the few samples you have, the authors use a technique called Manifold Mixup. Imagine taking two photos of a cat and a dog, and blending them together in a blender to create a "cat-dog" hybrid.
• Why it helps: They don't just mix the raw photos; they mix the internal thoughts (embeddings) of the AI. This creates thousands of "synthetic" hybrid examples from just a few real ones. It's like having a magic photocopier that can generate infinite variations of a single page, giving the AI enough "practice" to learn how to forget without needing the original data.

2. The "Gradient Ascent" (The Reverse Drive)

Once they have these blended, synthetic examples, they perform a special dance called Gradient Ascent.

• Normal Training: The AI tries to get the answer right. (Driving forward).
• Unlearning: The AI tries to get the answer wrong specifically for the person being forgotten. (Driving in reverse).
• The Magic: Because they used the "blended" examples, the AI learns to reverse its thinking about that specific label very quickly. It's like practicing a dance move in reverse so well that you can unlearn the choreography in seconds.

3. The "Recovery Phase" (The Safety Net)

When you erase a memory, you might accidentally forget some other things too.

• The Fix: After the "erasing" dance, they do a quick "recovery" step. They take a tiny bit of the good data (the people they want to keep) and gently nudge the AI back to being smart about those people.
• Result: The AI forgets the specific person you wanted gone but remembers everyone else perfectly.

Why This is a Big Deal

1. Privacy First: The most important part is that the "Active Party" (who holds the labels) never has to tell the "Passive Parties" (who hold the clues) which specific person is being deleted. They just send a generic signal based on the blended examples. It's like telling your friends, "Let's forget the concept of 'red' for a moment," without pointing at a specific red apple.
2. Speed: Old methods might take hours or days to retrain the model. This method does it in seconds.
3. Efficiency: It works even if you only have a handful of samples (a "few-shot" approach). You don't need the whole database to delete one entry.

The Bottom Line

This paper introduces a way to delete sensitive information from a collaborative AI system without exposing who is being deleted, without slowing everything down, and without ruining the AI's ability to help everyone else.

Think of it as a secure, instant "Undo" button for AI, where you can erase a specific memory without the rest of the team knowing what that memory was. It's a huge step forward for privacy in the age of collaborative AI.

Technical Summary

1. Problem Statement

The paper addresses the critical challenge of Machine Unlearning within Vertical Federated Learning (VFL) settings.

• Context: In VFL, multiple parties hold different features of the same samples (e.g., a bank holds financial data, a hospital holds medical records, and a third party holds the labels). The "Active Party" holds the labels, while "Passive Parties" hold features.
• The Gap: Existing unlearning research focuses heavily on Horizontal Federated Learning (HFL) or removing entire passive clients in VFL. There is a lack of methods for Label Unlearning in VFL, where specific labels (often highly sensitive, e.g., HIV status or loan approval) must be removed from the model without retraining the entire system.
• Key Challenges:
  1. Privacy Leakage: Traditional unlearning methods (like retraining or gradient-based approaches) often require exchanging intermediate data (embeddings/gradients) that can inadvertently reveal which specific samples or labels are being deleted to passive parties.
  2. Data Scarcity: Unlearning usually requires access to the specific data to be removed. In VFL, the active party may not want to share the raw unlearning data with passive parties, or the unlearning set might be small (Few-Shot).
  3. Efficiency: Full retraining is computationally prohibitive in distributed VFL environments due to synchronization constraints.

2. Methodology

The authors propose a Few-Shot Label Unlearning Framework that operates in three distinct phases. The core innovation is the use of Manifold Mixup at the representation level to generate synthetic embeddings, allowing effective unlearning with a minimal public dataset.

Phase 1: Vertical Manifold Mixup (Data Augmentation)

Instead of mixing raw features (which is impossible in VFL as parties hold disjoint features), the method mixes hidden embeddings.

• Process: The active party collects embeddings from passive parties. It then applies Manifold Mixup to these embeddings and the corresponding labels.
• Formula: For embeddings $H$ and labels $y$, the mixup is defined as:
  $$\text{Mix}_\lambda(a, b) = \lambda \cdot a + (1 - \lambda) \cdot b$$
  where $\lambda \in [0, 1]$.
• Purpose: This generates a synthetic dataset ($\vec{H}_u, \vec{y}_u$) from a small public subset of unlearning data ($D_{p,u}$). This flattens the state distribution and provides richer signals for the subsequent gradient steps, compensating for the small sample size.

Phase 2: Vertical Gradient-Based Label Unlearning

This phase removes the influence of the target labels from both active and passive models using Gradient Ascent.

• Active Party: Performs gradient ascent on the mixed embeddings and labels to maximize the loss, effectively "unlearning" the association between the features and the target label.
  $$\omega \leftarrow \omega + \eta \nabla_\omega \ell(F_\omega(\vec{H}_u), \vec{y}_u)$$
• Passive Parties: The active party computes gradients with respect to the embeddings ($\frac{\partial \ell}{\partial \vec{H}_k}$) and sends them to the passive parties. The passive parties update their local models using these gradients (also via ascent) to forget the corresponding representations.
  $$\theta_k \leftarrow \theta_k + \eta \nabla_{\vec{H}_k} \ell \cdot \nabla_{\theta_k} \vec{H}_k$$
• Theoretical Guarantee: The paper proves (Theorem 1) that the gradient direction derived from the manifold-mixed public data is positively aligned with the gradient direction of the full unlearning dataset, ensuring effectiveness even with few samples.

Phase 3: Remained Accuracy Recovery

Gradient ascent on the unlearning set can degrade performance on the retained data.

• Process: The active party performs standard Gradient Descent on a small set of retained data ($D_{p,r}$) using the mixed embeddings of the retained samples.
• Goal: This step refines the model to restore accuracy on the non-unlearned classes without retraining from scratch.

3. Key Contributions

1. First VFL Label Unlearning Method: This is the first work to specifically address label unlearning in Vertical Federated Learning, a setting previously overlooked compared to HFL.
2. Few-Shot Efficiency: The method achieves effective unlearning using only a tiny public dataset (e.g., ~40 samples per label) by leveraging Manifold Mixup, avoiding the need for full dataset access or retraining.
3. Process Privacy (Transcript-Level): The paper introduces the concept of Process Privacy, quantifying how much information about the deletion set is leaked to passive parties during the unlearning protocol.
   • Unlike retraining (which reveals 100% of the deletion set), the proposed method limits leakage to the embeddings of a small public subset.
   • Empirical results show a reduction in membership leakage from 100% (Retraining) to as low as 4.04% (CIFAR-100) and 14.38% (CIFAR-10).
4. Computational Efficiency: The method is extremely fast, completing unlearning in seconds (16x to 1200x faster than baselines like Fine-Tuning or Fisher Forgetting).

4. Experimental Results

The authors evaluated the method on diverse datasets (MNIST, CIFAR-10/100, ModelNet, Brain Tumor MRI, COVID-19 Radiography, Yahoo Answers) and architectures (ResNet18, VGG16, MixText).

• Utility Preservation ($D_r$): The method consistently maintains high accuracy on retained data (often >98% of the baseline), outperforming baselines like Fisher Forgetting (which causes severe degradation) and Amnesiac Unlearning.
• Unlearning Effectiveness ($y_u$): The accuracy on the unlearned labels drops to near-random levels (e.g., from ~90% to <2%), effectively "forgetting" the target class.
• Attack Success Rate (ASR): The method achieves low ASR (close to retrained models), indicating it does not suffer from the "Streisand effect" (where the model over-corrects and reveals the unlearned class by consistently misclassifying it as a specific wrong label).
• Scalability: Performance remains stable regardless of the number of passive parties (tested up to 8) and under privacy-preserving mechanisms like Differential Privacy and Gradient Compression.
• Multi-Label Unlearning: The method scales effectively to unlearning 2 or 4 labels simultaneously, where other baselines fail to preserve utility.

5. Significance

• Regulatory Compliance: Provides a practical solution for the "Right to be Forgotten" (GDPR/CCPA) in vertical data collaborations, which are common in banking, healthcare, and e-commerce.
• Privacy Paradigm Shift: By defining and measuring "Process Privacy," the paper highlights a new dimension of privacy risk in federated unlearning that previous works ignored.
• Efficiency: Demonstrates that high-quality unlearning does not require expensive retraining, making it viable for real-time or resource-constrained VFL deployments.
• Generalizability: The approach works across different modalities (images and text) and model architectures, suggesting broad applicability in collaborative AI systems.

In summary, this paper presents a robust, efficient, and privacy-preserving framework for removing sensitive label information from Vertical Federated Learning models using a novel few-shot manifold mixup strategy, setting a new standard for unlearning in vertical settings.