🤖 machine learning

Privacy Leakage via Output Label Space and Differentially Private Continual Learning

This paper identifies that the changing output label space in continual learning acts as a privacy side-channel and proposes a new formalization of differential privacy for continual learning, along with methods to mitigate this leakage while maintaining high model accuracy.

Original authors: Marlon Tobaben, Talal Alrawajfeh, Marcus Klasson, Mikko Heikkilä, Arno Solin, Antti Honkela

Published 2026-04-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Marlon Tobaben, Talal Alrawajfeh, Marcus Klasson, Mikko Heikkilä, Arno Solin, Antti Honkela

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a librarian in a highly secretive, high-security library. This library contains sensitive information about people’s medical records. To protect everyone, you follow a strict rule called Differential Privacy (DP). This rule is like a "blurring filter": whenever you share information, you add just enough "noise" or static to the data so that no one can tell if a specific person’s record was included in the books or not.

However, this paper identifies a sneaky "backdoor" that even the best security guards might miss.

The Problem: The "Label Leak" (The Secret Menu Analogy)

Imagine you run a secret restaurant. You use a "blurring filter" to make sure no one knows exactly what ingredients are in your soup (this is the DP training). You think you are safe.

But then, you release your Menu.

If your menu yesterday only had "Tomato Soup" and "Onion Soup," but today your menu suddenly adds "Lobster Bisque," a spy doesn't need to see your ingredients to know something happened. They can look at that new menu item and say, "Aha! You must have bought lobster today!"

Even though the recipe was blurred, the mere existence of the label "Lobster" leaked the secret.

In machine learning, this is called a Privacy Side-Channel. In "Continual Learning" (where a model learns new things over time), the model constantly adds new categories (labels). If the model suddenly learns to recognize "Type 2 Diabetes," the attacker knows that the new data being fed into the system contains information about Type 2 Diabetes. The "Menu" (the list of things the model can recognize) has leaked the secret.

The Solution: Two Ways to Fix the Menu

The researchers propose two ways to stop this leak:

1. The "Noisy Menu" (The Blurred Label Method)

Instead of releasing a perfect, crystal-clear menu, you release a blurry menu.

How it works: Before you tell the world what you can cook, you use a mathematical process to decide which labels to show. If a new ingredient (like Lobster) only appears once, you might decide to hide it or call it "Special Seafood" to keep the secret safe.
The Trade-off: It’s like a restaurant that sometimes hides its best dishes to stay secret. You gain privacy, but you might lose some "utility" (the ability to accurately name everything).

2. The "Master Menu" (The Public Label Method)

Instead of changing your menu every time you get new ingredients, you start with a massive, giant menu that contains almost everything in the world.

How it works: Your menu already has "Tomato," "Onion," "Lobster," "Pizza," and "Sushi" on it from day one. When you actually get lobster, you don't add a new item to the menu; you just start using the "Lobster" item that was already there.
The Benefit: Since the menu never changes, a spy looking at the menu sees nothing new. The "side-channel" is closed.

The Results: Does it actually work?

The researchers tested these ideas using advanced AI models (like those used in image recognition). They found that:

They beat the old ways: Their methods were more accurate and more private than previous attempts at "secret learning."
The "Master Menu" is great: Using a large, pre-existing set of labels is a very effective way to keep the model's growth a secret.
Efficiency matters: They showed that you can use "pre-trained" models (AI that already knows a lot about the world) to make this process much faster and more accurate, even while keeping the privacy filter turned on.

Summary in one sentence:

The paper proves that even if you hide the details of your data, simply announcing the names of the new things you've learned can leak secrets—and they provide a mathematical way to "blur" those names or use a "master list" to keep everything hidden.

Technical Summary: Privacy Leakage via Output Label Space and Differentially Private Continual Learning

1. Problem Statement

The paper addresses a critical security flaw in Differentially Private (DP) machine learning systems: privacy side-channels. While standard DP training (e.g., DP-SGD) protects the model weights, the authors identify that the output label space (the set of classes a model is capable of predicting) can act as a side-channel that leaks sensitive information.

This problem is particularly acute in Continual Learning (CL). In CL, models learn from a sequential stream of tasks where new classes appear over time. If a model naively updates its output label space to include new classes found in the sensitive training data, an attacker can observe the change in the label space to infer the presence of specific new classes with 100% accuracy. This effectively invalidates the formal DP guarantees of the model weights.

2. Methodology

The authors propose a formalization of DP for CL and introduce two primary methods to eliminate the label-space side-channel:

Formalization of DP-CL: They define Task-wise Adjacency and Task-wise DP, allowing for a rigorous mathematical framework to reason about privacy across a sequence of tasks, ensuring that the privacy budget does not degrade indefinitely as new tasks are added.
Proposed Solutions for Label Protection:
1. $S_{learned}$ (DP Label Release): Instead of releasing the exact labels from the dataset, the model uses a separate DP mechanism (inspired by the private partition selection problem) to decide which labels to include in the output space. This involves splitting the privacy budget between "learning the labels" and "training the weights."
2. $S_{prior}$ (Data-Independent Prior Labels): This method decouples the label space from the sensitive data by using a large, public, pre-defined set of labels. Any sensitive labels not in this set are remapped to existing public labels or dropped, ensuring the output space does not change based on the private data.
DP-CL Architectures using Pre-trained Models: To maintain high utility, the authors adapt two architectures to the DP-CL setting:
- Cosine Similarity Classifier: Uses a frozen pre-trained feature extractor and accumulates noisy, per-class feature sums (prototypes) to perform classification via cosine similarity.
- PEFT Ensemble: Uses Parameter-Efficient Fine-Tuning (specifically FiLM adapters) to create an ensemble of task-specific models, fine-tuned using DP-SGD.

3. Key Contributions

Identification of a New Side-Channel: Proving that the output label space in classification tasks is a non-DP component that can lead to catastrophic privacy failure.
New DP-CL Framework: Providing a formal mathematical definition for task-wise differential privacy in sequential learning settings.
Novel DP-CL Methods: Introducing the first DP-compatible CL methods that leverage pre-trained models (Cosine Classifier and PEFT Ensemble) while simultaneously protecting the label space.
Comprehensive Benchmarking: Evaluating these methods on complex datasets like Split-CIFAR-100 and Split-ImageNet-R under varying levels of "blurriness" (class overlap between tasks).

4. Results

Superiority over Baselines: The proposed methods consistently outperform existing DP-CL baselines (such as those using episodic memory buffers) without the need for storing non-DP sensitive data.
Trade-off Analysis:
- $S_{learned}$ is highly effective when classes are large, but small classes may be dropped due to the noise required for DP.
- $S_{prior}$ is robust and easy to implement but its utility depends on the quality of the prior label set.
Architecture Performance:
- PEFT Ensemble achieves higher accuracy in most settings, especially when the domain shift from pre-training data is large.
- Cosine Classifier is more robust and efficient under strict privacy constraints ( $\epsilon=1$ ) and when the domain shift is minimal.
Robustness to Blurriness: The models maintain reasonable utility even when classes are "blurry" (distributed across multiple tasks), though utility decreases as the privacy budget becomes tighter.

5. Significance

This work is significant because it closes a theoretical and practical gap in the deployment of private machine learning. By demonstrating that "privacy-preserving" models can still leak data through their metadata (the labels), the authors provide a more holistic and realistic standard for what constitutes a truly private machine learning system. Their integration of pre-trained models with DP-CL offers a viable path toward deploying high-utility, privacy-compliant AI in dynamic, real-world environments where data arrives sequentially.