💻 computer science

CoLA: A Choice Leakage Attack Framework to Expose Privacy Risks in Subset Training

This paper challenges the assumption that subset training reduces privacy risks by introducing CoLA, a framework demonstrating that the data selection process itself creates new attack surfaces (Training-membership and Selection-participation MIAs) capable of leaking sensitive information about both training data and the broader data-model supply chain.

Original authors: Qi Li, Cheng-Long Wang, Yinzhi Cao, Di Wang

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qi Li, Cheng-Long Wang, Yinzhi Cao, Di Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef preparing a massive banquet for a city. You have a warehouse full of ingredients (the full dataset), but cooking with everything would take forever and might ruin the dish with too much noise. So, you decide to be smart: you pick only the best, most representative ingredients to make your soup. This is called Subset Training.

For years, everyone thought, "Great! If we use fewer ingredients, we're safer. We're hiding the fact that we didn't use the rest of the warehouse."

This paper says: "Wrong. You're actually leaving a bigger trail of breadcrumbs."

Here is the story of CoLA (Choice Leakage Attack), the new framework that exposes this hidden danger.

The Big Misconception: "Less Data = Less Risk"

The common belief is that if you throw away 80% of your data and only train your AI on the remaining 20%, you are protecting the privacy of the 80% you threw away. You think, "The AI never saw those other ingredients, so it can't tell anyone about them."

The Reality: The act of choosing which ingredients to keep and which to toss is itself a secret message. The way you select your data leaks information about everything in the warehouse, not just the stuff you kept.

The Two Types of "Leaks"

The authors define two ways an attacker can spy on your process:

TM-MIA (The "Who Ate the Soup?" Leak):
- The Question: "Did this specific ingredient end up in the final soup?"
- The Old Way: Attackers used to look at the soup's taste (the model's output) to guess if an ingredient was in it.
- The Problem: When you use a smart selection method, the soup tastes so similar whether an ingredient was in it or not that this method fails.
SP-MIA (The "Who Was in the Kitchen?" Leak):
- The Question: "Did this ingredient even enter the kitchen to be considered, even if it was later thrown in the trash?"
- The New Danger: This is the paper's big discovery. The selection process itself (the "kitchen") leaves a fingerprint. If an ingredient was almost picked but rejected, or if it was picked every single time you tried to make the soup, that pattern reveals secrets.
- The Metaphor: Imagine a bouncer at a club. Even if he turns you away, the fact that you tried to get in, and the specific way you were turned away, tells the bouncer (or a spy watching) that you exist and what kind of person you are.

How CoLA Works: The "Sticky Fly" Analogy

The authors built a tool called CoLA to catch these leaks. How does it work?

Imagine you have a sticky fly trap (the Selector) and a bag of 1,000 flies (the Data).

The Old View: You think the trap only catches the flies that stick to it.
CoLA's View: CoLA realizes that the trap has a "preference." It keeps catching the same 200 flies over and over again, no matter how you shuffle the bag.

The Attack Strategy:

The Side-Channel Attack (The Spy with a Blueprint):
The attacker knows the recipe (the selection algorithm) and the ratio (e.g., "keep 20%"). They run the selection process themselves many times with slightly different batches of data.
- Result: If a specific data point keeps getting picked every single time, CoLA knows, "Aha! This data point is special. It was definitely in the original pool." It's like noticing that a specific fly always lands on the trap, no matter how you move it.
The Black-Box Attack (The Spy with Only the Soup):
The attacker doesn't know the recipe. They only see the final AI model.
- Result: CoLA looks at the "shape" of the data. It groups similar data points together (like clustering flies by color). It notices that certain data points are always the "center" of their group, meaning the selection process kept gravitating toward them. Even without seeing the selection process, CoLA can guess, "This data point was likely in the mix because it's a 'hub' that the selector loves."

Why This Matters in the Real World

This isn't just about soup or flies. It's about real privacy:

Medical Records: Imagine a hospital selects patients for a study on a rare disease. Even if a patient is rejected from the study, the fact that they were considered (and perhaps rejected for having a specific symptom) leaks that they might have that symptom.
Credit Scores: A bank filters loan applicants. If the filtering process consistently rejects people with a certain financial pattern, an attacker can infer who applied, even if they were denied.
The Whole Supply Chain: The risk isn't just the final AI model; it's the entire pipeline from raw data collection to the final model. The "choice" of data is now a vulnerability.

The Takeaway

Subset training does not make you safer; it makes you more exposed.

By trying to be efficient and pick only the "best" data, you inadvertently create a pattern. That pattern tells a clever attacker exactly which data was in the mix and which was left out.

CoLA is the flashlight that shows us this dark corner. It proves that to truly protect privacy, we can't just protect the final model; we have to protect the decision-making process of how we choose our data in the first place.

1. Problem Statement

Modern machine learning pipelines increasingly rely on subset training—selecting a carefully curated portion of data rather than using the full dataset—to improve efficiency, reduce noise, and enhance model quality. The prevailing intuition suggests that training on fewer samples inherently reduces privacy risks.

The Paper's Challenge:
The authors challenge this assumption, arguing that subset training introduces new privacy surfaces. The decision-making process of which data to include or exclude encodes sensitive signals. These "choice signals" can be exploited by adversaries to infer membership, not just for the final training set, but for the entire dataset pool involved in the selection process.

Key Definitions:
The paper defines two distinct membership inference attack (MIA) surfaces unique to subset training:

Training-membership MIA (TM-MIA): The traditional attack where the adversary tries to determine if a sample was part of the final training set ( $I$ ).
Selection-participation MIA (SP-MIA): A novel, broader attack surface where the adversary tries to determine if a sample participated in the selection process (i.e., was in the candidate pool $I \cup E$ , regardless of whether it was ultimately selected for training). This expands the privacy risk from the model output to the entire data-model supply chain.

2. Methodology: CoLA (Choice Leakage Attack)

The authors propose CoLA, a unified framework that leverages "choice signals" rather than relying solely on model overfitting (which is often insufficient in subset training due to distribution shifts).

Core Insight:
Subset selectors (algorithms that choose data) exhibit inclusion stability. They tend to consistently select specific samples that match their implicit criteria (e.g., high informativeness, low noise) across different candidate sets. CoLA exploits this stability as a membership signal.

Attack Scenarios:
CoLA operates under two threat models:

A. Subset-aware Side-channel Attack

Adversary Knowledge: The adversary knows the selection algorithm ($Sel$), the selection ratio ( $r$ ), and has access to the target model's outputs.
Mechanism:
1. The adversary constructs multiple overlapping "windows" (candidate subsets) from the original dataset.
2. They simulate the selection process on each window to see if a specific sample $x$ is chosen.
3. They calculate an inclusion count ( $t(x)$ ), representing how many times $x$ was selected across the windows.
4. A scoring function $s(x)$ is derived based on the deviation of $t(x)$ from the expected random selection count ( $r \cdot n$ ). Samples with high inclusion stability receive higher scores.

B. Black-box Attack

Adversary Knowledge: The adversary only has access to the trained model's outputs. The selection algorithm and metadata are hidden.
Mechanism:
1. The adversary creates overlapping windows of data.
2. For each window, they use an embedding model to cluster the data (unsupervised).
3. They identify "representative" samples within each cluster (those closest to the centroid).
4. The logic assumes that if a sample is consistently a geometric representative (stable inclusion) across different windows, it is likely part of the original selection pool.
5. The score combines the inclusion count with the average distance to the cluster centroid.

3. Key Contributions

Systematic Definition of Privacy Risks: The paper is the first to formally define and explore the SP-MIA threat model, demonstrating that data excluded from training is still vulnerable to privacy leakage due to the selection mechanism.
The CoLA Framework: A novel, data-oriented attack framework that unifies different attack settings (side-channel and black-box) by leveraging "choice leakage" (inclusion stability) rather than model memorization.
Empirical Validation: Extensive experiments on both Vision models (ResNet-18, VGG19 on CIFAR-10/100) and Language models (Pythia, GPT-Neo) demonstrating that CoLA significantly outperforms existing baselines.

4. Experimental Results

The authors evaluated CoLA against standard baselines (NN, LiRA, Min-K%, etc.) across various selection methods (coreset selection, pruning, deduplication).

Performance in Side-channel Setting:
- SP-MIA: Baseline methods failed significantly (AUC $\approx$ 50%, equivalent to random guessing) because they rely on model outputs which do not distinguish well between included and excluded data in subset training.
- CoLA: Achieved high AUC scores (e.g., 96.86% for SP-MIA on ResNet-18 with heavy selection intensity).
- TM-MIA: CoLA also outperformed baselines (e.g., 83.77% vs 69.86% for LiRA).
- Observation: Privacy risk increases as the selection ratio (intensity) increases.
Performance in Black-box Setting:
- Vision Models: CoLA consistently outperformed baselines by ~5% in AUC across different selection methods (Cal, Craig, Uncertainty).
- Language Models: The gap was even more pronounced. All baselines except CoLA performed worse than the "Bag of Words" golden baseline (AUC < 50-60%), while CoLA achieved AUC > 80% (e.g., 80%+ on Pythia-160M under SP-MIA).
- Ablation Studies: The framework proved robust to variations in window size, embedding model choice, and clustering numbers.

5. Significance and Implications

Paradigm Shift in Privacy: The paper overturns the belief that "less data = less risk." It proves that the selection process itself is a privacy leak.
Expanded Attack Surface: Privacy risks are no longer confined to the trained model but extend to the entire data-model supply chain. Even data that is "discarded" can reveal sensitive information about the dataset's composition or the selection criteria.
Real-world Impact:
- Stigma and Discrimination: Inferring that a record was in a candidate pool (even if rejected) can reveal sensitive participation in medical trials, HIV screening, or credit applications.
- Supply Chain Vulnerability: Adversaries can learn the "preferences" of a data pipeline, enabling targeted poisoning attacks or data manipulation.
Call to Action: The authors argue that privacy protections (like Differential Privacy) must be applied not just to model training, but to the data curation and selection stages of the ML pipeline.

In conclusion, CoLA demonstrates that subset training is not privacy-free. The "choices" made during data curation create a stable, exploitable signal that allows adversaries to reconstruct membership information with high accuracy, necessitating a re-evaluation of privacy standards in modern ML workflows.