🤖 machine learning

TabPATE: Differentially Private Tabular In-Context Learning Without Public Data

The paper introduces TabPATE, a differentially private framework for tabular in-context learning that leverages teacher-student models and synthetic queries generated from feature ranges or privatized marginals to protect against membership inference attacks without requiring public in-distribution data.

Original authors: Dariush Wahdany, Matthew Jagielski, Jesse C. Cresswell, Adam Dziedzic, Franziska Boenisch

Published 2026-07-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dariush Wahdany, Matthew Jagielski, Jesse C. Cresswell, Adam Dziedzic, Franziska Boenisch

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart, pre-trained "oracle" (a Tabular Foundation Model) that can predict things like loan approvals or medical diagnoses just by reading a few examples you give it. This is called In-Context Learning (ICL). You don't need to retrain the oracle; you just whisper a few examples into its ear, and it figures out the pattern.

The problem? Those examples you whisper might contain private secrets (like someone's credit card number or medical history).

The Problem: The "Leaky Whisper"

The paper starts by proving that if you use private data as these examples, the oracle accidentally "leaks" the secrets.

Think of it like this: You ask the oracle, "Is this person's credit score good?"

If the oracle has seen that specific person's data in its "whispered examples" before, it might answer with a slightly different confidence or speed than if it hadn't seen them.
A sneaky attacker (a "membership inference attack") can listen to these tiny differences and guess, "Aha! That specific person's data was in the examples you gave me!"

The authors tested this and found that even a passive listener could guess correctly about 10% of the time (which is way better than random chance), and a more aggressive attacker could guess correctly 75% of the time. This is a privacy disaster.

The Old Solution: The "Public Library" Problem

To fix this, researchers previously used a method called PATE (Private Aggregation of Teacher Ensembles).

How it worked: They split the private data among several "teacher" oracles. These teachers would vote on the answer to a question. To protect privacy, they added a little bit of "static noise" to the vote before announcing the result.
The Catch: To ask the teachers the right questions, you needed a public library of similar, non-private data to generate those questions.
The Reality: In sensitive fields like healthcare or finance, you often don't have a public library of similar data. You can't just ask a public hospital for "fake" patient records that look exactly like your private ones.

The New Solution: TabPATE (The "Imagination Engine")

The authors introduce TabPATE, a new way to protect privacy that doesn't need a public library.

Here is the analogy:
Instead of needing a public library of fake patients to ask questions, TabPATE uses the rules of the game to imagine them.

The Rules are Known: Tabular data (like spreadsheets) has strict rules. We know a "Height" column must be between 0 and 10 feet. We know "Age" is a number. We know "Gender" is a specific category.
Generating Synthetic Queries: TabPATE uses these known rules (the "bounds") to imagine (generate) fake, synthetic questions.
- Option A (The Pure Imagination): It just picks random numbers within the allowed ranges (e.g., "Age: 45, Height: 5.8"). This costs zero privacy money.
- Option B (The Guided Imagination): If the data is tricky (like a rare disease), it spends a tiny bit of privacy budget to learn the general "shape" of the data (e.g., "Most people are between 30 and 50") and then generates questions that fit that shape.
The Voting Process: It asks the "teacher" oracles (who hold the private secrets) about these imagined questions.
The Safe Answer: The teachers vote, and the system adds noise to the vote to ensure no single person's secret can be reverse-engineered.
The Result: The system releases a new list of "Imagined Questions + Safe Answers." This list becomes the new "context" for the oracle.

Why This is a Big Deal

No Public Data Needed: You don't need to find a public dataset that looks like your private one. The system builds its own "practice questions" from scratch using the known rules of the data.
Privacy is Preserved: The authors tested this and found that with TabPATE, the sneaky attacker's success rate drops to near-random guessing (basically 50/50). The private secrets are effectively hidden.
It Still Works Well: Even with all this privacy protection, the oracle can still make accurate predictions. In their tests, TabPATE performed almost as well as the non-private version and better than other privacy methods that didn't use public data.

Summary

TabPATE is a privacy shield for AI that reads private spreadsheets. Instead of needing a public dataset to practice on, it uses the known limits of the data (like "age must be positive") to generate its own practice questions. It then asks a group of private "teachers" to vote on these questions, adds a little noise to hide the individual secrets, and releases a safe, labeled dataset. This allows the AI to learn from private data without ever leaking who that data belongs to, and without needing any outside public data to help.

Technical Summary: TabPATE: Differentially Private Tabular In-Context Learning Without Public Data

1. Problem Statement

Tabular Foundation Models (TFMs) enable accurate predictions via In-Context Learning (ICL), where a small labeled dataset is placed in the model's context to guide inference without parameter updates. While effective in domains like healthcare and finance, this paradigm introduces a critical privacy vulnerability: the records placed in the context may contain sensitive private information.

The authors first demonstrate that this risk is not merely hypothetical. By adapting standard Membership Inference Attacks (MIAs) to the tabular ICL setting, they show that model predictions leak information about whether a specific record was included in the private demonstration context.

Threat Models: The study evaluates four threat models ranging from passive black-box access (label-only) to active grey-box access (with context manipulation/poisoning).
Findings: Even passive grey-box adversaries achieve a 9.9% True Positive Rate (TPR) at a 1% False Positive Rate (FPR). Active attacks, such as those using "Truth Serum" poisoning, can reach a 75.4% TPR, indicating that private context records are significantly exposed through model behavior.

Standard Differential Privacy (DP) training methods like Differentially Private Stochastic Gradient Descent (DPSGD) are unsuitable here because TFMs are used as frozen inference-time models. Existing PATE (Private Aggregation of Teacher Ensembles) approaches, such as PromptPATE, rely on unlabeled public in-distribution data to generate queries for teacher models. In sensitive tabular domains (e.g., finance, health), such public data is often unavailable, creating a gap for practical private deployment.

2. Methodology: TabPATE

The paper introduces TabPATE, a central-DP framework designed for tabular ICL that eliminates the requirement for public in-distribution data. It follows a teacher-student architecture:

A. Teacher Partitioning and Query Generation

Partitioning: The private sensitive dataset $D$ is partitioned into $K$ disjoint subsets. Each subset serves as the demonstration context for a separate "teacher" TFM (the same frozen model prompted with different data).
Synthetic Query Generation (The Core Innovation): Unlike standard PATE which queries teachers on public data, TabPATE generates synthetic tabular queries ( $\tilde{x}$ $\tilde{x}$ ) without public data. This leverages the specific properties of tabular data (bounded, typed, low-dimensional features):
- Data-Independent Prior ( $\alpha=0$ ): Queries are sampled from a fixed prior based on known feature bounds (e.g., uniform or Gaussian distributions over feature ranges). This incurs zero privacy cost.
- DP Marginal Statistics ( $\alpha>0$ ): A portion of the privacy budget ( $\alpha\epsilon$ ) is used to release noisy marginal statistics (means and variances) of the private data. Queries are then sampled from the resulting product distribution. This is beneficial for imbalanced datasets or regression tasks.

B. Private Aggregation

The teachers are queried on the synthetic records. Their predictions are aggregated using the Confident-GNMax algorithm:

If the teachers agree sufficiently on a label, a noisy plurality label is released.
If confidence is low, the mechanism abstains.
The resulting labeled synthetic pairs $(\tilde{x}, \tilde{y})$ form the student context ( $\tilde{D}$ ).

C. Privacy Guarantee

The released student context satisfies $(\epsilon, \delta)$ -Differential Privacy. The privacy budget is split between the optional marginal release and the label aggregation. Since downstream predictions rely only on the released student context, they incur no additional privacy cost.

3. Key Contributions

Empirical Verification of Risk: The authors adapt MIAs to tabular ICL, demonstrating that private context records leak through predictions, with active attacks causing severe leakage.
TabPATE Framework: A central-DP PATE-style defense for tabular ICL that replaces the need for public in-distribution transfer data with synthetic tabular queries derived from feature bounds or DP marginals.
Empirical Validation: Evaluation across multiple tabular benchmarks showing that TabPATE preserves competitive utility while reducing membership inference success to near-random levels.

4. Experimental Results

The authors evaluated TabPATE on five standard classification datasets (e.g., German Credit, Diabetes) and eight regression datasets, comparing it against non-private ICL, DP-SGD, DP-Synthetic data, Query-Time aggregation, and PromptPATE.

Utility: At a privacy budget of $\epsilon=10$ , TabPATE achieved 71.2% balanced accuracy, outperforming other public-data-free baselines (e.g., Query-Time at 61.3%) and approaching PromptPATE (68.6%), which requires public data.
Privacy Protection: TabPATE successfully mitigated membership inference. For the non-private TabPFN, LiRA attacks achieved an AUC of ~0.76. TabPATE reduced this to near-random performance (AUC $\approx$ 0.5) at both $\epsilon=1$ and $\epsilon=10$ .
Regression: The method was extended to regression tasks by aggregating bounded, noisy teacher predictions. While regression proved more challenging than classification, allocating a small budget to marginal query generation ( $\alpha > 0$ ) improved performance over purely data-independent queries.
Baseline Comparison: TabPATE consistently outperformed DP-SGD and DP-Synthetic data baselines, which struggled to maintain utility under privacy constraints.

5. Significance and Claims

The paper claims that TabPATE provides a practical path to private tabular ICL in scenarios where public in-distribution data is unavailable. By exploiting the structured nature of tabular features (bounded ranges and types), the method generates useful queries without compromising privacy or requiring external data.

The authors conclude that TabPATE offers a viable defense mechanism that:

Preserves competitive utility for downstream tasks.
Reduces membership inference risks to near-random guessing.
Enables the deployment of TFMs in sensitive domains (like healthcare and finance) where data sharing for public training sets is restricted.

The work suggests that public-data-free PATE-style knowledge transfer is a feasible route for securing foundation models in structured data environments.