🤖 AI

SynBench: A Benchmark for Differentially Private Text Generation

This paper introduces SynBench, a unified benchmarking framework that reveals the significant quality degradation and invalidated privacy guarantees of current differentially private text generation methods, particularly when models are pre-trained on data overlapping with the private datasets they aim to generate.

Original authors: Yidan Sun, Viktor Schlegel, Srinivasan Nandakumar, Iqra Zahid, Yuping Wu, Yulong Wu, Hao Li, Jie Zhang, Warren Del-Pinto, Goran Nenadic, Siew Kei Lam, Anil Anthony Bharath

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yidan Sun, Viktor Schlegel, Srinivasan Nandakumar, Iqra Zahid, Yuping Wu, Yulong Wu, Hao Li, Jie Zhang, Warren Del-Pinto, Goran Nenadic, Siew Kei Lam, Anil Anthony Bharath

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a vault full of incredibly sensitive secrets: medical records, legal case files, and private financial transactions. You want to share these secrets with researchers to help them build better tools, but you can't let anyone see the actual names or specific details because that would violate privacy laws.

The solution proposed by many is Synthetic Text Generation. Think of this as a master chef who tastes the secret ingredients from the vault, memorizes the flavor profile, and then cooks up a brand new dish that tastes exactly like the original but uses completely different ingredients. The goal is to get the "taste" (the data's usefulness) without serving the "original ingredients" (the private data).

To make this safe, researchers use a mathematical shield called Differential Privacy (DP). Imagine DP as a strict recipe rule: "You can taste the soup, but you must add a specific amount of 'noise' (like a pinch of salt you can't taste) to every spoonful so that no one can figure out exactly which specific vegetable was in the pot."

Enter "SynBench": The Ultimate Taste Test

The authors of this paper, Yidan Sun and their team, built a new testing ground called SynBench. They realized that while people were claiming their "privacy-safe recipes" were delicious, no one was checking if they were actually safe or if they were just cheating.

Here is what they found, using simple analogies:

1. The "Cheating Chef" Problem (Pre-training Contamination)

The biggest surprise in the paper is that many of these "privacy-safe" chefs are actually cheating.

Imagine a chef who claims to have cooked a new dish using only the secret ingredients from your vault. But, it turns out, this chef had already eaten a very similar dish at a public restaurant before they ever saw your vault. When they cook the "new" dish, they aren't actually learning from your vault; they are just regurgitating what they ate at the public restaurant.

The Paper's Finding: The researchers checked if the AI models had already seen the "private" data during their initial training (pre-training). They found that when the AI had already "eaten" parts of the private data, the synthetic text looked amazing and scored high on quality tests.
The Catch: This high quality wasn't because the AI learned from the private data safely; it was because it was just remembering what it already knew. This is dangerous because the AI might accidentally reveal the private details it memorized earlier, even if the "privacy recipe" (DP) says it shouldn't.

2. The "No Free Lunch" Reality

The paper shows a frustrating trade-off.

If you want high quality: The AI often needs to rely on what it already knows (which might be the private data itself), which breaks the privacy rules.
If you want strict privacy: The AI has to add so much "noise" (the salt) that the resulting dish becomes bland, random, or nonsensical. It fails to capture the complex, specialized language needed for things like medical diagnoses or legal arguments.

The authors tested this on nine different "flavors" of data (healthcare, finance, law) and found that as the data got more complex and specialized, the AI struggled more to keep it both private and useful.

3. The "Spy Test" (Membership Inference Attacks)

To see if the privacy shield was actually working, the researchers played the role of a spy. They didn't just trust the math; they tried to hack the system.

The Test: They asked the spy: "Is this specific sentence from the original secret vault, or is it just a fake copy?"
The Result: In many cases, the spy could guess correctly much more often than the "privacy math" promised they should be able to.
The Analogy: It's like a security guard saying, "I've locked the door, so no one can get in." But when you test the door, you find it's wide open because the guard forgot to lock the back window (the pre-training data). The math said the door was secure, but the reality was a breach.

4. Bigger Isn't Always Better

The team tested AI models of different sizes (small, medium, and huge).

The Finding: Bigger models didn't necessarily make better "privacy-safe" dishes. In fact, bigger models were sometimes worse at privacy because they were better at memorizing the secret ingredients they had already eaten, making it harder to hide the fact that they knew the secrets.

The Bottom Line

The paper concludes that we don't have a working solution yet.

Currently, the field is like a group of people trying to bake a cake without flour, but they keep secretly using flour they found in a neighbor's pantry. When they say, "Look, this cake is safe and delicious!" they are ignoring the fact that the flour came from the wrong place.

The authors argue that we need to stop trusting the "math promises" alone. We need to actually test if the AI is leaking secrets by checking if it had "eaten" the data before. Until we can generate high-quality, specialized text (like medical or legal reports) without accidentally leaking the original secrets, this remains a major unsolved challenge.

Technical Summary: SynBench – A Benchmark for Differentially Private Text Generation

Problem Statement

The proliferation of Large Language Models (LLMs) has created a critical bottleneck: the exhaustion of publicly available, high-quality training data. While vast amounts of sensitive data exist in specialized domains (healthcare, finance, law), privacy and regulatory constraints prevent their use. Synthetic data generation offers a potential solution, but current approaches face three fragmented challenges:

Lack of Standardization: Existing evaluations rely on "pick-and-choose" datasets and metrics, often using simple open-domain data (e.g., product reviews) that fail to capture domain-specific complexities like technical jargon or long-context dependencies.
Pre-training Contamination: Many benchmark datasets used to evaluate differentially private (DP) generators may have been encountered during the pre-training of the underlying LLMs. This leads to an overestimation of generation quality, as models may be leveraging memorized information rather than learning from the private dataset.
Insufficient Privacy Auditing: Current research often relies solely on formal $(\epsilon, \delta)$ guarantees without rigorous empirical verification. Consequently, privacy violations caused by pre-training exposure remain undetected, and the "public pre-training and private generation" paradigm may invalidate claimed privacy bounds in practice.

Methodology: The SynBench Framework

The authors introduce SynBench, a unified benchmark and auditing framework designed to evaluate DP text generation along three axes: Utility, Fidelity, and Privacy Leakage.

1. Dataset Curation

SynBench utilizes nine curated datasets spanning high-stakes domains, moving beyond open-domain benchmarks:

Healthcare: HOC (cancer literature), N2C2'08 (clinical summaries), PSYTAR (social media drug effects), and MIMIC (triage summaries).
Finance: DMSAFN, LC37FND, and ZSTFNS (news and tweets for sentiment).
Legal: ASYLEX (reasoning) and EURLEX (multi-label classification).
These datasets feature long documents, gated access, specialized terminology, and complex label spaces.

2. Evaluation Baselines

The framework benchmarks three distinct DP generation paradigms:

AUG-PE: An API-based method that estimates privatized feature statistics and iteratively queries an LLM to match them.
DP-Gen (DP-transformers): End-to-end neural generation using DP-SGD (gradient clipping and noise injection) to train Transformer models.
DP-Diffusion (DP-Dfs): A diffusion-based generator learning a reverse denoising process with DP constraints.

3. Evaluation Protocol

Utility: Measured by training downstream classification models on synthetic data and evaluating performance on held-out original data.
Fidelity: Assessed via MAUVE (semantic alignment), Named Entity Recognition (NER) distribution divergence, and text length statistics.
Privacy Auditing (MIA): The authors propose a novel Synthetic Text Membership Inference Attack (MIA). Unlike standard MIAs that assume access to model logits, this attack operates in a black-box setting where the adversary only has the synthetic dataset.
- Process: The pipeline identifies outlier records in the private dataset, partitions data into private and auxiliary sets, generates synthetic data, and trains n-gram models to detect membership signals.
- Metric: The Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) is used to quantify leakage.

4. Contamination Analysis

To address pre-training leakage, the authors use the Infini-gram API to measure exact overlap between private training sets and public web-scale corpora (proxied by RedPajama). They correlate this leakage fraction with synthetic data quality and privacy risks.

Key Results

1. Utility and Fidelity Challenges

Domain Dependency: Current DP methods struggle to match real-data baselines, particularly in complex, domain-specific settings. High utility is often dataset-dependent rather than method-dependent.
Privacy-Utility Trade-off: Stricter privacy budgets ( $\epsilon$ ) do not always correlate linearly with decreased performance, but high-quality generation under strong DP constraints remains elusive.
Model Collapse: DP-Diffusion exhibits brittleness under strong privacy constraints (e.g., $\epsilon=0.5$ ), often collapsing into degenerate outputs (empty or random strings).
Fidelity Deterioration: MAUVE scores and distributional metrics show that synthetic data distributions deviate significantly from real data, a gap that widens under DP constraints.

2. The Impact of Pre-training Contamination

Inflated Scores: A strong positive correlation exists between the fraction of leaked samples (overlap with pre-training data) and measured synthetic data quality (both utility and fidelity).
PSYTAR Case Study: On the gated-access PSYTAR dataset, the authors flagged 416/5102 samples as potentially contaminated. Removing these samples monotonically reduced downstream utility, proving that "high scores" were driven by memorized pre-training data rather than genuine learning from the private dataset.

3. Privacy Audit Findings

Bound Violations: Empirical MIA results frequently exceed theoretical $(\epsilon, \delta)$ bounds, particularly for datasets with high pre-training leakage (e.g., PSYTAR, DMSAFN).
Non-Monotonicity: The mapping from privacy budget $\epsilon$ to empirical leakage is non-monotonic. Tighter budgets do not always reduce leakage; in some cases (e.g., PSYTAR), AUC increased as $\epsilon$ decreased from 4 to 0.5.
Pre-training Risk: Models trained on synthetic data generated without DP (or with weak DP) can be more vulnerable to membership inference than models trained directly on private data with DP, suggesting complex interactions between pre-training exposure and the synthetic pipeline.

Significance and Claims

The paper argues that DP synthetic text generation remains an unsolved challenge for high-quality, domain-specific applications. Its primary contributions are:

Unified Framework: It provides the first standardized benchmark for DP text generation that integrates utility, fidelity, and rigorous privacy auditing across diverse, high-stakes domains.
Contamination Awareness: It demonstrates that ignoring pre-training contamination leads to a fundamental overestimation of generation quality and a false sense of security regarding privacy guarantees.
Empirical Validation: It provides quantitative evidence that the "public pre-training and private generation" paradigm can invalidate real-world privacy bounds. The authors assert that utility and fidelity must be interpreted alongside rigorous privacy audits, rather than inferred solely from privacy budgets or intuition.
Call for Auditing: The work motivates the adoption of contamination-aware reporting and standardized, multi-attack model-based audits to ensure the practical validity of DP guarantees in synthetic data generation.

The authors conclude that while DP offers a principled approach to data sharing, current methods cannot yet reliably balance utility, fidelity, and privacy without accounting for the complex interplay between pre-training data and private generation tasks.