📊 statistics

Phantoms and Disclosures: a Causal Framework for Auditing Synthetic Data

This paper introduces a model-agnostic, resource-efficient empirical auditing framework that distinguishes between true and phantom data disclosures in synthetic datasets using statistical hypothesis testing on holdout controls, thereby providing tighter privacy leakage bounds without requiring model access or reference training.

Original authors: Kareem Amin, Rudrajit Das, Alessandro Epasto, Adel Javanmard, Dennis Kraft, Mónica Ribero, Sergei Vassilvitskii

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kareem Amin, Rudrajit Das, Alessandro Epasto, Adel Javanmard, Dennis Kraft, Mónica Ribero, Sergei Vassilvitskii

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you run a hospital and want to share patient records with researchers to study diseases. But you can't share the real records because they contain private secrets like names and phone numbers. So, you use a super-smart AI to write fake patient stories that look and act just like the real ones, but (hopefully) don't contain any actual secrets.

The big worry is: Did the AI accidentally copy a real patient's secret into the fake story?

This paper introduces a new "Privacy Detective" tool to answer that question. Here is how it works, explained simply:

1. The Problem: Real Leaks vs. "Phantom" Leaks

When you check the fake data, you might find a phone number that matches a real patient.

True Disclosure: The AI looked at Patient A's file, memorized their phone number, and wrote it down in the fake story. This is a privacy breach.
Phantom Disclosure: The AI just happened to write a phone number that looks like Patient A's, but it didn't learn it from them. Maybe the AI knows that phone numbers usually start with "212," so it guessed one that just happened to match.

The Analogy: Imagine a magician (the AI) trying to guess a card you picked from a deck.

If he guesses your card because he peeked at it earlier, that's a True Disclosure.
If he guesses your card because he knows you usually pick the Ace of Spades, and he just got lucky, that's a Phantom Disclosure.

Previous tools counted every match as a leak, which made the AI look much worse than it actually was. This paper says: "We need to tell the difference between the magician peeking and the magician just getting lucky."

2. The Solution: The "Control Group" Experiment

To solve this, the authors created a simple experiment that doesn't need to see the AI's brain (the code). They only need the fake data and a secret list of real data that was never shown to the AI.

Here is the setup:

Split the Real Data: Take all the real patient records and split them into two piles: Pile A (Training) and Pile B (Holdout).
Train the AI: Show the AI only Pile A. Let it write its fake stories.
The Test: Now, look at the fake stories and compare them to both Pile A and Pile B.

The Logic:

If the AI is leaking secrets, it should match Pile A (the data it saw) much more often than Pile B (the data it never saw).
If the AI is just "getting lucky" (Phantom Disclosures), it will match Pile A and Pile B at roughly the same rate.

By comparing the two piles, the tool can mathematically prove if the AI actually memorized something or if it was just a coincidence.

3. What Makes This Tool Special?

The paper highlights three main superpowers of this new detective:

No "Canary" Traps Needed: Old methods required the hospital to plant fake "trap" records (like a fake patient named "John Doe" with a fake phone number) into the training data to see if the AI spat them out later. This new tool doesn't need traps. It just looks at the natural data. It's like checking if a thief stole a specific watch by seeing if the watch is in the thief's pocket, rather than setting a fake watch trap.
No "Shadow" Models: Old methods required building a whole second AI to guess what the first AI was thinking. This new tool is much lighter and faster. It's like solving a crime by looking at the evidence at the scene, rather than hiring a whole new police force to simulate the crime.
It Finds "Ghost" Leaks: The authors discovered that a huge chunk of what people thought were privacy leaks were actually just "Phantoms" (coincidences). In their tests, over 35% of the "leaks" were just the AI getting lucky. Their tool filters these out so you don't panic over false alarms.

4. The Results

The team tested this on real-world data (like emails, tweets, and financial records).

For "Naive" AI: When they used an AI that wasn't protected, the tool found many True Disclosures. The AI had indeed memorized real secrets.
For "Protected" AI: When they used an AI trained with special privacy rules (Differential Privacy), the tool found that the matches were mostly Phantoms. The AI wasn't leaking secrets; it was just guessing.

Summary

This paper gives us a way to audit AI-generated data without needing to see how the AI was built. It separates real privacy breaches (where the AI stole a secret) from coincidences (where the AI just guessed right). It's a faster, cheaper, and more accurate way to ensure that when we share "fake" data, we aren't accidentally sharing real secrets.

Technical Summary: Phantoms and Disclosures: a Causal Framework for Auditing Synthetic Data

Problem Statement

The rapid adoption of Generative AI and Large Language Models (LLMs) has positioned synthetic data as a privacy-preserving alternative to sensitive real-world datasets. However, generating high-utility synthetic data carries the inherent risk of memorizing and regurgitating private information from the training corpus. A critical challenge in auditing these systems is distinguishing between true disclosures (where the system directly reproduces a user's information due to memorization) and phantom disclosures (where the system incidentally generates a user's data due to generalization, inductive bias, or model contamination, regardless of whether that user's data was in the training set).

Prior auditing methods often rely on model access, the insertion of artificial "canaries" into the training set, or the training of shadow models. These approaches are computationally expensive, require domain expertise to design effective canaries, and often fail to account for the baseline rate of accidental data generation (phantoms), leading to an overestimation of privacy risks.

Methodology

The authors propose a customizable, empirical, black-box auditing framework that requires only the synthetic output, the original private data, and a held-out control set. The framework operates without access to the model internals, without inserting canaries, and without training reference models.

Core Components

Data Partitioning: The private dataset $D$ is partitioned into a training set ( $D_{train}$ ) and a holdout set ( $D_{hold}$ ). A synthetic dataset $Y$ is generated from $D_{train}$ using a mechanism $M$ .
Disclosure Definition: A disclosure is defined as information that is rare in the source corpus but appears in the synthetic output. The framework supports three specific classes:
- PII Leakage: Strings identifying a user (e.g., phone numbers) that are rare in the source.
- Regurgitation: Rare token sequences reproduced verbatim.
- Semantic Reconstruction: Records where the meaning is preserved in the synthetic data despite being semantically distinct in the source.
Phantom vs. True Disclosures:
- True Disclosures: Rare features found in $Y$ that originate from $D_{train}$ .
- Phantom Disclosures: Rare features found in $Y$ that originate from $D_{hold}$ . These serve as a baseline for accidental generation.
Statistical Hypothesis Testing: The framework treats the audit as a controlled experiment to determine if observed disclosures in $D_{train}$ $D_{t r ain}$ exceed the baseline established by $D_{hold}$ $D_{h o l d}$ .
- Zero-Learning Test: Tests the null hypothesis that the mechanism $M$ does not learn from the input at all (consistent with 0-Differential Privacy). It checks if the distribution of disclosures in the training set is statistically indistinguishable from the holdout set.
- DP-Bounded Learning Test: Tests the null hypothesis that the mechanism satisfies $\epsilon$ -Differential Privacy. It assesses whether the gap between training and holdout disclosures is consistent with a specific privacy budget $\epsilon$ .
Membership Inference Attacks (MIA): The framework constructs a membership inference attack using the disclosure scores. It predicts whether a record belongs to the training set based on its similarity to the synthetic data, measuring success via Area Under the Curve (AUC).

Key Contributions

Phantom Disclosure Formalization: The paper identifies and quantifies "phantom disclosures," demonstrating that a significant fraction (e.g., >35% in some experiments) of apparent privacy violations are incidental. Ignoring these leads to naive overestimation of risk.
Model-Agnostic, Black-Box Auditing: The framework functions without model access, canary insertion, or shadow-model training, requiring only the synthetic data and a held-out sample.
Causal Framework: By partitioning data into treatment (training) and control (holdout) groups, the method establishes a causal link between training-set membership and excess disclosures, controlling for Type I error.
Formal Privacy Bounds: The approach provides empirical lower bounds on the privacy parameter $\epsilon$ and offers rigorous statistical tests (with controlled Type I error) to reject zero-learning or specific DP guarantees.
Interpretability: Unlike black-box MIA scores, this framework identifies specific features (e.g., specific phone numbers or semantic embeddings) that constitute the leakage, allowing auditors to understand the nature of the risk.

Experimental Results

The authors evaluated the framework on six public datasets (Finance, NYT Comments, Panorama, Panorama+, Postings, Tweets) using three generation techniques:

Rewrites: Prompting an LLM to rewrite records.
Fine-tuning (SFT): Standard fine-tuning on private data.
DP-SGD: Fine-tuning with Differential Privacy constraints.

Key Findings:

Non-Private Models: For Rewrites and SFT, the framework successfully rejected the zero-learning hypothesis, detecting significant true disclosures. The estimated $\epsilon$ values were high (e.g., up to 5.3 for NYT rewrites), indicating substantial privacy leakage.
DP Models: For models trained with DP-SGD, the framework failed to reject the zero-learning hypothesis. Observed disclosures were statistically indistinguishable from phantom disclosures, validating the efficacy of the formal privacy protections.
Comparison to Baselines: Compared to the state-of-the-art canary-based audit by Meeus et al. [32], the proposed method:
- Detected tighter disclosure measures for non-private data.
- Achieved comparable or better results for DP data.
- Required orders of magnitude fewer computational resources (no shadow model training).
- Detected semantic leakage (via embeddings) that verbatim n-gram baselines missed (e.g., AUC of 0.93 for NYT rewrites vs. 0.51 for the baseline).

Significance and Claims

The paper claims that rigorous auditing of synthetic data must decouple from specific generation algorithms and account for the baseline rate of accidental data generation. By formalizing the distinction between true and phantom disclosures, the framework provides a more accurate, interpretable, and efficient method for quantifying privacy leakage.

The authors emphasize that their approach offers a "turn-key" deployment for internal privacy auditors, enabling them to quantify leakage risks prior to data release without needing to design complex canaries or access proprietary model weights. The framework provides formal statistical guarantees (Type I error control) and empirical lower bounds on privacy loss, offering a robust alternative to prior model-based or canary-dependent auditing techniques. The paper concludes that while non-private synthetic data often exhibits significant leakage, formal privacy protections (like DP-SGD) effectively mitigate true disclosures, rendering them statistically indistinguishable from random coincidences.