Privacy Auditing Synthetic Data Release through Local Likelihood Attacks

Technical Summary: Privacy Auditing Synthetic Data Release through Local Likelihood Attacks

Problem Statement
Auditing the privacy leakage of synthetic data remains an unresolved challenge. Existing frameworks for privacy auditing often rely on heuristics (such as distance or density metrics) or unrealistic assumptions regarding model access (e.g., white-box or shadow-box settings). These approaches offer limited ability to detect or describe privacy exposure, particularly when generative models exhibit specific failure modes like overfitting. Furthermore, many existing attacks lack a theoretical foundation explaining what they measure or under what conditions they should succeed, making it difficult to distinguish principled design from benchmark-specific coincidences.

Methodology: Generative Likelihood Ratio Attack (Gen-LRA)
The authors propose Gen-LRA, a novel Membership Inference Attack (MIA) designed for the No-Box threat model. In this setting, the adversary has no access to the generative model's architecture, parameters, or internal mechanisms. The adversary only possesses:

1. The released synthetic dataset ($S$).
2. An independently collected reference dataset ($R$) drawn from the same underlying population distribution ($P$).

Core Mechanism
Gen-LRA frames membership inference as a measurement of the influence a test observation ($x^*$) has on the estimated likelihood of the synthetic dataset. The attack operates on the hypothesis that if a generative model overfits, the inclusion of a training point $x^*$ in the reference set will increase the estimated likelihood of the synthetic data $S$.

The attack proceeds in the following steps:

1. Surrogate Modeling: A surrogate density estimator (specifically a Gaussian Kernel Density Estimator, KDE) is fit on the reference set $R$ to obtain $\hat{p}_R$.
2. Augmentation: A second surrogate is fit on the augmented set $R \cup \{x^*\}$ to obtain $\hat{p}_{R \cup \{x^*\}}$.
3. Local Likelihood Ratio: The attack score is calculated as the sum of log-likelihood ratios over the $k$-nearest synthetic neighbors of $x^*$ (denoted $S_k(x^*)$):
   $$\hat{I}(x^*; R, S) = \sum_{s \in S_k(x^*)} \left[ \log \hat{p}_{R \cup \{x^*\}}(s) - \log \hat{p}_R(s) \right]$$
4. Decision: A high score indicates that adding $x^*$ to the reference set significantly increased the likelihood of the nearby synthetic points, suggesting $x^*$ was part of the training set.

Theoretical Framework
The paper establishes three key theoretical results supporting Gen-LRA:

1. Invariance: The log-likelihood-ratio score is invariant to invertible reparametrizations of the data, ensuring robustness against preprocessing choices like scaling or encoding.
2. Closed-Form Characterization: The authors prove that the Gen-LRA score admits a closed-form characterization as a localized density-ratio statistic. Unlike single-point density ratio attacks (e.g., DOMIAS), Gen-LRA aggregates evidence over a local region, capturing diffuse memorization signals.
3. Mean-Score Gap: Under a general model of local overfitting, the authors prove a provable gap between the expected scores of members and non-members. The magnitude of this gap depends linearly on the overfitting strength ($\epsilon$) and is governed by the overlap between the surrogate's bandwidth and the generator's memorization scale ($\rho$).

Key Contributions

1. Novel Attack: Gen-LRA is the first MIA for tabular synthetic data to explicitly adapt an influence-function framework to the No-Box setting, formulating membership inference as the influence of a test point on a surrogate model's likelihood ratio.
2. Theoretical Justification: The work provides a rigorous theoretical account of what the attack measures (localized density ratios) and predicts when it succeeds (under local overfitting).
3. Simulation Validation: Controlled simulations instantiate the overfitting model, validating that Gen-LRA is uniquely tunable across overfitting regimes and outperforms existing methods, particularly when a generator approximately memorizes training data.
4. Empirical Benchmark: The authors evaluate Gen-LRA against a comprehensive benchmark comprising 35 datasets, 9 generative architectures (including GANs, VAEs, Diffusion models, and Transformers), and 9 competing attacks. This results in 1,525 unique synthetic dataset configurations and over 10,000 attack runs.

Results

• Aggregate Performance: Gen-LRA consistently dominates competing MIAs. It achieves the highest Top-1 ranking rate on every metric (AUC and True Positive Rate at fixed False Positive Rates) across all 1,525 runs. Specifically, it ranks first in over 38% of runs (AUC) and 42% of runs (TPR@FPR=0), which is more than double the rate of the next best baseline.
• Low False Positive Rates: Gen-LRA shows particularly strong gains at low false positive rates, a critical metric for practical privacy auditing.
• Vulnerability Regimes: The attack is most effective in the "approximate memorization" regime, where synthetic points retain a relationship to training data but are not exact copies. In this regime, single-point attacks (like DOMIAS) and distance-based attacks (like DCR) collapse, while Gen-LRA aggregates evidence across multiple neighbors to succeed.
• Surrogate Choice: Contrary to trends in high-dimensional density estimation where deep learning models often outperform KDE, the authors find that Gaussian KDE outperforms Block Neural Autoregressive Flows (BNAF) for Gen-LRA. This is attributed to KDE's explicit control over local smoothing (bandwidth), which matches the memorization scale, whereas BNAF optimizes for global accuracy and introduces stochastic noise that obscures the single-point influence signal.

Significance and Claims
The paper claims that current No-Box privacy auditing tools significantly underestimate privacy risks because they test incorrect hypotheses or rely on heuristics that fail in the approximate-memorization regime common in modern tabular generators.

Gen-LRA provides a principled, computationally efficient tool for third parties to audit synthetic data releases without requiring model access. The authors argue that the "No-Box" setting is the most realistic for privacy auditing, as defenders following best practices will withhold model details, rendering shadow-box attacks ineffective. By demonstrating that generative model overfitting poses significant privacy risks even in the absence of model access, the paper highlights the need for stronger auditing tools to ensure the responsible release of synthetic data in sensitive domains like healthcare and finance. The authors conclude that while no single attack is uniformly optimal across all conditions, Gen-LRA offers the most robust and consistent performance for detecting privacy leakage in real-world synthetic data releases.