📊 statistics

SynthGuard-ReleaseBench: Locked-Audit Evidence for Synthetic Tabular Data Releases

The paper introduces SynthGuard-ReleaseBench, a reproducible audit framework that establishes pre-locked criteria for evaluating synthetic tabular data releases by comparing real and synthetic workflows to provide finite-sample bounds and distinct evidence for specific use cases, populations, and threat models rather than making blanket claims about privacy or safety.

Original authors: Jeffery Opoku, David

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jeffery Opoku, David

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world of data, organizations often face a dilemma: they need to share information to fuel research and innovation, but they cannot risk exposing the private details of the people behind those numbers. To solve this, statisticians and computer scientists have developed a method called synthetic data. Instead of releasing a spreadsheet containing real people's records, they use complex computer programs to generate a new, fake dataset that looks and behaves statistically like the real one. This fake data contains no actual individuals, yet it preserves the patterns, relationships, and trends found in the original information. For years, the standard way to judge whether this fake data was good enough to release involved checking if it looked realistic or if it could train a machine-learning model to make accurate predictions. However, a new approach argues that looking realistic is not the same thing as being safe to release, and that a single score cannot answer the specific question of whether a dataset is ready for a specific purpose.

A team of researchers has introduced a new framework called SynthGuard-ReleaseBench, which treats the release of synthetic data not as a simple pass-or-fail test, but as a rigorous, locked-down audit. Imagine a scientist who wants to use a new dataset to study a specific health trend in a particular city. Under this new system, the scientist must declare exactly what they intend to do, what level of error they can tolerate, and which specific checks the data must pass before the data is even generated. The researchers then run a series of strict tests comparing how well a model trained on the real data performs versus how well a model trained on the fake data performs on a hidden, protected set of records. If the fake data fails to match the real data within the declared limits, or if the testing process itself was tampered with, the data is blocked from release. This method does not promise that a specific computer program is perfect; instead, it provides a reproducible, evidence-based record that says, "For this specific use, with this specific level of risk, this specific dataset has been proven to work."

The researchers tested this framework across a wide variety of real-world scenarios, including public census data from California and New York, and several different types of records from universities, such as bank marketing lists and hospital patient records. They pitted simple, transparent statistical methods against more complex, modern artificial intelligence models. In many cases, the simpler methods passed the strict audit, while the more complex artificial intelligence models failed when they were given limited computing power or small amounts of data to learn from. For instance, in a test using a dataset of online shoppers, a sophisticated AI model could not produce data that was good enough to match the real-world patterns within the tight limits set by the audit, whereas a simpler statistical approach succeeded. The study found that the success of a synthetic dataset is not a universal property of the tool used to create it; a model that works well for one type of data or one specific task might fail completely for another.

Crucially, the framework includes a safety mechanism that prevents the testing process from being manipulated. The researchers demonstrated that if a developer is allowed to look at the test results and then tweak their model to pass, the test becomes meaningless. To prevent this, the SynthGuard system locks the rules, the data splits, and the specific tests before the data is even generated. If a developer tries to reuse the test data to improve their model after seeing the results, the system flags this as a violation. The study also showed that even if a model passes the utility tests, it can still be blocked from release if the organization cannot prove that the testing was done independently and that the data was not tampered with. In one demonstration, a dataset passed all the technical math checks but was still rejected because the required paperwork proving the integrity of the process was missing.

The researchers also explored how the amount of data available affects the results. They found that artificial intelligence models that failed with small amounts of data could pass the same strict tests once they were given much larger datasets to learn from. This suggests that the failure was not because the models were fundamentally broken, but because they simply did not have enough information to learn the complex patterns. However, the study also revealed that the order in which models perform can change depending on the random starting point of the computer program, meaning that a single test run is not enough to declare one model superior to another. To address this, the framework recommends running multiple tests with different starting points to ensure the results are stable.

In the end, the work does not claim to have found the perfect way to generate fake data. Instead, it offers a new way to decide when a dataset is safe to share. It shifts the focus from asking "Is this model the best?" to "Can we prove this specific dataset works for this specific job?" By locking down the rules and requiring independent verification, the system ensures that the evidence supporting a data release is honest, specific, and resistant to manipulation. The researchers conclude that a benchmark earns trust not by making every candidate pass, but by making it difficult to produce unsupported claims of safety. This approach provides a clear, defensible path for organizations to share data responsibly, ensuring that the benefits of data science can be realized without compromising the privacy of the individuals involved.

Technical Summary: SynthGuard-ReleaseBench

Problem Statement
Current synthetic tabular data evaluation often relies on aggregate scores for realism, privacy, or downstream utility. The paper argues these metrics fail to answer whether a specific data release is supported for a named use, population, and threat model. Existing benchmarks (e.g., SDGym, SDMetrics) and formal privacy proofs (e.g., Differential Privacy) address different questions: the former compares generators under shared metrics, while the latter guarantees mechanism properties without addressing specific analysis errors. There is a lack of a framework that locks the analytical use, candidate panel, tolerances, and audit schedule before evaluation to provide simultaneous, finite-sample evidence for a specific release decision.

Methodology: The Audit Protocol
The authors introduce SynthGuard-ReleaseBench, an audit framework that treats release evidence as a "fail-closed" statistical decision record. The core methodology involves:

Pre-Registration and Locking: Before accessing protected audit data, the protocol locks the declared analytical use, tolerance ( $\tau$ ), candidate generator panel ( $G$ ), analysis panel ( $A$ ), and audit schedule. This prevents post-hoc tuning or "p-hacking."
Simultaneous Finite-Sample Certificates: The primary statistical target is the expected loss gap ( $\Delta_{g,a}$ $Δ_{g, a}$ ) between a workflow trained on real data ( $f_{R,a}$ $f_{R, a}$ ) and one trained on synthetic data ( $f_{S,g,a}$ $f_{S, g, a}$ ). The framework uses Hoeffding's inequality combined with a union bound to provide simultaneous confidence intervals for all candidates and analyses.
- Proposition 1 establishes that with probability $1-\alpha$ , the true gap lies within a half-width $h_{a,l}$ for all declared candidates and analyses.
- Corollary 2 ensures validity even if a candidate is selected based on development data, provided the selection rule is independent of the protected audit.
Controls and Governance:
- Negative Control: An independent-marginal resampling model that deliberately destroys dependencies. It must fail the audit; if it passes, the audit calibration is invalid (e.g., tolerance is too loose).
- Positive Control: A row-resampling model that leaks data, used to verify that attack batteries can detect known leaks.
- Governance Gate: A separate human decision layer that verifies audit independence and lifecycle integrity. A passing technical grid does not automatically authorize release.
Extensions to Dependence:
- Variance-Adaptive Instruments: Empirical Bernstein bounds and anytime-valid confidence sequences (Proposition 8) tighten the uncertainty bounds by factors of 2–10 without sacrificing validity, provided they are pre-registered.
- Temporal/Clustered Data: For time-ordered or clustered data, the framework uses "buffered-block" certificates (Proposition 10), discarding buffer rows to ensure independence between retained blocks, though this requires a declared (and unverifiable) dependence horizon.

Key Contributions

Formal Audit Target: A use-specific equivalence test with simultaneous finite-sample certificates for bounded, paired loss gaps, including rules for sufficient audit size and variance-adaptive tightening.
Lower Bounds on Audit Integrity:
- Proposition 5 & 6: Demonstrates that unrestricted reuse of a protected audit allows an adversary to reconstruct the audit labels (database reconstruction) once the query budget matches the audit size. This necessitates strict query counting and capping.
- Proposition 7: Proves that the multiplicity correction (penalizing for the number of candidates) is necessary; ignoring it causes simultaneous coverage to collapse as the panel grows.
Benchmark Implementation: A reproducible workflow with four completed public studies (American Community Survey), five non-ACS technical records, and a sealed-audit prototype. The implementation includes hash-checked evidence registries, containerization, and continuous integration.
Separation of Concerns: The framework explicitly decouples utility certification, empirical privacy risk, mechanism claims (e.g., $\epsilon$ -DP), and human release authority.

Results
The paper reports outcomes across multiple domains (ACS, UCI datasets) under strict locking conditions:

California (DP Frontier): Transparent baselines (target-conditional marginals) and DP histograms passed the locked utility checks, while the dependence-destroying negative control failed.
UCI Adult (Learned Models): Under a strict, low-budget (5-epoch) CPU constraint, compact learned models (CTGAN, TVAE) failed to meet the tolerance, while the Gaussian copula passed.
Post-Audit Scaling: When the learned models were re-fit with larger data (16,000 rows) and higher compute (300 epochs), they passed the same locked tolerance. This demonstrates the criterion discriminates based on evidence rather than merely rejecting learned models.
Exclusions: A health-table case (Diabetes 130-US Hospitals) was excluded because its negative control passed, indicating the tolerance was not discriminating. A sealed prototype blocked a release despite passing technical checks due to missing governance attestations.
Privacy Diagnostics: In the Florida study, a specific membership inference attack detected a known leak (positive control) but failed to distinguish other candidates. A second "similarity discriminator" attack failed to detect the leak, illustrating that a single attack metric is insufficient; a battery must be validated against known leaks.

Significance and Claims
The paper modestly claims to provide a reproducible workflow for use-specific release evidence, not a universal guarantee that any generator is private or safe.

Not a Generator Contest: The framework does not rank generators generally; it determines if a specific proposed release has evidence for a specific use.
Fail-Closed Design: The system is designed to block releases when evidence is insufficient, controls fail, or governance conditions are unmet.
Honesty about Uncertainty: The contribution includes the explicit reporting of failures, invalidations (e.g., non-record-disjoint experiments), and excluded cases.
Operational Reality: The paper emphasizes that a "passing" utility metric is insufficient for release; the framework requires a "locked" evidence packet that separates technical performance from the authority to release data.

The authors conclude that the benchmark earns trust not by making every candidate pass, but by making unsupported release claims difficult to produce. The work is an open-source, installable package (SynthGuard-ReleaseBench v0.1.14) intended to be adopted by data stewards to formalize their release decisions.

More like this