Towards Useful and Private Synthetic Omics: Community Benchmarking of Generative Models for Transcriptomics Data

This paper presents a community benchmark of 11 generative models for synthetic bulk RNA-seq data, revealing that while deep learning models offer high utility, they often face greater privacy risks compared to differentially private or simpler statistical approaches, highlighting the need to balance utility, biological fidelity, and privacy based on specific use cases.

The Gist

Imagine you are a doctor with a massive library of patient medical records. These records are incredibly valuable for training AI to predict diseases and save lives. However, there's a huge problem: you can't just hand these records to anyone. They contain private information about real people, and sharing them could violate their privacy.

The Solution: "Fake" but Realistic Data
To solve this, scientists decided to create "synthetic" data. Think of this like a master chef creating a perfect fake meal. The fake meal looks, smells, and tastes exactly like the real one, but it's made from scratch using a recipe. If someone eats the fake meal, they get the same nutritional experience, but no one gets sick because there are no actual ingredients from a specific person's kitchen.

In the world of genetics, this means creating fake gene-expression data (like a recipe for how genes turn on and off) that mimics real patients so well that AI can learn from it, without ever seeing a real patient's private data.

The Big Challenge: The "Goldilocks" Problem
The paper describes a massive competition (called the CAMDA 2025 Health Privacy Challenge) where different teams tried to build the best "fake meal" generators. But they faced a tricky balancing act, like trying to find a chair that is just right:

1. Too Fake: If the fake data is too simple, the AI learns nothing useful. It's like giving a student a blank notebook; they can't learn math from it.
2. Too Real: If the fake data is too perfect, it might accidentally include a real person's specific details. It's like the chef accidentally leaving a real customer's name tag on the fake burger. A clever hacker could look at the fake data and say, "Aha! This fake burger looks exactly like the one John ate last Tuesday, so John must have been in the kitchen!"
3. Just Right: The goal is to make data that is useful for science but safe for privacy.

The Contest: Blue Team vs. Red Team
The researchers set up a game to test these generators:

• The Blue Team (The Creators): They built 11 different types of AI models to generate the fake gene data. Some used simple math, while others used complex, deep-learning "neural networks."
• The Red Team (The Attackers): Their job was to try to break the Blue Team's work. They used "Membership Inference Attacks"—basically, they tried to guess if a specific person's real data was used to train the fake generator. If they could guess correctly, the privacy protection failed.

What They Discovered
After testing everything on thousands of patient records, here are the main takeaways, explained simply:

• The "Deep Learning" Trap: The most complex, powerful AI models (like the "Deep Generative Models") were the best at making data that looked real and helped AI learn new things. However, they were also the most dangerous. Because they memorized so much detail, hackers could easily tell if a specific person was in the training data. They were like a photocopier that made such a perfect copy it accidentally included the fingerprint of the person who held the original paper.
• The "Simple Math" Surprise: Simpler statistical models (like the Multivariate Normal distribution) were surprisingly good. They weren't as fancy as the deep learning models, but they produced data that was still very useful for science and much harder for hackers to attack. They were like a sketch artist: the drawing wasn't a photograph, but it captured the essence perfectly without revealing the subject's secrets.
• The Privacy Tax: The team tried using "Differential Privacy," which is like adding a little bit of "static noise" to the data to blur the edges. This made the data much safer from hackers, but it also made the "fake meal" taste a bit bland. The AI had a harder time learning from it. It's a trade-off: more safety means less flavor (utility).
• One Size Does Not Fit All: There was no single "winner."
  • If you need to study complex gene networks, you might need the powerful (but risky) deep learning models.
  • If you just need to predict cancer types and want to be safe, the simpler models might be better.
  • If you are in a high-risk situation, you might need to accept that your data will be a bit "blurry" (less useful) to ensure total privacy.

The Bottom Line
This paper teaches us that creating safe, fake medical data isn't about finding one perfect tool. It's about making a choice. You have to decide: How much privacy do I need? How much detail do I need for my research?

Just like you wouldn't use a sledgehammer to crack a nut, you shouldn't use the most complex AI model if a simpler one will do the job safely. The key is to test your data from every angle—checking if it's useful, if it looks real, and if it's truly safe from hackers—before you share it with the world.

Technical Summary

1. Problem Statement

The sharing of real-world genomic data (specifically bulk RNA-seq) is critical for advancing machine learning in healthcare but poses significant patient privacy risks. While synthetic data generation offers a promising solution to create anonymized datasets that approximate real distributions, current methods face a complex, multi-dimensional trade-off:

• Utility: Can the synthetic data train models that perform well on downstream tasks (e.g., disease classification)?
• Biological Plausibility: Does the synthetic data preserve complex biological signals, such as differential expression (DE) and gene co-expression networks?
• Privacy: Is the synthetic data resilient to Membership Inference Attacks (MIA), where an adversary attempts to determine if a specific individual's data was used in the training set?

Existing literature often evaluates these dimensions in isolation or focuses primarily on differentially private (DP) models. There is a lack of systematic, community-driven benchmarking that quantifies the trade-offs between these axes across diverse generative model families (statistical, deep learning, adversarial) in high-dimensional transcriptomic contexts.

2. Methodology

The study builds upon the CAMDA 2025 Health Privacy Challenge, conducting a systematic benchmark of 11 generative methods across two large-scale cancer cohorts:

• Datasets:
  • TCGA-BRCA: ~1,000 breast cancer samples (5 molecular subtypes).
  • TCGA-COMBINED: ~5,000 samples across 12 cancer types.
  • Features: Restricted to 978 LINCS L1000 landmark genes (VST-normalized).
• Generative Models Evaluated:
  • Statistical Baselines: Multivariate Normal (MVN), Non-negative Matrix Factorization (NMF) and its DP variant (DP-NMF).
  • Probabilistic Graphical Models: Private-PGM (P-PGM).
  • Latent-Variable Autoencoders: Conditional VAE (CVAE), CVAE with Gaussian Mixture Model (CVAE-GMM), and DP-CVAE.
  • Adversarial Models: CTGAN, DP-CTGAN, and WGAN-GP.
  • Diffusion Models: Embedded Diffusion (with noise injection).
• Evaluation Framework (Four Dimensions):
  1. Distributional Fidelity: Measured via Maximum Mean Discrepancy (MMD), Kullback-Leibler (KL) divergence, distance-to-closest record, and discriminator-based scores.
  2. Downstream Utility: Assessed via the Train-on-Synthetic, Test-on-Real (TSTR) paradigm. Metrics included relative AUROC, F1, and feature overlap (preservation of top predictive genes).
  3. Biological Plausibility:
     • Differential Expression (DE): Recovery of true positive DE genes at controlled False Positive Rates (FPR).
     • Co-expression: Recovery of gene-gene correlation networks (True Positive Rate vs. False Edge Rate).
  4. Privacy Risk: Evaluated using Black-Box Membership Inference Attacks (MIA). Methods included distance-based (GAN-leaks, Monte Carlo), density-based (LOGAN-D1, DOMIAS-KDE), and confidence-based (Random Forest) attacks. Performance was measured by AUC-ROC and True Positive Rate at fixed False Positive Rate (TPR@FPR=0.1).

3. Key Contributions

• Comprehensive Multi-Dimensional Benchmark: The first large-scale comparison of 11 distinct generative architectures specifically for bulk RNA-seq, evaluating the interplay between fidelity, utility, biology, and privacy.
• Quantification of Trade-offs: The study explicitly maps how model architecture dictates the trade-offs between privacy and utility. It demonstrates that high utility often correlates with higher privacy risk, but this is not a universal rule.
• Insight into Biological Signal Preservation: It reveals that preserving global distributional fidelity (MMD) does not guarantee the recovery of specific biological signals (DE or co-expression), and that different model classes excel at different biological tasks.
• Privacy Proxy Validation: The authors investigate whether "distance-to-closest record" is a valid proxy for privacy risk, finding it correlates strongly with distance-based attacks but fails to predict confidence-based attacks.

4. Key Results

A. Distributional Fidelity

• Statistical models (MVN) achieved high fidelity in global distribution (high inverted MMD/KL) and were difficult for discriminators to separate from real data.
• Deep generative models (e.g., Embedded Diffusion, CVAE) generally showed strong global alignment but sometimes struggled with gene-wise marginal distributions (low KL inversion).
• DP Models: P-PGM showed high MMD (good multivariate structure) but low KL inversion, indicating it matched overall structure but failed to reproduce individual gene distributions accurately.

B. Downstream Utility

• Top Performers: CVAE-GMM, Embedded Diffusion, and MVN achieved the highest relative utility (TSTR/TRTR ratios close to 1) and best feature overlap.
• Poor Performers: CTGAN and DP-CTGAN showed significant utility loss, likely due to architectural mismatches with high-dimensional continuous RNA-seq data and the cost of DP noise.
• DP Impact: DP methods generally suffered utility loss, though P-PGM performed competitively among DP models.

C. Biological Plausibility

• Differential Expression (DE): Deep generative models (CVAE-GMM, Embedded Diffusion) and MVN recovered DE signals best. DP constraints (especially in P-PGM) reduced DE recovery, particularly for subtle signals, though larger datasets (COMBINED) mitigated this loss.
• Co-expression Networks:
  • Deep Models: High True Positive Rate (TPR) for recovering edges but introduced many spurious edges (false positives).
  • Statistical Models (MVN): Produced sparse networks with very few spurious edges but missed many true connections under strict correlation thresholds.
  • DP Models: Generally failed to recover complex co-expression networks reliably.

D. Privacy Risk (MIA)

• High Risk: CVAE and CVAE-GMM exhibited the highest vulnerability to MIAs (TPR > 0.5), suggesting that expressive models memorize training data to achieve high fidelity.
• Low Risk (Effective Privacy): DP-CVAE, DP-NMF, and P-PGM consistently operated at random-guessing levels (TPR ~ 0.1), confirming the effectiveness of formal DP guarantees.
• Low Risk (Underfitting): CTGAN showed low MIA risk but also poor utility, suggesting its privacy "safety" was due to failure to learn the data structure rather than robust protection.
• Intermediate Risk: MVN and WGAN-GP occupied a middle ground, offering moderate utility with moderate privacy risk.

E. Trade-off Analysis

• Utility vs. Privacy: A positive correlation exists between high downstream utility/DE recovery and high MIA risk for non-DP deep models. However, MVN and P-PGM demonstrate that competitive utility can be achieved with moderate-to-low privacy risk.
• Fidelity vs. Utility: High global fidelity (MMD) does not guarantee high downstream utility.
• Biological vs. Utility: DE recovery is strongly correlated with downstream utility (since classification relies on mean shifts), but co-expression recovery is largely independent of utility.

5. Significance and Implications

• No "One-Size-Fits-All" Model: The choice of a generative model must be driven by the specific downstream task and privacy requirements.
  • For high utility/biology: Use CVAE-GMM or Embedded Diffusion (accepting higher privacy risk).
  • For strict privacy: Use P-PGM or DP-NMF (accepting reduced biological network recovery).
  • For a robust baseline: MVN offers a surprising balance of utility, speed, and moderate privacy.
• Multi-Dimensional Evaluation is Mandatory: Relying on a single metric (e.g., MMD or AUROC) is insufficient. Benchmarks must simultaneously assess statistical fidelity, biological signal recovery, and empirical privacy risk.
• Dataset Size Matters: The utility loss from Differential Privacy is less pronounced in larger datasets (5,000 samples vs. 1,000), suggesting DP is more viable for large-scale cohort studies.
• Future Directions: The authors advocate for moving beyond simple rankings to trade-off visualizations, expanding benchmarks to single-cell and multi-omics data, and incorporating fairness assessments across demographic subgroups.

In conclusion, this study provides a critical roadmap for researchers and regulators, demonstrating that while synthetic omics data is viable, its quality is inherently multi-dimensional, and model selection requires a careful balancing of biological fidelity, predictive utility, and privacy guarantees.