Synthetic but Not Realistic: The Evaluation Challenge in Generative Modelling for Structured Electronic Medical Records

Technical Summary: Synthetic but Not Realistic

Problem Statement
Synthetic healthcare data is increasingly proposed as a privacy-preserving substitute for real electronic medical records (EMRs) to facilitate data sharing and model development. However, the evaluation of this synthetic data remains dominated by statistical similarity metrics (e.g., marginal distribution matching) and downstream predictive performance. The authors argue that these standard metrics fail to reflect clinical validity. Specifically, existing evaluation practices often overlook whether synthetic data preserves the structural properties required for reliable epidemiological inference, such as accurate subgroup distributions, calibrated risk estimates, and correct causal dependency structures. Consequently, synthetic data may appear "realistic" under standard metrics while failing to support valid clinical or scientific conclusions.

Methodology
To address this misalignment, the authors propose a multi-dimensional evaluation framework grounded in epidemiology, assessing synthetic data across three distinct dimensions corresponding to descriptive, predictive, and causal research questions:

1. Descriptive Fidelity: Evaluates the reproduction of observable statistical properties, including marginal distributions and, crucially, stratified summaries across predefined subpopulations (e.g., age groups and socioeconomic status quintiles) to detect subgroup-specific distortions.
2. Clinical Utility: Assesses the data's ability to support predictive tasks using time-to-event analysis. The authors employ the Cox proportional hazards model to evaluate:
   • Hazard Ratios (HRs): Preservation of covariate effect sizes.
   • Calibration: The agreement between predicted event probabilities and observed frequencies, measured via the distance-to-1 metric ($D_{21}$), both at the population level and across strata.
3. Structural Validity: Evaluates the preservation of higher-order dependency structures using causal discovery. The Greedy Equivalence Search (GES) algorithm is applied to both real and synthetic data to recover directed acyclic graphs (DAGs). Performance is measured using Structural Hamming Distance (SHD), adjacency precision/recall/F1, and orientation precision/recall/F1.

Experimental Setup
The framework was applied to PRIME-CVD, a simulated cohort of 50,000 adults generated from a hand-specified Directed Acyclic Graph (DAG) with known ground-truth structure. This dataset allows for rigorous validation of statistical, clinical, and causal properties. The authors evaluated four representative generative paradigms under a unified representation:

• WGAN-GP: Generative Adversarial Networks with gradient penalty, augmented with a correlation regularizer.
• WGAN-GP + VAE: A hybrid approach replacing the fixed prior with a learned latent distribution via a Variational Autoencoder.
• DDPM: Denoising Diffusion Probabilistic Models using iterative denoising.
• MCM (Masked Conditional Model): A masking-based autoencoding approach learning conditional reconstruction.

Key Results
The evaluation revealed that while all models successfully reproduced marginal distributions at the population level, none simultaneously preserved subgroup structure, effect estimates, and dependency structure:

• Subgroup Distortion: All models exhibited increased deviations in conditional distributions across socioeconomic (IRSD) quintiles compared to the population level. Notably, DDPM and MCM showed reduced variance and sharper distributions for continuous variables.
• Clinical Utility:
  • Hazard Ratios: GAN-based methods and MCM produced HRs closest to the ground truth, whereas DDPM produced substantially larger HRs for multiple covariates.
  • Calibration: While WGAN-GP achieved the best population-level calibration ($D_{21} = 0.048$), all models exhibited significant calibration errors when stratified by socioeconomic status (e.g., WGAN-GP errors ranged from 0.15 to 0.35 across quintiles).
• Structural Validity:
  • Graph Recovery: No model successfully identified both Age and IRSD as exogenous nodes in the recovered graphs.
  • Spurious Relationships: Synthetic data generally yielded denser graphs than the ground truth, indicating spurious relationships. MCM achieved the lowest Structural Hamming Distance (16) and highest F1 scores for adjacency and orientation, while DDPM showed the largest structural deviation (32).

Significance and Claims
The paper claims that current evaluation practices overestimate the quality of synthetic healthcare data. The authors demonstrate that high distributional fidelity does not guarantee clinical utility or structural validity. Specifically:

• Models with strong marginal fidelity can exhibit poor calibration and distorted relationships, leading to unreliable inference.
• The three evaluation streams (descriptive, predictive, causal) are not interchangeable; a model may succeed in one dimension while failing in others.
• The findings challenge the prevailing "realism-privacy trade-off," suggesting that fidelity issues are currently the primary bottleneck rather than privacy risks.
• The authors conclude that incorporating unrealistic synthetic data into decision models risks introducing bias that could ultimately harm real patients. They advocate for domain-informed assessment based on the ability to support valid clinical and scientific conclusions rather than relying solely on statistical similarity metrics.

The study is modest in its scope, acknowledging that the evaluation was conducted on a single dataset (PRIME-CVD) with known ground truth, which may not capture all complexities of real-world EMR data. However, the results serve as a motivation to shift evaluation standards toward epidemiologically grounded frameworks.