Synthetic tabular healthcare data in health informatics: fixed-size substitution and incremental-value controls for predictive utility Application Paper

Technical Summary: Synthetic Tabular Healthcare Data in Health Informatics

Problem Statement
Health informatics teams require realistic data for predictive modeling and software testing, yet patient records are sensitive and difficult to share. While synthetic data generators aim to preserve useful statistical structures without releasing source records, evaluations can be affected by two factors that may produce overoptimistic assessments of predictive utility:

1. Training Size Confounding: Reported improvements after adding synthetic data may reflect changes in training-set size as well as the contribution of the synthetic records; without controls, these effects cannot be isolated.
2. Partition Leakage: Repeated predictor profiles (exact duplicates) split across training and test sets can artificially inflate performance metrics.
   Existing literature lacks a graded, fixed-size substitution benchmark that isolates the contribution of synthetic data from the effects of enlarging the training set.

Methodology
The authors designed a leakage-controlled framework utilizing three public binary-outcome healthcare datasets: CDC Diabetes Health Indicators (253,680 rows), Diabetes 130-US Hospitals (101,766 rows), and Early Stage Diabetes Risk Prediction (520 rows). The study employed a rigorous experimental design involving:
• Generators: Gaussian Copula, Conditional Tabular Generative Adversarial Network (CTGAN), and Tabular Variational Autoencoder (TVAE).
• Predictive Models: Logistic Regression, Random Forest, and Histogram Gradient Boosting.
• Experimental Scenarios:

1. Fixed-Size Substitution: The training size (N) was held constant. Real rows were replaced by synthetic rows at rates of 0% (Real-Only), 25%, 50%, 75%, and 100%.
2. Incremental-Value Control: A baseline of N rows was compared with two arms of size 2N:
   • RR-N: N real rows.
   • RB-2N: N real rows plus N bootstrap-sampled rows from those same records (not new independent patients).
   • RSADD-2N: N real rows plus N synthetic rows.
   • Data Integrity Measures: The Early Stage dataset was split by exact predictor signature, keeping repeated profiles in one train, validation, or test partition. Diabetes 130-US kept records from the same patient together.
   • Evaluation Metrics: The primary metric was the Area Under the Receiver Operating Characteristic Curve (AUC). Secondary metrics included AUPRC, F1, balanced accuracy, and Brier score. Statistical significance was assessed using paired Wilcoxon tests with Holm correction across 10 replications. Those replications reused one fixed data split; variability across alternative splits was not assessed.

Key Results
Under the evaluated conditions, synthetic data did not provide a general predictive-utility benefit:
• Fixed-Size Substitution: Replacing real data with synthetic data at a fixed training size resulted in significantly lower AUC in 89 of 108 model-specific contrasts after Holm correction. Significant improvements were found in only 4 contrasts, with 15 showing no significant difference. The mean AUC difference (ΔAUC) relative to real-only training was negative across all substitution levels, worsening as the proportion of synthetic data increased (e.g., -0.1469 at 100% substitution).
• Incremental-Value Control: Adding synthetic data to a full set of real rows (RSADD-2N) performed significantly worse than the real-only baseline (RR-N) in 21 of 27 contrasts and worse than the bootstrap-real control (RB-2N) in 16 of 27 contrasts.
• Isolated Gains: While some favorable effects occurred (e.g., Gaussian Copula on the Early Stage dataset), they were isolated and context-specific rather than systematic.
• Fidelity and Privacy: Gaussian Copula demonstrated the smallest mean Hellinger distance (0.056) compared to CTGAN and TVAE. Privacy-risk proxies indicated varying rates of near-duplicates and proximity to training records, though these were noted as screening proxies rather than definitive anonymity guarantees.
• Impact of Leakage Correction: A pre-submission audit revealed that the "Early Stage" dataset originally suffered from leakage due to row-wise splitting of duplicate profiles. Correcting this by grouping splits by predictor signature reduced the Real-Only AUC estimates (e.g., Random Forest dropped from 0.9933 to 0.9782), confirming that previous high performance was inflated by leakage.

Significance and Claims
The paper claims to provide a practical framework for separating the value of synthetic data composition from the effects of sample size enlargement and partition leakage. Its primary contributions are:

1. Methodological Rigor: It shows why reported predictive gains from augmenting datasets with synthetic rows should be interpreted against appropriate training-size controls, without attributing all observed effects to data composition alone.
2. Negative Evidence: The study demonstrates that, under the tested conditions, synthetic data substitution or addition offers no systematic improvement in predictive utility (AUC) for healthcare tabular data.
3. Reproducibility: The authors provide a comprehensive reproducibility package (Online Resource 5) containing code, configurations, and validated outputs to support independent verification.

The authors explicitly state that these findings do not establish privacy guarantees, fairness, clinical equivalence, or deployment suitability. They conclude that while low-loss synthetic mixtures may support teaching and software testing, clinical use requires real-data validation, calibration, and subgroup review. The study emphasizes that duplicate-safe splitting is essential to prevent inflated performance estimates in datasets with repeated predictor profiles.