💻 computer science

Synthetic tabular healthcare data in health informatics: fixed-size substitution and incremental-value controls for predictive utility Application Paper

This study finds no systematic predictive-utility benefit from synthetic tabular healthcare data under the evaluated conditions. Rigorous comparisons controlling for training-set size and partition leakage showed predominantly lower model performance than real-only or bootstrap-real controls, alongside a few context-specific improvements.

Original authors: Miguel Álvaro Andrade Oliveira, Miguel Ricou, Alberto Freitas

Published 2026-11-04✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Miguel Álvaro Andrade Oliveira, Miguel Ricou, Alberto Freitas

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In modern medicine, computers are increasingly used to predict health outcomes and help researchers understand health risks. To teach these computers, researchers need vast amounts of patient records. However, real medical data is deeply personal and protected by strict privacy laws, making it difficult to share between hospitals or research teams. To address this, scientists have developed a way to create "synthetic" data. Imagine a machine that reads thousands of real patient records, learns the patterns of how different health factors connect, and then writes new, computer-generated records designed to resemble the real ones, rather than describing newly observed patients. Such records still require privacy checks. The hope has been that by adding these synthetic records to the real ones, researchers could give their computer models more practice material, making them smarter and more accurate.

A team of researchers from the University of Porto in Portugal decided to test whether this hope was actually true. They wanted to know if adding synthetic records genuinely improved the computer's ability to predict health outcomes, or if the perceived improvement might partly reflect giving the computer more rows of data to study. They focused on three different public datasets related to diabetes, ranging from large national health surveys to hospital records and a smaller questionnaire dataset. Using three different methods to generate the synthetic data, they ran a series of rigorous experiments. In one set of tests, they kept the total number of training records exactly the same, swapping out real patient rows for synthetic ones to see whether the composition of the data mattered independently of quantity. In another set, they added synthetic rows to the real ones and compared the results against a control group that reused existing real training records sampled with replacement to see if the synthetic data offered any unique value.

The results were clear and somewhat surprising. When the researchers kept the total number of training records fixed and replaced real data with synthetic data, the computer models generally performed worse. In the vast majority of their tests, models trained with some or all real rows replaced by synthetic data were less able to distinguish diabetes-related outcomes than those trained only on real data. The more real data they replaced with synthetic data, the more average performance dropped. This suggests that although synthetic data can reproduce patterns found in real records, the tested training sets did not consistently preserve the information these predictive models needed. This finding concerns the evaluated prediction tasks, not the overall value of synthetic data.

The researchers also looked at what happens when you simply add synthetic data to the existing real data, increasing the total size of the training set. Here, the synthetic data did not provide a consistent advantage. In many cases, adding synthetic rows made the models perform worse than training on the original real records alone or reusing those records through bootstrap sampling. While there were a few isolated instances where the synthetic data helped a specific model in a specific situation, these were exceptions rather than the rule. The study found no evidence of a systematic boost to predictive power under the tested conditions. However, it does not establish why previous studies reported improvements: comparing larger augmented datasets with smaller real-only datasets may confound the effects of data composition and training size.

This work provides an important methodological check for the field of health informatics. It demonstrates that simply generating more data is not a guaranteed solution for improving medical predictions. The study highlights that the value of synthetic data is highly dependent on the specific context and cannot be assumed to be beneficial just because it increases the size of a dataset. For researchers and doctors, the takeaway is that real data validation remains essential. Although suitable synthetic mixtures may support teaching or software testing when task-specific limitations are acceptable, those applications were not directly evaluated here. Using synthetic data to develop clinical prediction models requires real-data testing, calibration, subgroup assessment and governance. Better health predictions depend on reliable real-world evidence and careful evaluation of generated data, not just the quantity of training rows.

Technical Summary: Synthetic Tabular Healthcare Data in Health Informatics

Problem Statement
Health informatics teams require realistic data for predictive modeling and software testing, yet patient records are sensitive and difficult to share. While synthetic data generators aim to preserve useful statistical structures without releasing source records, evaluations can be affected by two factors that may produce overoptimistic assessments of predictive utility:

Training Size Confounding: Reported improvements after adding synthetic data may reflect changes in training-set size as well as the contribution of the synthetic records; without controls, these effects cannot be isolated.
Partition Leakage: Repeated predictor profiles (exact duplicates) split across training and test sets can artificially inflate performance metrics.
Existing literature lacks a graded, fixed-size substitution benchmark that isolates the contribution of synthetic data from the effects of enlarging the training set.

Methodology
The authors designed a leakage-controlled framework utilizing three public binary-outcome healthcare datasets: CDC Diabetes Health Indicators (253,680 rows), Diabetes 130-US Hospitals (101,766 rows), and Early Stage Diabetes Risk Prediction (520 rows). The study employed a rigorous experimental design involving:

Generators: Gaussian Copula, Conditional Tabular Generative Adversarial Network (CTGAN), and Tabular Variational Autoencoder (TVAE).
Predictive Models: Logistic Regression, Random Forest, and Histogram Gradient Boosting.
Experimental Scenarios:
1. Fixed-Size Substitution: The training size ( $N$ ) was held constant. Real rows were replaced by synthetic rows at rates of 0% (Real-Only), 25%, 50%, 75%, and 100%.
2. Incremental-Value Control: A baseline of $N$ $N$ rows was compared with two arms of size $2N$ $2 N$ :
  - $RR-N$ : $N$ real rows.
  - $RB-2N$ : $N$ real rows plus $N$ bootstrap-sampled rows from those same records (not new independent patients).
  - $RSADD-2N$ : $N$ real rows plus $N$ synthetic rows.
Data Integrity Measures: The Early Stage dataset was split by exact predictor signature, keeping repeated profiles in one train, validation, or test partition. Diabetes 130-US kept records from the same patient together.
Evaluation Metrics: The primary metric was the Area Under the Receiver Operating Characteristic Curve (AUC). Secondary metrics included AUPRC, F1, balanced accuracy, and Brier score. Statistical significance was assessed using paired Wilcoxon tests with Holm correction across 10 replications. Those replications reused one fixed data split; variability across alternative splits was not assessed.

Key Results
Under the evaluated conditions, synthetic data did not provide a general predictive-utility benefit:

Fixed-Size Substitution: Replacing real data with synthetic data at a fixed training size resulted in significantly lower AUC in 89 of 108 model-specific contrasts after Holm correction. Significant improvements were found in only 4 contrasts, with 15 showing no significant difference. The mean AUC difference ( $\Delta$ AUC) relative to real-only training was negative across all substitution levels, worsening as the proportion of synthetic data increased (e.g., $-0.1469$ at 100% substitution).
Incremental-Value Control: Adding synthetic data to a full set of real rows ( $RSADD-2N$ ) performed significantly worse than the real-only baseline ( $RR-N$ ) in 21 of 27 contrasts and worse than the bootstrap-real control ( $RB-2N$ ) in 16 of 27 contrasts.
Isolated Gains: While some favorable effects occurred (e.g., Gaussian Copula on the Early Stage dataset), they were isolated and context-specific rather than systematic.
Fidelity and Privacy: Gaussian Copula demonstrated the smallest mean Hellinger distance (0.056) compared to CTGAN and TVAE. Privacy-risk proxies indicated varying rates of near-duplicates and proximity to training records, though these were noted as screening proxies rather than definitive anonymity guarantees.
Impact of Leakage Correction: A pre-submission audit revealed that the "Early Stage" dataset originally suffered from leakage due to row-wise splitting of duplicate profiles. Correcting this by grouping splits by predictor signature reduced the Real-Only AUC estimates (e.g., Random Forest dropped from 0.9933 to 0.9782), confirming that previous high performance was inflated by leakage.

Significance and Claims
The paper claims to provide a practical framework for separating the value of synthetic data composition from the effects of sample size enlargement and partition leakage. Its primary contributions are:

Methodological Rigor: It shows why reported predictive gains from augmenting datasets with synthetic rows should be interpreted against appropriate training-size controls, without attributing all observed effects to data composition alone.
Negative Evidence: The study demonstrates that, under the tested conditions, synthetic data substitution or addition offers no systematic improvement in predictive utility (AUC) for healthcare tabular data.
Reproducibility: The authors provide a comprehensive reproducibility package (Online Resource 5) containing code, configurations, and validated outputs to support independent verification.

The authors explicitly state that these findings do not establish privacy guarantees, fairness, clinical equivalence, or deployment suitability. They conclude that while low-loss synthetic mixtures may support teaching and software testing, clinical use requires real-data validation, calibration, and subgroup review. The study emphasizes that duplicate-safe splitting is essential to prevent inflated performance estimates in datasets with repeated predictor profiles.

Technical Summary: Synthetic Tabular Healthcare Data in Health Informatics

More like this