🤖 AI

Synthetic Data Generation for Augmenting Small Samples

This paper demonstrates that synthetic data generation significantly improves machine learning prognostic performance on small health datasets, particularly those with specific characteristics like low baseline AUC and balanced outcomes, by increasing data diversity more effectively than simple resampling.

Original authors: Dan Liu, Samer El Kababji, Nicholas Mitsakakis, Lisa Pilgram, Thomas Walters, Mark Clemons, Greg Pond, Alaa El-Hussuna, Khaled El Emam

Published 2026-08-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dan Liu, Samer El Kababji, Nicholas Mitsakakis, Lisa Pilgram, Thomas Walters, Mark Clemons, Greg Pond, Alaa El-Hussuna, Khaled El Emam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of medical research, data is the fuel that powers the engines of modern prediction. Scientists gather information about patients—their symptoms, their history, their test results—to train computer programs that can forecast future health outcomes, such as whether a treatment will work or if a disease will return. However, a persistent problem plagues this field: often, there simply is not enough data. Many studies rely on small collections of patient records, sometimes containing only a few hundred entries. When a computer model tries to learn from such a tiny sample, it struggles to find the true patterns that exist in the wider world. Instead, it memorizes the few examples it has seen, leading to predictions that fail when applied to new, unseen patients. This is known as overfitting, and it leaves researchers with tools that are unstable and unreliable. To solve this, scientists have long looked to a technique called data augmentation. Imagine a chef who has only a few ingredients but needs to create a vast menu; instead of buying more, they might learn to combine those few ingredients in new, creative ways to simulate a wider variety of dishes. In the digital realm, this means using computer algorithms to generate new, fake patient records that look and act like real ones, effectively expanding the dataset without needing to find more actual patients. While this trick has been a standard practice for years in fields like image recognition, where computers learn to identify cats or cars by generating thousands of slightly different photos, its value for the messy, complex tables of numbers found in health records has remained largely unproven.

A team of researchers set out to test whether this strategy of creating synthetic data actually works for tabular health information, and if so, under what conditions. They began by taking thirteen large, real-world health datasets, which included records ranging from hospital discharges and insurance claims to surveys on maternal health and adverse drug events. From these large pools, they deliberately carved out small samples, mimicking the scarcity found in many real studies. They then fed these small samples into four different types of computer programs designed to generate new, synthetic patient records. These programs ranged from methods that build decision trees to complex neural networks that learn the statistical relationships between variables. The researchers then trained prediction models on these augmented datasets and tested how well they performed on unseen data, measuring their success with a standard score that reflects accuracy. The results were clear: augmentation was not a magic bullet that worked everywhere, but it was a powerful tool in specific situations. The technique significantly improved the accuracy of the prediction models, but only when the starting dataset was small, the data was complex with many different types of variables, and the initial model was not already performing perfectly. For datasets that were already large or simple, adding synthetic records offered no benefit and could even make the model worse by introducing unnecessary noise.

The study revealed that the value of this approach comes not merely from having more numbers, but from having more variety. To prove this, the researchers compared their synthetic data against a simpler method called resampling, where the computer just copies and pastes existing records to make the list longer. They found that while simply increasing the number of records did not consistently improve the model's ability to predict outcomes, the synthetic data did. This is because the generative models created new, diverse examples that filled in the gaps of the original data, helping the computer model understand the full shape of the population rather than just the few examples it happened to see. The improvement was substantial; in tests on seven small, real-world datasets involving conditions like breast cancer and diabetic retinopathy, the augmented data boosted the accuracy of the models by an average of over fifteen percent, with some cases seeing gains as high as forty-three percent. In contrast, the models trained on simply resampled data often performed no better, or sometimes worse, than the original small datasets.

Recognizing that not every dataset is a candidate for this treatment, the researchers also built a decision support tool to help other scientists know when to try it. By analyzing the characteristics of a dataset—such as its size, how balanced the outcomes are, and how complex the variables are—they created a simple guide that predicts whether augmentation will be helpful. In their simulations, this guide was able to correctly recommend the strategy about seventy-six percent of the time. The study concludes that while generating synthetic data is not a universal fix, it is a highly effective method for rescuing small, complex health studies from poor performance. It allows researchers to build more robust models from limited resources, provided they use the right tools and know when to apply them. The work suggests that the future of small-sample medical research may lie not in waiting for more data to appear, but in intelligently expanding what is already there.

Technical Summary: Synthetic Data Generation for Augmenting Small Samples in Tabular Health Data

Problem Statement
Machine learning (ML) models in health research frequently suffer from suboptimal generalization performance due to small training datasets. Literature indicates that clinical prediction models often operate with significantly fewer events per predictor variable than the recommended thresholds required for stability. While data augmentation is a standard practice for imaging and time-series data, its efficacy and specific benefits for tabular health data remain under-evaluated. There is a lack of comprehensive guidance on whether augmentation improves predictive performance for tabular data, which generative models are most effective, and whether the benefits stem from increased sample size or increased data diversity.

Methodology
The study employed a two-part simulation and evaluation framework using 13 large real-world health datasets and 7 smaller application datasets.

Simulation Framework (Part 1):
- Data: 13 heterogeneous datasets (ranging from ~500k to 4M observations) covering domains such as hospital discharges, insurance claims, and public health surveys.
- Process: Large datasets were split into training (70%) and testing (30%) sets. From the training set, 40 base datasets of varying sizes ( $n_0$ ) were drawn via outcome-stratified random sampling.
- Augmentation: Four synthetic data generation (SDG) models were evaluated: Sequential Decision Trees (SEQ), Bayesian Networks (BN), Conditional Generative Adversarial Networks (CTGAN), and Variational Autoencoders (TVAE). Synthetic records were generated in geometric series to create augmented datasets of varying sizes ( $n = n_0 + n'$ ).
- Modeling: Binary Gradient Boosted Decision Trees (LGBM) were trained on augmented datasets and evaluated on the held-out test set using Area Under the Curve (AUC).
- Decision Support: A binary classification model was trained to predict whether augmentation would improve performance for a given dataset based on input features: base dataset size ( $n_0$ ), imbalance factor, degrees of freedom (predictor complexity), and baseline AUC.
Application and Diversity Evaluation (Part 2):
- Datasets: Seven small, realistic clinical datasets (ranging from 92 to 1,151 observations) were used to test the decision support tool and augmentation workflow.
- Diversity vs. Size: To isolate the impact of diversity from sample size, the study compared generative augmentation against bootstrap resampling (sampling with replacement). Resampling increases sample size but minimally impacts diversity.
- Metrics: Performance was measured by AUC improvement. Diversity was quantified using a novel metric based on outlier scores derived from an Extended Isolation Forest model trained on the base dataset, measuring the "contamination rate" of outliers in augmented data relative to the base.

Key Contributions

Empirical Evaluation of Tabular Augmentation: The study provides a systematic evaluation of four distinct SDG methods on tabular health data, identifying specific conditions under which augmentation yields benefits.
Decision Support Tool: Development of a logistic regression-based decision model that helps analysts determine a priori if augmentation is likely to be beneficial for a specific dataset.
Diversity Mechanism Validation: Demonstration that the performance gains from augmentation are driven by increased data diversity rather than merely increased sample size, as generative models outperformed resampling methods.
Optimization Workflow: A defined process for selecting the optimal level of augmentation ( $n'$ ) and the best-performing generative model for a specific dataset context.

Results

Performance Conditions: Augmentation significantly improved prognostic performance (measured by AUC) for datasets that are:
- Small to medium in size (benefits diminish or reverse for very large $n_0$ ).
- Have lower baseline AUC (indicating a ceiling effect for already high-performing models).
- Contain high-cardinality categorical variables.
- Have more balanced outcome distributions.
Generative Models: No single generative model consistently outperformed the others across all datasets. The optimal model is dataset-dependent, necessitating the evaluation of multiple models.
Decision Support Model: The logistic regression model achieved an AUC of 0.77 in predicting whether augmentation would be useful. Key predictors included baseline AUC (negative coefficient), dataset size (negative coefficient), imbalance factor (negative coefficient), and degrees of freedom (positive coefficient).
Application Results: On the seven small application datasets, augmentation resulted in relative AUC improvements ranging from 4.31% to 43.23%, with an average improvement of 15.55% ( $p=0.0078$ ).
Diversity vs. Resampling: Augmented datasets using generative models demonstrated significantly higher diversity than those created via resampling ( $p=0.046$ ). Furthermore, the AUC of generative augmentation was significantly higher than resampling ( $p=0.016$ ), confirming that diversity is the primary driver of performance gains.

Significance and Claims
The paper claims that data augmentation using generative models offers a marked benefit for machine learning models trained on small, complex tabular health datasets, but only when specific criteria regarding data size, complexity, and baseline performance are met. The study emphasizes that augmentation acts as a form of regularization that increases data diversity, thereby improving generalization to unseen data.

The authors assert that their decision support model provides a practical, low-cost mechanism for analysts to determine the utility of augmentation before incurring the computational cost of training multiple generative models. The findings challenge the assumption that simply increasing sample size (via resampling) is sufficient; instead, the introduction of diverse synthetic observations is critical for performance gains in small-sample scenarios. The study concludes that while augmentation is not universally beneficial, it is a powerful tool for specific subsets of health data problems where data scarcity and complexity hinder model stability.

More like this