🤖 machine learning

Accelerating Reproducible Research in Synthetic EHR Generation

This paper introduces a lightweight, end-to-end benchmarking framework built on PyHealth that unifies codebases, training, and evaluation protocols to enable reproducible, architecture-agnostic comparison of synthetic EHR generation models, specifically targeting longitudinal ICD diagnosis codes.

Original authors: Jalen Jiang, Chufan Gao, Ethan Rasmussen, Stephen Z. Xie, Jimeng Sun

Published 2026-06-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jalen Jiang, Chufan Gao, Ethan Rasmussen, Stephen Z. Xie, Jimeng Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Recipe" Mess

Imagine a group of chefs trying to invent a new type of cake (synthetic patient data) that tastes exactly like a real one but uses no real ingredients (protecting patient privacy).

The problem is that every chef has their own kitchen, their own measuring cups, and their own recipe books.

Chef A uses a specific brand of flour that only works with their mixer.
Chef B measures in cups, while Chef C measures in grams.
Chef D wrote their recipe in a language that no one else speaks anymore.

Because of this mess, no one can fairly compare who made the best cake. If you try to taste Chef A's cake against Chef B's, you don't know if the difference is because the cake is better, or just because they used different measuring tools.

The Solution: A Unified "Taste Test" Kitchen

The authors of this paper built a standardized kitchen (a benchmarking framework) where every chef must use the same tools, the same ingredients, and the same measuring cups.

They didn't just build a new cake; they built the entire testing facility so that future cakes can be compared fairly. They used a popular, community-maintained tool called PyHealth (think of it as a universal kitchen appliance) to make sure everything runs smoothly.

What They Did: Fixing the Old Recipes

The team took the most famous "cake recipes" (generative models) from the past and fixed them so they could work in this new kitchen:

MedGAN & CorGAN: These were old recipes that had been simplified. The authors restored their full capabilities, allowing them to handle the entire list of medical codes (like every specific type of diabetes), not just the top 3 or 4 most common ones.
PromptEHR & HALO: They updated these modern, complex recipes to work perfectly with the new equipment.
The GPT-2 Baseline: They added a simple, general-purpose recipe (GPT-2) to see if a "generalist" chef could compete with the "specialist" medical chefs.

The Big Discovery: The "Long Tail" Problem

In the world of medical codes, a few diseases are super common (like the common cold), but thousands of rare diseases exist (the "long tail").

The Old Way: Previous researchers often ignored the rare diseases to make the math easier. They would say, "Let's just look at the top 1,000 codes." The authors argue this is like judging a chef only on how well they make a sandwich, ignoring that they can't make a soufflé.
The New Way: This paper forces the models to try to generate all 6,955 codes, including the very rare ones.

The Results:

The "Flat" Models (Old Recipes): These models were great at getting the count right (e.g., "10% of people have diabetes"). However, they failed to understand the story of the patient. They couldn't figure out which diseases happen together or in what order. They were like a chef who knows exactly how many eggs to use but doesn't know how to mix them.
The "Sequential" Models (New Recipes): These models (like HALO and GPT-2) were much better at understanding the story. They knew that if a patient has a heart condition, they might also have high blood pressure, and that these happen in a specific order over time.
The Surprise: The simple, general-purpose GPT-2 model performed surprisingly well, often beating the complex, specialized medical models at capturing the "story" of the patient, even if it wasn't perfect at the rarest details.

The Safety Check: Privacy

The team also ran strict safety tests to make sure the fake data didn't accidentally reveal who the real patients were.

The Test: They tried to trick the models into "remembering" real patients.
The Result: All the models passed. None of them leaked private information. The fake data was indistinguishable from real data in terms of privacy risk.

The Takeaway

This paper isn't about inventing a single "perfect" AI doctor. Instead, it's about building a fair playing field.

Before this, comparing different AI models was like comparing apples to oranges because everyone used different rules. Now, the authors have built a single, standardized track where every model must run the same race, using the same full list of medical codes. This allows researchers to finally see which models are actually good at creating realistic, safe, and useful synthetic patient records.

In short: They fixed the broken measuring cups, forced everyone to use the full ingredient list (including the rare spices), and proved that while some specialized chefs are good, a simple, well-tuned general chef can sometimes bake a better cake.

Technical Summary: Accelerating Reproducible Research in Synthetic EHR Generation

1. Problem Statement

The generation of high-fidelity synthetic Electronic Health Records (EHR) is essential for advancing medical research while preserving patient privacy. However, the field faces a reproducibility crisis driven by systemic barriers:

Disjointed Infrastructure: Existing generative models (e.g., MedGAN, CorGAN, PromptEHR) are implemented in disparate codebases with incompatible dependencies, hard-coded dataset dimensions, and procedural designs that hinder head-to-head comparison.
Evaluation Fragmentation: There is no unified benchmarking framework. Metrics are reported inconsistently, and modern standardized tools (e.g., sdmetrics) often fail to handle the high dimensionality and sequential nature of EHR data.
Loss of Granularity: To manage sparsity, researchers frequently truncate ICD-9 codes from the full vocabulary (6,955 codes) to three-digit categories (1,071 codes). This practice discards clinically vital long-tail information (e.g., distinguishing complicated vs. uncomplicated diabetes) and masks the poor performance of models on rare codes.
Architectural Inconsistency: Implementations of the same algorithm often diverge in capability (e.g., some PyTorch ports of MedGAN support only binary matrices, omitting the original count-matrix support).

2. Methodology

The authors introduce a lightweight, end-to-end benchmarking framework built on the community-maintained PyHealth library. The framework unifies the pipeline from data ingestion to evaluation, ensuring all models consume identical inputs and produce outputs in a standardized format.

2.1. Unified Data Representation

The framework adopts a visit-level long-format schema (Subject ID, Visit ID, Code), consistent with community standards like MEDS. This schema preserves:

Temporal Structure: Visit boundaries and the order of visits.
Code Repetitions: Frequency of codes within visits.
Granularity: Full ICD-9 vocabulary (6,955 codes) rather than truncated subsets.

All synthetic outputs, regardless of the generator's internal representation (static matrices or dynamic sequences), are converted into this normalized relational structure prior to evaluation.

2.2. Reimplemented Baselines

The authors re-implement and scale four strong baselines to support full ICD-9 granularity:

MedGAN: Restores the original count matrix support (omitted in previous PyTorch ports) to model unbounded integer counts per diagnosis.
CorGAN: Extends the convolutional architecture from 6 to 8 layers to natively handle the 6,955-dimensional output space, replacing the original 3-digit truncation.
PromptEHR: A BART-based sequence-to-sequence model conditioned on demographics. The authors introduce frequency-guided generation (blending model logits with empirical code frequencies) and demographic-conditioned trajectory initialization to improve clinical realism without augmenting data with real patient records.
HALO: A hierarchical autoregressive model (visit-level and code-level modules) that was already production-ready and required minimal modification.
GPT-2 Baseline: A lightweight, general-purpose autoregressive transformer (8 layers, 512 embedding dimension) implemented to test if specialized architectures are necessary.

2.3. Architecture-Agnostic Evaluation Suite

The framework employs a rigorous, model-agnostic evaluation suite:

Fidelity (Utility):
- Statistical Prevalence: Measures marginal distribution alignment using $R^2$ , Pearson correlation, and RMSE.
- Discriminator-Based Fidelity: Trains a binary classifier (LSTM for sequential data, Random Forest for flattened data) to distinguish real from synthetic data. The Discriminative Score ( $1 - 2|0.5 - \text{Accuracy}|$ ) quantifies indistinguishability.
Privacy:
- Nearest Neighbor Adversarial Accuracy Risk (NNAAR): A sequence-aware set-distance metric (modified Hamming distance) that measures if synthetic samples are closer to training data than to held-out test data.
- Membership Inference Attack (MIA): Tests if an adversary can distinguish training members from non-members based on distance to synthetic data.

3. Key Contributions

Architectural Restoration and Scaling: The authors restore lost capabilities (MedGAN count mode) and extend architectures (CorGAN layers) to support the full ICD-9 vocabulary, moving beyond the truncated subsets used in prior work.
Standardized Benchmarking: They provide a unified pipeline that enables fair, side-by-side comparison of GAN-based and Transformer-based generators using consistent privacy and utility metrics.
Extensible Infrastructure: Built on PyHealth, the framework offers a single pip install package and interactive Colab workflows, lowering the engineering barrier for future research.
Long-Tail Analysis: The framework explicitly evaluates performance on rare codes, revealing failure modes that truncated benchmarks hide.

4. Results

The evaluation was conducted on MIMIC-III data with 10,000 patients.

Fidelity vs. Utility Trade-off:
- Sequential Models (HALO, PromptEHR, GPT-2): Achieved superior Discriminative Scores (closer to 0.5, indicating high indistinguishability). They successfully captured joint distributions and temporal dynamics.
- Flat Models (MedGAN, CorGAN): Achieved high Statistical Utility (high $R^2$ on marginal prevalence) but poor Discriminative Scores. They failed to capture complex joint correlations, making them easily distinguishable from real data.
Privacy: All models demonstrated robust privacy protection, with MIA accuracy near random guessing (0.5) and negligible NNAAR scores, indicating no significant memorization of training data.
GPT-2 Baseline Performance: Surprisingly, the lightweight GPT-2 baseline achieved the highest Discriminative Score (0.355) among sequential models, outperforming specialized architectures in realism, though it lagged slightly in marginal prevalence ( $R^2$ ) compared to HALO.
Long-Tail Performance:
- As the evaluation scope expanded to the 99th percentile of rare codes, HALO and PromptEHR maintained high $R^2$ , demonstrating the ability to model rare dependencies.
- CorGAN collapsed entirely ( $R^2 \approx 0$ ) on rare codes.
- GPT-2 performed well up to the 97.5th percentile but degraded sharply at the 99th percentile, highlighting the difficulty of generating extreme long-tail codes without specialized mechanisms.
- Privacy Risks: The inclusion of rare codes increased MIA risks for all models, suggesting rare codes act as quasi-identifiers.

5. Significance and Claims

The paper claims to provide a starting point for community-driven reproducibility rather than a final, immutable benchmark. Its significance lies in:

Lowering Barriers: By unifying disparate baselines under a single pipeline, it allows researchers to focus on scientific innovation rather than re-implementing code or managing dependency conflicts.
Exposing Hidden Failure Modes: The framework reveals that while models may appear successful on truncated, marginal metrics, they often fail to capture the joint distribution and long-tail reality of EHR data.
Challenging Specialization: The strong performance of the general-purpose GPT-2 baseline suggests that complex, domain-specific architectures may not always be necessary for high-fidelity EHR synthesis.
Future-Proofing: The authors position the framework as a "living substrate" intended to be extended by the community to support ICD-10, additional modalities (labs, vitals), and stronger privacy metrics.

The authors explicitly state they do not claim prior baselines were irreproducible at their release, nor that this framework will permanently withstand technical debt. Instead, they argue that the current state of disjointed repositories makes current comparison nearly impossible, and this work provides the necessary substrate to enable fair, scalable, and reproducible research moving forward.