🤖 machine learning

Generating Benchmark Health Data Using a Tabular Diffusion Transformer

This paper proposes a two-stage framework that transforms heterogeneous raw tables into standardized statistical representations and utilizes a diffusion transformer to generate synthetic statistical tables, which are then reconstructed into realistic synthetic raw tables to overcome the limitations of existing methods in cross-tabular data generation.

Original authors: Hao Yan, Lisa Pilgram, Dan Liu, Linglong Kong, Fida Dankar, Khaled El Emam

Published 2026-08-17

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Hao Yan, Lisa Pilgram, Dan Liu, Linglong Kong, Fida Dankar, Khaled El Emam

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Great Data Detective Game

Imagine you are a detective trying to solve a mystery, but instead of a single crime scene, you have thousands of different ones scattered across the globe. Some are messy scribbles on napkins, others are neat spreadsheets, and some are missing pages entirely. In the world of computer science, this is the challenge of synthetic data generation. Scientists want to create fake but realistic datasets that look and act exactly like real data. Why? Because real data often contains private secrets (like medical records) that can't be shared freely. If researchers can generate "fake" data that behaves just like the real thing, they can test their new AI tools, train their algorithms, and solve problems without ever seeing a single person's private information.

To do this, computers usually try to learn the "rules" of the data. Think of it like a chef trying to recreate a dish. If the chef only tastes one specific bowl of soup, they might learn to make that exact bowl perfectly, but they won't know how to make a new bowl that still tastes like soup. Most computer chefs today are stuck making just one bowl at a time. They struggle when asked to learn from a whole pantry of different soups, stews, and sauces all at once. This is where the new research comes in, aiming to teach computers how to be master chefs who can taste a thousand different recipes and then invent brand-new, delicious dishes that have never existed before.

The Paper: Teaching Computers to Cook with a Thousand Recipes

In this paper, a team of researchers from Canada and Germany introduces a clever two-step recipe for teaching computers how to generate realistic, fake health data from a massive collection of different real-world tables. They call their method Cross-Tabular Data Generation (CTDG). Instead of trying to memorize every single row of data (which is like trying to memorize every word in a library), they teach the computer to understand the flavor profile of the data first.

Step 1: The "Flavor Profile" Transformation

The first problem the team tackles is that real-world data tables are messy and different from each other. One table might have columns for "Age" and "Blood Pressure," while another has "Height" and "Weight." A computer can't easily learn from these if they don't look the same.

To fix this, the researchers invented a way to turn every single table into a standardized "statistical table." Imagine taking a complex, messy painting and turning it into a simple color palette card.

The Marginal Recipe: For every column (like "Age"), they don't keep the actual numbers. Instead, they figure out the shape of the distribution. Is it a bell curve? Is it skewed? They use a mathematical tool called a Beta-Binomial distribution to capture this shape using just three numbers: two shape parameters ( $\alpha$ and $\beta$ ) and the number of unique categories (cardinality). They also note how many missing values (blank spots) exist in that column.
The Relationship Map: Data columns often depend on each other (e.g., taller people tend to weigh more). To capture this, they calculate how the columns relate to one another. Instead of keeping a giant, messy grid of numbers, they use a mathematical trick called eigen-decomposition to compress this relationship map into a fixed-size vector of numbers.

The result? Every single table, no matter how different it was originally, gets transformed into a neat, uniform list of numbers. It's like turning a thousand different languages into a single, universal code that the computer can read.

Step 2: The Diffusion Transformer Chef

Once all the tables are converted into these uniform "statistical tables," the researchers train a special AI model called a Tabular Diffusion Transformer (TDT).

Think of this model as a sculptor who starts with a block of noise (random static) and slowly chips away at it to reveal a statue. In the world of AI, this is called a diffusion model. The model learns to take random noise and gradually turn it into a realistic "statistical table" that looks like the ones it studied.

The Training: The model was first "pretrained" on a huge library of 4,095 general datasets (from economics, games, engineering, etc.) to learn general patterns. Then, it was "fine-tuned" on 144 health-related datasets to learn the specific "flavor" of medical data.
The Generation: Once trained, the model can generate brand-new statistical tables. These aren't just copies; they are new combinations of patterns the model learned.
The Reconstruction: Finally, the computer takes these new statistical tables and reverses the process. It uses the numbers to sample new data points, effectively "reconstructing" a fake raw table that looks and acts like real health data.

What They Found

The researchers tested their method rigorously to see if it actually worked.

1. The Reconstruction Test:
First, they checked if they could turn a real table into a statistical table and then turn it back into a raw table without losing the "soul" of the data. They used a metric called dimension-Normalized Wasserstein Distance (dNWD) to measure the difference between the original and the reconstructed data.

The Result: The average difference was incredibly small: 0.095 (with a standard deviation of 0.061). Most datasets had a score below 0.2.
The Lesson: This suggests the statistical tables are a very faithful representation of the original data. The computer didn't just guess; it captured the essential structure.

2. The "Ablation" Test (What if we skip the relationships?):
To prove that understanding relationships between columns is crucial, they ran two experiments where they removed the relationship data.

Scenario A (No Relationships): They told the computer to assume all columns were independent (like rolling dice). The error jumped to 0.304. The data looked okay, but the connections were broken.
Scenario B (Random Relationships): They gave the computer random, made-up relationships. The error exploded to 0.636, and the data became a mess.
The Conclusion: The paper explicitly rules out the idea that you can just look at columns individually. You must capture how they relate to each other to get good results.

3. The Domain Test (Did it learn Health, or just Noise?):
The team generated 2,000 new synthetic health tables and compared them to real tables from Health, Economics, Statistics, Engineering, Finance, and Games. They used a metric called Closest Wasserstein Distance (CWD) to see which real domain the fake data looked most like.

The Result: The synthetic health tables were closest to real health data, with an average CWD of 0.72.
The Comparison: They were much further away from other domains: Economics (0.97), Statistics (0.98), Engineering (1.35), Finance (1.55), and Games (1.75).
The Proof: Statistical tests confirmed that the synthetic data is significantly closer to real health data than to any other type of data. The model didn't just memorize the training set; it learned the underlying "health" pattern.

4. The Diversity Test:
Finally, they asked: "Is the model just copying the same few tables over and over?" They checked how many unique real tables the synthetic ones were close to.

The Result: The nearest neighbors of the synthetic tables covered 77% of the real tables (at $k=1$ ) and over 95% when looking at the top 10 neighbors.
The Conclusion: The model is diverse. It isn't stuck in a loop; it's exploring the whole landscape of health data.

Why This Matters

The authors suggest that this method could be a game-changer for creating benchmark datasets. Right now, researchers often struggle to find good, diverse, and realistic data to test their new medical AI tools. This method can generate an unlimited number of realistic, fake health datasets that capture the complexity of the real world without exposing any real patients' privacy.

However, the paper is careful to note its limits. The method works best for tables with up to 256 columns. If a dataset is massive and has thousands of columns, this specific model might struggle. Also, because the method smooths out the data to find patterns, extreme outliers (very rare, weird data points) might get lost in the translation. The authors suggest that if a study needs those extreme outliers, they would have to be added back in manually.

In short, this paper proposes a way to teach computers to understand the "grammar" of data across many different sources, allowing them to write new, realistic "stories" (datasets) that are perfect for testing and training, all while keeping real people's secrets safe.

Technical Summary: Generating Benchmark Health Data Using a Tabular Diffusion Transformer

Problem Statement

Cross-Tabular Data Generation (CTDG) aims to learn a generative model from multiple heterogeneous tables to produce new synthetic tabular datasets. While existing Synthetic Tabular Data Generation (STDG) methods have advanced significantly, they are largely restricted to single-input-table scenarios. These methods struggle to effectively handle multiple heterogeneous tables with diverse feature sets, column names, and data types. Consequently, they cannot learn a unified generative model capable of capturing shared statistical characteristics and structural patterns across a domain. This limitation hinders the creation of diverse, realistic benchmark datasets that reflect the plurality of real-world data, particularly in healthcare where data is often distributed across non-aligned tables. Current benchmark repositories often lack diversity, rely on arbitrary dataset selection, or use hypothetical distributions that do not reflect actual health data structures.

Methodology

The authors propose a two-stage framework for CTDG that transforms heterogeneous raw tables into a homogeneous representation, learns a unified distribution, and reconstructs synthetic raw tables.

Stage 1: Transformation to Statistical Tables

To address heterogeneity, each real raw table ( $X_i$ ) is converted into a standardized "statistical table." This process abstracts away specific column names and values, retaining only the statistical properties necessary for generation:

Marginal Statistics:
- Categorical Features: Unique elements are sorted by frequency and mapped to integer indices. A Beta-Binomial distribution is fitted to these indices, yielding parameters $(\alpha, \beta)$ and the column's cardinality.
- Numerical Features: Values are discretized into a fixed number of intervals ( $I_N = 100$ ). The same sorting, mapping, and Beta-Binomial fitting procedure is applied, resulting in parameters $(\alpha, \beta)$ and a cardinality equal to $I_N$ .
- Missing Values: The missing rate for each column is recorded.
- Result: Each column is represented by a tuple $(\alpha, \beta, \text{cardinality}, \text{missing rate})$ .
Correlational Structure:
- To preserve dependencies without the computational intractability of sampling from multivariate Beta-Binomial distributions, columns are transformed to a normal distribution via the probability integral transform.
- Pairwise correlations are computed on these transformed variables to form a correlation matrix $\Sigma_i$ .
- Eigen-decomposition: To handle varying column counts across tables, $\Sigma_i$ is decomposed into $U\Lambda U^\top$ . The authors adopt an over-parameterized setting with a fixed dimension $K=256$ . The matrix $A = U\Lambda^{1/2}$ serves as a compact representation where each row vector $\mathbf{a}_i$ corresponds to a column's correlation parameters.
- Unified Representation: Each column in the statistical table is represented by a tuple of $K+4$ parameters: $(\alpha, \beta, \text{cardinality}, \text{missing rate}, \mathbf{a})$ .
Reconstruction: Raw tables are reconstructed by sampling from a multivariate Gaussian distribution using the recovered correlation matrix, mapping samples back to Beta-Binomial distributions via inverse probability integral transform, and introducing missing values based on recorded rates.

Stage 2: Tabular Diffusion Transformer (TDT)

A diffusion transformer model is trained on the set of homogeneous statistical tables to learn their joint distribution.

Architecture: The model uses a standard Transformer encoder. Each row of a statistical table is treated as an embedding (analogous to a token in a sentence), and the table itself is a sequence.
Standardization: Since parameters have diverse scales (e.g., unbounded $\alpha, \beta$ vs. bounded correlation values), global quantile transformations are applied to each of the $K+4$ columns across all tables to map them to a standard normal distribution.
Training: The model follows the Denoising Diffusion Probabilistic Model (DDPM) objective, learning to reconstruct noise added to the input statistical tables.
Strategy: A pretraining-finetuning approach is employed. The model is first pretrained on 4,095 general OpenML datasets to learn diverse statistical patterns, then fine-tuned on 144 health-related datasets (131 OpenML Health + 13 specific healthcare datasets).
Generation: The trained model generates synthetic statistical tables via a $T$ -step denoising process. These are then reconstructed into synthetic raw tables.

Key Contributions

Cross-Tabular Framework: The proposal of a two-stage CTDG framework that enables learning from multiple heterogeneous tables, overcoming the single-table limitations of existing STDG methods.
Statistical Table Representation: A novel method to convert heterogeneous raw tables into homogeneous statistical tables using Beta-Binomial marginal fitting and spectral decomposition of correlation matrices, enabling unified modeling.
Tabular Diffusion Transformer: The adaptation of a diffusion transformer architecture for tabular data generation, specifically designed to handle variable-length sequences of statistical parameters.
Benchmark Generation: The generation of 1,000 synthetic health datasets (10,000 rows each, 10–256 columns) intended for use as realistic benchmarks, addressing the lack of diverse, preprocessed, and domain-specific benchmark repositories.

Results

The study evaluates the framework on 144 health-related datasets using dimension-Normalized Wasserstein Distance (dNWD) and Closest Wasserstein Distance (CWD).

Reconstruction Fidelity:
- Reconstructing raw tables from their extracted statistical representations yielded an average dNWD of 0.095 (SD 0.061).
- Ablation studies confirmed that accurate correlation modeling is critical. Removing correlations (identity matrix) increased dNWD to 0.304, while using random correlations increased it to 0.636, demonstrating that the proposed correlation representation is essential for fidelity.
Synthetic-to-Real Evaluation:
- Using 5-fold cross-validation, synthetic tables generated after health-domain fine-tuning were significantly closer to real Health data (Avg CWD: 0.72) than to other domains (Economics: 0.97, Statistics: 0.98, Engineering: 1.35, Finance: 1.55, Games: 1.75).
- Statistical testing (Wilcoxon signed-rank tests combined via Fisher's method) confirmed that synthetic Health tables are significantly closer to real Health tables than to any other domain ( $p < 0.05$ after Holm correction).
- The synthetic-to-real distance hierarchy mirrored the intrinsic real-to-real distance hierarchy, indicating the model preserves domain-specific structural characteristics and relative cross-domain proximity.
Diversity Analysis:
- Diversity was measured by the "Top- $k$ Ratio," the proportion of unique real tables covered by the nearest neighbors of synthetic tables.
- For $k=1$ , 77% of real tables were covered; for $k=10$ , coverage exceeded 95%. This indicates the model avoids mode collapse and generates tables broadly distributed across the domain.

Significance and Claims

The paper claims that the proposed approach successfully learns a unified generative model from multiple heterogeneous tables, producing synthetic datasets that exhibit both high fidelity to real health data and meaningful diversity. The authors assert that this method provides a foundation for generating realistic benchmark datasets that reflect the plurality of real-world data, addressing issues of arbitrary dataset selection and lack of standardization in current benchmarks.

Regarding privacy, the authors note that because the generative model is trained on aggregate statistical tables (distributions and correlations) rather than individual records, and because there is no one-to-one mapping between training records and generated data, the risk of membership or attribute disclosure is mitigated. The generated datasets are intended to support reproducible benchmarking, simulation studies, and the testing of machine learning pipelines without exposing protected health information. The authors acknowledge limitations, such as the potential masking of extreme outliers and a column limit of 256, but position the work as a significant step toward scalable, domain-specific synthetic data generation.