🤖 machine learning

A Unified Framework for Tabular Generative Modeling: Loss Functions, Benchmarks, and Improved Multi-objective Bayesian Optimization Approaches

This paper presents a unified framework for tabular generative modeling that introduces a novel correlation- and distribution-aware loss function to improve data fidelity, proposes an iterative objective refinement Bayesian optimization (IORBO) strategy for superior hyperparameter tuning, and validates these advancements through comprehensive benchmarks across twenty real-world datasets.

Original authors: Minh H. Vu, Daniel Edler, Carl Wibom, Tommy Löfstedt, Beatrice Melin, Martin Rosvall

Published 2026-05-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Minh H. Vu, Daniel Edler, Carl Wibom, Tommy Löfstedt, Beatrice Melin, Martin Rosvall

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, messy spreadsheet of real-world data—like patient records, credit histories, or customer habits. This data is valuable, but sharing it is risky because it contains private information. You want to create a "fake" version of this spreadsheet that looks and acts exactly like the real one, so researchers can test new ideas without ever seeing the real private data.

This paper presents a Unified Framework (a complete toolkit) to make creating these fake spreadsheets much better. The authors argue that current tools are like clumsy chefs: they can copy the ingredients, but they often mess up the recipe, resulting in fake data that doesn't taste right or breaks when you try to cook with it.

Here is how their new toolkit fixes the problem, explained through three main ingredients:

1. The "Taste-Test" Recipe (The New Loss Function)

In machine learning, a "loss function" is like a scorecard that tells the computer how bad its fake data is. Usually, the computer just tries to make the numbers look similar to the real ones.

The authors realized this isn't enough. Real data has hidden relationships. For example, in a health dataset, "age" and "blood pressure" might be linked. If your fake data has old people with low blood pressure and young people with high blood pressure, the relationships are broken, even if the numbers look right.

The Analogy: Imagine trying to recreate a complex orchestra recording. A standard tool might just make sure the volume of the violins and drums matches the original. But if the violins are playing while the drums are silent (breaking the rhythm), the music sounds wrong.
The Fix: The authors added a special "Correlation-and-Distribution-Aware" rule to the scorecard. This rule forces the computer to check two things:
1. The Rhythm (Correlations): Do the variables still dance together the way they did in the real data?
2. The Shape (Distributions): Does the overall shape of the data (the highs, lows, and averages) match the original?
The Result: The fake data generated with this new rule is much more "faithful." It preserves the secret relationships between variables, making it a much better stand-in for the real thing.

2. The "Fair Judge" (Iterative Objective Refinement Bayesian Optimization)

Once the computer starts making fake data, you need to tune its settings (hyperparameters) to get the best result. Usually, you have to judge the data using many different metrics: some measure how close the numbers are (like a math test), while others measure how well a machine learning model performs on the fake data (like a driving test).

The Problem: Comparing a math score (0 to 100) with a driving score (0 to 1) is like trying to add "apples and oranges." Standard methods often just average them, which can be misleading. If one metric is huge and another is tiny, the tiny one gets ignored.
The Analogy: Imagine a talent show with judges scoring singing, dancing, and comedy. If you just add the scores, a judge who gives 100 points for singing might drown out a judge who gives 10 points for comedy. You need a way to say, "Who was the best singer? Who was the best dancer?" and then rank the contestants fairly.
The Fix: The authors created a new method called IORBO. Instead of adding the scores directly, it ranks them. It asks, "Out of all the attempts we've seen so far, which one was the best at singing? Which was best at dancing?" It then updates the computer's settings based on these rankings.
The Result: This method finds better settings faster and more reliably than standard methods, especially when the metrics are different types.

3. The "Grand Exam" (The Benchmarking Framework)

Finally, the authors built a massive testing ground to prove their ideas work. They didn't just test on one dataset; they tested on 20 different real-world datasets (ranging from small medical records to huge credit card databases) and compared their method against 10 different existing AI models.

The Analogy: Instead of testing a new car on just one smooth track, they drove it on 20 different terrains: dirt roads, icy highways, and steep hills. They also compared it against 10 other popular car models.
The Result: Their new "recipe" (loss function) and "fair judge" (IORBO) consistently beat the other models. The fake data they produced was better at helping other AI programs learn (a task called "TSTR" or Train-Synthetic-Test-Real) and was better at improving models when mixed with real data (augmentation).

Summary

The paper argues that to make good fake data, you can't just look at the numbers in isolation. You need a system that:

Respects the relationships between variables (the new loss function).
Treats different types of success fairly when tuning the system (the new optimization method).
Tests rigorously across many different scenarios (the benchmark).

By combining these three parts into one unified framework, they created a more reliable way to generate synthetic tabular data that behaves like the real thing, without needing to share the actual private data.

Technical Summary: A Unified Framework for Tabular Generative Modeling

Problem Statement

Deep generative models (DGMs) have achieved significant success in synthesizing structured data like images and text. However, their application to tabular data faces unique challenges that hinder direct transfer of techniques from other domains. Tabular data is characterized by mixed continuous and discrete variables, complex non-linear interactions, class imbalances, and intricate statistical dependencies.

Current approaches suffer from three primary limitations across the modeling lifecycle:

Training: Existing DGMs often fail to preserve feature correlations and marginal distributions, sometimes failing to approximate basic statistics like mean and variance, particularly in limited-data settings. They frequently rely on unguided likelihood or adversarial objectives that do not explicitly enforce key statistics.
Hyperparameter Tuning: Standard Bayesian Optimization (SBO) struggles with multi-objective aggregation. Combining heterogeneous metrics (e.g., classification accuracy and regression error) with different units and scales via simple averaging often leads to suboptimal parameter selections.
Evaluation: Rigorous assessment remains fragmented. Existing methods often focus on narrow metric subsets, making it difficult to assess model performance across diverse datasets and obscuring whether improvements are genuine or artifacts of selective testing.

There is a lack of a unified framework that integrates training, hyperparameter tuning, and evaluation to address these interconnected challenges.

Methodology

The authors propose a unified framework comprising three core components:

1. Correlation- and Distribution-Aware Loss Function

To address the failure of DGMs to capture statistical dependencies, the authors introduce a novel loss function designed as an auxiliary regularization term. It consists of two parts:

Correlation-Aware Loss ( $L_{correlation}$ ): This term minimizes the squared difference between the sample correlation matrices of the real data and the generated data. It utilizes variance-based statistics (diagonal covariance) to ensure scalability, avoiding the computational cost of full covariance matrix estimation while capturing pairwise dependencies.
Distribution-Aware Loss ( $L_{distribution}$ ): This term aligns the synthetic data distribution with the real data by matching statistical moments (raw first, central second, and standardized higher-order moments up to order $H$ ). Unlike distance-based metrics (e.g., Wasserstein), this approach uses moment matching for computational efficiency and stability. It handles both continuous and discrete variables uniformly by normalizing moments.

These losses are integrated into the primary objectives of three DGM architectures:

GANs: Added to the generator's loss.
VAEs (specifically TVAE): Added to the reconstruction and KL divergence loss.
Diffusion Models (TabDDPM): Adapted to handle continuous and discrete features separately, with specific terms for discrete feature correlations and continuous feature noise distributions.

Theoretical Guarantees: The paper provides theoretical analysis establishing that the gradients of these loss functions are bounded under sub-Gaussian tail assumptions, ensuring stability during stochastic optimization.

2. Iterative Objective Refinement Bayesian Optimization (IORBO)

To resolve the challenges of aggregating heterogeneous metrics in hyperparameter tuning, the authors propose IORBO.

Mechanism: Unlike SBO, which computes a static objective value for each sample based on a fixed aggregation function, IORBO uses a rank-based aggregation strategy.
Process: At each iteration $p$ , the objective value for a sample $u$ ( $u \le p$ ) is updated based on the ranking of its performance against all samples evaluated up to that point. The surrogate model is refitted with these revised ranks.
Benefit: This iterative refinement allows the optimization process to dynamically adjust the relative standing of hyperparameter configurations as more data is gathered, mitigating biases caused by metrics with different scales or units.

3. Comprehensive Benchmarking Framework

The authors establish a rigorous evaluation pipeline involving:

Datasets: 20 real-world tabular datasets ranging from small (768 rows) to large (~277k rows), covering regression, binary classification, and multiclass classification tasks.
Baselines: 10 established tabular DGMs (including CTGAN, TVAE, TabDDPM, CopulaGAN, etc.).
Evaluation Metrics:
- Statistical Similarity: KL divergence, Pearson's Chi-Square, Kolmogorov–Smirnov test, and Dimension-wise Probability (DWP) for distribution matching; Pearson and Cramer's V for correlations.
- ML Performance (TSTR): Train-Synthetic-Test-Real, where models trained on synthetic data are tested on real data.
- ML Performance (Augmentation): Models trained on a mix of real and synthetic data are compared against those trained on real data alone.
Statistical Analysis: Use of Friedman tests and Nemenyi post-hoc tests to determine statistical significance across models and datasets.

Key Contributions

Novel Loss Function: A correlation- and distribution-aware loss function that acts as a regularizer for DGMs, significantly improving the fidelity of synthetic tabular data and downstream ML performance.
IORBO Algorithm: A new multi-objective optimization strategy that uses iterative rank-based refinement to handle heterogeneous evaluation metrics more robustly than standard Bayesian optimization.
Unified Benchmarking Framework: An open-source framework evaluating 10 DGMs across 20 datasets using a comprehensive suite of statistical and ML performance metrics, providing a standardized protocol for future research.

Results

The framework was validated through extensive experiments:

Loss Function Performance: The proposed loss function significantly outperformed vanilla loss functions in TSTR (Win Rate: 0.611) and Augmentation (Win Rate: 0.551) tasks. While statistical similarity metrics showed mixed results (often statistically indistinguishable from vanilla), the downstream ML utility was consistently improved.
- Models like CTGAN, CopulaGAN, and DP-CGANS showed highly significant gains.
- CTAB-GAN was an exception, showing a statistically significant decrease in performance with the proposed loss, suggesting a need for specialized tuning for that architecture.
Optimization Performance: IORBO consistently outperformed SBO (using mean or median aggregation) in hyperparameter selection, with win rates of 0.591 and 0.561 respectively. It demonstrated superior robustness in handling metrics with different units.
Ablation Studies:
- The correlation loss was identified as the most impactful individual component.
- The distribution loss provided modest gains alone but offered crucial complementary benefits when combined with correlation loss, enhancing robustness.
- The combination of IORBO and the full loss function (IOR-CD) achieved the best overall performance, balancing high win rates with consistent generalization.
Computational Cost: IORBO incurred negligible additional computational overhead compared to SBO, with per-iteration times remaining nearly identical.

Significance and Claims

The paper claims to advance tabular generative modeling by moving beyond isolated method improvements to a unified pipeline. By tightly coupling training, tuning, and evaluation, the framework addresses the "cascade" of issues where failures in one stage (e.g., poor correlation capture) undermine downstream utility.

The authors emphasize that their approach:

Enhances Fidelity: The proposed loss function ensures synthetic data faithfully represents underlying distributions and correlations, leading to better downstream ML performance.
Improves Optimization: IORBO provides a principled way to navigate the complex trade-offs inherent in multi-metric evaluation.
Standardizes Evaluation: The benchmarking framework offers a reproducible, domain-agnostic tool to assess generative models, helping to distinguish genuine advances from artifacts of selective testing.

The work is presented as a step toward more reliable and effective synthetic data generation for tabular data, with the code made available to facilitate further research. The authors note limitations regarding dataset scale (up to ~277k rows) due to computational constraints but argue that the methods scale linearly with features and batch size, suggesting robustness for larger scales in future work.