🤖 machine learning

Measuring the Dependency Gap: Diagnosing Inter-Column Fidelity in Tabular Generative Models

This paper reveals that standard metrics fail to detect inter-column dependency gaps in synthetic tabular data, introducing a new diagnostic that shows even state-of-the-art generators struggle to preserve these dependencies despite increased capacity, due to a lack of direct dependency supervision rather than structural limitations.

Original authors: Jie Zhang

Published 2026-07-27

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jie Zhang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a chef trying to recreate a complex, multi-layered cake for a blind taste test. You have a list of ingredients: flour, sugar, eggs, and chocolate. A "marginal" recipe tells you exactly how much of each ingredient to use. If you follow that recipe perfectly, your cake will have the right amount of sugar and the right amount of flour. But here's the catch: a cake isn't just a pile of ingredients sitting in a bowl; it's about how they interact. The sugar needs to be mixed with the flour, and the chocolate needs to be folded in just right. If you just dump the right amounts of each ingredient into the bowl without mixing them, you have the right ingredients, but you don't have a cake. You have a messy pile.

This is the world of "tabular data," which is just a fancy name for spreadsheets full of information. In fields like spotting credit card fraud or predicting patient health risks, the most important clues aren't found in a single column (like "transaction amount" or "age"). The real secrets are hidden in the relationships between columns. A young person making a huge transaction might be normal, but a young person making a huge transaction at 3 AM in a different country is a red flag. The data generator's job is to create fake spreadsheets that look real, not just by getting the numbers right, but by getting the relationships right. If the fake data gets the relationships wrong, it might look okay on the surface, but it will fail miserably when used to catch the rare, tricky cases that matter most.

The Paper's Story: The "Blind" Test and the Stubborn Gap

The researchers in this paper, led by Jie Zhang, decided to investigate a big problem: How do we know if a computer program generating fake data has actually learned the secret relationships between the columns? They found that the tools everyone was using to check the data were basically blind to these relationships.

The "Blind" Test
Imagine you have a robot judge that is supposed to tell the difference between a real cake and a fake one. The researchers showed that the current "robot judges" (standard metrics like the Logistic-Regression C2ST and Trend scores) were terrible at their jobs. They tested a "degenerate" generator—a program that was so lazy it just took the right amount of flour, sugar, and eggs, but dumped them into the bowl without mixing them at all. It destroyed every single relationship between the ingredients. Surprisingly, the standard robot judges gave this unmixed pile a near-perfect score, saying it was indistinguishable from a real cake! The judges were only checking if the amounts were right, not if the ingredients were actually together.

The New Detective: The XGB-C2ST
To fix this, the authors built a new, much smarter detective: a "gradient-boosted tree" classifier (called XGB-C2ST). Think of this detective as a master baker who doesn't just weigh the ingredients but tastes the texture and structure of the cake. When this new detective looked at the unmixed pile of ingredients, it immediately screamed, "This is fake! The relationships are broken!"

Using this new detective, the researchers tested a state-of-the-art generator called TabbyFlow. They found a "dependency gap." Even though TabbyFlow was a very smart generator, it still had a small but real problem: it wasn't perfectly capturing the complex relationships between columns.

Why This Gap Matters
The paper showed that this gap isn't just a theoretical annoyance; it has real consequences. When they used the unmixed "lazy" data to train a system to find rare fraud cases (the "minority class"), the system failed completely. It was useless. The TabbyFlow generator was much better, but it still had a tiny gap compared to perfect real data, and this gap meant it was slightly less effective at catching the rare fraud cases than it could be.

The Big Mystery: Is the Generator Too Small?
The researchers then asked a crucial question: Why does this gap exist?

Is the math broken? They checked if the type of math the generator uses (mean-field objectives) was fundamentally incapable of learning relationships. They found this wasn't true. In theory, with infinite power, this math can learn the relationships perfectly.
Is the computer too weak? They wondered if the generator just wasn't big or powerful enough. To test this, they made the generator 16 times bigger (increasing its capacity). You would expect a 16x bigger brain to solve the problem. But it didn't. The gap remained exactly the same.

The Real Culprit: Missing Instructions
Since the math can do it, and a bigger brain didn't help, the authors concluded the problem is the "teacher." The generator is being trained with instructions that only tell it to get the individual columns right. It is never explicitly told, "Hey, make sure Column A and Column B move together!" Because the training objective doesn't directly supervise the relationships, the generator doesn't know it needs to fix them.

The "No Easy Fix" Conclusion
Finally, the researchers tried to fix the problem with some clever tricks. They tried adding special "dependency modules" to the generator's brain and tried fixing the data after it was generated (using something called copulas). None of these cheap fixes worked. The gap is a "higher-order" problem, meaning it's about very subtle, complex patterns that simple fixes can't see.

The Takeaway
The paper delivers a sobering but important message: You can't just make a generator bigger or add a fancy new module to fix relationship errors. If the training instructions don't explicitly tell the AI to learn the connections between columns, it won't learn them, no matter how hard you try. The only way to close the gap is to change the training objective itself to directly reward the AI for getting the relationships right. Until then, we have to be careful not to trust our current "blind" tests, because they might be telling us a pile of unmixed ingredients is a perfect cake.

Technical Summary: Measuring the Dependency Gap in Tabular Generative Models

1. Problem Statement

Synthetic tabular data is increasingly used for privacy-preserving sharing and augmenting minority classes in imbalanced domains (e.g., fraud detection, clinical risk). The utility of such data relies not only on preserving the marginal distribution of individual columns but, crucially, on maintaining the inter-column dependencies that carry discriminative signals for minority classes.

The paper identifies a critical flaw in current evaluation practices: standard metrics used to certify synthetic data quality are largely blind to inter-column dependency.

Logistic-Regression C2ST: A baseline generator that models every column independently (destroying all dependency) achieves a score indistinguishable from real data (0.995–1.000 on the SDMetrics scale).
Trend/Shape Scores: These pairwise metrics show only mild sensitivity to destroyed dependency (scoring the independent baseline at 0.69–0.78).

Consequently, practitioners may be misled into believing a generator is faithful when it has actually "washed out" the joint structure required for minority-class utility.

2. Methodology: A Dependency-Aware Fidelity Diagnostic

To address this, the authors introduce a diagnostic framework that decomposes the fidelity of a generator into specific components using a strong classifier two-sample test (XGB-C2ST). Unlike linear models, Gradient-Boosted Trees (XGB) are sensitive to high-order feature interactions.

The Decomposition Protocol:
Let $P$ be the real data distribution and $Q$ be the generator's distribution. The authors define three metrics based on the XGB-C2ST AUC (where 0.5 is indistinguishable and 1.0 is fully separable):

**Full Score ($full $):**$ c2st(P, Q)$. The total discriminability.
**Marginal Score ($marg $):**$ c2st(\Pi(P), \Pi(Q))$, where $\Pi$ independently permutes rows within each column. This destroys all inter-column dependency while preserving marginals.
Dependency Component ($dep$): Calculated as $full - marg$. This isolates the discriminability arising specifically from incorrect joint structures.
Cross Component ( $dep_{cross}$ ): Isolated via block permutation (shuffling numerical and categorical blocks independently) to measure numerical–categorical coupling specifically.

Anchors:
The diagnostic is anchored between two references derived from real data:

Fully-Factorized (FF) Lower Bound: A reference where columns are resampled independently from empirical marginals. It has correct marginals but zero dependency.
Oracle Upper Bound: A fresh subsample of real training data, representing the ideal joint distribution.

3. Key Contributions

Diagnostic Framework: A decomposition of XGB-C2ST into marginal, dependency, and numerical–categorical cross components, anchored by FF and Oracle bounds.
Empirical Characterization: Evidence that standard metrics (LR-C2ST, Trend) fail to detect dependency loss, while XGB-C2ST reveals a significant dependency gap in state-of-the-art (SOTA) generators.
Structural Analysis: An investigation into whether the dependency gap stems from the mean-field objective's structural limitations or capacity constraints.
Negative Results: Systematic testing showing that increasing model capacity (16×), adding in-model dependency mechanisms, or applying post-hoc copula corrections fail to close the gap.

4. Experimental Results

The diagnostic was applied to TabbyFlow (a flow-matching generator based on Exponential-Family Variational Flow Matching, EF-VFM) across five class-imbalanced datasets (Adult, Default, Bank, Magic, Shoppers).

Metric Blindness Confirmed: The FF baseline (zero dependency) scored near-perfect on LR-C2ST (0.995–1.000) and only mildly on Trend, yet XGB-C2ST correctly identified it as highly separable from real data (0.956–0.995).
Existence of a Dependency Gap: TabbyFlow exhibited a significant dependency gap across all datasets. For example, on the Adult dataset, the dependency component was $+0.050 \pm 0.007$ (statistically significant, $p < 10^{-3}$ ).
Impact on Utility: The dependency gap directly correlates with minority-class utility.
- The FF baseline (destroyed dependency) collapsed minority-class F1 scores (e.g., 0.102 on Adult vs. 0.709 for the Oracle).
- TabbyFlow maintained high utility (0.663) but trailed the Oracle by a consistent margin (0.012–0.046), mirroring its residual dependency gap.
Capacity Scaling Failure: Increasing the model width by 16× (from ~10M to ~168M parameters) did not significantly reduce the dependency component. The gap remained statistically unchanged ( $p > 0.05$ ), ruling out insufficient capacity as the primary cause.
Intervention Failure:
- In-Model Mechanisms: Adding a discrete-aware cross-coupling head to the generator produced no significant improvement in dependency scores or minority F1.
- Post-Hoc Corrections: Copula-based corrections (Gaussian, Vine) failed to supply the missing higher-order structure and often degraded utility.
- Higher-Order Nature: The residual gap is "higher-order"; the generator already matches second-order cross-moments (covariance) near the Oracle floor, meaning covariance-matching fixes have nothing to correct.

5. Significance and Claims

The paper argues that the dependency gap is not a structural blind spot of the mean-field objective. Recent theoretical results (Eijkelboom et al., 2026) suggest that variational flow matching can asymptotically recover the true joint distribution even with factorized objectives, provided the per-dimension posterior means are learned accurately.

Instead, the authors conclude the gap arises because the objective provides no direct gradient signal to allocate capacity toward dependency. The deficit lies in what the objective rewards (per-dimension means), not how the architecture represents data or its size.

Key Takeaways for Practitioners:

Standard metrics (LR-C2ST, Trend) are insufficient for certifying dependency; they must be supplemented with dependency-aware diagnostics like XGB-C2ST decomposition.
Simply scaling up model capacity or adding dependency modules does not guarantee the recovery of inter-column structure.
The "binding constraint" appears to be the lack of direct dependency supervision in the training objective.
The paper offers a "caution" against assuming architectural fixes or brute-force scaling will resolve these issues, suggesting that future progress likely requires objectives that explicitly reward dependence.

The authors remain modest in their claims, noting that while the gap is significant and persistent, the analysis is limited to specific datasets and the current state of flow-matching objectives. They do not propose a new generator but rather a diagnostic tool and a characterization of the current limitations in the field.