Cascaded Flow Matching for Heterogeneous Tabular Data with Mixed-Type Features

1. Problem Statement

The paper addresses the challenge of generating high-fidelity synthetic data for heterogeneous tabular datasets. These datasets contain a mix of:

• Categorical features (discrete states).
• Continuous numerical features.
• Mixed-type features: Numerical features that exhibit both continuous distributions and discrete point masses (e.g., zero-inflated, one-inflated, or missing values).

Key Challenges Identified:

• Structural Mismatch: Existing diffusion and flow-based models often force a unified generative objective across all feature types. However, categorical and continuous data rely on different structural assumptions (discrete support vs. continuous density, probability mass vs. density functions).
• Implicit Reweighting: Combining these types under a single objective leads to implicit feature reweighting, where one type dominates the learning process, degrading the quality of the other.
• Mixed-Type Complexity: Standard models struggle to generate features with discrete outcomes (like missing values or inflated zeros) embedded within a continuous distribution. They often fail to capture the specific probability mass required for these discrete states.
• Loss Balancing: Previous approaches require manual tuning of relative loss weights between categorical and numerical features to prevent one from dominating, which is difficult to scale and optimize.

2. Methodology: TabCascade

The authors propose TabCascade, a novel cascaded flow matching framework that decomposes the generation task into low-resolution and high-resolution stages.

A. Conceptual Framework: Resolution in Tabular Data

Instead of visual resolution (pixels), the authors define resolution based on feature granularity:

• Low-Resolution ($x_{low}$): Comprises all categorical features ($x_{cat}$) and a discretized, categorical representation ($z$) of the numerical features ($x_{num}$).
• High-Resolution ($x_{high}$): The fine-grained, continuous details of the numerical features.

B. The Cascaded Pipeline

The joint distribution is factorized as:
$$p_\theta(x_{cat}, x_{num}) = \sum_{z} p_\theta^{high}(x_{num} | z, x_{cat}) \cdot p_\theta^{low}(z, x_{cat})$$

1. Low-Resolution Model ($p_\theta^{low}$):

   • Learns the joint distribution of categorical features and the discretized numerical latents ($z$).
   • Encoder: Uses a Distributional Regression Tree (DT) (or Gaussian Mixture Model) to map continuous numerical values $x_{num}$ into discrete categories $z$.
   • Handling Mixed Types: The discretization $z$ explicitly encodes special states such as "missing" ($c_{miss}$) or "inflated" ($c_{infl}$). If $z$ indicates a missing state, the high-resolution model deterministically outputs a missing value, removing the need to learn this complex probability mass during the high-resolution phase.
   • Model Choice: The authors utilize CDTD (Continuous Diffusion for Mixed-Type Tabular Data) as the low-resolution generator.

2. High-Resolution Model ($p_\theta^{high}$):

   • Generates the detailed numerical values $x_{num}$ conditioned on the low-resolution output ($x_{low} = (x_{cat}, z)$).
   • Masking: If $z$ indicates a discrete state (e.g., missing), the loss for that specific feature is masked, allowing the model to focus capacity on learning the continuous details of non-discrete features.
   • Flow Matching: Uses Flow Matching to learn a vector field that transforms a source distribution to the target distribution.

C. Novel Technical Innovations

To improve the high-resolution generation, TabCascade introduces two key mechanisms:

1. Data-Dependent Coupling:

   • Instead of sampling the source noise $x_0$ from a standard Gaussian $N(0, I)$, the source is conditioned on $z$.
   • $x_0 = \mu(z) + \sigma(z)\epsilon$, where $\mu(z)$ and $\sigma(z)$ are the mean and variance of the specific Gaussian component in the DT encoder that generated $z$.
   • Benefit: This moves the source distribution closer to the target distribution in the latent space, significantly reducing the transport cost (the distance the flow must travel). The authors formally prove this tightens the transport cost bound compared to independent coupling.

2. Guided Conditional Probability Path:

   • Introduces learnable, feature-specific time schedules $\gamma_t(x_{low})$.
   • The probability path is defined as:
     $$x_t = \gamma_t(x_{low})x_1 + (1 - \gamma_t(x_{low}))[\mu(z) + \sigma(z)\epsilon]$$
   • This allows the model to learn non-linear trajectories tailored to specific feature groups defined by the low-resolution information, rather than using a fixed linear path.

3. Key Contributions

1. First Cascaded Diffusion Model for Tabular Data: Proposes the first framework to apply a cascaded (low-to-high resolution) approach to tabular data generation.
2. Mixed-Type Feature Generation: The first diffusion model explicitly designed to handle mixed-type features (continuous + discrete point masses like missing/inflated values) by separating discrete state generation from continuous refinement.
3. Theoretical Guarantee: Provides a formal proof that the proposed data-dependent coupling tightens the transport cost bound, theoretically justifying the efficiency gains.
4. Loss-Free Balancing: By separating the tasks, the model avoids the need for hyperparameter tuning of relative loss weights between feature types, as the low-resolution model handles the "easy" categorical/discrete structure, and the high-resolution model focuses solely on numerical details.

4. Experimental Results

The model was evaluated on 12 diverse datasets (including Adult, Airlines, Diabetes, etc.) against state-of-the-art baselines (CTGAN, TVAE, TabDDPM, TabSyn, TabDiff, CDTD).

• Detection Score (Main Metric): TabCascade achieved a 51.9% improvement in the detection score compared to the best baseline (CDTD). This metric measures how well a classifier can distinguish real from synthetic data; a higher score indicates the synthetic data is indistinguishable from real data.
• Distributional Fidelity:
  • Numerical Features: Significant improvements in Wasserstein Distance (WD) and Shape scores for numerical features, indicating better capture of continuous distributions and mixed-type structures.
  • Categorical Features: Performance remained competitive with CDTD, confirming that the low-resolution model effectively handles categorical data.
• Feature Dependencies: Improved Trend (mixed) scores, demonstrating that the cascaded approach successfully captures dependencies between categorical and numerical features without the "divide-and-conquer" penalty often seen in sequential models.
• Downstream Utility: Achieved a 30% lower Machine Learning Efficiency (MLE) score, meaning models trained on TabCascade's synthetic data performed closer to models trained on real data in downstream tasks.
• Ablation Studies: Confirmed that both the data-dependent coupling and non-linear paths are critical for performance. The Distributional Regression Tree (DT) encoder outperformed the Gaussian Mixture Model (GMM) encoder due to finer granularity and less overlap in clusters.

5. Significance and Impact

• Realism: TabCascade sets a new state-of-the-art for generating realistic synthetic tabular data, particularly for complex datasets containing missing values, zeros, or other discrete anomalies in numerical columns.
• Efficiency: By decomposing the problem, the model learns the difficult "details" (numerical distributions) more efficiently, as it does not waste capacity trying to learn coarse categorical structures simultaneously.
• Generalizability: The framework is modular; the low-resolution component can be swapped with other categorical generators (e.g., ARF), and the approach generalizes to arbitrary mixed-type structures.
• Privacy Trade-off: The paper notes that higher fidelity (better detection scores) naturally brings samples closer to the original data distribution, which may increase privacy risks (measured via Membership Inference Attacks). However, the authors emphasize that the model itself is a tool for generation, and privacy guarantees would require additional mechanisms like differential privacy.

In summary, TabCascade represents a paradigm shift in tabular generative modeling by treating data generation as a hierarchical process, leveraging low-resolution structural information to guide high-resolution detail generation, thereby solving the long-standing issue of mixed-type feature generation.