Evaluating Inter-Column Logical Relationships in Synthetic Tabular Data Generation

Technical Summary: Evaluating Inter-Column Logical Relationships in Synthetic Tabular Data Generation

Problem Statement
Current evaluations of synthetic tabular data predominantly focus on modeling joint distributions and assessing low-order statistics (e.g., density estimation, correlation scores) or high-order statistics (e.g., $\alpha$-Precision, $\beta$-Recall). However, these metrics often overlook the preservation of realistic event sequences and coherent entity relationships across columns. Tabular data is inherently heterogeneous, featuring diverse variable types and complex interdependencies (Wang et al., 2024). Existing generative models, including GANs (e.g., CTGAN), diffusion models (e.g., TabDDPM, TabSyn), and autoregressive models (e.g., GReaT), struggle to explicitly capture specific conditional dependencies such as $P(x_1 | x_2)$. For instance, diffusion models often assume isotropic noise addition, which fails to account for the non-linear, non-ordered semantic dependencies found in tabular data, unlike the spatial correlations in images. Similarly, while Large Language Models (LLMs) leverage pretrained knowledge, they may not explicitly model marginal distributions, leading to biased sampling. Consequently, there is a lack of fine-grained, context-aware evaluation metrics to assess whether synthetic data preserves critical logical relationships, such as hierarchical structures (e.g., city $\to$ country) or mathematical/temporal dependencies.

Methodology
To address the gap in evaluating logical consistency, the authors propose three new evaluation metrics designed to quantify inter-column relationships:

1. Hierarchical Consistency Score (HCS): This metric quantifies the preservation of hierarchical relationships across columns. It is defined as the ratio of rows where attribute tuples within a specific group $G_k$ (e.g., city, state, country) belong to a set of valid combinations $C_{k,j}$.
   $$HCS = \frac{1}{M \times N} \sum_{k=1}^{N} \sum_{j=1}^{M} \mathbb{1}((x_{i,j})_{i \in G_k} \in C_{k,j})$$
   where $M$ is the number of rows, $N$ is the number of consistency groups, and $\mathbb{1}(\cdot)$ is the indicator function.

2. Multivariate Dependency Index (MDI): This index evaluates multivariate dependencies, including temporal or mathematical relationships. It measures the proportion of rows where attributes satisfy a specific Boolean dependency function $F$ relative to other attributes in a dependency group $G_g$.
   $$MDI = \frac{1}{M \times N} \sum_{g=1}^{N} \sum_{j=1}^{M} \mathbb{1}(D_{g,j})$$
   where $D_{g,j}$ represents the condition that attributes in group $G_g$ satisfy the dependency function $F$.

3. Distributional Similarity Index (DSI): To capture non-linear relationships where explicit rules are difficult to define, DSI compares the log-likelihoods of Gaussian Mixture Models (GMM) fitted to synthetic versus real data rows. It calculates the relative difference in log-likelihoods to reflect fine-grained distributional differences.
   $$DSI = \frac{1}{K} \sum_{i=1}^{K} \left( 1 - \frac{|\log L(\hat{X}^*_{syn,i}) - \log L(\hat{X}^*_{real})|}{|\log L(\hat{X}^*_{real})|} \right)$$

Experimental Setup and Results
The study evaluated five representative generation methods on the DataCo dataset, a large-scale real-world industrial dataset containing approximately 140,000 samples with complex, high-dimensional features and diverse inter-column relationships. The evaluated methods included:

• Interpolation-based: SMOTE
• Latent Space Representation-based: CTGAN, TabDDPM, TabSyn
• LLM-based: GReaT

Each method generated 140,000 samples, and evaluations were repeated ten times to ensure robustness.

Key Findings:

• Statistical Metrics: SMOTE and TabSyn achieved the best performance in low-order statistical accuracies (e.g., Density Estimation, Correlation Score, $\alpha$-Precision). TabSyn excelled in modeling joint distributions, while SMOTE outperformed others in balancing data categories ($\beta$-Recall).
• Logical Consistency (HCS): SMOTE (98.09%) and GReaT (98.01%) significantly outperformed latent space-based methods (CTGAN: 39.23%, TabDDPM: 16.07%) in preserving hierarchical consistency.
• Multivariate Dependencies (MDI): GReaT achieved the highest accuracy (97.37%) in capturing temporal and mathematical dependencies, followed by SMOTE (87.03%). Latent space-based methods struggled significantly, with TabDDPM scoring 59.08% and CTGAN 38.87%.
• Distributional Similarity (DSI): GReaT (85.55%) and TabSyn (83.62%) performed best, while CTGAN (11.10%) showed poor distributional alignment.
• Qualitative Analysis: Detailed inspection of generated samples revealed that CTGAN and TabDDPM frequently produced logical errors (e.g., incorrect geographical hierarchies, violated temporal sequences, and broken mathematical formulas). In contrast, GReaT and SMOTE maintained near-perfect logical consistency, though GReaT occasionally generated unseen attribute values (e.g., incorrect city names), indicating potential issues with controllability.

Significance and Contributions
The primary contribution of this work is the introduction of HCS, MDI, and DSI as specialized metrics to evaluate the preservation of logical relationships in synthetic tabular data, moving beyond standard distributional fidelity. The experimental results demonstrate that existing state-of-the-art methods, particularly those based on latent space representations, often fail to rigorously maintain hierarchical consistency and multivariate dependencies, which are crucial for the fine-grained realism of real-world data.

The paper concludes that while current methods can model general distributions, they lack the explicit mechanisms to enforce logical constraints. Future pathways suggested by the authors include:

• For LLM-based methods: Utilizing knowledge graphs or Bayesian networks to reorder tokenization sequences or restructure column serialization to leverage prior knowledge.
• For latent space-based methods: Employing LLM reasoning to analyze column names and descriptions for semantic relationships, or grouping data by logical relationships before embedding them into a shared latent space.
• Hybrid approaches: Incorporating interpolation techniques like SMOTE to balance data classes, particularly for learning minority logical relationships.

The authors emphasize that these directions are necessary to design generative models that effectively capture inter-column logical relationships, a critical step for advancing synthetic tabular data generation.