Self-Improving Tabular Language Models via Iterative Group Alignment

1. Problem Statement

The paper addresses two fundamental limitations in using Large Language Models (LLMs) for tabular data generation:

1. Static Training Paradigm: Current LLM-based tabular generators (e.g., GReaT) rely on static supervised fine-tuning (SFT). Once trained, they cannot learn from their own generated samples or self-correct statistical errors that emerge during generation. The model's capability is fixed post-training.
2. Misalignment of Objectives: Autoregressive objectives optimize for local token coherence (next-token prediction) but fail to preserve global statistical properties essential for tabular data, such as column distributions, feature correlations, and joint distributions. This leads to samples that look plausible row-by-row but exhibit systematic deviations at the dataset level.

While Reinforcement Learning (RL) and alignment methods (like DPO) have succeeded in text and image generation, they face prohibitive barriers in tabular data:

• Lack of Instance-Level Preferences: Tabular quality is inherently distributional (statistical), not instance-based. It is difficult to define "better" or "worse" for a single row without human expertise or complex statistical metrics.
• Reward Design Difficulty: Designing reward functions that balance competing objectives (fidelity vs. utility vs. privacy) is complex and often leads to training instability or mode collapse.
• Data Leakage Risks: Traditional RLHF requires human feedback or external reward models, which is impractical for private tabular data and risks exposing real records during the alignment process.

2. Methodology: TabGRAA

The authors propose TabGRAA (Tabular Group-Relative Advantage Alignment), a self-improving framework that automates the feedback loop without human intervention.

Core Components

1. Automated Quality Signal (The "Critic"):
   Instead of human preferences, TabGRAA uses an automated signal to evaluate synthetic data. The default signal is a Distinguishability Classifier (e.g., Random Forest or XGBoost) trained to distinguish between real data ($D_{real}$) and newly generated synthetic data ($D_{synth}$).

   • The classifier output is transformed into an indistinguishability score $s(\tilde{x}) \in [0, 1]$. A score near 1 indicates the sample is hard to distinguish from real data (high quality); a score near 0 indicates it is easily separable (low quality).
   • Alternative: A distance-based score (Distance to Closest Record) can also be used.
   • Privacy: The classifier is retrained at every iteration on the current synthetic batch, ensuring the LLM is only fine-tuned on self-generated signals, minimizing the risk of memorizing real records beyond the initial SFT stage.

2. Sample Stratification:
   Generated samples are ranked by their quality scores. The framework creates contrastive signals by pairing samples from the top half (high-quality/indistinguishable) and the bottom half (low-quality/distinguishable) of the distribution.

3. Group-Relative Advantage Alignment (GRAA):
   Unlike standard alignment methods (DPO, KTO) that operate on individual sample pairs, TabGRAA operates on groups.

   • Group Aggregation: Samples are aggregated into two groups: $B_{high}$ (high-quality) and $B_{low}$ (low-quality).
   • Implicit Reward: The method computes the average implicit reward (log-ratio of policy to reference model) for each group: $\bar{r}_{high}$ and $\bar{r}_{low}$.
   • Loss Function: The objective is to maximize the probability that the high-quality group is preferred over the low-quality group using a sigmoid loss:
     $$L_{GRAA} = \sigma(\bar{r}_{low} - \bar{r}_{high})$$
   • Bidirectional Gradient Flow: Crucially, the loss keeps both group terms in the computation graph. This simultaneously suppresses low-quality patterns and reinforces high-quality patterns, preventing the distributional collapse often seen in methods like KTO (which detaches one side of the gradient).

4. Iterative Cycle:
   The process repeats for $T$ iterations:

   • Generate synthetic data using the current policy.
   • Train a new classifier to distinguish real vs. synthetic.
   • Score samples and form groups.
   • Update the policy using the GRAA loss.
   • The reference model ($\pi_{ref}$) remains fixed (anchored to the initial SFT distribution) to prevent excessive deviation.

3. Key Contributions

• First Self-Improving Framework for Tabular Data: TabGRAA enables continuous quality enhancement through automated feedback, transforming static training into an iterative refinement process.
• Group-Relative Advantage Objective: Introduces a novel alignment objective that operates on quality-stratified groups rather than individual preferences. This aligns better with the distributional nature of tabular data and provides more stable gradient estimates.
• Modular and Privacy-Preserving: The quality signal is modular (classifier or distance-based) and computed against self-generated data, mitigating data leakage risks associated with exposing real records during alignment.
• Theoretical Guarantees: The paper provides proofs for bounded gradient norms, $O(1/B)$ variance reduction due to group averaging, and reward-agnostic stability.

4. Experimental Results

The authors evaluated TabGRAA on five diverse UCI datasets (Adult, Default, Shoppers, Magic, Beijing) against:

• Baselines: GReaT (SFT only), GReaT-FT+, and alignment variants (TabDPO, TabNPO, TabKTO).
• State-of-the-Art (SOTA): Diffusion-based (TabDDPM, TabSyn, TabDiff) and GAN-based (CTGAN, TVAE) synthesizers.

Key Findings:

• Superior Performance: TabGRAA consistently outperformed all alignment baselines (DPO, NPO, KTO) across all metrics (Fidelity, Utility, Privacy).
• Competitive with Diffusion: TabGRAA achieved performance comparable to or exceeding diffusion-based synthesizers in terms of Fidelity (CDE, PCC, $\alpha$-Precision) and Privacy (Distinguishability Attack AUC $\approx$ 0.5).
• Stability: Unlike TabKTO, which suffered catastrophic model collapse after a few iterations, TabGRAA showed monotonic improvement over 5–20 rounds.
• Ablation Studies:
  • Group vs. Pair: Random sampling within high/low groups performed as well as explicit top-vs-bottom pairing, confirming the method relies on group statistics, not specific pairings.
  • Bidirectional Flow: Removing the bidirectional gradient (detaching one side) led to instability, validating the design choice.
  • Signal Necessity: Random scoring or random pairing resulted in no improvement, proving the necessity of a meaningful quality signal.

5. Significance

TabGRAA represents a paradigm shift in tabular data synthesis. It moves the field from static statistical replication (where models are fixed after training) to dynamic, self-improving generation. By leveraging automated distinguishability signals and group-level alignment, it solves the "reward design" problem inherent in tabular data without requiring human feedback. This makes it a highly practical solution for privacy-sensitive domains where high-fidelity synthetic data is needed for downstream machine learning tasks, bridging the performance gap between LLM-based generators and diffusion models while retaining the flexibility of autoregressive architectures.