📊 statistics

GEAR: Generative Expansion and Real Anchoring for Two-Stage Distillation of Tabular Foundation Models

The paper proposes GEAR, a modular two-stage distillation framework that converts resource-intensive Tabular Foundation Models into lightweight, high-performance predictors for commodity CPUs by combining synthetic query expansion with real-data re-anchoring, achieving significant gains in accuracy and efficiency over supervised baselines.

Original authors: Qi Qin, Jiajie Zhu, Dali Chen, Yuzhao Zhang, Jia-Xing Han, Yu Su, Peng Zhang, Ying Yan, Yifan Sun

Published 2026-08-20

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Qi Qin, Jiajie Zhu, Dali Chen, Yuzhao Zhang, Jia-Xing Han, Yu Su, Peng Zhang, Ying Yan, Yifan Sun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of data science, information often comes in the form of spreadsheets: rows of numbers and categories describing everything from loan applications to patient health records. For decades, the most reliable way to find patterns in these tables has been to train specialized computer programs on the specific data at hand. These programs learn by example, adjusting their internal settings until they can predict the outcome for a new row based on the patterns they saw before. However, a new generation of models has emerged that works differently. Instead of learning a fixed set of rules, these "foundation models" act like a universal expert that can look at a labeled table and instantly predict what happens next, simply by reading the context provided. While this approach is incredibly powerful and accurate, it comes with a heavy price. To make a prediction, the model must keep the entire original table in its memory and process it every single time, which is slow, expensive, and often impossible for everyday devices.

Researchers have long sought a way to keep the intelligence of these powerful experts while shedding their heavy baggage. The goal is to teach a small, fast, and simple computer program to mimic the expert's behavior so that it can make predictions on its own, without needing the original table or the massive model. A team of scientists has now introduced a method called GEAR to solve this problem. Their approach does not try to force the small program to learn everything at once. Instead, they break the process into two distinct steps. First, they generate thousands of new, synthetic data points that look like the real data and use them to teach the small program how the expert thinks. Then, they bring the program back to the real data to fine-tune its understanding, ensuring it stays grounded in reality. This two-step process allows the small program to learn the expert's secrets without ever needing to see the expert again during actual use.

The core challenge the researchers faced was that the real data available for training is often limited. If a small program only learns from the few rows of data it has, it might miss the broader patterns the expert has mastered. To fix this, the team's first step involves creating a vast amount of new, fake data that mimics the structure of the real world. They feed these synthetic rows to the powerful expert model to see what it predicts, and then they teach the small program to copy those predictions. This expands the small program's experience, allowing it to practice on a much wider variety of scenarios than it could with real data alone. However, relying solely on fake data is risky; the small program might learn to mimic the expert too well on the fake data but fail when faced with the messy reality of actual records.

To solve this, the researchers added a second step, which they call "real anchoring." Once the small program has learned from the synthetic data, they return it to the real, original table. Here, they do not let the program simply memorize the answers. Instead, they use a clever technique where the program learns from the expert's predictions on data that the expert has never seen before. This prevents the program from relying on memorized answers it is supposed to learn. By mixing the real answers with the expert's guidance, the small program corrects any mistakes it made while learning from the synthetic data. The result is a model that has the broad experience of the synthetic training but the precise accuracy of the real world.

The team tested this method on a wide range of tasks, from binary choices like "yes or no" to more complex classifications with many possible outcomes. They found that the small programs trained with this two-step method were significantly better than those trained on real data alone. On binary tasks, the new method improved the accuracy by nearly two percentage points compared to standard training, and on more complex tasks, it still gained over one percentage point. These improvements held true even when the researchers used different types of small programs, including those based on decision trees, which are common in industry. The small models were not only more accurate but also vastly more efficient. In terms of speed, the new method made predictions between 57 and 2,866 times faster than the original large models. It also reduced the amount of computer memory needed to make a prediction by nearly three times, allowing these powerful tools to run on standard computer processors rather than requiring expensive, specialized hardware.

The researchers also explored how much synthetic data was actually needed. They found that adding more fake data helped at first, but only up to a certain point. Once the small program had seen enough synthetic examples, adding more did not improve performance and sometimes even hurt it if the fake data did not perfectly match the real world. This confirmed that the second step, the return to real data, was essential. Without it, the small program would remain stuck in the world of the synthetic data, unable to handle the nuances of reality. The study showed that simply training longer or starting with a different type of pre-trained model did not produce the same results. The specific combination of expanding the training scope with synthetic data and then grounding it with real data was the key to success.

This work demonstrates that it is possible to distill the intelligence of massive, context-dependent models into lightweight, independent tools without losing much of their power. The method does not require the small program to have access to the original expert or the training data at the moment of prediction. This opens the door for deploying these advanced models in settings where speed and privacy are critical, such as on mobile devices or in secure environments where data cannot be shared. The researchers showed that by carefully balancing the use of generated data with real-world verification, they could create models that are both smart and practical. The findings suggest that the future of powerful data analysis may not lie in building bigger models, but in teaching smaller ones how to think like the giants, using a mix of imagination and reality.

Technical Summary: GEAR for Tabular Foundation Model Distillation

1. Problem Statement

Tabular Foundation Models (TFMs) such as TabPFN, TabICL, and TabDPT have demonstrated strong few-shot performance through in-context learning (ICL). However, their deployment is hindered by significant latency and memory costs. Unlike standard supervised models, TFMs require the entire labeled context table to be available at inference time to condition predictions. This context-dependent inference scales poorly with context size and query workload, often leading to out-of-memory failures and high inference latency, making them unsuitable for large-scale, real-time applications on commodity hardware.

While recent approaches attempt to mitigate this by optimizing context selection or using mixture-of-experts, they still rely on a large teacher model and an inference-time context. Knowledge distillation (KD) offers a path to compress the teacher's task-adaptive behavior into a lightweight, context-free student (e.g., an MLP or tree-based model). However, standard KD faces two primary challenges in this setting:

Query Coverage: Real-world tables provide only a finite number of query locations, which is particularly restrictive in small-sample regimes.
Self-Labeling Leakage: If a teacher predicts on a row that is also part of its own context, the target label leaks into the distillation target, biasing the student.

2. Methodology: GEAR

The authors propose GEAR (Generative Expansion and Real Anchoring), a modular two-stage framework designed to distill frozen TFMs into efficient predictors while addressing coverage and leakage issues.

Stage 1: Generative Expansion

Objective: Expand the coverage of teacher-query locations beyond the observed real data.
Mechanism: A generator (e.g., GAN, diffusion model, or energy-based model) produces synthetic covariates. These synthetic rows are treated solely as query locations for the frozen teacher; their generated labels are discarded.
Training: The student model is trained via pure teacher imitation using soft targets generated by the TFM on these synthetic covariates. This stage aims to acquire broad knowledge from the teacher's behavior across a wider feature space.

Stage 2: Real Anchoring

Objective: Re-align the student with the target domain distribution and correct for any bias introduced by the synthetic data.
Mechanism: The student from Stage 1 is initialized and further trained on the real labeled dataset.
Leakage Prevention: To avoid self-labeling leakage, the framework employs Out-of-Fold (OOF) teacher predictions. The real data is partitioned into $K_{cf}$ folds; for each fold, the teacher is conditioned on the remaining data (excluding the current fold) to generate soft targets.
Loss Function: A mixed objective is optimized, combining hard ground-truth labels with OOF soft teacher targets:
$\mathcal{L}_\lambda(p, y, q) = (1-\lambda)\ell(p, y) + \lambda\ell_T(p, q)$
where $\ell$ is the standard supervised loss and $\ell_T$ is the soft-label loss.

Theoretical Guarantee
The paper derives a risk certificate characterizing the trade-off between the volume of generated queries ( $M$ ) and the fidelity of the generator ( $Q_X$ vs. real distribution $P_X$ ). The analysis shows that while increasing query volume reduces finite-sample error (decaying as $M^{-1/2}$ ), it cannot overcome a "fidelity floor" caused by distribution mismatch. Stage 2 is theoretically necessary to debias the student when the synthetic distribution does not perfectly match the real one.

3. Key Contributions

Framework Design: Introduction of GEAR, a two-stage protocol that decouples generative query expansion from real-data anchoring, supporting diverse generator families, TFM teachers, and student architectures (MLPs, LightGBM, XGBoost).
Empirical Performance: Extensive experiments on TALENT and TabArena benchmarks demonstrating that GEAR significantly outperforms both supervised baselines and real-data-only distillation methods.
Theoretical Insight: Derivation of a risk certificate that explains the diminishing returns of synthetic expansion and the critical role of real-data anchoring in mitigating distribution shift.

4. Experimental Results

Experiments were conducted using TabICL, TabPFN, and TabDPT as teachers, with TabPFGen, Copula, CTGAN, and TabDiff as generators.

Performance Gains (MLP Students):
- Binary Tasks: GEAR MLPs outperformed supervised MLPs by 1.81–2.00 AUC points and surpassed real-data-only distillation by 1.76–2.19 points.
- Multiclass Tasks: GEAR MLPs outperformed supervised MLPs by 1.19–1.35 AUC points and real-data-only distillation by 2.09–2.40 points.
- Tree Models: On binary tasks, two-stage LightGBM and XGBoost models also showed gains over supervised training and outperformed CatBoost (the strongest non-TFM baseline) in mean AUC.
Efficiency:
- Inference Time: GEAR reduced median inference time by 57–2,866× compared to direct TFM inference.
- Memory: Peak prediction memory was reduced by 1.9–3.3×.
- Accuracy vs. Cost: GEAR students retained higher AUC than matched supervised baselines while operating at a fraction of the cost.
Ablation Studies:
- Generator Impact: Higher-fidelity generators (e.g., TabPFGen, TabDiff) yielded larger gains. Lower-fidelity generators (e.g., Copula) showed diminishing returns, consistent with the theoretical fidelity floor.
- Two-Stage vs. Mixed: Separating the stages (GEAR) provided greater stability and higher median gains compared to mixed optimization (training on real and synthetic data simultaneously).
- Mechanism: The gains were confirmed to stem from the informative query locations provided by synthetic data, not merely from longer training schedules or generic warm starts.

5. Significance

The paper claims that GEAR provides a practical solution to the deployment bottleneck of Tabular Foundation Models. By successfully distilling the in-context learning capabilities of large, context-dependent TFMs into lightweight, context-free models, GEAR enables the use of foundation model performance on commodity CPUs with millisecond-scale latency. The work highlights that while synthetic data can expand the teacher's effective coverage, a dedicated real-data anchoring stage is essential to correct distributional shifts and prevent self-labeling leakage, offering a robust pathway for deploying TFMs in high-stakes domains like finance and healthcare.