🤖 machine learning

End-to-End Compression for Tabular Foundation Models

The paper introduces TACO, an end-to-end tabular compression model that significantly reduces inference time and memory usage compared to state-of-the-art transformer-based tabular foundation models while maintaining or improving predictive performance on large-scale datasets.

Original authors: Guri Zabërgja, Rafiq Kamel, Arlind Kadra, Christian M. M. Frey, Josif Grabocka

Published 2026-06-01

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Guri Zabërgja, Rafiq Kamel, Arlind Kadra, Christian M. M. Frey, Josif Grabocka

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Library of Everything"

Imagine you have a brilliant librarian (the Tabular Foundation Model) who can predict almost anything about a customer, a machine, or a patient. To do this, the librarian doesn't just guess; they read the entire history of every single person who ever walked through the door (the training dataset) before making a prediction for a new person.

In the past, this librarian was slow because they had to read every single book in the library one by one. Recently, a new type of librarian arrived who is super fast at reading, but they have a major flaw: they get overwhelmed by the size of the library.

If the library has 100 books, the librarian is fine. But if the library has 100,000 books, the librarian's brain (the computer's memory) explodes. They can't hold all the books in their head at once to make a prediction. This is the "quadratic complexity" problem mentioned in the paper: as the data grows, the time and memory needed to process it grow explosively, making it impossible to use these smart models on huge, real-world datasets.

The Solution: TACO (The "Smart Summarizer")

The authors of this paper created a new tool called TACO. Think of TACO as a super-efficient summarizer that sits between the massive library and the librarian.

Here is how it works in three simple steps:

The Compression (The Summarizer): Before the librarian ever sees the data, TACO takes the massive library of 100,000 books and condenses it into a single, tiny, 100-page "Greatest Hits" booklet. It doesn't throw away the important stories; it learns the patterns and essence of the data and writes them down in a compact format.
The Prediction (The Librarian): The librarian then reads this tiny 100-page booklet instead of the massive library. Because the booklet is so small, the librarian can make predictions almost instantly.
The Result: The librarian still knows almost everything they need to know, but they are doing it with a fraction of the effort.

What TACO Achieves (The Magic Numbers)

The paper tested this system against the best existing models (like TabPFN) and found some incredible results:

Speed: TACO is up to 94 times faster at making predictions. If the old model took 10 seconds to think, TACO takes a fraction of a second.
Memory: TACO uses up to 97% less memory. Imagine trying to carry a heavy suitcase (the old model) versus a small envelope (TACO). The old model often crashes because the suitcase is too heavy for the computer to hold; TACO fits easily in a pocket.
Accuracy: Despite shrinking the data down to just 1% of its original size, TACO didn't lose much accuracy. It performed just as well as the models that tried to read the whole library.

How They Did It (The Secret Sauce)

The paper explains that TACO isn't just a simple filter; it's a jointly trained team.

Imagine a Compressor (the summarizer) and a Predictor (the librarian) training together in a gym.
They practice together: The Compressor learns how to summarize the data specifically so the Predictor can understand it best. The Predictor learns how to read these summaries effectively.
If they trained separately, the Compressor might summarize things the Predictor can't understand. But because they train together (end-to-end), they speak the same language.

The "Chunking" Trick (Handling the Unmanageable)

The paper also solved a problem for datasets so huge they are bigger than any computer's memory (like 1.5 million rows).

The Analogy: Imagine trying to summarize a 1,000-page novel, but your notepad only holds 10 pages.
The Strategy: TACO breaks the novel into small chapters (chunks). It summarizes Chapter 1, then Chapter 2, then Chapter 3. Finally, it stitches these small summaries together into one final "Master Summary."
This allowed them to run predictions on a dataset with 1.5 million rows, a task that caused other models to crash immediately due to memory limits.

The Bottom Line

The paper claims that TACO allows us to use the super-smart "Foundation Models" on massive, real-world datasets without needing a supercomputer. It turns a slow, memory-hungry process into a fast, lightweight one, all while keeping the predictions accurate. It effectively solves the "scaling problem" that was stopping these AI models from being used on the biggest data problems.

Technical Summary: End-to-End Compression for Tabular Foundation Models

Problem Statement

Tabular foundation models (TFMs), particularly those utilizing in-context learning (ICL) with decoder-only transformer architectures, have recently challenged the dominance of gradient-boosted decision trees. However, these models face a fundamental scaling bottleneck: their inference complexity scales quadratically ( $O(N^2)$ ) or linearly ( $O(N)$ with KV caching) with the number of training instances $N$ . This quadratic dependency on the conditional context size creates prohibitive memory and compute overheads, limiting TFMs to small or medium-sized datasets. Consequently, performance often degrades on large, information-rich datasets, and the models struggle to handle real-world scenarios requiring massive context windows.

Methodology: TACO

To address this bottleneck, the authors propose TACO (Tabular Compression), an end-to-end modular architecture designed to compress the training dataset into a latent space before prediction. The framework consists of two jointly trained components:

Compressor ( $g$ ): A transformer-based module that maps the original training dataset $D_{train}$ (with $N$ rows) into a compact, lower-rank representation $D_{mini-train}$ containing $K$ rows, where $K \ll N$ . The compressor is trained to learn a condensed embedding that captures prototypical patterns of the tabular data. It utilizes a "dummy" table initialized from a subset of the data, which is iteratively refined to represent the full dataset.
Predictor ( $f$ ): A standard tabular transformer (based on the TabPFNv2 architecture) that performs in-context prediction. Instead of attending to the full $N$ -row dataset, the predictor conditions on the compressed context $D_{mini-train}$ and the test batch features.

Training Strategy:
The compressor and predictor are meta-learned jointly in an end-to-end manner. The objective is to minimize the prediction loss over a prior of datasets, incentivizing the compressor to produce a representation that maximizes the predictor's performance. The training pipeline involves:

Synthetic Pretraining: Using structural causal models (SCMs) to generate diverse synthetic datasets.
Curriculum Learning: Progressively increasing sequence lengths during training (from 1k to 60k rows).
Real-Data Adaptation: Fine-tuning on real tabular datasets using self-supervised tasks (bootstrapping rows/features).
Multi-Rate Training: The model is trained to handle various compression rates ( $r \in \{1, 2, 4, 8, 16\}\%$ ) simultaneously, allowing a single model to operate under different resource constraints.

Scalability Mechanism:
For datasets exceeding the memory capacity of a single forward pass (e.g., >1 million rows), TACO employs a chunking-and-stitching strategy. The dataset is partitioned into disjoint chunks, each compressed independently, and the resulting summaries are concatenated to form a global context for the predictor.

Key Contributions

Novel Architecture: Introduction of TACO, the first end-to-end method for compressing the training context of tabular foundation models, decoupling predictive quality from the raw size of the conditional context.
Efficiency Gains: Empirical demonstration that TACO achieves up to a 94× speedup in inference time and 98% memory savings compared to state-of-the-art predictor-only transformers (POT), while maintaining performance at a 1% compression rate.
Scalability: A validated strategy for ingesting datasets with over one million rows (e.g., the MetroPT-3 dataset with ~1.5M rows) by compressing them to a few thousand latent rows, a task infeasible for standard TFMs due to memory constraints.
Joint Optimization: Evidence that joint training of the compressor and predictor is superior to freezing the predictor, as it allows the predictor to adapt its representation space to the compressed inputs.

Experimental Results

The authors evaluated TACO on the TabArena benchmark and additional large-scale datasets (TabFSBench, TableShift, MetroPT-3):

Performance vs. Speed: At a 1% compression rate, TACO achieves a 94× speedup in subsequent predictions and reduces memory consumption by 97.6% compared to the uncompressed POT baseline. The performance degradation is statistically insignificant (ROC-AUC difference near zero) compared to the full-context baseline.
Memory Efficiency: In scenarios with KV caching, POT frequently encounters out-of-memory (OOM) errors on datasets with 32,000+ rows, whereas TACO remains feasible. TACO reduces VRAM usage by up to 92% in KV caching settings.
Baseline Comparisons: TACO significantly outperforms simple compression baselines like random sampling and k-Nearest Neighbors (kNN) subsampling across all compression rates.
Large-Scale Handling: On the MetroPT-3 dataset (~1.5M rows), TACO compressed the training set to ~1,200 rows and achieved superior precision and AUPRC compared to baselines and XGBoost, while POT and TabPFNv2 failed to run.
Robustness: On the TableShift benchmark, TACO successfully operated on 12 out of 15 datasets where standard TFMs failed due to memory limits, maintaining competitive accuracy under distribution shifts.

Significance and Claims

The paper claims that TACO fundamentally reframes tabular in-context inference by learning data representations that remain predictive under strict inference-time budgets. The authors assert that their method enables a principled accuracy-latency trade-off, allowing in-context tabular predictors to scale to real-world, large-scale datasets that were previously inaccessible to transformer-based approaches.

The work highlights that the quadratic complexity of attention mechanisms is not an insurmountable barrier if the context is intelligently compressed. By reducing the effective context size by a factor of $N/K$ , TACO linearizes the scaling of memory and compute costs, making high-performance foundation models viable for industrial applications involving massive tabular data. The authors conclude that while TACO advances scalability, future work should address diverse priors for missing values, distribution shifts, and extensions to regression and time-series tasks.