🤖 machine learning

Speedrunning Tabular Foundation Model Pretraining

This paper introduces a community-driven "speedrun" challenge for the nanoTabPFN model, where contributors compete to drastically reduce pretraining time and data requirements to achieve a fixed downstream performance target, thereby accelerating the iteration cycle for tabular foundation model research.

Original authors: Salih Bora Ozturk, Alexander Pfefferle, Frank Hutter

Published 2026-06-03

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Salih Bora Ozturk, Alexander Pfefferle, Frank Hutter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to make decisions based on spreadsheets (tables of data). Usually, this is like trying to fill a giant swimming pool with a single teaspoon: it takes a long time, costs a fortune in electricity, and by the time you're done, you're too tired to try a new idea.

This paper introduces a fun, competitive solution to that problem called a "Speedrun."

The Big Idea: The "Speedrun" Analogy

Think of video game speedruns, where players try to beat a game as fast as possible using clever tricks. The authors created a similar competition for training AI models on tabular data.

The Goal: Instead of trying to build the "perfect" model, the goal is simply to reach a specific, modest performance level (beating a standard "Random Forest" algorithm) as quickly as possible.
The Rules: Everyone uses the same computer (one NVIDIA L40S graphics card) and the same starting code. The only thing you can change is how you train the model.
The Prize: The winner gets to show off the fastest time on a public leaderboard.

The Starting Line: The "Slow Baseline"

The authors started with a standard, educational version of a tabular AI model called nanoTabPFN.

The Problem: Using the standard settings, it took 74.32 minutes to train the model enough to hit the target performance. It also needed to "read" over 80,000 fake practice datasets to learn the rules.
The Metaphor: This is like a student studying for a test by reading every single page of a textbook, one word at a time, taking two hours to finish.

The Race: How They Got Faster

The community (researchers and enthusiasts) took turns modifying the training script to shave off time. Here is how they sped things up, using simple analogies:

Better Study Techniques (The Muon Optimizer):
They swapped out the standard learning method for a smarter one called "Muon."
- Result: Time dropped to 54 minutes. It was like switching from reading a book aloud to skimming the key points.
Upgrading the Hardware & Math (SDPA, bf16, Width):
They changed how the computer does math (using a faster type of calculation called "bf16") and adjusted the size of the model's "brain" (making it wider but with fewer channels).
- Result: A massive jump! Time dropped to 10 minutes. This is like upgrading from a bicycle to a sports car.
Batching and Compiling:
They grouped tasks together so the computer didn't have to stop and start as often, and they "compiled" the code to run smoother.
- Result: Time dropped to 9 minutes.
The "Thinking Rows" Trick:
They added 16 special, invisible "thinking rows" to the data. These are like little sticky notes the model can write on to organize its thoughts before answering.
- Result: Time dropped to 3.88 minutes. It's like giving the student a whiteboard to brainstorm before taking the test.
Grouping Features:
They taught the model to look at columns of data in overlapping groups (like looking at a puzzle piece and its neighbors simultaneously) to stop it from getting confused.
- Result: Time dropped to 2.15 minutes.
The AI Coach (Autoresearch HPO):
Finally, they used an AI agent (a robot coach) to automatically tweak the settings and find the perfect combination of tricks.
- Result: 0.92 minutes.

The Final Score

The current record holder trained the model in less than one minute (0.92 minutes).

Speed: This is 81 times faster than the original method.
Efficiency: The model needed to see 22 times fewer fake datasets to learn the same thing.

Why This Matters (According to the Paper)

The paper isn't claiming this specific model is now the best at solving real-world problems. Instead, the format is the invention.

The "Protocol": They created a simple, fair way for the whole community to test new ideas. If someone has a new trick, they just run it, log the time, and see if it beats the record.
Stacking Improvements: Because everyone builds on the same script, we can see exactly which tricks work and which don't.
Future Potential: The paper notes that if you take this super-fast recipe and let it run for the full 74 minutes (instead of stopping early), it actually becomes a very powerful model, suggesting these speed tricks also make the AI smarter, not just faster.

In short, the paper turned a slow, expensive, and isolated research process into a fast, open, and collaborative game where the community races to find the most efficient way to teach AI about spreadsheets.

Technical Summary: Speedrunning Tabular Foundation Model Pretraining

Problem
Pretraining tabular foundation models (TFMs) is a computationally expensive bottleneck that slows the iteration cycle for new architectures, priors, and optimization strategies. While significant efforts have been made to accelerate Large Language Model (LLM) pretraining, no equivalent, community-driven framework exists for TFMs. The high cost of pretraining limits rapid prototyping and the accumulation of speedup techniques within the tabular learning community.

Methodology
The authors introduce a public "speedrun" competition centered around nanoTabPFN, a minimal, educational reimplementation of TabPFN v2. The competition establishes a fixed protocol where contributors modify a single-file training script to reach a specific downstream performance target in the shortest possible wallclock time.

Target: The goal is to pretrain a neural network that achieves an average validation ROC AUC exceeding that of a Random Forest baseline on subsampled TabArena datasets (38 classification tasks).
Constraints: All runs must use a single NVIDIA L40S GPU. The reported time is the median of multiple verification runs to suppress cluster noise. Records must not load pretrained weights and must use a fixed evaluation pipeline.
Progression: The speedrun is cumulative; each new record builds upon the techniques of the previous one, isolating the impact of specific optimizations.

Key Contributions and Techniques
The paper does not propose a new model family but rather a reproducible protocol and a sequence of records demonstrating how various techniques transfer from language modeling to tabular data. The progression from the baseline (74.32 minutes) to the current best (0.92 minutes) involved the following key technical interventions:

Optimizer Switch: Replacing the schedule-free AdamW optimizer with the Muon optimizer for 2D weight matrices significantly improved sample efficiency.
Architecture and Precision:
- Implementation of Scaled Dot-Product Attention (SDPA).
- Switching from post-norm to pre-norm architectures.
- Adoption of mixed precision training (bf16) and TF32 matrix multiplication.
- Adjusting hyperparameters (increasing learning rate to $10^{-3}$ ) and widening the embedding dimension (192 $\to$ 256) while reducing attention heads.
Throughput Optimizations: Batching the Newton–Schulz iteration for Muon and compiling the transformer encoder forward pass.
Architectural Modifications:
- Residual Decay: Exponentially down-weighting inputs to earlier transformer blocks ( $0.95^i$ ).
- RMSNorm: Replacing LayerNorm with lower-precision RMSNorm.
- Thinking Rows: Prepending 16 (later 24) learnable "thinking rows" to the input, which the data rows attend to, significantly boosting performance.
- Repeated Feature Grouping: Embedding columns jointly with specific offsets (e.g., $j, j+1, j+3$ ) to prevent representation collapse in features with similar distributions.
Advanced Training Strategies:
- Latest Weight Averaging (LAWA): Averaging the last 10 checkpoints for evaluation only.
- Weight Decay: Applying weight decay to both AdamW and Muon parameters.
- Decoder Modification: Feeding the mean of feature embeddings (rather than target embeddings) to the decoder.
- Autoresearch: The final record utilized an LLM-driven hyperparameter and architecture search loop (autoresearch) to discover the optimal configuration.

Results
The current best record achieves the target downstream performance in 0.92 minutes on a single NVIDIA L40S GPU. This represents:

An 81× speedup in wallclock time compared to the 74.32-minute baseline.
A 22× reduction in the number of synthetic datasets required (from 80,576 to 3,648).
When trained for the full baseline budget (256,000 datasets), the optimized model reaches an average ROC AUC of 0.82, approaching the performance of TabPFN v2.5 (0.8286) without preprocessing or ensembles.

Significance
The paper claims that the primary contribution is the speedrun format itself, which provides a simple, open protocol for the community to add, verify, and stack pretraining improvements. By fixing the synthetic prior and evaluation pipeline, the leaderboard isolates which practical optimizations are effective for TFMs. The authors note that the current best record holds the synthetic prior fixed, suggesting that the prior itself remains an "untouched direction" for future improvement. The work demonstrates that techniques successful in language modeling (such as Muon, SDPA, and specific architectural tweaks) can be effectively transferred to tabular foundation models to drastically reduce training costs.