🤖 machine learning

Scaling Laws for Classical Machine Learning on Tabular Data: A Benchmark Study

This study presents a large-scale, classroom-distributed benchmark of 11,536 training runs across 18 datasets and 6 model families, demonstrating that while power laws effectively describe learning curves for most tabular models with approximate shared exponents, significant variance persists due to implementation details rather than random seeds.

Original authors: Kaihua Ding

Published 2026-07-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaihua Ding

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to make decisions, like predicting if a customer will buy a product or if a loan will be paid back. For a long time, scientists have been obsessed with "scaling laws" for giant neural networks—the massive, brain-like computers that power things like chatbots and image generators. They discovered a magical rule: if you feed these giants more data, more brain cells, and more computing power, their mistakes drop in a perfectly predictable way, like a slide going down a hill. This rule has become so important that companies spend millions of dollars just to calculate how much data they need to buy.

But here is the twist: most of the real world doesn't run on giant neural networks. It runs on "tabular data." Think of a spreadsheet: rows of customers, columns of their age, income, and purchase history. This is the bread and butter of business, used for everything from insurance pricing to sales forecasting. For these tasks, simpler, older tools like decision trees and linear models often work better than the fancy giants. The big question was: do these simpler tools follow the same magical "more data = fewer mistakes" rule? And if they do, is the rule the same for everyone, or does it change depending on the specific spreadsheet you are using? Until now, no one had tested this on a massive scale to see if the rule holds up when different people try to run the same experiment.

This paper is a giant, classroom-sized experiment designed to answer exactly that. Instead of a single research team running a few tests, the author organized 127 students in a graduate machine learning course to act as independent scientists. Each student was given a specific set of rules and assigned three different real-world datasets (like credit card defaults or housing prices) to work on. They had to train six different types of "classical" machine learning models on these datasets, starting with tiny amounts of data and slowly adding more, to see how the error rates changed. In total, they ran over 11,000 training sessions, creating a massive library of results to see if a simple mathematical formula could predict how well these models would learn.

The results were surprisingly clear, but with a few important caveats. First, the "magic formula" (a power law) actually works very well for most of these models. The authors found that for about 78% of the experiments, the formula fit the data so well that it could accurately predict how much the error would drop as more data was added. However, not all models are created equal. The "tree-based" models (like Boosting and Random Forests) were the champions, consistently outperforming the others and reaching the lowest error rates. On the other hand, some models, particularly Lasso regression, failed to follow the rule entirely, often just guessing the average and ignoring the data size.

The study also tackled a fascinating question: Is there one single "learning speed" for a whole family of models, regardless of the dataset? The answer is a "sort of." For five out of the six model families, the students found that a single average "learning speed" (an exponent) could predict how the models would behave across different datasets almost as well as calculating a unique speed for each one. It's like saying all cars of a certain brand accelerate at roughly the same rate, even if the specific road conditions vary. However, this isn't a perfect universal law; the fit isn't exact, and for some models like Ridge regression, the "speed" was so unstable that it couldn't be trusted.

Perhaps the most surprising finding wasn't about the data, but about the humans doing the work. Even though every student was given the exact same instructions and the same random seed (a digital starting point) to ensure they should get identical results, their final answers still varied slightly. The authors calculated that this "human implementation noise"—caused by tiny, unavoidable differences in how people handle missing numbers or encode text—created a variation of about 14% in the results. This suggests that when a single research team claims to have discovered a specific scaling law, there is a built-in "fuzziness" of about 14% that comes just from the way the experiment was set up, not from the data itself.

In the end, this paper gives us a practical map for the world of tabular data. It confirms that for most business problems, you don't need a supercomputer; a well-tuned tree model will likely win. It provides a "shopping list" for data, estimating that for some common problems, you might only need a few hundred rows of data to reach a high level of accuracy, while others might need tens of thousands. But it also warns us to be humble: the rules are approximate, not absolute, and the way we prepare our data can introduce as much uncertainty as the data itself. It's a reminder that in the world of machine learning, even the simplest spreadsheets have their own complex, slightly messy, but ultimately predictable rhythms.

Technical Summary: Scaling Laws for Classical Machine Learning on Tabular Data

Problem Statement
While empirical scaling laws (power-law decay of test loss with data size) have become central to designing large neural networks, their behavior in classical machine learning (ML) on tabular data remains under-characterized at scale. Prior work on learning curves for tree, linear, and kernel models exists but is limited to small scales (single teams, handful of datasets). There is a lack of cross-replicator evidence determining whether classical models follow predictable power laws, whether exponents are shared across datasets within a model family, and how much variance exists in these estimates due to implementation differences.

Methodology
The study employs a distributed, classroom-scale replication protocol involving 127 graduate students.

Datasets: 18 curated tabular datasets (10 classification, 8 regression) ranging from 303 to 53,940 rows, covering domains like credit risk, healthcare, and e-commerce.
Models: Six classical model families from scikit-learn: Boosting, Random Forest, SVM (RBF), Linear/Logistic Regression, Ridge, and Lasso.
Protocol:
- Fixed hyperparameters (no per-N tuning) to isolate data-size effects.
- A single 80/20 train/test split (seed=42) per dataset.
- Training on seven nested subsets of the training fold (fractions: 0.01 to 1.00).
- Error metrics: $1 - \text{AUROC}$ for classification; normalized RMSE for regression.
Fitting: A three-parameter additive power law, $\text{error}(N) = aN^{-b} + c$ , was fitted to the seven data points for each (student, dataset, model) triple using non-linear least squares.
Aggregation: Results were aggregated via pooled fits (concatenating all student points) and per-fraction means. Cross-student variance was measured as the coefficient of variation (CV) of the exponent $b$ .

Key Contributions

Large-Scale Empirical Benchmark: The study generates 11,536 training runs and 1,648 successful scaling curves, providing the first cross-replicator view of classical ML scaling.
Approximate Predictive Compressibility: It establishes that while a single "universal" exponent does not strictly exist (per-dataset fits are statistically preferred), a single family-level exponent can predict cross-dataset curves with minimal loss in predictive power ( $R^2$ gap < 0.011 for 5 of 6 families). This is framed as "approximate predictive compressibility" rather than dataset-independent universality.
Quantification of Implementation Variance: By fixing random_state=42, the study isolates variance induced by unconstrained protocol elements (preprocessing, encoding, missing-value handling) rather than random seed variance. It finds a mean CV of the exponent $b$ of 0.144 across students.
Practical Data-Requirement Table: The authors provide pooled-fit estimates for the sample size ( $N^\star$ ) required to reach a target error of 0.15, flagging cases where the irreducible error floor ( $c$ ) makes the target unreachable.

Results

Fit Quality: The additive power law fits well, with $R^2 > 0.8$ in 77.7% of cells. Tree ensembles (Boosting, Random Forest) and SVMs show the cleanest fits. Ridge and Lasso perform poorly; Lasso often fails to fit a power law due to shrinking coefficients to zero in low-data regimes, and Ridge exhibits non-monotonic behavior in specific cases.
Scaling Exponents ( $b$ ):
- Tree ensembles exhibit moderate exponents (Random Forest median $\tilde{b} \approx 0.325$ ; Boosting $\tilde{b} \approx 0.480$ ).
- Linear models on classification show larger exponents ( $\tilde{b} \approx 0.830$ ), indicating faster error reduction per data point relative to their high initial error.
- The median exponent across all good fits is 0.495.
Irreducible Error ( $c$ ): Tree ensembles achieve the lowest error floors (e.g., Random Forest median $c \approx 0.110$ ). Linear models have low floors but only because their high $b$ drives error down; in absolute terms, they often underperform tree ensembles at full data.
Model Ranking at Full Data: Boosting wins on 50% of datasets, followed by Random Forest (33%). Linear models and SVMs are mid-pack, while Ridge and Lasso win 0%.
Shared Exponents: A joint fit sharing a single exponent $b_{\text{model}}$ across datasets performs nearly as well as unconstrained fits for Boosting, Random Forest, SVM, and Linear models (LODO $R^2$ gaps $\le 0.012$ ). Ridge is fragile (LODO CV 0.96), and Lasso fails entirely.
Variance: The 14.4% CV in exponents across students highlights that implementation details (not just seeds) significantly impact scaling law estimates. Variance is highest at small $N$ .

Significance and Claims
The paper claims its primary contribution is the scale of replication and the protocol-level reproducibility lens, rather than the novelty of fitting power laws to classical models.

For Practitioners: Default-tuned tree ensembles dominate tabular tasks, offering the best trade-off between scaling exponent and irreducible error floor. The study provides concrete data-requirement estimates (e.g., ~200–500 rows for specific tasks) and warns that for some cells, more data cannot overcome a high error floor ( $c \ge 0.15$ ).
For Theory: Classical tabular models exhibit scaling exponents ( $b \approx 0.3\text{--}0.8$ ) that are generally larger than those of large neural networks ( $b \approx 0.07\text{--}0.35$ ), suggesting each row yields more error reduction in classical models than each token in transformers. However, the irreducible-error floor is the binding constraint in tabular ML, unlike in overparameterized language models where the floor is negligible.
Methodological Insight: The study treats the variance of reported exponents across nominally compliant re-implementations as a first-class quantity. It suggests that single-group scaling studies should report uncertainty intervals wide enough to accommodate a ~14% floor in exponent estimation due to protocol drift.

The authors explicitly state they do not claim universality of exponents; rather, they demonstrate that shared exponents offer "approximate predictive compressibility" that is useful for planning, even if per-dataset heterogeneity remains statistically significant.

Technical Summary: Scaling Laws for Classical Machine Learning on Tabular Data

More like this