💻 computer science

What Counts as Compute? An Empirical Analysis of Accounting Conventions in Compute-Optimal Transformer Training

This paper demonstrates that while varying accounting conventions for counting parameters and compute significantly alter the fitted exponents of the compute-optimal training frontier, the resulting practical impact on training efficiency is modest due to the flatness of the loss landscape, with the specific accounting choices being predictable from simple architectural ratios.

Original authors: Alexander Memming

Published 2026-07-28

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alexander Memming

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to bake the perfect cake, but you don't have a recipe. You know that the taste depends on two things: how much flour you use (the size of the cake) and how long you bake it (the amount of time). In the world of artificial intelligence, scientists are trying to find the perfect "recipe" for training giant brain-like computers called Transformers. They want to know: if I have a fixed amount of money to spend on electricity and computer time, should I build a tiny brain and feed it a mountain of data, or a huge brain and feed it just a little bit?

To figure this out, researchers have to do some math. They count the "ingredients" (the brain's size) and the "effort" (the computer work required). But here's the tricky part: just like a baker might count "cups of flour" differently depending on whether they include the bowl's weight or just the flour itself, AI scientists count "ingredients" and "effort" in different ways. Some count every single part of the brain, while others only count the main thinking parts. Some count the raw math operations, while others count the actual seconds the computer takes to work. For years, everyone assumed these different ways of counting didn't matter much because the brains they were studying were so huge that the extra bits were tiny. But what if you are baking a small cupcake instead of a giant wedding cake? Does the way you count the ingredients suddenly change the recipe?

This paper, written by independent researcher Alexander Memming, asks exactly that question. The author decided to stop guessing and start measuring. Instead of looking at massive, expensive AI models that only giant companies can afford, Memming built a grid of 35 smaller, manageable AI models. He trained them all on the same educational text, using the same rules, but he analyzed the results four different times, changing only how he counted the "size" and the "cost."

The findings are a bit like discovering that your kitchen scale is lying to you, but only when you weigh small amounts. The paper found that when you use the old, simple way of counting (which ignores the "output head" of the brain), the math suggests you should build a model that is about 19 times smaller than what a more precise count suggests. In other words, depending on which "ruler" you use, the recommended recipe changes wildly. The exponent—a number that tells you how fast the model should grow—shifted from 0.25 to 0.42 just by changing the accounting method. This is a huge difference, suggesting that two scientists looking at the exact same training runs could publish completely different "optimal" recipes simply because they used different definitions for their numbers.

However, there is a twist that makes this less scary for anyone actually building these models. Even though the recipes looked very different on paper, the cakes tasted almost the same. The author found that the "valley" of the best performance is very flat. This means that if you follow the wrong recipe (the one based on the old, simple counting method), you don't waste your whole budget. You might waste about 18% of your compute, but the final result is still very close to the best possible outcome. The paper concludes that while we need to be very careful about how we report our numbers so scientists can compare apples to apples, the practical cost of getting the accounting slightly wrong is surprisingly small. The "perfect" recipe is a bit of a moving target, but the range of "good enough" recipes is much wider than we thought.

Technical Summary: What Counts as Compute? An Empirical Analysis of Accounting Conventions in Compute-Optimal Transformer Training

Problem Statement
Deriving a compute-optimal training recipe for transformers involves two critical, yet often implicit, choices: defining "model size" (what counts as a parameter) and defining "compute" (what counts as the cost of training). The two most cited studies in this domain, Kaplan et al. [2020] and Hoffmann et al. [2022], made divergent choices: the former counted only non-embedding parameters, while the latter counted total parameters. Both relied on the approximation $C \approx 6ND$ (where $N$ is parameters and $D$ is tokens), ignoring the computational cost of the output softmax projection and attention mechanisms.

While these approximations hold for frontier-scale models (e.g., GPT-3, Chinchilla) where embedding and output-head overheads are negligible, they fail at the smaller scales accessible to academic and smaller-scale research. At these scales, the output projection can consume the majority of training FLOPs, and embedding parameters can dominate the parameter count. This paper asks: How much does the reported compute-optimal frontier (specifically the exponent $a$ in $N^* \propto C^a$ and the tokens-per-parameter ratio) shift when the accounting convention changes, holding the underlying training runs fixed? Furthermore, what is the practical loss incurred by following a suboptimal convention?

Methodology
The authors conducted a controlled empirical sweep to isolate the effect of accounting conventions.

Model Family: 35 decoder-only transformers were trained on FineWeb-Edu. Models ranged from 0.056M to 5.98M non-embedding parameters ( $N_{ne}$ ), with widths $d \in \{48, \dots, 288\}$ and depths tied to width ( $L = \max(2, d/48)$ ).
Vocabulary and Compute: Experiments were run at two BPE vocabulary sizes ( $V=2048$ and $V=8192$ ) to vary the ratio of embedding/output-head overheads by a factor of 16. Training compute varied across a 5.8-fold range.
Controls: To eliminate confounders identified in prior scaling studies (e.g., Porian et al. [2024]), the authors calibrated learning rates via their own sweep, matched cosine schedules to run lengths, ensured no data repetition, and used consistent optimizers and tokenizers.
Four Accounting Conventions: The same 35 runs were analyzed under four different definitions of size ( $N$ $N$ ) and cost ( $C$ $C$ ):
1. K (Kaplan): $N = N_{ne}$ , $C = 6N_{ne}D$ .
2. H (Hoffmann): $N = N_{tot}$ (total parameters), $C = 6N_{tot}D$ .
3. F (Exact FLOP): $N = N_{ne}$ , $C = C_{exact}$ (including attention matmuls and output softmax).
4. W (Wall-clock): $N = N_{ne}$ , $C = C_{sec}$ (measured device time).
Analysis: The authors fitted a standard additive loss surface ( $L = E + A/N^\alpha + B/D^\beta$ ) to the data and derived the compute-optimal frontier for each convention. They also derived a closed-form relation predicting the shift in exponents based on two elasticities computable from the architecture alone.

Key Results

Magnitude of the Shift: Changing the accounting convention on the same dataset caused the fitted exponent $a$ to vary significantly, ranging from 0.25 to 0.42. The implied tokens-per-parameter ratio shifted by a factor of 19 across conventions.
The "Head" Dominance: The primary driver of the discrepancy at small scales is not attention, but the output softmax projection. At the smallest models ( $d=48, V=8192$ ), the output head consumes 83% of training FLOPs, making the $6ND$ approximation underestimate true cost by up to 8.5×.
Predictive Accuracy: A closed-form relation derived from two elasticities ( $\kappa_c$ and $\gamma_c$ ) successfully predicted the fitted exponents for three of the four conventions within 0.105 of the numerical fit. This confirms that the shift is a deterministic arithmetic consequence of the accounting choice, not noise.
Ordering: The conventions consistently ordered as $a_H < a_K < a_F$ (where $H$ is total parameters, $K$ is non-embedding with $6ND$, and $F$ is exact FLOPs), matching theoretical predictions.
Practical Cost (The Asymmetry): Despite the large disagreement in reported exponents and optimal ratios, the practical cost of following the "wrong" convention is modest. Because the IsoFLOP loss valley is flat, following a suboptimal recommendation wastes at most 18% of the compute budget (0.031 nats per token). The three FLOP-based conventions were within 5% of each other in compute-equivalent terms.

Significance and Claims
The paper claims that reported compute-optimal exponents are not comparable across studies unless the specific accounting convention (parameter definition and cost metric) is explicitly stated. The divergence is largest precisely in the regime where academic scaling experiments are typically conducted, where overhead ratios are non-negligible.

However, the paper is modest regarding the absolute values of the exponents, noting they are properties of the specific data, architecture, and optimization protocol used, not universal constants. The primary contribution is structural:

It provides a conversion rule (Eq. 12) allowing researchers to translate exponents between conventions.
It offers a diagnostic (checking if $a+b=1$ ) to identify if a study assumed the $6ND$ approximation.
It demonstrates that while bookkeeping choices create large numerical discrepancies in reported metrics, the loss landscape is sufficiently flat that the practical penalty for these discrepancies is small.

The authors conclude that the field should explicitly report accounting conventions to ensure reproducibility and comparability, but that practitioners need not panic over minor deviations in recommended recipes, as the "compute-optimal valley" is forgiving.

More like this