🤖 AI

On the Smallness of the Large Language Models Scaling Exponents

This paper argues that current Large Language Model scaling exponents indicate an unsustainable energy regime, a conclusion that persists even after accounting for the "pedestal effect" of non-zero loss limits, while also drawing analogies to fluid turbulence to discuss the influence of data smoothness on these exponents.

Original authors: Sauro Succi, Peter V. Coveney, Alex Hansen

Published 2026-06-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sauro Succi, Peter V. Coveney, Alex Hansen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Bigger is Better" Problem

Imagine you are trying to teach a robot to speak perfectly. The current rule in the AI world is: "The bigger the robot, the better it speaks." This idea, called the "no-wall" finding, suggests that if you just keep adding more data and more computer power, the robot will keep getting smarter forever.

However, the authors of this paper argue that this strategy is unsustainable. They say we are hitting a wall of energy consumption, even if we haven't hit a wall of intelligence yet.

The Core Problem: Diminishing Returns

The paper uses a simple math concept to explain why this is a problem. Imagine you are trying to clean a very dirty window.

The Goal: Make the window perfectly clear (zero errors).
The Current Reality: To make the window just half as dirty, you don't need twice the effort. You need 1,000 times the effort.

The authors point out that current Large Language Models (LLMs) have a "scaling exponent" (a number that measures how fast they get better) that is very small (around 0.05 to 0.1).

The Analogy: If you want to improve the AI's performance by a tiny bit, you have to feed it a mountain of new data and burn a massive amount of electricity. It's like trying to push a boulder up a hill that gets steeper the higher you go. The paper argues this is a dead end for energy resources.

The "Pedestal" Defense: Why We Can't Just Wait

Some critics say, "Don't worry! We don't actually need the AI to be perfect (zero error). We just need it to be 'good enough' to stop talking nonsense. We stop training before it hits the bottom."

The authors call this the "Pedestal Effect." They imagine a floor (the pedestal) that the AI's errors can never go below, no matter how much data you give it.

The Paper's Rebuttal: Even if we accept that we don't need perfection, the math still doesn't work. The authors show that because the "floor" is so high relative to the starting point, the "good enough" zone is reached so quickly that the tiny scaling exponent still applies.
The Metaphor: Imagine you are trying to fill a bathtub. You argue, "I don't need it full to the brim; I just need it to have a few inches of water." The authors say, "Even with that low goal, the faucet is dripping so slowly that it will take you a million years to get those few inches, and you'll run out of water (energy) long before you get there."

Why Are the Numbers So Small? (The "Roughness" of Data)

The paper asks: Why is the AI getting better so slowly?

They compare AI training to fluid turbulence (like water swirling in a river or smoke rising).

Smooth vs. Rough: If you are trying to learn a smooth, predictable pattern (like a straight line), you learn fast. But real-world data (language, images, complex systems) is "rough" and chaotic, like a stormy ocean.
The Fractal Connection: The authors use an analogy from physics. In a storm, the energy isn't spread out evenly; it's concentrated in tiny, chaotic swirls. To understand the storm, you have to look at those tiny swirls.
The Result: Because the data is "rough" and chaotic, the AI has to work much harder to find the patterns. The paper suggests that the "roughness" of the data forces the scaling exponent to stay very low.

The "Hidden Map" Theory

The paper references a theory by Sharma and Kaplan (SK) which suggests that the speed of learning depends on the Intrinsic Dimension of the data.

The Analogy: Imagine a library.
- The Embedding Space (the size of the library) is huge—maybe millions of shelves.
- The Intrinsic Dimension (the actual number of books that contain the real story) is much smaller, but still very large.
The Problem: Even though LLMs are amazing at finding the "real story" hidden in the massive library (compressing the data), the "real story" is still too complex (too many dimensions) for the AI to learn efficiently without burning up the world's energy supply.

The Conclusion: Stop Chasing "Bigger"

The authors conclude that the "bigger is better" approach is a recipe for energy burnout.

The Verdict: No matter how smart the AI gets, if the data is complex and "rough," the energy required to make it slightly smarter will always be too high.
The Suggested Path: Instead of just building bigger, dumber models that eat more electricity, we need to go back to foundational physics-aware models (called "world models"). These are systems that understand how the world works (like gravity or cause-and-effect) rather than just memorizing patterns in a massive database.

In short: We are trying to solve a complex puzzle by throwing more and more people at it, but the puzzle is so hard that we are running out of coffee (energy) before we solve it. We need to change the strategy, not just add more workers.

Problem Statement
The paper addresses the sustainability of the current trajectory in Large Language Model (LLM) development, which is driven by the "no-wall" finding: the observation that LLM performance continues to improve as model size and data volume increase. While this has led to the mantra "bigger is better," the authors argue that the scaling exponents governing this improvement are dangerously small (typically $\alpha \approx 0.05 \div 0.10$ ). In practical terms, an exponent of 0.1 implies that reducing the loss function by a factor of two requires a $2^{10} = 1024$ -fold increase in computational resources. The authors contend that this regime of "diminishing returns" renders the current scaling strategy unsustainable regarding energy consumption, regardless of whether the loss function is viewed as a strict numerical error or an empirical metric.

Methodology and Theoretical Framework
The authors employ a theoretical analysis combining scaling laws, numerical analysis analogies, and phenomenological models from fluid dynamics to critique and refine the understanding of LLM scaling exponents.

Rebuttal of the "Pedestal Effect" Critique: The paper first addresses a counter-argument suggesting that the small scaling exponents are a numerical bias caused by ignoring a non-zero "pedestal" (a non-zero limit of the loss function as data size $N \to \infty$ ). By modeling the loss function as $L(x) = L_0 + (L_1 - L_0)x^\alpha$ (where $x=1/N$ ), the authors derive a "running scaling exponent." They demonstrate that even with a small pedestal, the transition from "high" scaling regimes (like the Chinchilla scaling of $\alpha \approx 1/3$ ) to the observed "low" regimes ( $\alpha \approx 0.05 \div 0.1$ ) occurs at dataset sizes well below those used in modern LLMs. Thus, the pedestal effect explains the observed low exponents but does not resolve the underlying unsustainability.
Theoretical Lower Bound (Sharma-Kaplan Model): The authors adopt the theoretical framework proposed by Sharma and Kaplan (SK), which posits that scaling exponents are bounded by the intrinsic dimension ( $d$ ) of the data manifold: $\alpha \sim 4/d$ . This relation suggests that LLMs function similarly to adaptive grid refinement methods in a $d$ -dimensional feature space, where $d$ is significantly smaller than the embedding dimension $D$ .
Analogy with Fluid Turbulence: To explain the magnitude of these exponents, the paper draws an analogy with the phenomenology of fluid turbulence.
- Smoothness vs. Roughness: The authors connect the scaling exponent to the smoothness (Hurst exponent $h$ ) of the target signal. For smooth signals, $h=1$ , but for non-smooth processes like turbulence, $h < 1$ .
- Fractal Dimension: Using Kolmogorov's 1941 (K41) theory and multifractal models, they show that for turbulent flows, the effective scaling exponent is reduced further due to the roughness of the signal and the fractal nature of energy dissipation.
- Implication for LLMs: The paper argues that many complex systems (and likely the data LLMs process) are non-smooth. This "roughness" effectively magnifies the intrinsic dimension by a factor of $1/h$ , leading to even smaller scaling exponents ( $\alpha \sim 4h/d$ ).

Key Contributions and Results

Validation of Unsustainability: The authors confirm that the small scaling exponents ( $\alpha < 0.1$ ) are not merely artifacts of early stopping or numerical bias (the pedestal effect) but reflect a fundamental limitation in the efficiency of scaling current LLM architectures.
Quantification of the Pedestal Effect: Through analytical derivation and graphical representation (Figures 1 and 2), the paper shows that the "pedestal" (non-zero loss limit) dominates the scaling behavior for realistic dataset sizes, effectively masking any potential for higher scaling exponents (like the Chinchilla $\alpha \approx 1/3$ ) in large-scale applications.
Dimensional Compression Limits: While acknowledging the impressive ability of LLMs to compress high-dimensional embedding spaces ( $D \sim$ thousands) down to intrinsic manifolds ( $d \sim 100$ ), the authors calculate that for any dataset with an intrinsic dimension $d > 40$ , the scaling exponent will inevitably fall below 0.1.
Impact of Data Roughness: The paper establishes that non-smooth data (characterized by $h < 1$ ) exacerbates the problem, leading to even lower scaling exponents and greater resource inefficiency compared to smooth data.

Significance and Conclusion
The paper concludes that the "bigger is better" strategy, driven by the "no-wall" excitement, is a "sure recipe for energy burnout." The authors assert that the current scaling regime is mathematically unsustainable for datasets with intrinsic dimensions exceeding 40, a condition likely met by most complex real-world applications.

The significance of this work lies in its shift of focus from merely increasing model size to seeking "physical insight and compliance with world models." The authors suggest that the path forward requires a move away from purely muscular leverage of parameters and data volume toward foundational, physics-aware AI models (so-called "world models") that can better handle the intrinsic dimensionality and roughness of complex data without relying on unsustainable scaling laws.