🤖 machine learning

Scaling Laws of Global Weather Models

This paper analyzes empirical scaling laws in global weather models, revealing that performance is maximized by prioritizing wider architectures and larger training datasets over deeper models or increased parameter counts, with compute-optimal strategies favoring data expansion under fixed budgets.

Original authors: Yuejiang Yu, Langwen Huang, Alexandru Calotoiu, Torsten Hoefler

Published 2026-06-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuejiang Yu, Langwen Huang, Alexandru Calotoiu, Torsten Hoefler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to predict the weather. For decades, scientists have used complex physics equations to simulate the atmosphere, like running a giant, virtual wind tunnel. But recently, a new approach has emerged: instead of physics, we use "data-driven" models. These are like super-smart students who learn to predict the weather just by reading millions of pages of past weather reports.

This paper is a study guide for those students. The researchers asked: "If we give these weather-studying computers more brainpower (parameters), more textbooks (data), or more time to study (compute), how much better do they actually get?"

They tested five different "students" (models named Aurora, GraphCast, Pangu, SFNO, and AIFS) to see how they scale up. Here is what they found, explained simply:

1. The "More Data" Rule: Aurora is the Best Reader

In the world of language models (like the ones that write essays), adding more data helps, but there's a limit. The researchers found that for weather models, more data is a superpower, but some models are better at reading it than others.

The Analogy: Imagine two students taking a test. One (GraphCast) is very smart but only reads a few books. The other (Aurora) is a voracious reader.
The Finding: When they increased the amount of training data by 10 times, Aurora improved its predictions by a massive amount (up to 3.2 times better). It is the most efficient at turning raw data into weather knowledge. Other models improved too, but not as dramatically.

2. The "Brain Size" Rule: GraphCast is the Efficient Genius

The researchers also looked at how much "brainpower" (parameters) each model needed to get good.

The Analogy: Think of model size as the size of a library. GraphCast is like a tiny library that somehow knows everything it needs to know. It is incredibly efficient; it doesn't need a massive building to store its knowledge.
The Finding: GraphCast is the most "parameter-efficient." It achieves great results with a relatively small model size. However, there's a catch: while it's smart, it's slow to run on computers because of how it processes information (like a genius who thinks very hard but moves slowly).

3. The "Shape" Surprise: Wide and Shallow is Better

This is the biggest surprise. In the world of language models (like chatbots), making a model "deeper" (adding more layers of thinking) usually helps. But for weather, depth doesn't matter as much as width.

The Analogy: Imagine building a house.
- Deep: Building a skyscraper with 50 floors but only one room per floor.
- Wide: Building a ranch-style house with only 2 floors, but 50 rooms on each floor.
The Finding: For weather prediction, the wide house wins. The researchers found that models with more "width" (more processing power happening at the same time) performed better than models with more "depth" (more steps in a chain). Even models with just one or two layers of "thinking" worked surprisingly well. It turns out that predicting the weather for the next 6 hours is more about having a broad view of the whole picture than taking many small, deep steps.

4. The "Budget" Strategy: Study Longer, Don't Just Get Bigger

If you have a fixed amount of money (or computer power) to spend, how should you spend it? Should you buy a bigger brain (more parameters) or buy more textbooks (more data)?

The Analogy: You have a budget for a student. Do you hire a bigger professor (bigger model) or give the current professor more years to study the library (more data)?
The Finding: The paper says give them more data and let them study longer. Under a fixed budget, it is better to train a slightly smaller model on a massive amount of data than to train a huge model on a small amount of data. The "sweet spot" for weather models is to prioritize data volume over model size.

5. Not All Variables Are Created Equal

The researchers also noticed that the models didn't get better at everything at the same speed.

The Analogy: Imagine a student who is amazing at math but struggles with history.
The Finding: Some weather variables (like wind speed) improved differently than others (like temperature). A model might be the best at predicting wind but only "okay" at predicting temperature. This means you can't just look at one overall score; you have to check how the model handles each specific part of the weather.

6. The "Hardware" Reality Check

Finally, the paper looked at how well these models actually run on real computers.

The Analogy: A Ferrari engine (GraphCast) might be powerful, but if it's stuck in traffic (inefficient hardware usage), it won't go fast. A reliable truck (Aurora) might not be a Ferrari, but it drives smoothly and uses the road efficiently.
The Finding: Aurora was incredibly efficient at using computer hardware (using about 36 times more of the computer's potential power than GraphCast). GraphCast, while smart, was very wasteful with computer resources. This suggests that for real-world use, a model that is "efficient" on the hardware is just as important as a model that is "smart."

Summary

If you want to build the best weather predictor based on this paper:

Go Wide, Not Deep: Build models that are broad and flat, not tall and narrow.
Feed Them Data: Prioritize giving the model more training data rather than making the model itself bigger.
Pick Your Champion: Aurora is great at learning from data and using computer power efficiently. GraphCast is great at being small and efficient with its brain size, but it runs slowly on computers.
Check the Details: Don't just look at the average score; check how the model handles specific things like wind or temperature, as they behave differently.

Technical Summary: Scaling Laws of Global Weather Models

Problem Statement

Data-driven models are rapidly transforming weather forecasting, yet the field lacks a systematic understanding of how model performance scales with key resources. While scaling laws have successfully guided the development of Large Language Models (LLMs) and vision models by establishing power-law relationships between validation loss ( $L$ ) and model size ( $N$ ), dataset size ( $D$ ), and compute budget ( $C$ ), their applicability to global weather models remains unverified.

Weather forecasting presents unique challenges distinct from NLP and vision:

Physical Constraints: The atmosphere is a chaotic system with inherent limits of predictability.
Variable Heterogeneity: Models must predict hundreds of correlated but physically distinct variables (e.g., temperature, wind, pressure) with varying degrees of difficulty.
Architectural Diversity: Current state-of-the-art models utilize diverse backbones (Transformers, Graph Neural Networks, Fourier Neural Operators), and it is unclear if scaling behaviors are consistent across these architectures.

The paper addresses the urgent need to characterize these scaling laws to optimize resource allocation and guide the development of next-generation weather models.

Methodology

The authors conducted the first cross-model scaling analysis in weather prediction, benchmarking five state-of-the-art models: Aurora, GraphCast, AIFS, Pangu, and SFNO.

Experimental Setup

Data: All models were trained and evaluated on the ERA5 reanalysis dataset (1979–2020 for training, 2021 for validation) via WeatherBench 2.
Standardization: To ensure fair comparison, the authors enforced strict uniformity:
- Identical input/output variables (geopotential, temperature, wind components, humidity, etc.).
- Consistent spatial resolution ( $0.25^\circ \times 0.25^\circ$ global grid).
- Unified validation loss function (weighted MSE normalized by standard deviation and grid area).
- Consistent hyperparameter tuning (learning rate sweeps, optimizer settings) to ensure stable training dynamics.
Variables: The study systematically varied:
- Model Size ( $N$ ): By adjusting width (embedding dimension) and depth (number of layers/blocks).
- Dataset Size ( $D$ ): By training on subsets of the total data volume (ranging from ~3 TB to 90 TB).
- Compute Budget ( $C$ ): Analyzed via the relationship $C \approx N \times D$ (with model-specific adjustments for architecture).

Analysis Framework

The authors fitted empirical power laws of the form:

$L(D) = \alpha D^{-\beta}$ (Data scaling)
$L(N) = \gamma N^{-\delta}$ (Parameter scaling)
$L(C) = \lambda C^{-\epsilon}$ (Compute scaling)

They also investigated compute-optimal training (the optimal trade-off between $N$ and $D$ for a fixed $C$ ) and analyzed model shape (width vs. depth) and variable-specific performance.

Key Contributions and Results

1. Data-Scaling Efficiency ( $\beta$ )

The study confirms that validation loss follows power-law relationships with dataset size across all models.

Aurora exhibits the strongest data-scaling behavior with a coefficient $\beta \approx 0.51$ . Increasing the training dataset by $10\times$ reduces validation loss by up to $3.2\times$ .
GraphCast shows the lowest $\beta$ ( $\approx 0.36$ ), indicating it benefits less from massive data increases compared to Aurora.
Implication: Aurora is the most efficient model for scaling with larger datasets, while models like Pangu show a threshold effect where parameter scaling only becomes effective with sufficient data volume.

2. Parameter Efficiency and Hardware Utilization

GraphCast demonstrates the highest parameter efficiency (lowest loss for a given $N$ ) but suffers from severe hardware utilization issues. On H100 GPUs, GraphCast utilizes only 1.03% of peak single-precision performance due to memory-bound message-passing operations in its GNN architecture.
Aurora achieves 37.2% GPU utilization (approx. $36\times$ higher than GraphCast) due to its transformer-based attention mechanisms, which are better optimized for modern hardware.
Conclusion: Theoretical parameter efficiency does not guarantee practical deployment efficiency; engineering optimizations and hardware utilization are critical factors.

3. Model Shape: Width vs. Depth

Contrary to findings in NLP (where performance is largely insensitive to shape at fixed $N$ ), weather models show a strong preference for width over depth.

Across all five models, wider configurations consistently achieve lower validation loss than narrower, deeper counterparts with the same parameter count.
Depth Saturation: Models like GraphCast and SFNO perform well even at a depth of 1, suggesting that 6-hour weather prediction dynamics are well-approximated by linear models or shallow architectures, and deep non-linear stacking offers diminishing returns.
Recommendation: Future models should prioritize increasing width (representational capacity) rather than depth.

4. Compute-Optimal Allocation

Under fixed compute budgets, the optimal strategy for weather models differs from the standard NLP scaling rule ( $N \propto C^{0.73}, D \propto C^{0.27}$ ).

The analysis indicates that allocating resources to increase the training dataset ( $D$ ) yields greater performance gains than increasing model size ( $N$ ).
For a fixed $C$ , training smaller models for longer durations (more steps) is more effective than training larger models with insufficient data.

5. Variable-Specific Heterogeneity

Scaling behavior is not uniform across all atmospheric variables.

Aurora achieves the lowest RMSE for the 10m u-component of wind (10U).
GraphCast outperforms others for 2m temperature (2T), particularly at smaller parameter counts.
Significance: A single weighted validation loss is only a rough indicator of overall performance. Different variables scale differently, implying that model design may need to be tailored or hybridized for specific physical fields.

Significance and Claims

The paper claims to provide the first quantitative framework for atmospheric prediction scaling laws, revealing fundamental differences between weather models and language models.

Architectural Shift: The finding that weather models favor width over depth challenges the prevailing Transformer-centric design patterns derived from NLP.
Resource Strategy: The study argues that for practical weather forecasting, the priority should be larger effective training datasets and wider architectures, rather than simply scaling up model parameters.
Engineering Reality: The work highlights that theoretical scaling laws must be tempered by hardware utilization realities. A model with superior theoretical scaling (like GraphCast) may be less practical than a model with slightly lower scaling efficiency but vastly superior hardware utilization (like Aurora).
Future Direction: The authors suggest that future weather models should prioritize wider architectures and larger datasets to maximize predictive performance, while acknowledging that variable-specific tuning may be necessary to fully capitalize on scaling improvements.

The paper concludes that while scaling laws are predictable across diverse weather models, the specific coefficients and optimal strategies (width vs. depth, data vs. parameters) are domain-specific and distinct from those in NLP.