Crowdsourcing the Frontier: Advancing Hybrid Physics-ML Climate Simulation via a $50,000 Kaggle Competition

This paper demonstrates that a $50,000 Kaggle competition successfully crowdsourced diverse machine learning architectures for subgrid parameterization, which, when coupled with a full-physics climate model, achieved reproducible online stability and state-of-the-art performance, marking a significant milestone in advancing hybrid physics-ML climate simulations.

The Gist

Imagine trying to predict the weather for the next 50 years. To do this accurately, scientists use massive computer models of the Earth. But here's the catch: these models are like low-resolution maps. They can see the continents and oceans, but they are too "blurry" to see individual clouds, tiny storms, or the way heat rises from a specific patch of forest.

To fill in these blurry spots, scientists have to use "cheats"—mathematical guesses called parameterizations. For decades, these cheats have been hand-tuned by experts, but they often lead to errors, like predicting a drought when a flood is coming.

The Big Idea: Letting AI Fill in the Blanks

Recently, scientists realized that Artificial Intelligence (AI) could learn these "cheats" much better than humans can. Instead of guessing, an AI could look at high-resolution simulations (which are incredibly expensive to run) and learn the pattern of how clouds and storms behave. Then, it could act as a super-smart shortcut in the big climate model.

This is called a Hybrid Physics-ML Model: part traditional physics, part AI.

The Problem: The "Online" Nightmare

There was a huge hurdle. Scientists could train these AIs perfectly in a test tube (called "offline"), where they just guess the weather based on a snapshot of data. But when they tried to plug these AIs into a running climate model (called "online"), the models often crashed.

Think of it like this: You can teach a self-driving car to recognize a stop sign perfectly in a video game. But when you put it on a real road with wind, rain, and other cars, it might panic, freeze, or drive off a cliff. In climate modeling, the AI would sometimes get confused by its own predictions, causing the whole simulation to spiral out of control.

The Solution: The $50,000 Climate Challenge

To solve this, the scientists decided to crowdsource the problem. They launched a Kaggle Competition (a famous platform for data science contests) with a $50,000 prize.

They gave the world's best data scientists a massive dataset and said: "Build the best AI to predict these cloud patterns. We don't care how you do it, just make it accurate."

About 700 teams from around the world entered. They built all sorts of crazy, complex AI architectures (like Squeezeformers, ResLSTMs, and Pao Models) to win.

What This Paper Found

This paper is the "report card" after the competition. The researchers took the winning AI designs from the contest and tried to plug them into a real, running climate model to see if they would survive.

Here are the key takeaways, explained simply:

1. The "Crash" Problem is Solved (Mostly)
The biggest fear was that these new AIs would be too unstable to run for years. The paper found that yes, we can now run these hybrid models stably for five years without them crashing. This is a massive milestone. It's like finally getting that self-driving car to drive across the country without taking a detour into a lake.

2. Different Cars, Same Traffic Jams
Even though the 700 teams built very different AI "engines," they all ended up making the same types of mistakes when running the climate model.

• The Analogy: Imagine 100 different chefs trying to bake a cake. They use different ovens, different mixers, and different recipes. But when they all taste the cake, they all agree: "It's a little too dry near the top and too sweet near the bottom."
• The Science: The models consistently underestimated how much water vapor was in the tropics and had similar temperature errors. This suggests the problem isn't just the AI design; it's something deeper about how the data is structured or what information is missing.

3. More Data Helps, But Not for Everyone
The scientists tried giving the AIs more information (like adding "memory" of past weather or knowing the latitude).

• The Result: For some AI designs, adding more data made them run smoother. For others, it made them crash immediately. It's like giving a new driver more mirrors and sensors; for some, it helps them drive better; for others, it overwhelms them.

4. The "Best" AI Depends on What You Measure
No single AI won every prize.

• One AI was amazing at predicting temperature.
• Another was the best at predicting rain.
• A third was the fastest at running the simulation.
  It turns out, there is no "one-size-fits-all" AI yet.

Why This Matters

This paper proves that crowdsourcing works. By opening the problem to the global data science community, they found new ways to build climate models that are stable and accurate.

However, it also reveals that we haven't solved everything yet. The fact that all the different AIs make the same specific mistakes tells scientists that they need to look deeper. They need to figure out why the models are missing the water vapor in the tropics, perhaps by giving the AI better "eyes" to see the tiny details of cloud structures.

In a nutshell: We finally built a self-driving car that doesn't crash on the highway, but we still need to figure out why it keeps getting lost in the same specific neighborhood. The next step is to fix that neighborhood so the car can drive perfectly.

Technical Summary

Here is a detailed technical summary of the manuscript "Crowdsourcing the Frontier: Advancing Hybrid Physics-ML Climate Simulation via a $50,000 Kaggle Competition."

1. Problem Statement

Climate models currently rely on coarse resolutions (>25 km) and hand-tuned subgrid parameterizations to represent processes like convection and turbulence, leading to systematic biases and large uncertainties in warming projections. While high-resolution simulations (e.g., Multiscale Modeling Frameworks or MMF) offer better fidelity, they are computationally prohibitive for long-term climate projections.

Machine Learning (ML) offers a potential solution by emulating these expensive subgrid processes (Cloud Resolving Models or CRMs) within a Global Climate Model (GCM). However, a critical bottleneck remains: online instability. While ML models often perform well in "offline" testing (predicting tendencies from static data), they frequently fail when "online" (dynamically coupled to the GCM), leading to numerical crashes, drift, or unphysical biases. Previous attempts to solve this have been limited to small teams of domain scientists. This paper investigates whether crowdsourcing the problem to the broader data science community via a Kaggle competition can accelerate the discovery of stable, high-performance hybrid physics-ML architectures.

2. Methodology

The study serves as a systematic follow-up to the 2024 ClimSim Kaggle competition, which attracted ~700 teams. The authors evaluated the offline and online performance of architectures inspired by the top 5 winning teams, comparing them against a strong U-Net baseline (Hu et al., 2025).

• Datasets: The study uses the ClimSim dataset, generated by the Energy Exascale Earth System Model (E3SM) with an MMF setup. The focus is on the low-resolution real-geography version (~11.5° horizontal resolution, 60 vertical levels).
• Architectures Tested:
  1. U-Net: The baseline from Hu et al. (2025).
  2. Squeezeformer: (1st Place) Combines convolutional and transformer components.
  3. Pure ResLSTM: (2nd Place) Multi-layer bidirectional LSTM.
  4. Pao Model: (3rd Place) Custom architecture separating scalar and vertical profile processing.
  5. ConvNeXt: (4th Place) Modern 1D convolutional network.
  6. Encoder-Decoder LSTM: (5th Place) Latent representation learning followed by recurrent processing.
• Configurations: Each architecture was tested across five design configurations to isolate the impact of specific engineering choices:
  • Standard: Baseline inputs.
  • Confidence Loss: Adds a head to predict loss magnitude.
  • Difference Loss: Penalizes errors in vertical gradients.
  • Multirepresentation: Uses multiple normalization strategies (level-wise, column-wise, log-symmetric) simultaneously.
  • Expanded Variable List: Includes convective memory (tendencies at $t-1, t-2$), large-scale forcings, and latitudinal information.
• Experimental Setup:
  • Training: 90 models total (6 architectures × 5 configurations × 3 random seeds).
  • Online Coupling: All models were coupled to the E3SM-MMF using an FTorch binding for GPU acceleration.
  • Constraints: A microphysics constraint (diagnosing cloud mixing ratios from temperature) was applied to all models to ensure physical consistency, a key factor in previous stability successes.

3. Key Contributions

• Reproducible Online Stability: The study demonstrates that stable, long-term (multi-year) online simulations are now reproducible across diverse ML architectures in a low-resolution real-geography setting, provided specific constraints (like microphysics constraints) are used.
• Crowdsourcing Validation: It validates that "offline" improvements driven by the broader ML community (Kaggle) can translate to "online" gains, though not universally.
• Systematic Failure Mode Analysis: The paper identifies that while architectures differ in performance, they share universal offline and online biases (e.g., underestimation of tropical precipitable water, specific zonal mean temperature biases) that are independent of the model architecture.
• Efficiency Benchmarking: It provides a comparative analysis of computational efficiency (Simulation Years Per Day vs. RMSE), revealing that complex architectures do not necessarily yield better accuracy or speed.

4. Key Results

A. Offline Performance

• R2 Scores: When controlling for preprocessing and feature selection, the performance gap between the winning Kaggle architectures and the U-Net baseline nearly vanishes. The Squeezeformer showed slight superiority, while the Pao Model underperformed.
• Variable Expansion: Expanding the input variable list (adding convective memory and forcings) universally improved offline skill across all architectures.
• Normalization: The "Multirepresentation" configuration significantly improved offline skill for cloud mixing ratios (highly skewed distributions) but did not always translate to online stability.

B. Online Stability and Drift

• Stability is Architecture-Dependent: Stability was not guaranteed.
  • ConvNeXt was unstable in the "Standard" configuration but became the most stable and accurate in the "Expanded Variable List" configuration.
  • Squeezeformer and Pao Model suffered catastrophic crashes in the "Expanded Variable List" and "Multirepresentation" configurations, respectively.
  • Pure ResLSTM and Encoder-Decoder LSTM showed high sensitivity to random seeds, with some seeds crashing while others remained stable.
• Conclusion: Online stability is not an inherent property of an architecture but a complex interaction between the architecture, the input configuration, and random initialization.

C. Accuracy and Biases

• State-of-the-Art (SOTA) Gains: No single simulation beat the U-Net baseline across all metrics simultaneously. However, specific architectures achieved SOTA results for specific variables:
  • Temperature: 11.1% improvement.
  • Liquid/Ice Cloud: ~20% improvement.
  • Moisture: The U-Net with "Confidence Loss" remained the best; no other architecture surpassed it.
• Universal Biases:
  • Zonal Mean Bias: All architectures exhibited similar bias patterns (e.g., warm bias at high altitudes over poles), suggesting these are systemic issues of the MMF emulation problem rather than specific model failures.
  • Precipitable Water: All models systematically underestimated precipitable water, particularly in the tropics.
  • Precipitation Extremes: The "Expanded Variable List" configuration partially mitigated the underestimation of extreme precipitation, likely due to the inclusion of convective memory.

D. Computational Efficiency

• SYPD (Simulation Years Per Day): The ConvNeXt architecture was the fastest, followed by Encoder-Decoder LSTM and Pao Model. Surprisingly, the U-Net and Pure ResLSTM were slower despite having fewer trainable parameters, indicating that parameter count is not a perfect predictor of runtime efficiency in this context.
• Trade-offs: The Encoder-Decoder LSTM offered the best balance of accuracy and speed for most variables, making it a strong candidate for future baselines.

5. Significance and Future Directions

• Milestone Achievement: The paper marks a transition from "proof-of-concept" to "reproducible operational capability" for hybrid physics-ML climate models. The ability to run stable, multi-year simulations with diverse architectures is a major step forward.
• Limitations of Offline Benchmarks: The study highlights that optimizing for a single offline metric (like R2) does not guarantee online stability. The "offline-to-online" gap remains a critical challenge.
• Future Work:
  • Bias Penalties: Future loss functions should explicitly penalize systematic zonal mean biases (similar to spectral bias penalization in fluid dynamics).
  • Stochasticity: Incorporating stochastic elements or subgrid state information (currently excluded due to data volume) may improve the representation of precipitation extremes.
  • Beyond MMF: While MMF provides a clean scale separation for training, future work must address the lack of aerosol-cloud interactions in current GPU-enabled E3SM versions and move toward Global Cloud Resolving Models (GCRMs) where scale separation is less distinct.

In summary, this paper demonstrates that crowdsourcing ML architectures can successfully push the frontier of hybrid climate modeling, achieving stable online simulations and variable-specific SOTA results, while simultaneously revealing deep, systemic biases that require new physical constraints and loss function designs to resolve.