💻 computer science

Performance Evaluation of GraphCast for Medium-Range Weather Forecasting over Brazil

This study evaluates GraphCast against ECMWF IFS HRES across Brazil's diverse climates, revealing that while the AI model struggles with fast-propagating baroclinic systems during austral winter, it outperforms traditional numerical weather prediction in the extended range and during the summer wet season by effectively capturing large-scale patterns while dampening high-frequency convective noise.

Original authors: Wolfgang R. Rowell Jr., Lucas S. Kupssinskü

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Wolfgang R. Rowell Jr., Lucas S. Kupssinskü

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Two Weather Forecasters in a Race

Imagine two weather forecasters trying to predict the weather in Brazil for the next 10 days.

The Veteran (IFS HRES): This is the "old school" expert. It uses complex physics equations (like the laws of motion and thermodynamics) and runs on massive supercomputers. It's been the gold standard for decades.
The New AI (GraphCast): This is a "machine learning" model. Instead of solving physics equations, it studied decades of past weather data to learn patterns, much like a student who memorized thousands of history books to guess what happens next. It runs much faster and cheaper than the veteran.

The Question: The paper asks: Does the new AI student perform as well as the veteran expert when predicting the weather specifically over Brazil, a country with incredibly diverse climates?

The Setting: Brazil is a Challenging Classroom

Brazil is huge and has four very different "classrooms" (climatic regions):

The North (Amazon): Hot and wet, like a steam room.
The Northeast: Coastal and influenced by ocean winds.
The Center-West: A tropical savanna with distinct wet and dry seasons.
The South: A temperate region that gets cold, with real winters, frost, and fast-moving storm systems.

The researchers tested the AI in all four regions across four different times of the year (Summer, Autumn, Winter, Spring).

The Results: A Tale of Two Seasons

The study found that the AI is generally excellent, but it has a specific "blind spot" that depends on the season and the region.

1. The Summer Success (The "Smooth Operator")

During the Australian Summer (which is our winter, but summer in Brazil), the AI model was a superstar.

The Analogy: Imagine trying to predict the flow of a slow-moving, lazy river. The AI is great at this. It captures the big picture of moisture and temperature perfectly.
Why it works: In the tropics and during summer, weather changes slowly. The AI's ability to "smooth out" tiny, chaotic details actually helps it predict the big, steady patterns better than the physics model. It gets the "flying rivers" of moisture right, which is crucial for rain and energy planning.

2. The Winter Struggle (The "Fast-Moving Train")

During the Australian Winter (our summer, but winter in Brazil), specifically in the South of the country, the AI stumbled.

The Analogy: Imagine trying to predict a high-speed train racing through a mountain pass. The AI model, which is great at slow rivers, gets confused by the fast, violent twists and turns of the train.
What happened: In the South during winter, fast-moving storm systems (called baroclinic systems) zoom across the sky. The AI model failed to track these fast movements accurately between days 2 and 7 of the forecast. Its predictions were significantly worse than the veteran physics model during this specific window.
The Recovery: Interestingly, after day 7, the AI "woke up" and started performing well again, even outperforming the veteran model in the extended forecast.

Why Did the AI Fail in Winter?

The paper suggests a structural reason for this failure, using a simple metaphor: The Frame Rate.

The Video Game Analogy: Imagine watching a video game. If the game runs at 60 frames per second, you see smooth, fast motion. If it runs at 10 frames per second, fast-moving objects look like they are jumping or glitching.
The Reality: The AI model was trained on data that updates every 6 hours.
- In the tropics/summer, weather moves slowly. A 6-hour update is like watching a slow movie; you catch every detail.
- In the South/winter, weather moves very fast. A 6-hour update is like watching that fast train at 10 frames per second. The AI misses the rapid changes happening between the updates. It tries to guess the path, but because it missed the fast "jumps," its prediction gets worse and worse as the days go on.

The Bottom Line

The AI is not broken; it's just specialized. It is incredibly good at predicting the slow, steady, tropical weather that dominates most of Brazil and the summer months.
It has a specific weakness. It struggles with the fast, chaotic, cold-front storms that hit the South during winter, specifically in the middle of the forecast (days 2–7).
The Takeaway: You can trust this AI for most weather in Brazil, especially in the tropics and during summer. However, for predicting fast-moving winter storms in the South, the traditional physics-based supercomputer is still the more reliable choice for the middle of the week.

The paper concludes that we now know exactly where this AI works and where it doesn't. This helps scientists understand how to "tropicalize" or adjust these AI models so they can handle the fast-moving storms of the South in the future.

Technical Summary: Performance Evaluation of GraphCast for Medium-Range Weather Forecasting over Brazil

Problem Statement
While Machine Learning Weather Prediction (MLWP) models, such as Google DeepMind's GraphCast, have demonstrated remarkable skill on global benchmarks, their operational efficacy in the Global South remains under-verified. Specifically, there is a lack of regional benchmarks for complex, highly convective environments like Brazil, which spans diverse climatic regimes from the equatorial Amazon to the temperate South. Traditional Numerical Weather Prediction (NWP) models, such as the ECMWF IFS HRES, face limitations in computational cost and the parameterization of subgrid processes, particularly in tropical convection. The critical question addressed is whether the skill of data-driven MLWP models generalizes to specific, climatically complex regions or if a "generalization gap" exists that could impact weather-sensitive sectors in the Global South.

Methodology
The study presents a rigorous, regime-stratified benchmark of the operational GraphCast model against the deterministic ECMWF IFS HRES baseline.

Data and Ground Truth: Both models were evaluated against the operational ECMWF IFS analysis (the $t=0$ state) as ground truth. This approach avoids circular bias by not evaluating GraphCast against its own training distribution (ERA5). The study utilized 27 datasets comprising 145 GB of data sourced from ECMWF Open Data and Google's WeatherNext.
Spatial and Temporal Domains: The evaluation covered four distinct Brazilian climatic sub-regions (North, Northeast, Center-West/Southeast, and South) defined by 10°×10° bounding boxes to prevent error dilution across diverse regimes. The temporal scope included four seasonal windows: March 2024 (Summer-Autumn transition), July 2024 (Winter peak), October 2024 (Spring transition), and January 2025 (Summer peak).
Variables: Three tropospheric variables were assessed: Geopotential at 500 hPa ( $Z_{500}$ ), Temperature at 850 hPa ( $T_{850}$ ), and Specific Humidity at 850 hPa ( $Q_{850}$ ).
Pipeline and Metrics: A scalable, cloud-native pipeline was developed using the WeatherBench-X framework. Metrics included Root Mean Square Error (RMSE), Anomaly Correlation Coefficient (ACC), and a Normalized RMSE Difference (Skill Score) to quantify relative performance.

Key Contributions

First Regional Benchmark: This work provides the first regime-stratified, sub-regional benchmark of an operational MLWP model over Brazil.
Reproducible Pipeline: It introduces a cloud-native evaluation pipeline built on WeatherBench-X that harmonizes ECMWF Open Data and Google DeepMind WeatherNext outputs into a common analytical substrate.
Diagnostic Mapping: The study establishes a diagnostic mapping between Brazilian climatic regimes and the specific physical boundaries where the inductive ML paradigm gains or loses skill relative to the deterministic baseline.

Results
The evaluation reveals a regime-dependent skill profile with distinct seasonal and regional variations:

General Performance: In March, October, and January, GraphCast consistently outperformed the IFS HRES baseline across all variables, regions, and lead times (Skill Scores > 100%).
The Winter Deficit (July): A significant degradation was observed during the austral winter (July 2024) specifically in the South subregion for the $Z_{500}$ variable.
- Medium-Range Failure: For lead days 2–7 (48h–168h), GraphCast underperformed significantly. At 120 hours (5 days), the Skill Score dropped to 47.91%, representing a 52.09% deficit relative to the IFS HRES.
- Variable Specificity: This degradation was isolated to the dynamic circulation field ( $Z_{500}$ ). Thermodynamic variables ( $T_{850}$ and $Q_{850}$ ) maintained Skill Scores near or above parity even during July, indicating the deficit is not uniform across all forecast fields.
- Recovery: Performance recovered in the extended range (beyond day 7), where GraphCast's inherent smoothing of chaotic small-scale variability became beneficial under deterministic metrics.
Regional Asymmetry: While all regions showed some short-range challenges in July, only the South exhibited a deepening deficit rather than a recovery. The authors attribute this to the structural limitations of GraphCast's 6-hour timestep in resolving fast-propagating baroclinic systems characteristic of the extratropical winter, which rely on sub-synoptic divergent motions.

Significance and Claims
The paper claims to establish a baseline for Brazil and define the specific physical boundaries guiding future "tropicalization" efforts. The findings suggest that while foundational AI models offer superior efficiency and skill in many contexts, their current architecture faces structural limitations in resolving fast-propagating baroclinic systems in the Southern Hemisphere winter. The study emphasizes that optimizing these models for regional resilience requires understanding these specific physical constraints rather than assuming universal generalization. The work aims to support weather-sensitive sectors in the Global South by clarifying where and when MLWP models can be operationally trusted versus where traditional NWP remains superior.