💻 computer science

TCBench: A Benchmark for Tropical Cyclone Track and Intensity Forecasting at the Global Scale

TCBench is a global-scale benchmark that enables fair, model-agnostic evaluation of tropical cyclone track and intensity forecasts by leveraging the IBTrACS dataset to compare state-of-the-art physics-based and AI weather prediction models, ultimately aiming to democratize data-driven forecasting and improve extreme event prediction.

Original authors: Milton Gomez, Marie McGraw, Saranya Ganesh S., Frederick Iat-Hin Tam, Ilia Azizi, Samuel Darmon, Monika Feldmann, Stella Bourdin, Louis Poulain--Auzéau, Suzana J. Camargo, Jonathan Lin, Dan Chavas, Ch

Published 2026-06-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Milton Gomez, Marie McGraw, Saranya Ganesh S., Frederick Iat-Hin Tam, Ilia Azizi, Samuel Darmon, Monika Feldmann, Stella Bourdin, Louis Poulain--Auzéau, Suzana J. Camargo, Jonathan Lin, Dan Chavas, Chia-Ying Lee, Ritwik Gupta, Andrea Jenney, Tom Beucler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine trying to predict the path and strength of a hurricane. For decades, scientists have used massive, complex computer simulations based on the laws of physics (like fluid dynamics and thermodynamics) to do this. These are like super-accurate, but very heavy and expensive, "physics engines" that require giant supercomputers to run.

Recently, a new type of "AI engine" has emerged. These are neural networks that learn from past weather data to predict the future, much like how a chess player learns from thousands of past games rather than calculating every possible move from first principles. These AI models are incredibly fast and cheap to run.

The Problem:
While these AI models are great at predicting general weather, they have struggled to accurately predict the specific path and intensity of tropical cyclones (hurricanes/typhoons). Why?

Different Rules: Everyone was playing by different rules. Some researchers used one method to find the storm in the data, others used a different method. It was like comparing apples to oranges.
The "Intensity" Gap: AI models were good at saying where the storm would go, but bad at saying how strong it would get. They often underestimated the wind speeds or missed the "rapid intensification" (when a storm suddenly gets much stronger very quickly).
Data Mismatch: The AI models were often trained on "smoothed out" data that didn't capture the violent, chaotic details of a real hurricane.

The Solution: TCBench
The authors of this paper created TCBench, which acts like a universal referee and a standardized testing ground.

The "Rulebook": They took all the messy, different data sources (observations, physics models, and AI models) and forced them into a single, standardized format. They used a "ground truth" dataset called IBTrACS (the official record of where storms actually went) as the scorecard.
The "Race": They set up a fair race using data from 2023. They took the best physics-based models and the best new AI models and asked them the same question: "Given a storm exists right now, where will it be in 1, 2, 3, 4, or 5 days, and how strong will it be?"
The "Spotter": Since AI models sometimes "lose track" of a storm (they might predict the storm disappears or never existed), the benchmark has a safety net. If an AI model fails to predict a storm, it defaults to a simple "persistence" guess (assuming the storm just keeps doing what it's doing right now) so the comparison remains fair.

What They Found (The Results):

The Path (Track): The AI models are excellent at predicting where the storm will go. In fact, for predicting the path, the AI models are just as good as, and sometimes even better than, the traditional physics supercomputers. They are like a GPS that knows the road better than anyone else.
The Strength (Intensity): The AI models were initially bad at predicting how strong the storm would get. They often guessed the winds would be weaker than they actually were.
- The Fix: However, the authors showed that if you take the AI's raw guess and run it through a simple "post-processing" step (like a coach giving the player a quick pep talk and a few tactical adjustments), the AI's intensity predictions become very accurate.
The "Sudden Explosion" (Rapid Intensification): Predicting when a storm will suddenly get much stronger is the hardest challenge. The raw AI models mostly missed this. Only the AI models that were specifically tweaked or "post-processed" could spot these dangerous spikes in strength.

The Big Picture:
TCBench isn't just a list of scores; it's a toolkit. It provides the data, the rules, and the measuring sticks so that scientists and AI developers can stop arguing about who is right and start working together to make better predictions.

The paper concludes that while AI models are already winning the race for predicting where storms go, they need a little help (like post-processing or specific training) to get good at predicting how strong they will get. By lowering the barrier to entry and standardizing the tests, TCBench hopes to speed up the development of tools that can save lives by giving us better warnings about these destructive storms.

Technical Summary: TCBench – A Benchmark for Tropical Cyclone Track and Intensity Forecasting at the Global Scale

1. Problem Statement

Tropical cyclones (TCs) are globally devastating weather systems, causing trillions in damages and thousands of deaths. While improving TC forecasting is urgent for risk mitigation, current operational methods face significant limitations. Traditional physics-based models require supercomputing infrastructure to resolve TCs at sufficient spatial resolution, limiting accessibility. Conversely, emerging Artificial Intelligence Weather Prediction (AIWP) models (neural weather models) offer computational efficiency and the potential for large calibrated ensembles but often lack standardized evaluation frameworks specific to TCs.

A critical gap exists in the evaluation of AIWP models regarding TCs. Existing benchmarks often rely on divergent tracking algorithms, lack consistency in intensity definitions, or focus on global metrics that obscure performance on specific extreme events. Furthermore, while AI models have shown skill in TC track prediction, intensity forecasting—particularly Rapid Intensification (RI)—remains a major challenge due to the complex interplay of oceanic energy, vertical wind shear, and inner-core dynamics, which are often underrepresented in coarse-resolution global models or biased training data.

2. Methodology

The paper introduces TCBench, a benchmark designed to evaluate global, short-to-medium-range (1–5 days) forecasts of TC track and intensity. The methodology is built on the following pillars:

Data Integration and Standardization

Ground Truth: The benchmark uses the International Best Track Archive for Climate Stewardship (IBTrACS) as the observational "ground truth," specifically utilizing U.S. agency definitions for intensity (1-minute maximum sustained wind speed and minimum sea-level pressure).
Model Inputs: TCBench integrates heterogeneous data sources, including:
- Physics-based models: Hindcasts from the International Grand Global Ensemble (TIGGE), specifically the GEFS and IFS ensembles.
- AIWP models: Hindcasts from NVIDIA's FourCastNet v2, Huawei's Pangu-Weather, Google DeepMind's GenCast, FNV3, and ECMWF's AIFS v1.0.
- Reanalysis: ERA5 data is referenced for environmental fields but not redistributed.
Tracking Consistency: To ensure fair, model-agnostic comparison, TC tracks are consistently derived from model outputs using the TempestExtremes library. If a model does not explicitly forecast tracks, they are extracted from global fields. A "fallback-to-persistence" strategy is employed: if a model fails to predict a storm, the persistence baseline is used to ensure the evaluation set is not constrained by the most restrictive model.

Forecast Formulation

TCBench frames TC forecasting as predicting the time evolution of an existing tropical system conditioned on its initial position and intensity. The dataset is split by calendar year to prevent leakage:

Training: 2017–2020
Validation: 2021–2022
Testing: 2023 (held out from all baseline training and tuning).

Evaluation Metrics

The benchmark provides both deterministic and probabilistic metrics:

Deterministic Track: Direct Positional Error (DPE), Cross-Track Error (CTE), and Along-Track Error (ATE).
Deterministic Intensity: Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) for wind speed ( $V_{max}$ ) and pressure ( $p_{min}$ ).
Probabilistic: The fair variant of the Continuous Ranked Probability Score (CRPS) for both tracks and intensity.
Classification: Rapid Intensification (RI) is treated as a binary classification task (increase in $V_{max}$ of $\ge$ 30 kt in 24 hours), evaluated using the Critical Success Index (CSI).

3. Key Contributions

Unified Framework: TCBench bridges state-of-the-art neural and physics-based models with observational records, providing a standardized, reproducible evaluation framework.
Data Accessibility: It provides pre-processing pipelines, evaluation protocols, visualization tools, and baseline scores (including persistence and Mean Tendency by Lead & Basin) to lower barriers for AI practitioners and meteorologists.
Model-Agnostic Tracking: By using a consistent tracking algorithm (TempestExtremes) and a fallback mechanism, it allows for the comparison of a large number of models with different biases without constraining the evaluation set size.
Post-Processing Baselines: The benchmark includes baselines that apply post-processing (e.g., parametric distribution sampling) to raw AI model outputs to predict intensity, demonstrating how data-driven models can be refined.

4. Results

Evaluated on 2023 test cases, the paper reports the following findings:

Track Prediction: AIWP models (e.g., AIFS, GenCast, FourCastNet v2) demonstrate skillful track forecasting, outperforming persistence baselines at all lead times (up to 5 days). In deterministic metrics beyond 24 hours, the AIFS model notably outperforms others. However, physics-based ensembles (GEFS) still outperform AI models in probabilistic track predictions (CRPS).
Intensity Prediction: Raw AI models generally underperform compared to persistence baselines at short lead times and show only marginal improvement over physics-based models at longer lead times. This is attributed to coarse spatial resolutions and biases in training data (e.g., ERA5's negative intensity bias).
Impact of Post-Processing: Applying post-processing algorithms to raw AI predictions significantly improves intensity forecasts. For instance, a post-processed Pangu-Weather model achieved performance on par with or better than the GEFS model.
Rapid Intensification (RI): Most global AI and physics-based models show little to no skill in predicting RI. Only the post-processed Pangu-Weather model and the FNV3 model (trained specifically for extreme events) demonstrated any skill in detecting RI, particularly at 48–96 hour lead times.

5. Significance and Claims

The paper positions TCBench as a tool to democratize data-driven TC forecasting. Its significance lies in:

Standardization: It provides a common point of comparison centered on the prediction of TC behavior after system formation, addressing the lack of consistent evaluation in the literature.
Hybrid Potential: The results highlight the complementary strengths of neural and physics-based systems. While AI models excel in deterministic track forecasting, physics-based ensembles remain superior for probabilistic track and raw intensity forecasts.
Process Awareness: By lowering barriers to reproducible evaluation, TCBench equips tropical meteorologists with data-driven tools to improve prediction and process understanding.
Future Direction: The authors suggest that task-specific training and post-processing are necessary steps for AI models to achieve skillful intensity forecasts, pointing toward hybrid approaches as a promising future direction.

The authors maintain a modest tone, acknowledging limitations such as data uncertainty, the limited selection of sources for tractability, and inconsistencies in TC tracks across agencies. They emphasize that TCBench is designed for rapid iteration and extensibility to new models and predictive targets.