Do AI weather models forecast extremes better than GFS? A station-based verification over China with a common initialization source, 2022-2025
This study verifies that AI weather models (GraphCast, Pangu-Weather, and FourCastNet) generally outperform the operational GFS in forecasting temperature extremes across China when initialized from a common GFS source, though their advantage for heavy precipitation diminishes out-of-sample and is sensitive to initialization data and verification settings.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Weather forecasting has long been the domain of physics-based supercomputers, machines that solve complex equations describing how air moves, heats up, and cools down across the globe. For decades, these traditional systems have been the gold standard for predicting everything from gentle breezes to violent storms. However, a new generation of tools has emerged, powered by artificial intelligence. Instead of solving equations step-by-step, these AI models learn to predict the future by studying vast archives of past weather data, recognizing patterns in the way the atmosphere has behaved over time. The question now facing scientists is not just whether these AI models can predict average conditions, but whether they can handle the most dangerous moments: the extreme heatwaves, the sudden cold snaps, and the heavy downpours that cause the most damage. If an AI model misses a record-breaking storm or underestimates a heatwave, the consequences are far more severe than a small error in a daily temperature forecast.
A recent study set out to answer this question with a rigorous, real-world test over China. The researchers compared three leading AI weather models against the operational Global Forecast System, the standard physics-based model used by the US National Weather Service. To ensure a fair fight, they did not let the AI models start with their own best guesses of the current weather. Instead, they forced all the models, both the AI and the traditional one, to begin with the exact same snapshot of the atmosphere. This snapshot came from the Global Forecast System itself. By holding the starting point constant, the researchers could isolate the true predictive power of the AI algorithms, stripping away any advantage that might come from having a better initial map of the sky. They tested these models over 334 weather stations across China, a region that spans tropical jungles, arid deserts, and high mountain plateaus, covering a period from early 2022 through mid-2025.
The results revealed a nuanced picture where AI often outperforms the traditional model, but not in the way one might expect. For temperature, the AI models showed a clear and consistent advantage. When predicting the daily high and low temperatures, the AI models were significantly more accurate than the traditional system, often improving the forecast skill by double-digit percentages. This was true for both warm days and cold nights. The AI models were particularly good at capturing the tails of the distribution, meaning they were better at predicting the hottest days and the coldest nights than the traditional model was. However, this superiority was not absolute across every single hour. While the AI models excelled at daily forecasts, they sometimes struggled at six-hourly intervals, showing that their strength lies in the broader daily rhythm rather than every single moment.
The story changed when the researchers looked at wind and rain. For wind speed, the AI models offered only marginal improvements, and in some cases, they performed slightly worse than the traditional model, especially at shorter time scales. The most significant difference appeared in the prediction of heavy rainfall. While the AI model that provided precipitation data showed some skill in predicting rain, its advantage over the traditional system was much smaller and less reliable. When the researchers tested the AI's ability to predict extreme rain events using data from a time period it had never seen before, the advantage largely disappeared. The model could predict the general pattern of rain, but it did not consistently outperform the traditional physics-based system when it came to the heaviest downpours.
The study also highlighted that the performance of these AI models depends heavily on where you are. In the temperate regions of eastern China, where the climate is moderate and the weather patterns are well-represented in the training data, the AI models shone. But in the high-altitude mountainous regions and the arid deserts, the results were mixed. In the highlands, the AI models sometimes performed worse than the traditional system, particularly when predicting daily high temperatures. This suggests that the AI models, which learned from global data, may struggle to capture the complex, local effects of terrain and rapid temperature shifts that occur in extreme environments. The traditional physics-based model, which calculates the actual movement of air over mountains, still holds an edge in these difficult, rugged landscapes.
Perhaps the most critical finding of the study concerns the source of the initial data. The researchers discovered that the skill of the AI models is deeply sensitive to the quality of the starting snapshot. When the AI models were given a different, higher-quality starting point from a European forecasting system, their performance improved dramatically, often doubling their skill in the short term. This indicates that the AI models are not magic boxes that work perfectly regardless of the input; they are highly dependent on the quality of the data they are fed. The study concludes that while AI weather models have unlocked new potential for predicting temperature extremes, they are not yet a universal replacement for traditional physics-based models. Their ability to forecast the most dangerous weather events, like heavy rain, is still catching up, and their success varies significantly depending on the local climate and the quality of the initial data they receive. The future of forecasting likely lies not in choosing one over the other, but in understanding exactly where each tool excels and how they can be combined to provide the most reliable warnings for the public.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.