📊 statistics

Forecast Skill Is Not Decision Skill: Evidence from Weather-Dependent Decision Tasks

This paper introduces decision calibration as a framework to demonstrate that standard statistical forecast evaluations often fail to predict a model's actual utility in real-world decision-making, as model rankings can shift significantly when assessed based on their ability to improve specific weather-dependent decisions.

Original authors: Kornelius Raeth, Nicole Ludwig

Published 2026-09-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kornelius Raeth, Nicole Ludwig

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Weather forecasts are more than just a morning ritual of checking if an umbrella is needed; they are the invisible backbone of modern society, guiding everything from the flight paths of airplanes to the harvest schedules of farmers and the stability of our power grids. For decades, scientists have judged the quality of these predictions using statistical tools that measure how closely a forecast matches what actually happened in the atmosphere. If a model predicts a 90 percent chance of rain and it rains 90 percent of the time, the model is considered well-calibrated. This approach assumes that a statistically perfect forecast is automatically the most useful one for the people relying on it. However, the real world is rarely about perfect statistics; it is about specific choices made under uncertainty. A farmer deciding whether to cover crops before a frost, a city planner preparing for a heatwave, or a wind farm operator scheduling energy delivery all face different costs for being wrong. The question driving new research is whether the models that look best on a statistical chart are actually the ones that save the most money and lives when put to the test.

A team of researchers at the University of Tübingen and the University of Augsburg set out to answer this by shifting the focus from the forecast itself to the decision it supports. They compared two very different types of weather prediction systems: a traditional numerical model used by meteorologists for years, which relies on complex physics equations, and a newer machine learning model that learns patterns from historical data. Instead of simply checking which model predicted temperatures or wind speeds more accurately, the researchers built three specific scenarios where a forecast leads to a concrete action. They simulated a farmer deciding whether to protect crops from freezing, a public health official deciding whether to issue heat warnings, and a wind energy provider deciding how much power to promise to the grid. In each case, they assigned a cost to every possible outcome: the cost of taking unnecessary protective action versus the cost of failing to act when disaster strikes.

The results revealed a surprising disconnect between statistical accuracy and practical value. In the task of protecting crops from frost, the two models performed almost identically when judged by standard statistical measures. Yet, when the researchers applied the specific costs of farming, the machine learning model proved significantly better at guiding decisions for some types of frost risks, while the traditional model was superior for others. The difference depended entirely on the specific conditions, such as how sensitive the crops were to cold and how expensive the protection measures were. Similarly, for heat protection, the machine learning model showed a distinct advantage in predicting extreme heat events that trigger the most severe health risks, a nuance that standard statistical charts had completely missed. The researchers found that a model could be statistically "worse" overall but still make better decisions for a specific, high-stakes task because its errors happened in parts of the weather distribution that mattered less to that particular decision.

Perhaps the most telling example came from wind power dispatch, where energy providers must predict how much electricity they can generate. Here, the two models were so statistically similar that standard metrics could not distinguish between them. However, when the researchers introduced the financial penalties for under-delivering energy, the machine learning model again showed a slight edge in minimizing costs, particularly when the financial stakes were high. This suggests that the traditional way of evaluating weather models often hides the very differences that matter most to the people using them. The study concludes that there is no single "best" weather model for every situation. Instead, the right choice depends on the specific decision at hand. To truly know which forecast is most valuable, we must stop looking only at the numbers and start measuring how well those numbers help us make the right choices in a world where every decision carries a price.

Technical Summary: Forecast Skill Is Not Decision Skill

Problem Statement
Standard weather forecast evaluations predominantly focus on the forecaster's perspective, utilizing statistical metrics (e.g., CRPS, PIT histograms, spread-skill ratios) to assess the consistency between forecasts and observations. While these metrics evaluate distributional calibration, they often fail to capture the specific properties of a forecast that are relevant to end-users who utilize predictions to make decisions. The paper argues that a forecast may be well-calibrated in a global statistical sense yet perform poorly in specific downstream decision tasks, or conversely, perform well in a specific task despite suboptimal global metrics. Current verification practices do not reliably translate to performance in decision-making, creating a gap between forecast verification and the actual economic or operational value of a forecast.

Methodology
The authors employ the framework of decision calibration (Zhao et al., 2021) to evaluate probabilistic weather forecasts at the decision level rather than the forecast level.

Decision Framework: The study models decision-making as a process where a decision-maker selects an action $a$ from an action space $\mathcal{A}$ to minimize expected costs $E[c(a, Y)]$ based on a predictive distribution $F_X$ .
Decision Calibration Metric: Instead of comparing forecast distributions directly to observations, the framework evaluates the cost gap ( $C_{gap}$ ), defined as the absolute difference between the expected costs (simulated using the forecast distribution) and the observed costs (incurred using the true observation). A smaller cost gap indicates better decision calibration.
Models Compared: The study compares two state-of-the-art probabilistic weather prediction models:
1. IFS ENS: A classical Numerical Weather Prediction (NWP) ensemble model from ECMWF.
2. Arches: A Machine Learning Weather Prediction (MLWP) generative model based on diffusion models, trained on ERA5 reanalysis data.
Decision Tasks: Three weather-dependent decision tasks were formulated using parameterized cost functions to represent different decision-maker preferences:
1. Frost Protection: A binary decision (apply protection or not) based on 2-meter temperature, modeling agricultural crop loss vs. protection costs.
2. Heat Protection: A binary decision (issue warnings or not) based on 2-meter temperature, modeling public health interventions.
3. Wind Power Dispatch: A multi-action decision (promised power output) based on 10-meter wind speed, modeling penalties for under-delivery and opportunity costs for over-delivery.
Experimental Setup: Evaluations were conducted on 2021 data for Europe with lead times up to 15 days. The study computed standard calibration metrics (CRPS, PIT, SSR) alongside decision calibration metrics (cost gap and observed costs) for varying cost function parameters (thresholds and cost ratios).

Key Contributions

First Decision-Level Evaluation of ML vs. NWP: This is the first study to apply the decision calibration framework to compare state-of-the-art ML and classical NWP models across multiple weather-dependent sectors.
Demonstration of Non-Transferability: The paper provides empirical evidence that performance rankings at the forecast level (e.g., CRPS) do not reliably translate to performance rankings at the decision level.
Task-Dependent Model Rankings: The study demonstrates that model rankings can change even among seemingly similar decision tasks (e.g., varying the temperature threshold or cost ratio in frost protection).
Framework Generalizability: While focused on weather, the authors posit that the framework is applicable to any domain where probabilistic models inform downstream decisions.

Results

Frost Protection: Standard metrics (CRPS, PIT) indicated similar performance between Arches and IFS ENS, with Arches slightly worse at longer lead times. However, decision calibration results varied significantly based on task parameters. Arches showed up to a 19% improvement in cost gap (better decision calibration) for low threshold temperatures but up to a 15% degradation for high threshold temperatures compared to IFS. These differences were not detectable via global calibration metrics.
Heat Protection: Similar to frost protection, standard metrics showed comparable performance. Decision calibration analysis revealed that Arches often surpassed IFS in high-temperature scenarios (tail events), with up to an 18% smaller cost gap at 10-day lead times. Model rankings also varied geographically (e.g., Arches performed better in the Mediterranean than in Central Europe).
Wind Power Dispatch: Standard metrics showed very similar performance, with minor disagreements at 1-day lead times. Decision calibration differences were small, as the cost function required decisions across the entire output domain, effectively probing global calibration. However, at 1-day lead times with high under-delivery penalties, Arches outperformed IFS in decision calibration, likely due to IFS's slight overestimation of wind speeds at that specific lead time.
Observed Costs: Improvements in decision calibration generally correlated with lower observed costs, confirming that better decision calibration leads to better economic outcomes.

Significance and Claims
The paper claims that typical forecast evaluations are insufficient for selecting the optimal forecast model for a specific decision task. The authors assert that forecast skill is not decision skill.

Limitations of Current Metrics: Standard metrics may omit forecast properties relevant to specific decision tasks, particularly those involving tail events or asymmetric cost structures.
Necessity of Decision-Level Evaluation: To identify the best model for a specific application, evaluation must occur at the decision level using cost functions that reflect the specific decision-maker's constraints and preferences.
Future Directions: The authors suggest that while decision calibration requires well-defined cost functions (which can be derived via applied economics or cognitive science methods), future work could optimize ML weather prediction models directly for specific downstream tasks by using cost functions as loss functions during training or fine-tuning. This would align model optimization directly with the relevant decision task, moving beyond the current practice of optimizing for generic proper scoring rules like CRPS.

More like this