📊 statistics

Enforcing tail calibration when training probabilistic forecast models

This paper demonstrates that adapting loss functions through weighted scoring rules and tail miscalibration regularization can improve the calibration of probabilistic forecasts for extreme events, such as UK wind speeds, though this enhancement introduces a trade-off with the calibration of more common outcomes.

Original authors: Jakob Benjamin Wessel, Maybritt Schillinger, Frank Kwasniok, Sam Allen

Published 2026-05-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jakob Benjamin Wessel, Maybritt Schillinger, Frank Kwasniok, Sam Allen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a weather forecaster, but instead of just saying "It will rain," you are a chef predicting a meal. You don't just serve a single dish; you serve a probability menu. You tell your guests: "There's a 70% chance of spaghetti, a 20% chance of pizza, and a 10% chance of a surprise salad."

In the world of data science, this is called a probabilistic forecast. The goal is to make sure your menu matches reality. If you say there's a 10% chance of a salad, and over 100 days you actually serve a salad 10 times, your forecast is "calibrated." It's trustworthy.

However, there's a big problem: Extreme events.

The Problem: The "Surprise Salad" Disaster

Imagine your restaurant is in a stormy area. Most days, the weather is mild (spaghetti or pizza). But occasionally, a massive hurricane hits (the "surprise salad" that is actually a tornado).

The paper argues that even the smartest, most high-tech chefs (machine learning models) are great at predicting the mild days but terrible at predicting the extreme storms. They might say, "Oh, there's a tiny 1% chance of a tornado," when in reality, the storm is coming.

Why? Because these models are trained to be "good on average." They focus on getting the common days right (the spaghetti and pizza) because those happen 99% of the time. They ignore the rare, dangerous days because they don't want to waste "calibration points" on them.

The authors call this a lack of Tail Calibration. "Tail" refers to the very end of the probability curve—the extreme, rare events. If your forecast isn't "tail calibrated," you might be calm when you should be screaming, "Run for your lives!"

The Solution: Changing the "Scorecard"

In machine learning, models learn by trying to get the highest score on a test. Usually, the test is a standard "accuracy score" (like the CRPS or Log Score). This scorecard rewards you for being right about the common things.

The paper suggests we need to change the scorecard to force the models to pay attention to the storms. They propose two main ways to do this:

1. The "Weighted Score" (The VIP Pass)

Imagine the standard test gives you 1 point for getting a spaghetti prediction right, but only 0.1 points for getting a tornado prediction right. The model ignores the tornado.

The authors suggest giving the tornado prediction a VIP Pass. They use a Weighted Scoring Rule. Now, getting the tornado right is worth 100 points.

The Metaphor: It's like telling the chef, "If you get the rare, dangerous dish right, you get a gold star. If you get the boring pasta right, you get a regular star."
The Result: The chef starts paying attention to the tornado. But, because they are so focused on the tornado, they might start messing up the pasta predictions a little bit.

2. The "Miscalibration Penalty" (The Strict Inspector)

This is a more direct approach. Instead of just changing the points, the authors add a penalty to the scorecard.

The Metaphor: Imagine a strict health inspector (the Regularization Term) who doesn't care how tasty the food is, but only cares if the menu matches the kitchen's actual output. If the menu says "10% chance of salad" but the kitchen serves salad 50% of the time, the inspector slaps a huge fine on the chef.
The Twist: They created a special inspector for the Tail (the extreme events). This inspector only checks the tornado predictions. If the model is lying about the tornado, it gets fined heavily.
The Result: The chef is terrified of the fine, so they adjust their tornado predictions to be perfectly accurate.

The Trade-Off: You Can't Have It All

The paper's most important finding is a trade-off.

When you force the model to be perfect at predicting extreme storms (using these new scorecards), it often becomes slightly worse at predicting the calm, everyday weather.

Analogy: It's like a student who studies only for the hardest questions on the final exam. They might ace the difficult calculus problems (the storms) but forget the basic arithmetic (the sunny days).
The Paper's Claim: The authors tested this on UK wind speed data using three different types of "chefs" (simple math models, neural networks, and complex generative models). They found that all three failed to predict extreme winds accurately using standard methods. But when they applied their new "Tail Calibration" penalties, the models became much better at predicting the storms, even if they got slightly worse at predicting the gentle breezes.

Summary

The Issue: AI weather models are great at average days but terrible at extreme disasters because they are trained to be "average."
The Fix: The authors changed the training rules (the loss function) to heavily penalize the models if they get the extreme events wrong.
The Outcome: The models became much more reliable for extreme events (like high winds), but this came at the cost of being slightly less accurate for normal, everyday weather.
The Takeaway: If you need to make decisions about life-or-death extreme events, you must train your models specifically for those events, even if it means they aren't perfect for the rest of the time.

The paper concludes that while we can't perfectly predict everything, we can use these new "scorecards" to ensure that when the storm hits, our forecasts are actually telling the truth.

Technical Summary: Enforcing Tail Calibration when Training Probabilistic Forecast Models

Problem Statement
Probabilistic forecasts are typically generated by optimizing a proper scoring rule (e.g., CRPS or log score) over training data to estimate model parameters. While this approach yields accurate forecasts when the model class is well-specified, real-world applications often involve mis-specified models or limited data. In such cases, optimizing a standard proper scoring rule does not guarantee that the resulting forecasts are calibrated (reliable). This issue is particularly critical for extreme events, which often carry the largest socio-economic impacts. Recent work (Allen et al., 2025b) has demonstrated that a forecast can satisfy standard probabilistic calibration (uniformity of Probability Integral Transform, or PIT, values) while failing to be "tail calibrated"—meaning it provides unreliable predictions for extreme outcomes. Standard training procedures offer no guarantee of tail calibration, and while weighted scoring rules can target extreme outcomes, they do not inherently enforce calibration in the tails.

Methodology
The authors investigate two primary strategies to adapt the loss function during model training to enforce tail calibration:

Weighted Scoring Rules: The authors utilize weighted scoring rules (e.g., threshold-weighted CRPS or censored likelihood scores) to place higher emphasis on outcomes above a specific threshold $t$ . They propose combining these with unweighted scores (e.g., $CRPS + \gamma \cdot twCRPS$ ) to maintain strict propriety and avoid non-unique solutions that can arise when using weighted rules alone.
Regularization via Miscalibration Penalties: Drawing on Wilks (2018), the authors propose adding a regularization term to the loss function that directly penalizes miscalibration.
- MCB (Miscalibration): A penalty based on the Wasserstein-1 distance between the empirical distribution of PIT values and a standard uniform distribution.
- TMCB (Tail Miscalibration): A novel adaptation of the MCB concept for the tails. This metric penalizes the deviation of the Conditional PIT (CPIT) values (for outcomes exceeding threshold $t$ ) from uniformity, while also accounting for the forecast probability of threshold exceedance. The TMCB is defined as the Wasserstein-1 distance between the empirical quantile function of the tail forecasts and the identity function.

These methods are applied to a hierarchy of three increasingly flexible probabilistic wind speed models for the UK:

Ensemble Model Output Statistics (EMOS): A simple parametric approach (truncated normal distribution).
Distributional Regression Networks (DRNs): Non-linear extensions of EMOS using neural networks.
Conditional Generative Models (CGMs): Non-parametric, sample-based generative models.

The models are trained on UK Met Office wind speed data (10m wind speed) and evaluated against a threshold of 12.5 m/s (approx. 97.5th percentile).

Key Contributions

Empirical Demonstration of Tail Miscalibration: The study shows that state-of-the-art models (parametric, neural network, and generative), when trained with standard proper scoring rules, fail to issue tail-calibrated forecasts for extreme wind speeds, even if they appear well-calibrated overall.
Tail Calibration Regularization: The authors introduce and implement a TMCB regularization term that explicitly penalizes tail miscalibration during training.
Trade-off Analysis: The work quantifies the trade-off between improving tail reliability and maintaining overall forecast accuracy and calibration.

Results

Simulation Study: In a controlled simulation, penalizing tail miscalibration successfully shifted model parameters toward a solution that was tail-calibrated, whereas standard optimization failed to do so.
Weather Forecasting Application:
- EMOS: Penalizing tail miscalibration (TMCB) or using the threshold-weighted CRPS (twCRPS) significantly improved tail calibration (up to ~65% improvement in TMCB skill) but resulted in a deterioration of overall calibration (MCB skill decreased by ~187%) and a slight drop in overall CRPS.
- DRNs: Standard CRPS training produced highly variable tail calibration across different random initializations. Regularization with TMCB or twCRPS reduced this variability and improved tail calibration, though at the cost of overall calibration.
- CGMs: Similar to DRNs, standard training yielded miscalibrated tails. Regularization improved tail calibration and reduced variability across model runs, with TMCB regularization showing the strongest improvement in tail metrics.
General Finding: Across all model classes, enforcing tail calibration via regularization or weighted scores improves the reliability of extreme event forecasts but introduces a trade-off where overall probabilistic calibration and sometimes forecast skill (CRPS) are compromised.

Significance and Claims
The paper claims that current state-of-the-art probabilistic forecasting models do not inherently produce calibrated forecasts for extreme events. The significance of this work lies in demonstrating that suitable adaptations to the loss function during training can improve the reliability of forecasts for extreme events.

The authors modestly frame their contribution as a method to "weakly enforce" tail calibration. They acknowledge that this approach introduces a trade-off: improving tail reliability often comes at the expense of overall calibration and sharpness. They note that while their framework is effective for the studied wind speed data, the results are specific to the chosen thresholds and model classes. The paper suggests that future work should explore multi-objective optimization and stronger notions of calibration (e.g., conformal prediction) but emphasizes that the proposed regularization provides a practical, implementable method for practitioners needing more reliable extreme event forecasts.