📊 statistics

"Calibeating": Beating Forecasters at Their Own Game

This paper argues that forecasters should be evaluated by their Brier score rather than calibration alone, and demonstrates that any forecast can be improved to simultaneously enhance calibration without sacrificing expertise ("calibeating") through deterministic or stochastic online procedures.

Original authors: Dean P. Foster, Sergiu Hart

Published 2026-03-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dean P. Foster, Sergiu Hart

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a weather forecaster. Every day, you look at the sky and say, "There is a 70% chance of rain."

For decades, the standard way to judge if you are a "good" expert was to check your Calibration. This means: on all the days you predicted 70% rain, did it actually rain about 70% of the time? If yes, you are "calibrated."

The Problem:
The paper argues that being "calibrated" is actually a very low bar. In fact, it's so easy to pass that a complete fool can do it.

The "Fool's Strategy" (The Analogy):
Imagine a weather forecaster who knows absolutely nothing about the weather. They just guess "50% chance of rain" every single day, no matter what.

If it rains 50% of the time in the long run, this fool is perfectly calibrated.
But are they useful? No. If you are planning a picnic, knowing "50%" every day tells you nothing. You don't know if it's going to be a sunny Tuesday or a rainy Tuesday.

Now, imagine a real expert. They look at the clouds and say, "If the clouds look like this, it will rain 100% of the time. If they look like that, it will never rain."

This expert is also calibrated (100% of the time it rains when they say 100%).
But this expert is useful. They sorted the days into "Rainy" bins and "Dry" bins.

The Scorecard:
The paper introduces a better way to grade forecasters called the Brier Score. Think of this as a "Total Error" score.

Calibration Score: How wrong are your labels? (e.g., Did you say 70% but it rained 40%?)
Refinement Score (The "Expertise" Score): How well did you sort the days? Did you group similar days together?
- The "Fool" (50% every day) has a terrible Refinement Score because they put rainy days and sunny days in the same bucket.
- The "Expert" has a perfect Refinement Score because they kept rainy days and sunny days in separate buckets.

The Big Question:
Can we take a forecaster who is not calibrated (maybe they are too optimistic or too pessimistic) and fix them without ruining their expertise? Can we "beat" their calibration score while keeping their sorting skills intact?

The authors call this "Calibeating."

The Solution: "The Time-Traveling Correction"

The paper provides a simple, magical trick to "calibeat" any forecast. Here is how it works in plain English:

The Old Way (Offline/Retrospective):
Imagine you wait until the end of the year. You look at all the days the forecaster said "70%." You realize, "Oh, it only rained 40% of those days." You then go back and rewrite history, changing every "70%" prediction to "40%."

Result: The new forecast is perfectly calibrated. The error is gone. The sorting (bins) is the same. The "Expertise" is preserved.
Problem: You can't do this in real-time. You can't change yesterday's prediction based on tomorrow's weather.

The New Way (Online/Real-Time):
The paper proves you can do this live, day by day, without knowing the future.

The Trick: Instead of making a prediction based on your gut feeling, you simply look at the past average of what actually happened on days when the original forecaster made that same prediction.
Example:
- Day 1: The original forecaster says "70%." You look back. Has "70%" been called before? No. You guess randomly (or use a default).
- Day 2: The original forecaster says "70%" again. You look back at Day 1. It rained. So, the average for "70%" days so far is 100%.
- Your Prediction: You don't say "70%." You say "100%."
- Day 3: The original forecaster says "70%" again. It didn't rain on Day 2. The average for "70%" days is now (100% + 0%) / 2 = 50%.
- Your Prediction: You say "50%."

Why is this magic?
By simply replacing the original forecast with the historical average of that specific forecast, you automatically fix the calibration error.

If the original forecaster was too optimistic (saying 70% when it only rained 40%), your new forecast will slowly drift down to 40%.
Crucially: You didn't mess up their sorting! You still kept the "70%" days in their own bucket. You just fixed the label on the bucket.

The Results

You can always win: No matter how bad the original forecaster is, or how chaotic the weather is, this simple "look at the past average" method guarantees a better score (lower Brier score) than the original.
You can be a "Perfect" Expert: The paper shows you can even make a procedure that is both perfectly calibrated and keeps the original forecaster's sorting skills.
It works for many forecasters: You can do this for a whole team of forecasters at once, creating a "super-forecaster" that beats all of them simultaneously.

The Takeaway

In the world of predictions, Calibration is necessary but not sufficient. Being "right on average" isn't enough; you need to be "right for the right reasons" (sorting the days correctly).

This paper gives us a simple recipe: Don't trust the label; trust the history of that label. By constantly updating your prediction based on how the past of that specific prediction played out, you can "calibeat" any expert, fixing their mistakes without losing their insight. It turns a flawed forecaster into a near-perfect one, right in real-time.

1. Problem Statement

The paper addresses a fundamental issue in the evaluation of probabilistic forecasters: the inadequacy of calibration as the sole metric for expertise.

The Calibration Paradox: A forecaster is "calibrated" if, for every forecast $p$ , the realized frequency of the event matches $p$ in the long run. However, Foster and Vohra (1998) proved that any sequence of actions can be matched by a forecaster who is perfectly calibrated, regardless of whether the forecaster possesses any genuine expertise or predictive power.
The Brier Score Limitation: The standard measure of forecast accuracy is the Brier score ( $B$ ), defined as the mean squared error between forecasts and outcomes. The Brier score decomposes into two components:
$B = K + R$
Where $K$ is the Calibration Score (error due to misalignment between forecast and realized frequency) and $R$ is the Refinement Score (error due to the variance within bins of identical forecasts).
The Core Question: Since calibration ( $K$ ) can be artificially forced to zero without improving predictive power (expertise), and expertise is reflected in a low refinement score ( $R$ ), the authors ask: Can one improve a forecaster's Brier score by reducing the calibration error ( $K$ ) without increasing the refinement error ( $R$ )?

They term this process "Calibeating": beating a forecaster's Brier score by an amount equal to their calibration score, effectively achieving $B' \approx R$ (the theoretical lower bound for that specific binning).

2. Methodology and Framework

The authors develop an online framework where a new forecaster ( $c$ ) observes the forecasts of an existing forecaster ( $b$ ) and the history of outcomes, then generates a new forecast $c_t$ to "calibeat" $b$ .

Key Definitions

Binning: Grouping time periods where the forecast $b_s$ takes the same value.
Refinement Score ( $R$ ): The average variance of outcomes within these bins. It measures how well the forecaster sorts days into homogeneous groups.
Online vs. Offline:
- Offline: One can calculate the true bin average $\bar{a}_t(b)$ and replace every forecast $b$ with this average, achieving perfect calibration ( $K=0$ ) while keeping $R$ constant. This is impossible online because the future is unknown.
- Online: The new forecast $c_t$ must be based only on history up to time $t-1$ .

The "Online Refinement Score"

A crucial technical tool introduced is the Online Refinement Score ( $\tilde{R}_t$ ). Instead of using the final bin average $\bar{a}_t$ (which is unknown at time $t$ ), it uses the past average $\bar{a}_{t-1}$ of the bin to which the current forecast belongs.

Proposition 1: The difference between the offline refinement score ( $R_t$ ) and the online version ( $\tilde{R}_t$ ) is bounded by $O(\frac{\log t}{t})$ . This allows the authors to use online estimates to approximate the offline optimum.

3. Key Contributions and Results

The paper provides a hierarchy of results, moving from simple deterministic procedures to complex stochastic and multi-forecaster scenarios.

A. Simple Deterministic Calibeating (Theorem 3)

Procedure: For any finite set of forecasts $B$ , the new forecaster $c$ simply predicts the average of past outcomes for the specific forecast $b_t$ currently being made by the opponent.
$c_t = \bar{a}_{t-1}(b_t)$
Result: This procedure guarantees that the new Brier score $B^c_t$ is lower than the original Brier score $B^b_t$ by essentially the original calibration score $K^b_t$ .
$B^c_t \leq R^b_t + O\left(\frac{|B| \log t}{t}\right)$
Significance: This is a "universal" procedure that improves any non-calibrated forecaster without needing to know the underlying data distribution. It works against an "adversary" (arbitrary action sequences).

B. Calibeating with Calibration (Theorem 4 & 5)

The simple procedure in (A) is not necessarily calibrated itself. The authors address this by constructing a procedure that is both calibrated and calibeating.

Method: They utilize a Stochastic Fixed Point approach (specifically, the "outgoing minimax" tool from Foster and Hart, 2021).
Result: There exists a stochastic procedure $\zeta$ that is $\delta$ -calibrated and simultaneously calibeats any $B$ -forecasting procedure.
$E[B^c_t] \leq E[R^b_t] + \delta^2 + O\left(\frac{\log t}{t}\right)$
Implication: One can achieve perfect calibration ( $K=0$ ) while retaining the expertise (low $R$ ) of the original forecaster, provided one accepts a stochastic (randomized) forecast.

C. Deterministic Continuously Calibrated Calibeating (Theorem 6)

Challenge: Stochastic procedures are sometimes undesirable.
Solution: By weakening the requirement from "calibration" to "continuous calibration" (a concept introduced in Foster and Hart, 2021, which implies smoothness and is sufficient for equilibrium dynamics), the authors construct a deterministic procedure that is continuously calibrated and calibeating.
Tool: This relies on a Fixed Point (FP) theorem rather than a Minimax theorem.

D. Multi-Calibeating (Theorem 7)

Scenario: Simultaneously beating $N$ different forecasters ( $b^1, \dots, b^N$ ).
Result: The authors construct a single forecaster that calibeats all $N$ forecasters simultaneously.
Error Bound: The error term initially scales with the product of the sizes of the forecast sets ( $\prod |B_n|$ ), which is exponential in $N$ .
Optimization: In Appendix A.8, they provide improved procedures using Blackwell Approachability and Online Linear Regression that reduce the error dependence to polynomial in $N$ (e.g., $O(\sqrt{N/t})$ or $O(N \log t / t)$ ).

E. Extension to Logarithmic Scores (Appendix A.9)

The authors demonstrate that their framework extends beyond quadratic (Brier) scores to Logarithmic scoring rules (using Kullback-Leibler divergence). They define "Log-Calibeating" and prove analogous theorems, showing the robustness of the approach across different strictly proper scoring rules.

4. Technical Significance

Redefining Expertise: The paper rigorously argues that Refinement ( $R$ ) is the true measure of expertise, while Calibration ( $K$ ) is merely a property that can be gamed. Experts should be judged by the Brier score, not calibration alone.
Online Variance Estimation: The derivation of the online refinement score and the bound on the difference between online and offline variance (Proposition 1) is a significant contribution to online learning and game theory.
Minimax vs. Fixed Point: The paper clarifies the distinction between procedures requiring Minimax (stochastic, randomized) solutions and those solvable by Fixed Point (deterministic, continuous) methods, linking these to the specific requirements of calibration vs. continuous calibration.
Universality: The proposed procedures are "universal"; they do not require knowledge of the data generating process, the stationarity of the environment, or the specific strategy of the adversary. They work uniformly against any sequence of actions.

5. Conclusion

"Calibeating" provides a constructive method to transform any forecasting procedure into a superior one. By replacing a forecast with the historical average of outcomes for that specific forecast type, a new forecaster can eliminate the calibration error component of the Brier score without sacrificing the refinement (sorting) capability of the original forecaster.

The paper establishes that expertise (low refinement error) and calibration are compatible, but achieving both simultaneously often requires stochasticity or a relaxation to continuous calibration. The results have broad implications for the evaluation of experts in economics, machine learning (online regression), and game theory.