📈 economics

Proper Calibeating

This paper extends the concepts of calibration and calibeating from quadratic scoring rules to the broader class of proper scoring rules, establishing their theoretical relationships, providing methods to guarantee proper-calibeating, and demonstrating their equivalence to universal no-regret decision-making.

Original authors: Dean P. Foster, Sergiu Hart

Published 2026-05-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dean P. Foster, Sergiu Hart

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a weather forecaster. Every day, you predict the chance of rain. Sometimes you say "20%," sometimes "80%." At the end of the year, you want to know: Did you do a good job?

For decades, statisticians have used a specific "scorecard" (called the quadratic scoring rule) to grade forecasters. This paper takes that scorecard and asks a big question: What if we used a different scorecard? Would your grade still be good?

The authors, Dean Foster and Sergiu Hart, explore two main ways to judge a forecaster: Calibration and Calibeating. They then test if these concepts hold up when we change the rules of the game (the scoring system).

Here is the breakdown of their findings in plain English.

1. The Two Ways to Grade a Forecaster

To understand the paper, you need to know the two concepts they are testing:

Calibration (The "Truth-Teller" Test):
Imagine you say "It will rain 20% of the time" on 100 different days. If it actually rains on exactly 20 of those days, you are calibrated. You aren't necessarily smart about which days it will rain, but your numbers match reality perfectly.
- The Paper's Finding: If you are calibrated under the standard scorecard, you are automatically calibrated under any reasonable scorecard. This is great news! It means being a "truth-teller" is a universal skill.
Calibeating (The "Expert" Test):
This is a newer, stricter concept. Imagine you are competing against a "Reference Forecaster" (let's call him Bob). Bob is an expert who knows a lot, but he might be slightly off in his numbers (not perfectly calibrated).
- Calibeating means you do better than Bob. Specifically, you achieve a score that is at least as good as Bob's "expertise" (how well he groups days into categories), even if you don't match his exact predictions. You get the benefits of his expertise without his mistakes.
- The Paper's Finding: This is where things get tricky. If you "calibeat" Bob under the standard scorecard, you might fail to beat him under a different scorecard. Being an "expert" under one set of rules doesn't guarantee you are an expert under all rules.

2. The Big Discovery: The "Asymmetry"

The paper reveals a surprising asymmetry:

Calibration is Robust: If you are good at calibration, you are good at it no matter how you measure it. It's like being a perfect archer; if you hit the bullseye, you hit it regardless of whether the target is red, blue, or green.
Calibeating is Fragile: If you beat a reference forecaster under one scoring rule, you might lose under another. It's like winning a race on a track made of rubber; you might not win on a track made of ice.

The Analogy of the "Joint Bin":
Why does calibeating fail? The authors show that to guarantee you beat everyone under every rule, you can't just beat the reference forecaster on your own. You have to beat them together.

Imagine you and Bob are sorting socks into bins.

Standard Calibeating: You sort your socks into bins, and Bob sorts his. You win if your bins are "better" than his.
Proper Calibeating (The Solution): You and Bob sort your socks into bins simultaneously, creating a giant grid of "Joint Bins" (e.g., "Bob's Red Bin" + "Your Blue Bin").
The paper proves that if you can beat Bob in this Joint Bin scenario, you are guaranteed to be a winner under every possible scoring rule. It's not enough to just be good; you have to be good relative to the specific combination of your predictions and his.

3. How to Guarantee a Win (The "Proper" Solutions)

Since standard calibeating isn't enough, the authors propose three specific methods to ensure you are "Proper-Calibeating" (winning under all rules):

The "Joint" Strategy: Use a procedure that tracks the interaction between your forecasts and the reference forecaster's forecasts simultaneously. If you beat the "joint" history, you win universally.
The "Simple Average" Trick (for Smooth Rules): There is a very simple method where you just predict the average of what happened in the past for a specific category. The authors show this works perfectly for "smooth" scoring rules (rules that don't have sudden jumps), but it fails for "jagged" rules.
The "Continuous" Strategy: They prove there exists a complex, deterministic method that guarantees you win under all smooth rules while also being perfectly calibrated.

4. The Decision-Making Connection

Finally, the paper connects this to real-life decisions.

Imagine a doctor using your forecast to decide on a treatment.
Regret: This is the feeling of "I could have made a better choice if I had known more."
The authors show that Proper-Calibration is exactly the same thing as having "No Regret." If your forecasts are properly calibrated, a decision-maker using your advice will never regret their choices, no matter what their personal goals (utility function) are.

Summary

Calibration (being accurate) is universal. If you are calibrated, you are calibrated everywhere.
Calibeating (beating an expert) is not universal. Beating someone under one rule doesn't mean you beat them under another.
The Fix: To guarantee you beat an expert under all rules, you must beat them in a "Joint" scenario where you account for how your predictions and theirs interact.
The Result: If you achieve this "Proper-Calibeating," you ensure that anyone using your forecasts to make decisions will get the best possible outcome, regardless of how they measure "good."

The paper is a mathematical proof that while some forecasting skills are fragile, there are specific, robust strategies (like tracking joint bins) that make a forecaster universally reliable.

Technical Summary: Proper Calibeating

1. Problem Statement

The paper addresses the robustness of forecasting guarantees across different proper scoring rules. While the classic concepts of calibration (forecasts matching realized frequencies) and the more recent refinement of calibeating (achieving a Brier score no worse than a reference forecaster's refinement score) are well-established for the standard quadratic (Brier) scoring rule, their validity under the broader class of proper scoring rules remains an open question.

A proper scoring rule is one where the expected loss is minimized when the forecast equals the true underlying distribution. The authors investigate whether procedures that are calibrated or calibeating under the quadratic rule necessarily satisfy these properties under all bounded proper scoring rules. This is crucial because decision-making under uncertainty induces specific proper scoring rules; if guarantees are artifacts of the quadratic functional form, their practical relevance in general decision contexts is limited.

2. Methodology and Framework

The authors extend the framework of Foster and Hart (2023) to the class of bounded proper scoring rules.

Definitions:
- Proper-Calibration: A procedure is proper-calibrated if its calibration score converges to zero uniformly over all bounded proper scoring rules.
- Proper-Calibeating: A procedure is proper-calibeating if it beats a reference forecaster's refinement score uniformly over all bounded proper scoring rules.
- Lipschitz Variants: The paper also distinguishes between general bounded rules and those that are Lipschitz-continuous (proper-Li-calibeating).
Decomposition: The analysis relies on the decomposition of the Brier score ( $B$ ) into a calibration score ( $K$ ) and a refinement score ( $R$ ), such that $B = K + R$ . This decomposition holds exactly for the quadratic rule and approximately (within a bound dependent on the binning granularity) for Lipschitz rules under general binning sequences.
Decision-Theoretic Link: The paper connects forecasting performance to decision-making under uncertainty. It establishes that the calibration score under an induced scoring rule is equivalent to the regret incurred by a decision-maker who best-responds to the forecasts.

3. Key Contributions and Results

A. Calibration Transfers, Calibeating Does Not

The paper establishes a fundamental asymmetry between calibration and calibeating regarding their extension to proper scoring rules:

Calibration Implies Proper-Calibration: The authors prove that any procedure that is calibrated under the quadratic score is automatically proper-calibrated.
- Mechanism: The calibration score under any bounded proper scoring rule is bounded by a multiple of the square root of the quadratic calibration score. Consequently, existing calibrated procedures (stochastic or deterministic continuously calibrated) require no modification to achieve proper-calibration.
Calibeating Does Not Imply Proper-Calibeating: Unlike calibration, calibeating under the quadratic rule does not guarantee calibeating under other proper rules.
- Counterexample: The authors construct a specific sequence where a forecast $c$ calibeats a reference $b$ under the quadratic rule but fails to do so under the 2-spherical proper scoring rule (which is bounded and Lipschitz). Both sequences are perfectly calibrated; the failure arises because refinement scores do not move in tandem across different proper rules.

B. Achieving Proper-Calibeating

Since standard calibeating is insufficient, the paper provides three distinct methods to guarantee proper-calibeating:

Joint Binning (Stochastic): A stochastic procedure that calibeats the joint binning of the reference forecaster and the procedure itself is proper-calibeating.
- Insight: It is insufficient to simply be calibrated and calibeat the reference separately. One must calibeat the interaction between the two binning structures. This procedure is also proper-calibrated.
Simple Deterministic Procedure (Lipschitz Only): The simple deterministic procedure from Foster and Hart (2023), which forecasts the running average of actions within the reference forecaster's bins, is shown to be proper-Li-calibeating (valid for all Lipschitz proper scoring rules).
- Limitation: This specific procedure fails for general bounded proper scoring rules that are not Lipschitz (e.g., discontinuous rules).
Deterministic Continuously Proper-Calibrated Procedure: The authors establish the existence of a deterministic procedure that is simultaneously continuously proper-calibrated and proper-Li-calibeating. This is achieved via an approximate decomposition of the score that generalizes the quadratic case.

C. Extension to Multicalibeating

All positive results for proper-calibeating extend to the simultaneous proper-calibeating of multiple reference forecasters by cross-classifying across all their bins.

D. Equivalence to Universal No-Regret

The paper demonstrates a precise equivalence between proper-calibration and universal no-regret in sequential decision-making.

A forecasting procedure is proper-calibrated if and only if every decision-maker with a bounded utility function achieves no regret when best-responding to the forecasts.
Similarly, proper-calibeating yields gain guarantees across all utilities, ensuring that using the new forecasts yields utility gains at least as large as the reference forecaster's regret.

4. Significance

The paper clarifies the theoretical boundaries of forecasting guarantees. It confirms that while calibration is a robust property that holds across all bounded proper scoring rules, calibeating is a more fragile property that depends on the specific scoring rule unless specific, stronger conditions (like joint binning or Lipschitz continuity) are met.

By linking proper-calibration to universal no-regret, the paper unifies the concepts of forecast accuracy and decision-theoretic optimality. It provides constructive methods for generating forecasts that are robust to the specific choice of loss function in decision-making environments, ensuring that "expertise" (refinement) is preserved alongside calibration across a wide class of decision problems.

5. Limitations and Scope

Boundedness: The results apply to bounded proper scoring rules. The logarithmic scoring rule (unbounded) is explicitly excluded, though the authors note in footnotes that specific modifications exist for log-calibeating in other works.
Lipschitz vs. General: The simple deterministic calibeating procedure is only guaranteed for Lipschitz rules, not all bounded rules.
Counterexamples: The paper relies on specific constructed sequences to demonstrate the failure of calibeating under non-quadratic rules, rather than empirical data.