📊 statistics

Anytime-Valid Evidence for Prespecified Predictive Corrections

This paper introduces a framework for accumulating anytime-valid evidence that a prespecified predictive correction outperforms an uncorrected source distribution by utilizing a fixed nonnegative tilt to generate a conditional e-process, which remains statistically valid under optional stopping and arbitrary input sequences while quantifying the cumulative log-score advantage of the correction.

Original authors: Seungjin Choi

Published 2026-08-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seungjin Choi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a weather forecaster who has built a brilliant model to predict tomorrow's rain. You call this your "Source Model." But then, you hear a rumor: the local climate is shifting, or maybe the sensors on your weather station are drifting. You don't want to throw away your entire model and start from scratch; instead, you have a specific, pre-planned idea about how to tweak it. Maybe you think, "I bet the rain will be 10% heavier," or "I bet the storms will be more unpredictable." This is a predictive correction: a specific, pre-written note on how to adjust your old forecast to fit a new reality.

The big question in statistics is: How do you know if your tweak is actually working? Usually, you'd wait until you have a huge pile of new data to see if the old model was wrong and your new one is right. But what if the data arrives one by one, like a slow drip? What if you need to decide right now whether to switch to your new forecast, but you also don't want to panic and switch if it's just a fluke? This is where the concept of anytime-valid evidence comes in. Think of it like a safety harness for your decision-making. It allows you to check your progress every single second, stop whenever you want, and still be mathematically guaranteed that you haven't been tricked by random noise. It's the difference between guessing if a coin is fair after one flip versus having a magical rule that says, "No matter when you stop watching, if the coin looks this biased, you can be sure it's not just luck."

This paper, titled "Anytime-Valid Evidence for Prespecified Predictive Corrections," tackles exactly that problem. The author, Seungjin Choi, proposes a clever method to track whether a specific, pre-planned tweak to a prediction model is actually better than the original, un-tweaked version. They don't try to find any possible change; they test your specific idea.

Here is how the magic works, using a simple analogy. Imagine your original weather model is a "Source Map." You have a specific idea (your "Correction") that the map needs a little nudge. The paper creates a special scorecard (called an e-process) that starts at 1. Every time a new weather event happens (a new data point), you compare how well the Original Map predicted it versus how well your Corrected Map predicted it.

If your Correction was right, the scorecard goes up. If the Original Map was actually better, the scorecard goes down. The brilliant part is that this scorecard is built with a mathematical "safety net." The author proves that if your Correction is actually wrong (and the Original Map is still the truth), the scorecard will almost never climb high enough to trick you. Even if you check the score every second, stop whenever you feel like it, or even change your strategy based on what you've seen so far, the math guarantees that the chance of a "false alarm" (thinking your correction is great when it's not) stays below a tiny number you set at the start, like 5%.

The paper finds that this method is incredibly flexible. It works for all sorts of specific tweaks:

Label Shifts: Like realizing that in a new city, "Sunny" days are actually more common than "Rainy" days, so you just need to adjust the odds.
Mean Shifts: Like realizing the temperature is consistently 2 degrees warmer than your model thought.
Variance Shifts: Like realizing the weather is becoming more chaotic and unpredictable, so your forecast needs wider "maybe" ranges.

The author also shows that you can mix and match these ideas. If you aren't sure how much to adjust the temperature, you can create a "mixture" of several possible adjustments and let the scorecard tell you if any of them are working. They even show how to set a "tolerance" level: "I don't care about tiny changes; I only want to switch my model if the change is big enough to matter."

However, the paper is very careful about what this scorecard doesn't do. It is a referee for a specific match between your Original Map and your Corrected Map. It does not tell you if your Corrected Map is the true reality. It only says, "Your tweak is beating the original." It's possible that both maps are wrong, but your tweak is just less wrong. The author demonstrates this with simulations: if the real world changes in a way your tweak didn't predict (like a sudden storm instead of a temperature shift), your scorecard might still go up if the math happens to favor your tweak by accident. The paper warns that crossing the finish line doesn't mean you've discovered the cause of the change, only that your specific idea is currently the better predictor.

In short, this paper gives scientists and engineers a powerful, safe tool to monitor their predictions in real-time. It lets them say, "I have a specific idea for fixing my model, and I can prove, with mathematical certainty, that this idea is working better than the old one, even if I'm watching the data stream second-by-second." It turns the scary, uncertain process of adapting to change into a game with clear, unbreakable rules.

Technical Summary: Anytime-Valid Evidence for Prespecified Predictive Corrections

1. Problem Setup

The paper addresses the problem of confirming a prespecified predictive correction in a sequential setting. In many scientific and operational contexts (e.g., instrument recalibration, assay drift, or known interventions), a practitioner may possess a specific hypothesis about how an existing source predictive distribution, $p_0(y \mid x, D_{tr})$ , should be modified to reflect anticipated changes in future outcomes. This modification, or tilt $h(x, y)$ , defines a corrected predictive distribution $p_h(y \mid x, D_{tr})$ .

Unlike standard distribution shift problems where the goal is to estimate an unknown target distribution using abundant target data, this work focuses on confirmation: determining whether the specific, prespecified correction $p_h$ predicts incoming target outcomes $(X_i, Y_i)$ better than the original source $p_0$ . The setting assumes outcomes arrive sequentially, potentially under adaptive input selection, and the practitioner wishes to monitor evidence continuously without fixing a sample size in advance.

2. Methodology

2.1 The E-Process Construction

The core methodology constructs an anytime-valid e-process based on the corrected-to-source predictive likelihood ratio.

Tilt and Normalization: A nonnegative tilt $h: \mathcal{X} \times \mathcal{Y} \to [0, \infty)$ is prespecified. It is normalized by $Z_h(x, D_{tr}) = \int h(x, y)p_0(y \mid x, D_{tr}) dy$ to form the corrected predictive:
$p_h(y \mid x, D_{tr}) = \frac{h(x, y)p_0(y \mid x, D_{tr})}{Z_h(x, D_{tr})}$
Conditional E-Value: For each observation $i$ , the one-step evidence factor is defined as:
$e_i = \frac{p_h(Y_i \mid X_i, D_{tr})}{p_0(Y_i \mid X_i, D_{tr})} = \frac{h(X_i, Y_i)}{Z_h(X_i, D_{tr})}$
Under the source predictive null hypothesis $H_0^{pred}: Y_i \mid X_i, \mathcal{F}_{i-1} \sim p_0(\cdot \mid X_i, D_{tr})$ , the conditional expectation $E[e_i \mid \mathcal{G}_i] = 1$ , where $\mathcal{G}_i$ contains past data and the current input $X_i$ . Thus, $e_i$ is a conditional e-value.
The Wealth Process: The running product $M_t = \prod_{i=1}^t e_i$ forms a nonnegative martingale under $H_0^{pred}$ . By Ville's inequality, for any $\alpha \in (0, 1)$ :
$\sup_{P \in \mathcal{P}_0^{pred}} P\left( \sup_{t \ge 0} M_t > \frac{1}{\alpha} \right) \le \alpha$
This guarantees that the probability of ever falsely confirming the correction (crossing the boundary $1/\alpha$ ) is at most $\alpha$ , regardless of the stopping time or the input selection mechanism (including adaptive designs).

2.2 Evidence Interpretation

The logarithm of the wealth process, $\log M_t$ , represents the cumulative predictive log-score advantage of the corrected distribution over the source:
$\log M_t = \sum_{i=1}^t \left( \log p_h(Y_i \mid X_i, D_{tr}) - \log p_0(Y_i \mid X_i, D_{tr}) \right)$
A crossing of the boundary $1/\alpha$ (or $\log(1/\alpha)$ in log-space) provides anytime-valid relative evidence that $p_h$ outperforms $p_0$ on the observed stream.

2.3 Extensions and Variations

Refutation: If $h(x, y) > 0$ almost surely, the reciprocal process $M_t^\downarrow = 1/M_t$ allows for anytime-valid refutation of the corrected predictive in favor of the source.
Predictable Tilts: The framework extends to tilts $h_i$ that are updated predictably based on past observations, interpreting the result as evidence for an adaptive betting strategy rather than a single fixed correction.
Mixtures: To handle uncertainty over the magnitude or direction of the correction, the paper proposes mixing wealth paths over a prespecified family of tilts, $\int M_t(\theta) d\Pi(\theta)$ , which retains the e-process property.
Beyond-Tolerance: The method can be adapted to test whether a shift exceeds a specific tolerance boundary $\delta_{tol}$ by comparing an actionable predictive $p_{alarm}$ against $p_{tol}$ rather than the uncorrected source.

3. Key Contributions

Anytime-Valid Relative Confirmation: The paper constructs a conditional e-process for prespecified predictive corrections. Validity holds under optional stopping and arbitrary (including adaptive) input sequences. The process quantifies the cumulative log-score advantage of the correction over the source.
Drift Decomposition and Growth: The author derives a conditional drift decomposition showing that the expected log-growth rate is the difference in conditional Kullback-Leibler (KL) divergences:
$\Gamma_h(x) = D_{KL}(q(\cdot \mid x) \parallel p_0(\cdot \mid x, D_{tr})) - D_{KL}(q(\cdot \mid x) \parallel p_h(\cdot \mid x, D_{tr}))$
This clarifies that positive evidence accumulation requires only that the correction be closer to the true target than the source, not that it be perfectly specified.
False-Confirmation Control under Misspecification: The paper identifies a correction-dependent protected half-space of misspecified target distributions. If the true target $q$ satisfies a specific moment condition ( $E_q[h(X, Y)] \le Z_h(X)$ ), the original false-confirmation bound remains valid. This distinguishes between generic proximity to the source (which does not guarantee control) and specific moment-based robustness.
Overshoot Identity: An identity is derived explaining why the realized Type I error rate is often strictly below the nominal level $\alpha$ . The probability of crossing is inversely related to the expected overshoot of the wealth process at the crossing time.
Structured Corrections: The framework is instantiated for specific cases including label-shift, conditional mean/variance corrections, subgroup-specific adjustments, and general exponential-family tilts.

4. Results from Synthetic Experiments

The paper validates the theory using synthetic experiments with an oracle Gaussian source predictive:

Validity: Under the source null, the empirical false-confirmation rates were consistently below the nominal $\alpha = 0.05$ (e.g., 0.034), consistent with the overshoot identity.
Growth Rates: Correctly specified corrections (mean and variance shifts) accumulated evidence at rates matching the theoretical drift predictions. Underspecified corrections still accumulated evidence (positive drift) but at slower rates, while overspecified corrections could result in negative drift or reduced power.
Robustness and Failure Modes:
- Cross-Family Drift: A variance-correction process could be driven by a pure mean shift (if the mean shift increased the second moment sufficiently), leading to false confirmation of a variance change. Conversely, a mean-correction process could be driven by heavy tails or variance inflation. This demonstrates that a crossing supports the relative predictive performance of the correction but does not uniquely identify the physical mechanism.
- Moment Condition Failure: When the target distribution violated the moment condition (e.g., heavy-tailed targets where the tilt's expectation is infinite), the false-confirmation bound could be exceeded, even if the long-run drift was negative. This highlights that negative drift alone does not guarantee time-uniform protection.
Adaptivity: Adaptive input selection (e.g., selecting inputs with high KL separation) accelerated evidence accumulation without compromising the validity of the source-null guarantee.
Mixtures and Beyond-Tolerance: Mixtures over correction magnitudes provided robustness to uncertainty, and beyond-tolerance tests successfully controlled false alarms over a composite region of acceptable shifts.

5. Significance and Claims

The paper claims to provide a finite-sample, anytime-valid framework for evaluating prespecified predictive corrections without needing to estimate the full target distribution.

Relative vs. Absolute: The author emphasizes that the method provides relative confirmation. A boundary crossing indicates that the proposed correction is better than the source predictive for the observed data, but it does not prove the correction is the true target distribution or identify the specific mechanism of the shift.
Operational Utility: The framework is designed for settings where a plausible correction is available before monitoring begins (e.g., due to engineering analysis or policy changes), allowing for sequential monitoring with strict error control.
Limitations: The guarantees are conditional on the fitted source predictive $p_0$ . The paper does not claim to handle source-model misspecification or uncertainty arising from the model-fitting stage itself. Furthermore, the reciprocal refutation boundary is only calibrated under the corrected predictive null, not under a broad class of misspecifications.
Distinction from Estimation: The work explicitly distinguishes itself from distribution-shift estimation or adaptation methods. It does not learn an unrestricted target; it tests a specific hypothesis about how the source should be modified.

In summary, the paper offers a rigorous statistical tool for "betting" on a specific predictive correction against a source model, ensuring that the risk of false alarms is controlled at every step of a sequential monitoring process, while clarifying the interpretive limits regarding the identification of the underlying shift mechanism.