Sequential model confidence sets

Technical Summary: Sequential Model Confidence Sets

Problem Statement
In statistical prediction and estimation, practitioners often face the challenge of selecting the best model(s) from a set of candidates ($M_0 = \{1, \dots, m\}$) based on a user-specified loss function. The standard approach, Model Confidence Sets (MCS) proposed by Hansen et al. (2011), reduces the candidate set to a subset containing the best models with a given confidence level. However, the classical MCS relies on two restrictive assumptions that conflict with real-world forecasting scenarios:

1. Fixed Sample Size: It assumes a pre-specified, fixed evaluation period, whereas model selection and forecast evaluation are inherently sequential tasks where data arrives over time and decisions to continue or stop a study depend on interim outcomes.
2. Stationarity: It assumes the sequences of observed loss differences are stationary, an assumption often violated as models evolve, correct errors, or adapt to systematic biases over time.

Furthermore, classical MCS relies on bootstrap approximations and asymptotic Central Limit Theorem (CLT) properties, offering only asymptotic coverage guarantees and incurring high computational costs. There is a need for methods that allow continuous monitoring of model performance with time-uniform, non-asymptotic coverage guarantees.

Methodology
The authors propose Sequential Model Confidence Sets (SMCS), which generalize the MCS framework to a sequential setting using e-processes and confidence sequences. The methodology is built on "safe anytime-valid inference," ensuring validity regardless of when the analysis is stopped.

The core of the approach involves three distinct definitions of "superior" models, corresponding to different hypotheses about model performance:

1. Strongly Superior ($M_{s,\star}$): Models that outperform all others at every time point in terms of conditional expected loss differences ($\mu_{ij,t} \le 0$).
2. Uniformly Weakly Superior ($M_{uw,\star}$): Models that outperform all others on average across all time points ($\Delta_{ij,t} \le 0$ for all $t$).
3. Weakly Superior ($M_{w,\star,t}$): Models that outperform all others at a specific time $t$ based on the average loss up to that point.

Construction of SMCS:

• For Strong and Uniformly Weak Hypotheses: The authors utilize e-processes (specifically those by Howard et al., 2020, and Choe and Ramdas, 2023) under the assumption of conditionally bounded or uniformly bounded loss differences. They employ the closure principle with an arithmetic mean as an e-merging function to control the family-wise error rate across multiple pairwise comparisons. A model is excluded from the SMCS if the corresponding e-process exceeds a threshold ($1/\alpha$).
• For Weak Hypotheses: The construction relies on time-uniform confidence sequences. The authors define a confidence region for the vector of average loss differences. A model is excluded if the confidence region does not intersect with the half-space where the model is superior.
• Handling Unbounded Losses: For loss functions that are not uniformly bounded (e.g., CRPS with unbounded support), the authors propose a transformation to conditionally bounded differences, which are then scaled to be uniformly bounded, allowing the application of the derived e-processes.

Key Contributions

1. Sequential Extension: The paper extends the MCS concept to a sequential framework, allowing for continuous monitoring of model performance without requiring a fixed sample size.
2. Time-Uniform Guarantees: The proposed SMCSs provide time-uniform, non-asymptotic coverage guarantees. This means the probability that the true superior models are contained in the set is at least $1-\alpha$ for all time points simultaneously, a stronger guarantee than point-wise asymptotic coverage.
3. Minimal Assumptions: The methods rely on minimal distributional assumptions (sub-Gaussian or sub-exponential loss differences) and do not require computationally expensive bootstrap approximations.
4. Flexibility in Superiority Definitions: The framework accommodates three different notions of model superiority, allowing researchers to choose the hypothesis (strong, uniformly weak, or weak) that best fits their application context (e.g., nested information sets vs. evolving model performance).

Results
The paper validates the methodology through simulation studies and two real-world case studies:

• Simulations:
  • Simulation 1 (Strong Hypothesis): With 49 forecasters, the SMCS achieved a 100% coverage rate for the true best model, demonstrating conservative but valid performance. The average size of the SMCS decreased over time.
  • Simulation 2 (Uniformly Weak Hypothesis): In a scenario where the best model was only superior on average (not at every single time point), the SMCS correctly identified the uniformly weakly superior model, whereas the strong hypothesis would have yielded an empty set.
  • Simulation 3 (Weak Hypothesis): In a setting where the best model changed over time (worsening $\to$ biased $\to$ improving), the SMCS for the weak hypothesis successfully tracked the changing superior model, including and excluding models as their relative performance shifted.
• Case Study 1 (Covid-19 Forecasts): Applied to 69 models from the Covid-19 Forecast Hub. The SMCS (under the strong hypothesis) identified ensemble models as consistently superior and rejected specific epidemiological models (e.g., PSI-draft) relatively early. This demonstrated the utility of sequential monitoring for policy-making, allowing for early rejection of underperforming models without waiting for a fixed sample size.
• Case Study 2 (Wind Gust Predictions): Applied to 9 postprocessing methods for wind gusts. Under the weak hypothesis, the SMCS revealed that while some models were initially inferior, they became competitive again after updates to the underlying Numerical Weather Prediction (NWP) model. This dynamic behavior (models re-entering the confidence set) would have been missed by fixed-sample or strong-hypothesis approaches.

Significance and Claims
The paper claims that Sequential Model Confidence Sets offer a statistically rigorous alternative to traditional model selection, particularly in dynamic environments.

• Anytime-Validity: The primary significance is the ability to "stop whenever" without inflating Type I error rates, a crucial feature for ongoing monitoring in fields like epidemiology and meteorology.
• Handling Non-Stationarity: By distinguishing between strong, uniformly weak, and weak superiority, the method acknowledges that models may evolve. It allows for the detection of models that are superior on average or temporarily, rather than requiring dominance at every single instant.
• Practical Utility: The authors argue that their methods provide a theoretical justification for the common practice of regular model assessment in emergency situations (like the pandemic), offering a way to collect evidence sequentially and make decisions based on the current state of knowledge while respecting selection uncertainty.

The authors remain modest, noting that while they focused on bounded loss differences, the theory is not strictly limited to them, though further research is needed for general loss functions. They also acknowledge that while the methods control family-wise error rates, future work could explore false discovery rate (FDR) control for potentially higher power.