Bet on Features: Anytime-Valid and Feature-Aware Auditing of Conditional Quantile Forecasters

Technical Summary: Bet on Features

Problem Statement
Black-box conditional quantile forecasters are increasingly deployed for sequential decision-making under asymmetric costs, such as inventory planning and risk management. However, once deployed, these models face two critical challenges: (1) data streams often drift or change regimes, rendering standard fixed-horizon backtests invalid due to optional stopping; and (2) existing calibration tests often fail to account for the information asymmetry between the forecaster and the auditor. A forecaster may possess rich internal context (e.g., promotions, specific store identifiers), while an external auditor may only observe a coarser view (e.g., aggregate sales). Consequently, a forecast may appear calibrated to a marginal auditor while being systematically miscalibrated relative to specific, observable features. Current frameworks generally satisfy either the requirement for anytime-validity (safe under continuous monitoring) or the requirement for conditional sensitivity, but rarely both, and they often lack a mechanism to explicitly index the calibration null by the auditor's available information.

Methodology
The authors propose a distribution-free, game-theoretic framework for auditing conditional quantile forecasters based on "testing by betting."

1. Information-Indexed Calibration Nulls: The paper formalizes conditional quantile calibration relative to a specific monitoring information structure $\mathcal{H}$. Let $Z_t = \mathbb{1}\{Y_t \le \hat{q}_t\} - \alpha$ be the centered hit. The null hypothesis $P_0(\mathcal{H})$ is defined as the set of laws where $\mathbb{E}_P[Z_t | \mathcal{H}_{t-1}] = 0$ almost surely. The authors establish a hierarchy of nulls: marginal calibration ($\mathcal{H}^{marg}$) is the weakest, followed by single-feature calibration ($\mathcal{H}^w$), and full-feature calibration ($\mathcal{H}^\phi$). Crucially, $P_0(\mathcal{H}^\phi) \subseteq P_0(\mathcal{H}^w) \subseteq P_0(\mathcal{H}^{marg})$. This implies that a test valid for a coarser null (e.g., marginal) remains valid for richer nulls, but a coarser test may lack power to detect violations visible only in richer information sets.

2. Anytime-Valid E-Processes: The auditor acts as a "skeptic" in a sequential betting game. At each time $t$, the skeptic chooses a betting multiplier $\lambda_t$ based on their information $\mathcal{H}_{t-1}$ from a no-bankruptcy interval $\Lambda_\alpha = [-1/(1-\alpha), 1/\alpha]$. The wealth process $M_t = \prod (1 + \lambda_i Z_i)$ forms a test supermartingale (and thus an e-process) under the null. By Ville's inequality, this process provides anytime-valid evidence: if $M_t$ exceeds $1/\gamma$, the null is rejected with Type-I error probability at most $\gamma$, regardless of when the test is stopped.

3. Feature-Aware Contextual Betting: To achieve power against feature-specific miscalibration, the authors introduce contextual betting. The skeptic utilizes a predictable feature dictionary $\phi_t$ (e.g., calendar indicators, lagged errors) and employs an online convex optimization (OCO) learner to adapt the betting weights $\theta_t$. The bet is defined as $\lambda_t(\theta) = \langle \theta, \phi_t \rangle$. The learner minimizes the regret against a fixed comparator $\theta^*$, defined via the loss $g_t(\theta) = -\log(1 + \langle \theta, \phi_t \rangle Z_t)$.

4. Regret-to-Power Guarantees: The paper derives finite-time detection guarantees. If the alternative law $Q$ exhibits a "predictable edge" (i.e., the conditional miscalibration aligns with a linear feature direction such that the quadratic edge $b^Q_t(\theta^*) \ge \kappa$), the wealth of the contextual skeptic grows exponentially. Specifically, if the OCO learner achieves a pathwise regret bound $r_T$, the stopping time $\tau_\gamma$ is finite with probability 1 under the alternative, and the expected stopping time is bounded by a function of the regret and the edge strength.

Key Contributions

1. Information-Indexed Anytime-Valid Audits: The paper formulates continuous auditing as a sequential game indexed by the auditor's monitoring information. It proves that predictable no-bankruptcy bets generate valid e-processes for any monitoring structure without assuming i.i.d. data, stationarity, or parametric models.
2. Hierarchy of Validity and Power: It establishes that validity transfers from coarser to richer nulls, but power does not. A marginal audit can be perfectly valid yet blind to feature-specific violations.
3. Feature-Aware Contextual Betting: The authors develop a method where an online learner adapts linear bets over a feature dictionary. They prove that under persistent feature-aligned predictable edges, any OCO algorithm with sublinear regret yields finite-time detection guarantees.
4. Empirical Validation of Miscalibration: The framework is validated on synthetic data and real-world store sales (Rossmann dataset) using the Chronos-2 forecaster.

Results

• Validity: On a calibrated Negative-Binomial oracle, the empirical rejection rates of the betting audits across various quantile levels and skeptic configurations (marginal, single-feature, full-feature) align with the nominal 5% Ville level, confirming Type-I error control.
• Power Gap: In experiments with a controlled Markov hit process, marginal skeptics failed to detect violations (censored at the horizon), while skeptics aware of the previous hit ($Z_{t-1}$) rejected quickly, demonstrating the power gap predicted by the theory.
• Real-World Findings: When applied to Chronos-2 forecasts on Rossmann store-sales data and synthetic Negative-Binomial series, feature-aware skeptics detected significant conditional miscalibration that marginal audits missed. Specifically, the audits revealed that forecasts were systematically miscalibrated with respect to promotion status, Saturday indicators, and log-price. The marginal audit wealth remained flat, while feature-aware wealth crossed rejection thresholds.

Significance and Claims
The paper claims to provide the first framework that simultaneously satisfies anytime-validity and feature-awareness for black-box conditional quantile forecasters. By explicitly modeling the auditor's information set, the framework separates the concepts of validity (safety) and power (detectability). The authors argue that this is essential for operational settings where auditors have limited visibility compared to forecasters. The empirical results suggest that popular foundation models like Chronos-2, while potentially well-calibrated on average, exhibit systematic, predictable errors conditioned on specific business features (like promotions), which standard marginal backtests fail to detect. The work bridges game-theoretic statistics, online learning, and forecast evaluation, offering a tool for continuous, interpretable monitoring of probabilistic forecasts in non-stationary environments.