Technical Summary: Relevance-Aware Thresholding in Online Conformal Prediction for Time Series

Problem Statement

Machine learning applications in sensitive domains (health, finance, etc.) increasingly require robust uncertainty quantification. While Conformal Prediction (CP) offers distribution-free coverage guarantees, its classical formulation relies on the assumption of data exchangeability, which is frequently violated in time series data due to distribution shifts.

To address this, Online Conformal Prediction (OCP) methods have been developed to update prediction intervals dynamically based on observed errors. However, existing state-of-the-art OCP methods (such as PID and ECI) typically rely on a binary evaluation of prediction performance: whether the ground truth falls inside or outside the prediction interval. This binary approach discards valuable information regarding the magnitude of the error when the ground truth is outside the interval, or the degree of safety when it is inside. Consequently, these methods may produce overly conservative (wide) prediction intervals or suffer from abrupt threshold adjustments, leading to uninformative outputs.

Methodology

The paper proposes a Relevance-Aware Thresholding framework that replaces the binary error indicator with a continuous function quantifying the "relevance" of a prediction interval.

Core Concept

Instead of using the indicator function $1\{Y_t \notin \hat{C}_t\}$, the authors introduce a family of customizable functions, $f_{\omega, v, \mu_t}(s_t - q_t)$, which map the distance between the ground truth ($Y_t$) and the prediction bounds to a value in $[0, 1]$.

• Input: The quantity $s_t - q_t$, where $s_t$ is the non-conformity score (distance from prediction to ground truth) and $q_t$ is the current threshold.
• Behavior:
  • If the ground truth is far outside the interval ($s_t - q_t \gg 0$), the function approaches 1, signaling a need to significantly increase the threshold for the next step.
  • If the ground truth is far inside the interval ($s_t - q_t \ll 0$), the function approaches 0, signaling that the threshold can be decreased to tighten the interval.
  • If the ground truth is near the boundary, the function transitions smoothly around the target miscoverage level $\alpha$.

Function Design

The proposed function family is defined as:
$$f_{\omega,v,\mu_t}(s_t - q_t) = \sum_{i=1}^l \omega_i \cdot \sigma\left( \frac{v_i}{\mu_t} \cdot (s_t - q_t) - \ln\left(\frac{1-\alpha}{\alpha}\right) \right)$$
where:

• $\sigma$ is the sigmoid function.
• $\mu_t$ is a dynamic scaling factor (mean of recent absolute errors) ensuring scale independence.
• $\omega$ and $v$ are parameters allowing Decision Makers (DMs) to tune the function's shape (e.g., creating plateaus for specific error ranges) based on domain expertise.

Integration into OCP

The authors integrate these functions into two prominent OCP algorithms:

1. PID (Proportional-Integral-Derivative): The binary indicator in the update rule is replaced by the relevance function in various configurations (full replacement, saturation-only, or error-term-only).
2. ECI (Error-Correcting Inference): The smoothing term involving the gradient of a function is modified to use the gradient of the proposed relevance function.

Theoretical analysis (Theorems 1–3) demonstrates that under specific conditions regarding the saturation functions and learning rates, these modified updates preserve the long-run coverage guarantee (validity).

Key Contributions

1. Relevance-Aware Framework: Introduction of a general approach to leverage the distance between ground truth and interval bounds, moving beyond binary error tracking in OCP.
2. Customizable Function Family: A set of parameterized functions that allow DMs to tailor the penalty/reward mechanism for prediction errors, accommodating specific domain constraints and preferences.
3. Algorithmic Enhancements: Modification of state-of-the-art OCP methods (PID and ECI) to incorporate these functions, with theoretical proofs ensuring that validity (coverage) is maintained.
4. Empirical Validation: Demonstration that the modified methods achieve tighter prediction intervals (improved efficiency) while maintaining or improving coverage validity compared to original methods.

Experimental Results

The authors evaluated the modified PID and ECI methods on real-world datasets: Amazon, Google, and Microsoft stock prices, and Delhi temperature data.

• PID Results: On the Amazon dataset, the modified PID reduced the average interval width from 32.42 to 29.36 (AR regressor) while maintaining 90% coverage. On Google stock data, the modified PID significantly reduced average width (e.g., from 43.75 to 34.96 for AR) while maintaining coverage.
• ECI Results: The original ECI method struggled to achieve the target 90% coverage in the authors' experimental setup (achieving ~78-83%). The modified ECI successfully achieved the target 90% coverage across all datasets. While the modified ECI produced slightly wider intervals than the original ECI in some cases, the original method's narrower intervals were deemed unreliable due to under-coverage. In direct comparison with results reported in the original ECI paper, the modified method showed competitive or superior performance, particularly on the temperature dataset where it outperformed the original across all metrics.
• Stability: The use of relevance functions resulted in smoother threshold adjustments, preventing the abrupt changes often seen in binary-update methods.

Significance and Claims

The paper claims that by shifting from a binary error view to a relevance-based view, OCP methods can produce more informative (narrower) prediction intervals without sacrificing the critical requirement of coverage validity. This is particularly significant for time series applications where distribution shifts are common and decision-makers require robust uncertainty quantification.

The authors emphasize that this work contributes to the development of trustworthy AI for sensitive applications. They note that the approach allows for the incorporation of Decision Maker preferences through the parametrization of the relevance functions, bridging the gap between statistical guarantees and practical decision-making needs. The paper concludes modestly, suggesting future work should focus on optimizing hyperparameters and further investigating the impact of the scaling parameter $\mu_t$.