Technical Summary: Leave a Window Out (LWO) for Predictive Inference in Time Series

Problem Statement

Conformal prediction methods provide rigorous predictive inference (valid prediction intervals) under the assumption of data exchangeability and memoryless predictors. However, these assumptions are frequently violated in time-series settings due to temporal dependence and the necessity of using memory-based predictors (which utilize past observations to forecast future values).

While recent work has demonstrated that split conformal prediction is robust to these violations, it suffers from a "split tax": partitioning data into training and calibration sets reduces the effective sample size, leading to wider prediction intervals and lower statistical efficiency, particularly in data-scarce regimes.

Conversely, leave-one-out (jackknife) methods utilize the full dataset, offering greater statistical efficiency. However, the theoretical validity of the vanilla jackknife relies on exchangeability. This paper investigates whether the vanilla jackknife remains reliable under temporal dependence. Through numerical experiments on moving average (MA) processes, the authors demonstrate that the vanilla jackknife can suffer arbitrary loss of coverage (severe undercoverage) even in simple time-series models with mild dependence. This occurs because the leave-one-out calibration creates an asymmetry: proxy test points in the training set have "two-sided" information (past and future), whereas the actual test point at the end of the sequence only has "one-sided" information (past), leading to miscalibrated intervals.

Methodology: Leave-a-Window-Out (LWO)

To remedy the failure of the vanilla jackknife while retaining its data efficiency, the authors propose the Leave-a-Window-Out (LWO) method.

Core Mechanism

The LWO method modifies the standard jackknife calibration procedure to account for temporal dependence:

1. Training with a Gap: For each index $k$ in the training set, the predictor $\hat{f}_k$ is trained not just by removing the $k$-th point, but by removing a window of data points from index $k$ to $\min\{k+\tau, n\}$, where $\tau$ is a hyperparameter representing the window size.
2. Score Computation: The conformity score $s_k$ is computed on the $k$-th point using the predictor trained on the remaining data (which includes a "pre-window" and a "post-window" component).
3. Prediction Set: The final prediction set for the test point $X_{n+1}$ is constructed by centering on the predictor trained on all $n$ points and inflating it by the $(1-\alpha)$-quantile of the windowed leave-out scores.

Theoretical Framework

The validity of LWO is established through a novel theoretical framework based on cyclic exchangeability:

• Cyclic Embedding: The authors introduce the concept of embedding a dependent time series into a longer, cyclically exchangeable sequence. They define the cyclic embedding coefficient ($\rho_\tau$) to quantify the distance between the actual time series and the closest cyclically exchangeable sequence.
• Stability Assumptions: The proofs rely on an out-of-sample (OOS) stability condition. Specifically, the pair of the conformity score function and the training algorithm must be stable when a random block of training data is masked (removed). This is weaker than uniform stability but sufficient for the proposed bounds.
• Mixing Conditions: The authors relate the cyclic embedding coefficient to standard dependence measures, such as $\beta$-mixing coefficients and $\beta$-conditional-mixing coefficients. They show that for mixing processes (including approximately Markovian ones), the cyclic embedding coefficient is small if the window size $\tau$ exceeds the effective dependence range.

Key Contributions

1. Identification of Jackknife Failure: The paper rigorously demonstrates that the vanilla jackknife fails to provide valid coverage in time-series settings due to the asymmetry between training proxy points and the test point, even under mild dependence.
2. Proposal of LWO: A new method, LWO, is introduced that modifies the leave-one-out principle by removing a window of future data relative to the proxy test point. This restores the information structure of the calibration scores to match the test point.
3. Theoretical Guarantees:
   • Theorem 1 & 2: Finite-sample coverage bounds are derived for LWO. These bounds depend on the cyclic embedding coefficient and the OOS stability parameters.
   • Corollaries: The bounds are specialized for mixing processes (Corollary 1 for approximately Markovian processes; Corollary 2 for general mixing processes), showing that valid coverage is achievable if the window size $\tau$ is chosen appropriately relative to the mixing time.
4. New Coefficients: The paper introduces the cyclic embedding coefficient and utilizes switch coefficients (adapted from Barber and Pananjady, 2026) to measure departures from exchangeability in a cyclic context.

Experimental Results

The authors evaluate LWO against Split Conformal Prediction (Split CP) and the vanilla Jackknife (JK) on both synthetic and real-world datasets:

• Synthetic Data (MA(1) Process): In multidimensional moving average processes, the vanilla jackknife severely undercovers (often dropping below 70% coverage for a 90% target), especially in higher dimensions. Split CP maintains nominal coverage but produces significantly wider prediction intervals. LWO achieves coverage close to the nominal level (comparable to Split CP) while producing prediction regions that are substantially narrower (more efficient).
• Real Data (Traffic and Solar Energy): On real-world time-series benchmarks, LWO consistently maintains near-nominal coverage across various base predictors (Ridge, KNN, MLP, Decision Trees, Kernel Regression). In contrast, the vanilla jackknife fails to cover for specific predictor-data combinations (e.g., KNN on Solar Energy data). LWO avoids the efficiency loss of Split CP while guarding against the undercoverage of the vanilla jackknife.

Significance and Claims

The paper claims that LWO offers a principled solution to the trade-off between validity and efficiency in predictive inference for time series.

• Validity: Unlike the vanilla jackknife, LWO provides provable approximate coverage guarantees under natural dependence assumptions (mixing) and stability conditions.
• Efficiency: Unlike split conformal prediction, LWO utilizes the entire dataset for training, resulting in tighter prediction intervals.
• Conceptual Advance: The work introduces a new perspective on analyzing dependent data by embedding it into cyclically exchangeable sequences, providing a distributional tool (the cyclic embedding coefficient) that connects time-series dependence to conformal validity.

The authors remain modest regarding open questions, noting that their guarantees rely on OOS stability assumptions (which are satisfied by many standard algorithms like ridge regression and KNN, but not all). They also acknowledge that while theory suggests $\tau$ should exceed the dependence range, a fully data-driven method for selecting $\tau$ with valid coverage guarantees remains an open problem.