🤖 machine learning

RareCP: Regime-Aware Retrieval for Efficient Conformal Prediction

RareCP is a regime-aware retrieval method for time series forecasting that enhances conformal prediction efficiency by using mixture-of-experts to distinguish distinct error regimes and a drift-adaptive hypernetwork to retrieve relevant calibration examples for generating reliable, asymmetric prediction intervals.

Original authors: Manuel Heurich, Maximilian Granz, Tim Landgraf

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Manuel Heurich, Maximilian Granz, Tim Landgraf

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather for tomorrow. You have a super-smart computer model that says, "It will be 75°F." That's a point prediction. But in the real world, you need to know the range of possibilities. Will it be 70–80°F? Or could it be a wild 60–90°F?

This range is called a prediction interval. If the interval is too wide, it's not very helpful (it's like saying "it might rain or it might not"). If it's too narrow, you might get caught in the rain without an umbrella.

The paper introduces a new tool called RareCP (Regime-Aware Retrieval for Efficient Conformal Prediction). Its job is to take any existing weather model (or stock predictor, or traffic forecaster) and give it a much smarter, more efficient way to draw those prediction ranges.

Here is how RareCP works, broken down into simple concepts:

1. The Problem: "One Size Does Not Fit All"

Standard methods for drawing these ranges often act like a one-size-fits-all raincoat. They assume that the "mistakes" a model makes are the same all the time.

The Reality: Time series data (like weather or stock prices) is messy. Sometimes the model is very accurate (calm days), and sometimes it makes huge mistakes (stormy days). Sometimes the errors drift slowly over time; other times, the world suddenly switches to a completely different "mode" (a regime), like a sudden heatwave or a market crash.
The Flaw: Old methods either react too slowly to these changes or just make the raincoat bigger and bigger to be safe, which makes the prediction useless.

2. The Solution: A "Smart Librarian" with a "Chameleon"

RareCP acts like a smart librarian who doesn't just look at the current day, but goes back into the archives to find days that feel exactly like today.

The "Regime-Aware" Librarian: Instead of looking at all past days, RareCP knows that there are different "types" of days. It has a team of specialist librarians (called "experts").
- One specialist only looks at "Calm, Sunny Days."
- Another only looks at "Chaotic, Stormy Days."
- A third looks at "Slowly Drifting Seasons."
- When a new forecast comes in, a "Gatekeeper" decides which specialist is most relevant right now and asks them for help.
The "Retrieval" (Finding the Twins): Once the right specialist is chosen, RareCP searches the past for the top-k most similar days (the "twins"). It doesn't just grab random past data; it grabs the specific days where the model made mistakes that look exactly like the mistakes it might make today.
The "Chameleon" (Hypernetwork): The world changes. A "Calm Day" from 10 years ago might look different than a "Calm Day" today. RareCP uses a chameleon-like mechanism (a hypernetwork) that constantly reshapes its search criteria to match the current "vibe" of the data. It ensures the librarian is looking for the right kind of similarity, not just the same old patterns.

3. How It Builds the Interval

Once RareCP finds these "twin days" from the past:

It looks at the actual errors (the difference between the prediction and reality) on those twin days.
It calculates the range where 80% (or 90%) of those past errors fell.
It applies that range to the current prediction.

The Result:

On calm days, the "twin" days were also calm, so the errors were small. RareCP draws a tight, narrow interval.
On chaotic days, the "twin" days were also chaotic, with big errors. RareCP draws a wide, safe interval.
It doesn't just make the interval wider everywhere; it adapts locally.

4. Why It's Better (The "Efficiency" Claim)

The paper tested RareCP on a massive benchmark called GIFT-Eval, which includes 22 different datasets (energy, traffic, finance, etc.).

The Analogy: Imagine two people trying to guess the weight of a pumpkin.
- Person A (Old Method): Always guesses "Between 5 and 50 pounds." They are right 80% of the time, but the range is huge and unhelpful.
- Person B (RareCP): Looks at the pumpkin's size, color, and stem. If it looks like a small pumpkin, they guess "5 to 8 pounds." If it looks huge, they guess "40 to 50 pounds." They are also right 80% of the time, but their guesses are much more precise.

The paper shows that RareCP achieves the same "safety" (coverage) as other advanced methods but produces much tighter, more useful intervals. It does this without needing to retrain the main forecasting model; it just sits on top like a smart adapter.

Summary of Claims

It works with any model: You can use it with simple statistical models (ARIMA) or massive AI foundation models.
It handles change: It adapts to slow drifts and sudden shifts in data behavior.
It's efficient: It doesn't waste space on wide intervals when the data is predictable.
It's proven: In tests, it beat other top methods (like ACI, HopCPT, and ResCP) in making intervals that were both accurate and narrow.

In short, RareCP is a context-aware safety net that knows when to be tight and when to be loose, ensuring you get the most precise prediction possible without losing your safety guarantee.

Technical Summary: RareCP – Regime-Aware Retrieval for Efficient Conformal Prediction

Problem Statement

Time series forecasting requires reliable uncertainty quantification (UQ) for risk-aware decision-making. While Conformal Prediction (CP) offers a model-agnostic framework with finite-sample coverage guarantees under the assumption of exchangeability, this assumption is fundamentally violated by the sequential nature of time series data. Existing adaptations to handle temporal dependence and drift generally fall into two categories: (1) adaptive miscoverage methods that adjust the target coverage rate over time, and (2) weighted CP approaches that assign calibration weights based on sample relevance.

The authors argue that these existing methods fail to explicitly separate two central sources of nonstationarity in forecasting errors:

Smoothly drifting error distributions: Where residual similarities evolve as the underlying process changes.
Co-existing distinct error regimes: Where residual distributions switch between distinct contexts rather than varying smoothly along a single trajectory.

Standard methods often react to these issues by uniformly inflating intervals or relying on fixed similarity weighting, leading to inefficiency (overly wide intervals) or poor coverage during regime shifts.

Methodology: RareCP

RareCP (Regime-Aware Retrieval for Conformal Prediction) is introduced as a backbone-agnostic adapter for fixed forecasting models. It calibrates prediction intervals by retrieving past signed residuals from locally relevant error regimes. The architecture consists of three primary components:

1. Local Signed-Residual Calibration

Unlike methods using absolute residuals, RareCP calibrates signed residuals ( $r_i = y_i - \hat{y}_i$ ). This allows the lower and upper tails of the prediction interval to adapt separately, enabling asymmetric intervals. Given a query context $a_t$ , RareCP forms a weighted empirical cumulative distribution function (CDF) of residuals, $\hat{F}_t(\rho)$ , and extracts quantiles to define the interval $C_t = [\hat{y}_t + Q_t(\alpha/2), \hat{y}_t + Q_t(1-\alpha/2)]$ .

2. Regime-Aware Top- $k$ Retrieval

To address the "regime structure" of errors, RareCP employs a Mixture of Experts (MoE) retrieval mechanism:

Hypernetwork-Conditioned Experts: Several low-capacity retrieval experts are trained. Each expert uses a hypernetwork that takes the current query context ( $a_t$ ) and a dataset descriptor ( $\eta_d$ , containing dataset ID and statistics) to output an affine retrieval map. This allows the similarity metric to be specific to both the current time step and the dataset.
Cosine-Attention Similarity: Experts normalize their mapped keys and queries to the unit sphere, using cosine similarity to retrieve the top- $k$ most relevant calibration residuals.
Sparse Retrieval: By restricting the support to the top- $k$ neighbors, the method enforces locality, reducing the mass assigned to irrelevant regimes.
Gate Mixing: A lightweight gate network produces simplex weights to mix the local CDFs constructed by each expert. This allows the system to switch between clustered error regimes dynamically.

3. Training and Optimization

Objective: The model is trained to minimize a smooth relaxation of the Winkler interval score.
Teacher Anchoring: To prevent the learned retrieval maps from collapsing or becoming unstable, each student hypernetwork is anchored to a lightweight, pre-fitted teacher map via a parameter-space distillation loss.
Leave-One-Out (LOO): Training utilizes LOO calibration episodes where the query target is excluded from the retrieval set, mirroring the test-time scenario.
Coverage Correction: At test time, RareCP integrates with Adaptive Conformal Inference (ACI) to adjust the target miscoverage level $\alpha_t$ online, correcting for drift.

Key Contributions

Competitive Performance with Backbones: RareCP demonstrates that conformal prediction, when equipped with this retrieval mechanism, is competitive with native quantile-based prediction intervals of state-of-the-art time series foundation models (FMs), while remaining applicable to any point forecaster (including ARIMA) that lacks native UQ.
Efficiency-Coverage Trade-off via Retrieval: The paper introduces sparse residual retrieval for conformal calibration. It analyzes the trade-off controlled by $K_n$ (the number of retrieved residuals): smaller $K_n$ improves efficiency by focusing on local behavior, while larger $K_n$ supports asymptotic consistency.
Architecture for Structural Biases: The proposed linear cosine-attention retrieval architecture is explicitly designed to handle time-drifting similarities and clustered error regimes, avoiding the need for unconstrained calibration weights.

Experimental Results

The authors evaluated RareCP on the GIFT-Eval benchmark, covering 10 core datasets (Bench10) and a full suite of 22 datasets (Bench22) across energy, transport, climate, and finance domains.

Baselines: Comparisons included standard CP methods (SplitCP, ACI, HopCPT, ResCP, KOWCPI) and native UQ heads of foundation models (TiRex, Chronos, Moirai, TimesFM, etc.).
Performance: RareCP achieved the lowest average normalized Winkler score (nWink) across all four tested backbones (TiRex, TimesFM-2.5, TabPFN-TS, ARIMA) on Bench10. On Bench22, it remained competitive, trailing only ACI on one specific backbone.
Scaling: Performance gains increased with larger calibration streams (100k and full-series tests), suggesting that richer regime structures benefit more from retrieval-based calibration than uniform smoothing.
Ablations: The study confirmed that each component contributes to performance:
- Learned dense similarity improved nWink by ~13.9%.
- Adding hard top- $k$ retrieval added ~4.6%.
- Hypernetwork conditioning added ~1.7%.
- Regime mixtures (MoE) added ~2.1%.
- Teacher anchoring and ACI correction provided smaller but consistent gains.
Coverage: RareCP maintained empirical coverage close to the target (80%) across most datasets, whereas some baselines (e.g., KOWCPI) achieved narrow intervals at the cost of significant undercoverage.

Significance and Limitations

Significance:
The paper posits that conformal calibration for time series benefits from modeling drift and residual-regime structure separately. RareCP provides a retrieval-based mechanism to improve interval efficiency without sacrificing coverage guarantees, effectively bridging the gap between the flexibility of deep learning and the rigor of conformal prediction. It is the first method to amortize conformal weighting rule parameters using a hypernetwork with teacher-student distillation in parameter space.

Limitations:

Theoretical Scope: The theoretical analysis assumes a clean setting where the retrieval rule is fixed before calibration or learned from data distinct from calibration residuals. It does not fully resolve the finite-sample trade-off between learning a strong retrieval rule and reserving data for calibration.
Heterogeneous Drift: While effective on Bench10, coverage was weaker on the more heterogeneous Bench22 split, indicating that sparse or abrupt drift remains challenging.
Computational Cost: RareCP has a higher offline training cost compared to simpler conformal baselines due to the training of hypernetworks and the teacher anchoring process.

The authors conclude that future work should explore learning-calibration splits at larger scales and reducing training costs through cached embeddings or approximate retrieval, while preserving the core principle of calibrating intervals using residuals from the same local error regime.