🤖 machine learning

Online Irregular Multivariate Time Series Forecasting via Uncertainty-Driven Dual-Expert Calibration

This paper proposes Under-Cali, an uncertainty-driven dual-expert calibration framework that enables efficient and stable online forecasting for irregular multivariate time series by dynamically routing samples between reliable and unreliable experts based on estimated uncertainty, all while keeping the source model frozen.

Original authors: Haonan Wen, Hanyang Chen, Songhe Feng

Published 2026-05-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Haonan Wen, Hanyang Chen, Songhe Feng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a weather forecaster who has spent years studying the climate of a specific city. You have built a super-smart computer model that predicts the weather perfectly based on the data you've collected over the last decade. This is your "Source Forecaster."

Now, imagine you are sent to a new city. The weather patterns are slightly different, the sensors are placed in different spots, and sometimes the sensors go missing for hours at a time. If you just use your old model, it will start making mistakes because the "rules" of the weather have changed. This is the problem of distribution shift.

Most existing online learning methods are like a student who tries to relearn the entire subject every time they see a new fact. They get confused, forget what they knew before, and make a mess. This paper, Under-Cali, proposes a smarter way to handle this, specifically for messy, irregular data (like heart rate monitors that skip beats or traffic sensors that go offline).

Here is how Under-Cali works, explained through a simple analogy:

The Core Idea: The "Uncertainty-Driven" Team

Instead of trying to fix the main weather model (which is expensive and risky), Under-Cali adds a small, smart "calibration team" that sits in front of and behind the main model. This team has three parts:

1. The Uncertainty Estimator (The "Sniffer Dog")

Before the main model makes a prediction, this module acts like a sniffer dog. It looks at the new data and asks: "Does this look like the old data I know, or is it something weird and new?"

How it works: It calculates a "worry score" (uncertainty). If the data looks familiar, the score is low. If the data is strange (maybe a sensor broke or the weather pattern changed), the score is high.
Why it matters: In messy data, you can't rely on "time" to tell you what's happening because the data arrives at random times. So, the system relies on how confused the model is to decide what to do next.

2. The Dual-Expert Calibration (The "Specialist Doctors")

This is the heart of the system. Instead of having one doctor treat every patient, Under-Cali has two specialists:

The Reliable Expert (The General Practitioner): This doctor handles the "low worry" cases—data that looks normal and familiar. They make tiny, careful adjustments to keep the predictions accurate. They are very stable and don't change their mind easily.
The Unreliable Expert (The Specialist for Rare Diseases): This doctor handles the "high worry" cases—data that is weird, missing, or looks like a distribution shift. They are allowed to take bigger risks and learn from these difficult examples.

The Magic Trick: The system keeps these two doctors in separate rooms. The "weird" data goes to the Specialist, and the "normal" data goes to the General Practitioner. This prevents the weird, noisy data from confusing the General Practitioner and ruining their ability to predict normal weather.

3. The Adaptive Routing Module (The "Traffic Cop")

This module directs the traffic based on the "worry score" from the Sniffer Dog.

Low Worry? The Traffic Cop sends the data to the Reliable Expert.
High Worry? The Traffic Cop sends the data to the Unreliable Expert.
When to Update? The Traffic Cop also decides when to let the doctors study new cases. It doesn't let them update every single second (which would be chaotic). It waits until the "worry score" for a whole batch of data gets high enough to justify a lesson. This ensures the system learns efficiently without getting overwhelmed.

Why is this better than what we had before?

No "Catastrophic Forgetting": Old online learning methods often try to learn new things so hard that they forget the old things. Under-Cali protects the "Reliable Expert" from the messy data, so it keeps doing a great job on the normal stuff.
Handles "Messy" Data: Regular time series data is like a perfect clock ticking every second. Irregular data (like this paper focuses on) is like a clock that ticks at random times and sometimes skips hours. Under-Cali is specifically designed to handle these gaps and skips without breaking.
Lightweight: It doesn't need to retrain the giant main model. It just updates the small "calibration team," making it fast and cheap to run.

The Results

The authors tested this on real-world messy data, like:

Medical data: Heart monitors and ICU records (where sensors often drop out).
Human movement: 3D motion capture data.
Climate data: Weather records with missing days.

They found that when they added Under-Cali to existing forecasting models, the predictions got significantly better, especially when the data changed suddenly. It was like giving a tired forecaster a team of specialists who could instantly adapt to new conditions without losing their previous knowledge.

In short, Under-Cali is a smart, modular add-on that helps AI models stay accurate even when the world changes, the data gets messy, and the rules shift unexpectedly. It does this by listening to its own "gut feeling" (uncertainty) and assigning the right "expert" to handle the job.

Technical Summary: Online Irregular Multivariate Time Series Forecasting via Uncertainty-Driven Dual-Expert Calibration

Problem Definition
The paper addresses the challenge of Online Irregular Multivariate Time Series (IMTS) Forecasting. While significant progress has been made in offline IMTS forecasting using continuous-time models and attention mechanisms, these methods typically freeze parameters after training, rendering them unable to adapt to evolving data distributions. In real-world scenarios (e.g., healthcare, climate, finance), data distributions shift due to changes in subjects, devices, or environments. Existing online learning techniques for time series largely rely on dense temporal continuity and periodicity, assumptions that break down in IMTS where sampling is irregular, observation intervals are variable, and missingness patterns are non-stationary and informative. Consequently, directly applying existing online adaptation methods to IMTS leads to performance degradation. The authors formally define the problem of online IMTS forecasting, which requires models to generate sequential predictions, adapt to distribution shifts, and handle irregular sampling and dynamic missingness simultaneously.

Methodology: Under-Cali
The authors propose Under-Cali, an uncertainty-driven dual-expert calibration framework designed for efficient and stable online adaptation without modifying the source forecasting model. The framework consists of three core components:

Uncertainty Estimator (UE):
- Instead of relying on temporal continuity, the UE quantifies forecasting uncertainty as a proxy for distribution shift.
- It is defined operationally as the normalized prediction error (Mean Squared Error) under the current data distribution, rather than a Bayesian confidence estimate.
- The UE is a neural network ( $g_\phi$ ) trained offline to map input time series and predictions to an uncertainty score ( $\hat{u} \in [0,1]$ ). During online adaptation, it uses calibrated predictions from the "reliable expert" to compute scores, ensuring stability.
Dual-Expert Gated Distribution Calibrator (GDC):
- To prevent unstable adaptation, the GDC employs two isolated experts with separate parameters: a Reliable Expert ( $e_{\psi r}$ ) and an Unreliable Expert ( $e_{\psi ur}$ ).
- Reliable Expert: Handles low-uncertainty samples (familiar patterns) with fine-grained adjustments.
- Unreliable Expert: Handles high-uncertainty samples (anomalies or significant shifts) with cautious adaptation to prevent overfitting or corrupting stable patterns.
- Both experts utilize a variable-wise, gated temporal calibration model consisting of an input calibrator and an output calibrator. They are initialized to approximate identity mappings (zero-initialized weights) to preserve the source forecaster's original knowledge until shifts occur.
Adaptive Routing Module (ARM):
- The ARM governs sample allocation and adaptation triggering using two dynamic thresholds derived from Exponential Moving Averages (EMA) of uncertainty statistics:
  - Allocation Threshold ( $\tau_{alloc}$ ): Routes samples to either the reliable or unreliable expert based on their estimated uncertainty score.
  - Trigger Threshold ( $\tau_{trig}$ ): Determines whether an adaptation step is necessary for the current batch. Adaptation is triggered only if the batch's mean uncertainty exceeds this threshold, enabling sparse updates.
- Online Adaptation Process:
  - Inference: Samples are initially processed by the reliable expert. Uncertainty scores are computed. High-uncertainty samples are routed to the unreliable expert for secondary calibration.
  - Adaptation: If triggered, the reliable expert and UE are updated using low-uncertainty (reliable) samples, while the unreliable expert is updated using high-uncertainty (challenging) samples. This separation prevents catastrophic interference.

Key Contributions

Problem Definition: The paper is the first to formally define the problem of Online Irregular Multivariate Time Series Forecasting, distinguishing it from existing regular time series settings by highlighting the challenges of irregular sampling and dynamic missingness.
Framework Design: Introduction of Under-Cali, a model-agnostic framework that enables principled online adaptation. It features a dual-expert gated calibrator that performs differentiated calibration based on sample uncertainty.
Adaptive Mechanism: Development of an Adaptive Routing Module that autonomously triggers updates and allocates samples to distinct experts, ensuring stable online forecasting without modifying the source model.
Efficiency: The framework updates only a small fraction of parameters (the calibration modules and UE), keeping the source forecaster frozen, which ensures resource efficiency.

Experimental Results
The authors evaluated Under-Cali on four IMTS benchmarks (MIMIC, PhysioNet, Human Activity, USHCN) across 21 different source forecasters (including SOTA methods like tPatchGNN, HyperIMTS, mTAN, and Transformers).

Performance: Under-Cali consistently improved forecasting performance (measured by MSE) across all architectures and datasets. Notable improvements included an 8.41% MSE reduction for tPatchGNN and a 21.84% reduction for Warpformer on the Human Activity dataset.
Robustness: The method demonstrated superior stability in long-horizon forecasting and across varying lookback lengths compared to non-adaptive baselines.
Comparison: Under-Cali outperformed existing online learning and Test-Time Adaptation (TTA) methods (e.g., OneNet, FSNet, D3A, TAFAS). Notably, methods designed for regular time series (like D3A and TAFAS) often degraded performance on IMTS, whereas Under-Cali maintained or improved accuracy.
Efficiency: The framework introduced only marginal increases in GPU memory footprint and maintained inference latency within acceptable ranges (0.1–0.4s per batch), validating its practicality for deployment.
Ablation: Experiments confirmed that all components (UE, Dual-Expert GDC, and ARM) are necessary; removing any component or using random routing/triggers led to suboptimal performance.

Significance
The paper claims that Under-Cali provides a robust, model-agnostic solution to the critical gap in online IMTS forecasting. By leveraging uncertainty as a control signal and isolating calibration pathways for reliable and unreliable samples, the framework achieves stable adaptation in dynamic environments where traditional temporal assumptions fail. It enables existing offline IMTS models to continuously assimilate new data distributions without the computational cost of full retraining or the instability of direct fine-tuning. The work establishes a new standard for handling distribution shifts in irregular time series applications.