🤖 machine learning

Dynamic TMoE: A Drift-Aware Dynamic Mixture of Experts Framework for Non-Stationary Time Series Forecasting

The paper proposes Dynamic TMoE, a novel framework for non-stationary time series forecasting that dynamically manages heterogeneous expert pools and employs a temporal memory router to adaptively handle distribution shifts, achieving state-of-the-art performance across nine benchmarks.

Original authors: Jiawen Zhu, Shuhan Liu, Di Weng, Yingcai Wu

Published 2026-05-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiawen Zhu, Shuhan Liu, Di Weng, Yingcai Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather. If you used a single, rigid rulebook that said, "It is always sunny in July," you would fail miserably when a sudden storm hits or when the seasons change. This is the problem with many computer models that try to forecast time-based data (like stock prices, electricity usage, or traffic). They are built on the assumption that the future will look exactly like the past, but in the real world, data is non-stationary—it constantly shifts, drifts, and changes its behavior.

The paper introduces a new system called Dynamic TMoE to solve this. Here is how it works, using simple analogies:

1. The Problem: The "Static Team" vs. The "Chaos"

Most current forecasting models are like a static team of identical workers.

The Issue: If the job changes from "building a house" to "putting out a fire," a team of construction workers is useless. They can't adapt.
The Paper's View: Existing models have a fixed pool of "experts" (specialized algorithms) that never change. When the data suddenly shifts (a "regime shift"), these models get confused because they are trying to force old patterns onto new data.

2. The Solution: A "Living, Breathing" Team

Dynamic TMoE is like a dynamic, evolving task force that changes its members based on the situation. It has three main superpowers:

A. The "Drift Detector" (The Smoke Alarm)

Before the team can react, they need to know something has changed.

How it works: The system constantly compares the "old" data (what it knows) with the "new" data (what is happening right now). It uses a mathematical tool called MMD (Maximum Mean Discrepancy) to measure the distance between the two.
The Analogy: Think of this as a smoke alarm. If the air quality changes slightly (noise), it ignores it. But if the smoke suddenly gets thick (a real shift), the alarm goes off. This tells the system, "Hey, the rules have changed! We need a new strategy."

B. The "Evolving Expert Pool" (Hiring and Firing)

Once the alarm goes off, the system doesn't just panic; it reorganizes.

How it works: The system analyzes why the prediction failed. Was it a sudden spike? A new seasonal pattern? A weird fluctuation?
- Hiring: If the data shows a new trend (like a sudden heatwave), the system instantly creates a new expert specifically designed to handle heatwaves.
- Firing: If an expert hasn't been useful for a while, the system prunes (fires) them to save space and keep the team lean.
The Analogy: Imagine a restaurant kitchen. If the menu suddenly changes from "Italian" to "Sushi," the chef doesn't just keep cooking pasta. They hire a sushi chef and send the pasta chef home. The kitchen evolves to match the new demand.

C. The "Memory Router" (The Experienced Manager)

This is the most critical part. In older models, the person deciding which expert to use has no memory. They look at the current moment and guess, often flipping back and forth wildly between experts.

How it works: Dynamic TMoE uses a Temporal Memory Router. It remembers the history of decisions. It knows that if the data has been trending up for three days, it shouldn't suddenly switch to a "downward trend" expert just because of one noisy data point.
The Analogy: Think of a seasoned traffic controller. A new controller might switch lanes frantically every time a car moves. The experienced controller (our router) remembers the flow of traffic over the last hour and makes smooth, consistent decisions, preventing the system from getting "jittery."

3. The "Specialized Tools" (Heterogeneous Experts)

The experts in this system aren't all the same. They are built differently to handle different types of data shifts.

The Trend Expert: Good at seeing the big picture (like a long-term rise in temperature).
The Seasonality Expert: Good at spotting repeating cycles (like daily rush hour).
The Fluctuation Expert: Good at catching sudden, chaotic spikes (like a stock market crash).
The Analogy: Instead of giving every worker a hammer, this team has a hammer, a screwdriver, a wrench, and a saw. When the job changes, they pick the right tool for the specific problem.

The Results: Why It Matters

The authors tested this system on nine different real-world datasets (including electricity usage, traffic, and weather).

The Outcome: Dynamic TMoE beat all the previous "state-of-the-art" models.
The Numbers: It reduced prediction errors (MSE) by 10.4% and (MAE) by 7.8% compared to the best existing methods.
The Takeaway: By allowing the model to grow, shrink, and remember its history, it handles the chaos of the real world much better than rigid, static models.

In short, Dynamic TMoE is a forecasting system that doesn't just memorize the past; it learns, adapts, and reorganizes itself in real-time to handle a world that never stands still.

Technical Summary: Dynamic TMoE

Problem Statement

Time series forecasting is fundamental to critical decision-making systems in energy, healthcare, and finance. However, real-world signals are inherently non-stationary, characterized by continuous distribution shifts and evolving temporal dependencies. These shifts often manifest as regime changes, where a single time series transitions between distinct statistical distributions (e.g., from stable seasonality to volatile trends driven by external shocks).

Existing approaches struggle with this non-stationarity. Monolithic models often lack the flexibility to handle abrupt regime transitions. While Mixture-of-Experts (MoE) architectures offer a promising paradigm for decoupling complex drift patterns, current MoE-based time series models suffer from two primary limitations:

Temporal Rigidity: They rely on fixed expert pools and memoryless routing, failing to maintain temporal continuity or adapt to novel patterns emerging from severe distribution shifts.
Insufficient Specialization: They typically employ homogeneous experts, lacking the functional diversity required to disentangle distinct drift components (e.g., trend vs. seasonality), forcing the model to redundantly relearn entire shifted distributions.

Methodology: Dynamic TMoE

The authors propose Dynamic TMoE, a framework that unifies architectural evolution with temporal continuity. The system operates via a closed-loop Perception-Decision-Adaptation mechanism:

1. Perception: Distribution Shift Detection

The framework continuously monitors the divergence between historical and current data windows using Maximum Mean Discrepancy (MMD).

Mechanism: A kernel-based metric calculates the distance between distributions in a Reproducing Kernel Hilbert Space.
Adaptive Thresholding: Instead of a fixed value, the system uses a dynamic threshold ( $\epsilon = \mu_H + \lambda \cdot \sigma_H$ ) based on the mean and standard deviation of recent MMD scores to account for noise fluctuations.
Theoretical Basis: The paper provides a generalization bound proving that increasing MMD directly elevates the upper bound of target prediction risk, justifying the need for adaptation when MMD exceeds the threshold.

2. Decision: Temporal Memory Router

To address the "memoryless" limitation of conventional MoE, Dynamic TMoE employs a Temporal Memory Router.

Sequential State Modeling: The router utilizes a Gated Recurrent Unit (GRU) to maintain a hidden state ( $h_t$ ) that encapsulates historical routing context. This ensures that routing decisions are conditioned on the sequence of inputs rather than isolated patches, preventing erratic expert switching.
Anomaly State Repository: When a drift is detected, the router archives the corresponding hidden state. For recurring shifts, it retrieves these historical states to synthesize a context-aware reference prototype, accelerating adaptation.
Sparse Dispatching: A Top-k gating strategy activates only the top- $k$ experts, maintaining constant computational cost regardless of the pool size.

3. Adaptation: Evolvable Expert Manager & Heterogeneous Pool

The framework treats the expert pool as a dynamic system that evolves during the training phase.

Heterogeneous Expert Design: Unlike homogeneous pools, Dynamic TMoE employs functionally diverse experts tailored to specific temporal components:
- Trend Expert: Uses average pooling and MLP to capture global trends.
- Seasonality Expert: Operates in the frequency domain (FFT) to model periodic signals.
- Fluctuation Expert: Uses 1D causal convolutions to capture local volatility.
- Identity Expert: Preserves raw information.
Drift Pattern Profiler: Upon drift detection, this module analyzes prediction residuals to identify the dominant missing pattern (Trend, Seasonality, or Fluctuation) and instantiates a new expert of the matching type.
Post-Addition Alignment: New experts undergo a targeted alignment phase where existing parameters are frozen, and only the new expert and router head are fine-tuned on the drift data to prevent catastrophic forgetting.
Expert Pruning: An "Expert Usage Tracker" monitors routing weights and prunes underutilized experts to maintain efficiency and prevent overfitting.

Key Contributions

Framework Integration: The introduction of a framework combining heterogeneous experts with temporal memory-based routing, enabling specialized modeling of distinct temporal patterns while maintaining context-aware, temporally consistent selection.
Dynamic Capacity Scaling: An MMD-based drift detection mechanism that dynamically expands and prunes the expert pool, allowing model capacity to scale adaptively with distribution shifts.
State-of-the-Art Performance: Extensive evaluation on nine real-world benchmarks demonstrating significant improvements over competitive baselines.

Experimental Results

The model was evaluated on nine datasets (ETT, Electricity, Exchange, Traffic, Weather, ILI) covering domains from energy to healthcare.

Performance: Dynamic TMoE achieved state-of-the-art performance, ranking 1st in 11 out of 18 evaluation metrics and 2nd in 5 others.
Improvement: Compared to competitive baselines, the framework reduced MSE by 10.4% and MAE by 7.8% on average.
Comparison with MoE Baselines: Specifically against the leading MoE-based baseline (TFPS), Dynamic TMoE reduced average MSE by 5.9% and MAE by 3.6%.
Efficiency: While inference latency is slightly higher than memoryless MoE models due to the sequential nature of the GRU router, Dynamic TMoE demonstrates superior memory efficiency compared to other MoE-based models (e.g., DUET, FreqMoE) and avoids the parameter explosion seen in some baselines.

Significance and Claims

The paper claims that Dynamic TMoE addresses the fundamental structural rigidity of conventional MoE architectures. By treating the expert pool as an evolvable system rather than a static collection, the framework effectively handles the "inherent challenges of non-stationary forecasting."

The authors emphasize that the performance gains are driven by the synergy of architectural innovations (drift detection, memory routing, and heterogeneous specialization) rather than mere capacity scaling. Ablation studies confirm that removing the dynamic mechanism or using homogeneous experts leads to significant performance degradation. The work positions Dynamic TMoE as a robust solution for real-world scenarios where data distributions are not static, offering a path toward more adaptive and interpretable time series forecasting systems. Future work is suggested to explore test-time adaptation and parameter-efficient fine-tuning (PEFT) to further enhance scalability.