Regime-Aware Specialist Routing for Volatility Forecasting

This paper introduces a regime-aware specialist routing framework that dynamically combines different forecasting models based on market conditions, significantly reducing high-volatility and underprediction errors compared to a static rolling-best baseline.

The Gist

Imagine you are the captain of a ship navigating the ocean. Your goal is to predict the weather for tomorrow to keep your cargo safe.

In the world of finance, this "weather" is volatility (how wildly stock prices are swinging). If you predict the weather wrong, you might lose money or miss a great opportunity.

For a long time, financial experts tried to use one single "Super Captain" (a single mathematical model) to predict the weather for every situation. They hoped this one model would be perfect whether the ocean was calm and sunny or stormy and terrifying.

The Problem:
Just like a real captain, no single model is perfect at everything.

• A model that is great at predicting calm, sunny days often gets confused and panics when a hurricane hits.
• A model designed to survive a hurricane might be overly cautious and miss the fun on a calm day.
• When the market switches from "calm" to "chaos," the old single-model approach often fails, leading to bad predictions exactly when you need them most.

The Solution: The "Specialist Routing" Team

Tenghan Zhong's paper proposes a smarter way to captain the ship. Instead of relying on one person, they built a dynamic team of specialists with a smart manager (the "Router").

Here is how the system works, broken down into simple steps:

1. The Team of Specialists

The system doesn't just have one model; it has a pool of different experts, each with a different personality:

• The "Calm" Experts: These are good at predicting smooth sailing. They are relaxed and efficient when the market is quiet.
• The "Storm" Experts: These are built for chaos. They are tough, conservative, and good at predicting big waves and crashes.

2. The Smart Manager (The Router)

This is the magic part. The system has a "Manager" that constantly checks the ocean conditions (the market state).

• Is it a calm day? The Manager looks at the team and says, "Okay, let's listen mostly to the Calm Experts."
• Is a storm brewing? The Manager immediately switches gears: "Drop the Calm Experts! We need the Storm Experts right now!"

The Manager doesn't just guess; it uses a risk-sensitive score. It asks: "Who made the fewest mistakes recently, especially when things were scary?" If a model keeps underestimating the danger (predicting calm when it's actually stormy), the Manager penalizes it heavily.

3. The "Hybrid" Forecast

The system doesn't just pick one winner. It creates a blend.

• If the market is 80% calm and 20% stormy, the final prediction is 80% "Calm Expert" advice and 20% "Storm Expert" advice.
• This ensures that even if the market suddenly turns, the prediction doesn't crash because it already had a little bit of "storm safety" built in.

4. The Safety Net

Finally, the system has a "floor." If the prediction gets too weird or the market gets too crazy, it forces the prediction to stay above a certain safety level (based on a standard, reliable model). This prevents the system from predicting that the ocean will be perfectly flat when a tsunami is coming.

The Results: Why It Matters

The author tested this system on six different types of investments (like Gold, Tech stocks, and Bonds) over several years.

• The Old Way (Single Model): Often failed miserably during market crashes, underestimating the danger.
• The New Way (Specialist Routing):
  • In Calm Markets: It performed just as well as the best single models.
  • In Stormy Markets: It was a game-changer. It reduced the "missed danger" errors by about 22% and the overall prediction errors by 24% compared to the old methods.

The Big Takeaway

Think of this like a smart thermostat for your house.

• An old thermostat just turns the heat on or off based on a single rule.
• This new system is like a smart home that knows: "When it's summer, use the AC. When it's winter, use the heater. And if a blizzard hits, switch to emergency heating mode immediately."

The paper proves that in finance, adaptability is better than rigidity. By routing different experts to the right market conditions, you can navigate both calm seas and violent storms much more safely.

Technical Summary

1. Problem Statement

Volatility forecasting is critical for risk management and portfolio allocation but suffers from instability when market conditions shift. A model that performs well in calm markets often deteriorates during stress, while models designed for turbulent markets may be overly conservative in quiet periods.

• Core Challenge: No single forecasting model (econometric or machine learning) dominates across all market regimes.
• Gap: Existing methods often rely on a single "best" model, static averaging, or simple regime-switching rules that fail to adapt dynamically to the specific characteristics of the current market state.

2. Methodology: Regime-Aware Specialist Routing Framework

The authors propose an online, adaptive framework that combines multiple forecasting specialists based on real-time market conditions. The architecture consists of four distinct stages:

A. Input Construction & Model Pool

• Target: Next-day realized variance, proxied by the squared Garman–Klass volatility proxy.
• Inputs: A combination of ETF-specific data (lagged realized variance, returns) and macro-financial variables (VIX, yield curves, credit spreads).
• Model Pool: A diverse set of five forecasters:
  • Econometric: HAR-RV, GARCH-t, FIGARCH.
  • Machine Learning: GRU (Gated Recurrent Unit), XGBoost.

B. Online Risk-Sensitive Scoring

Instead of standard error metrics, the framework uses a risk-sensitive loss function to evaluate models:

• Loss Function: $L = \text{QLIKE} + \lambda_{\text{under}} \times U$
  • QLIKE: Quasi-likelihood loss (standard for volatility).
  • Underprediction Penalty ($U$): A specific term that heavily penalizes forecasts where the realized variance exceeds the forecast ($y > \hat{y}$). This is crucial for risk management to avoid underestimating risk.
• Scoring: Models are scored based on their excess loss relative to the best active model over a rolling window (252 days).
• Weighting: Scores are weighted by time decay and regime similarity. Models that performed well in market states similar to the current one receive higher weights.

C. State-Dependent Specialist Routing

The framework does not select a single winner but maintains a Routing Set of competitive models:

• Thresholding: A dynamic threshold $\tau_t$ determines which models enter the routing set. This threshold balances global performance with local (regime-specific) performance.
• Specialist Branches: The routing set feeds into two specialized branches:
  • Calm Branch: Prioritizes ML models (GRU, XGBoost) and HAR-RV.
  • Stress Branch: Prioritizes econometric models (GARCH-t, FIGARCH) and HAR-RV.
• Branch Aggregation:
  • Calm: Uses the median of routed forecasts.
  • Stress: Uses the 75th percentile (after winsorizing) to be more conservative.

D. Final Forecast Combination

The final forecast is a weighted combination of the two branches and a robust baseline:

1. Stress Score ($p_t$): A logistic transform of the market-state vector determines the weight given to the stress branch.
2. Two-Stage Gating:
   • A "Low-State" branch blends the Rolling-Best benchmark with the Calm branch.
   • A "High-State" branch blends the Combined forecast with the Stress branch.
   • A final gate ($\omega_t$) blends these two based on the stress score.
3. Safety Floor: A conditional HAR-RV floor is applied. If the stress score is high or forecast dispersion is large, the final forecast is capped at the HAR-RV forecast to prevent catastrophic underprediction.

3. Key Contributions

1. Risk-Sensitive Routing: Introduces a scoring mechanism that explicitly penalizes underprediction, aligning the model selection process with risk management objectives.
2. Empirical Evidence of Regime Dependence: Demonstrates that the "best" forecaster is not global but strictly regime-dependent (e.g., GRU dominates in calm markets; GARCH-t/FIGARCH dominate in stressed markets).
3. Adaptive Architecture: Proposes a practical framework that dynamically routes specialists to appropriate market states rather than relying on a single static model or simple averaging.

4. Experimental Results

The framework was tested on a daily panel of six ETFs (SPY, QQQ, IWM, EEM, GLD, TLT) from 2015 to 2025 using a rolling walk-forward design.

• Performance in High Volatility:
  • The proposed framework reduced high-volatility forecast loss (QLIKE) by ~24% compared to the Rolling-Best baseline.
  • It reduced underprediction loss by ~22% in high-volatility regimes.
  • Specifically for the TLT (long-term Treasuries) ETF, where baseline methods failed catastrophically (QLIKE > 100), the proposed model remained stable (~0.32), demonstrating superior robustness.
• Overall Performance:
  • The framework achieved the lowest median QLIKE (0.323) across all assets compared to single models and baselines.
  • While it incurred a slight cost in low-volatility regimes (trading off some calm-market precision), the gains in stressed regimes were substantial.
• Routing Diagnostics:
  • The system successfully identified that no single model is always best. The "Miss-Best Rate" (failing to include the absolute best model on a given day) was high (67%), but the Selected Regret (loss gap relative to the best active model) was low (0.21). This indicates the system successfully retains a set of competitive models rather than gambling on a single winner.
• Statistical Significance:
  • Diebold-Mariano tests showed the proposed model significantly outperformed the naive VIX-switch baseline across 5 of 6 assets.
  • It outperformed the Rolling-Best baseline significantly on IWM and SPY.

5. Significance and Conclusion

The paper argues that specialist routing is a superior strategy for financial forecasting in changing environments.

• Practical Impact: The framework provides a robust solution for risk management by ensuring that underprediction (a critical failure mode in finance) is minimized during market stress.
• Theoretical Insight: It validates the hypothesis that model performance is non-stationary and regime-dependent, necessitating adaptive architectures over static model selection.
• Future Work: The authors suggest extending the framework to alternative volatility targets, refining regime definitions, and incorporating more granular cross-sectional stress indicators.

In summary, the paper presents a sophisticated, risk-aware ensemble method that dynamically switches between machine learning and econometric specialists, proving that adaptability to market regimes is more valuable than seeking a single "global best" model.