An Adaptive Model Selection Framework for Demand Forecasting under Horizon-Induced Degradation to Support Business Strategy and Operations

This paper introduces AHSIV, an adaptive framework that addresses horizon-induced model ranking instability in demand forecasting by integrating horizon-aware error metrics, structural demand classification, and multi-objective optimization to provide robust, operationally coherent model selection for heterogeneous business environments.

The Gist

Imagine you are the manager of a massive supermarket chain. Every day, you have to decide how much of every single product—from fresh avocados to winter coats—to order for the next few weeks or months.

If you order too much, you waste money on storage and food goes bad. If you order too little, customers leave empty-handed, and you lose sales. To make these decisions, you need a crystal ball (a forecasting model) to predict future demand.

The problem? No single crystal ball works perfectly for every situation. Sometimes a simple guess works best; other times, a complex AI is needed. And the "best" guess often changes depending on whether you are looking 1 day ahead or 12 days ahead.

This paper introduces a new, smart system called AHSIV (Adaptive Hybrid Selector for Intermittency and Variability) to solve this headache. Here is how it works, explained simply:

1. The Problem: The "One-Size-Fits-All" Trap

In the past, businesses often picked one forecasting method and stuck with it for everything. It's like trying to use a sledgehammer to fix a watch, a car, and a house.

• The Issue: A model that predicts daily milk sales perfectly might fail miserably at predicting the sales of rare, expensive spices (which sell only once a month).
• The Horizon Problem: Predicting next Tuesday is easy. Predicting next December is hard. As you look further into the future, errors tend to grow. Most systems ignore this, assuming a model that works for "today" will work for "next month." It usually doesn't.

2. The Solution: A "Smart Matchmaker"

The authors built a system (AHSIV) that acts like a super-smart matchmaker. Instead of forcing one model to do all the work, it looks at each product and asks:

• "Is this product popular and steady (like milk)?"
• "Is this product rare and unpredictable (like a specific holiday decoration)?"
• "How far into the future are we trying to predict?"

Based on the answers, it picks the perfect crystal ball for that specific job.

3. The Secret Sauce: The "Horizon Degradation" Filter

This is the paper's most creative innovation. The authors realized that time is a thief. The further you look into the future, the more "fog" appears, and your predictions get blurrier.

They created a rule called MDFH (Metric Degradation by Forecast Horizon).

• The Analogy: Imagine you are looking at a landscape through a telescope. If you look at a mountain 1 mile away, it's clear. If you look at a mountain 100 miles away, it looks hazy.
• How it works: The system doesn't just look at how well a model did yesterday. It mathematically adjusts the score to account for the "haze" of the future. If a model is good for 1 week but terrible for 12 weeks, the system knows to downgrade its score for long-term planning. It prevents the system from being tricked by models that look great in the short term but fail later.

4. How It Decides: The "Taste Test"

The system uses a two-step strategy to pick the winner:

• For Steady Products (The "Regulars"): It uses a Pareto Front approach. Imagine a restaurant menu where you want the dish that is both the cheapest and the tastiest. Sometimes the cheapest isn't the tastiest. The system finds the "sweet spot" models that offer the best balance of accuracy and stability without sacrificing one for the other.
• For Chaotic Products (The "Wildcards"): For items that sell randomly (intermittent demand), it gets conservative. It picks the model that is simply the most stable, avoiding fancy tricks that might backfire.

5. The Result: Less Waste, Better Stock

The researchers tested this system on massive datasets (like Walmart's sales data and global competitions). They compared their "Smart Matchmaker" against:

1. The Simple Approach: Just picking the model with the lowest error on one specific metric.
2. The Average Approach: Trying to average out all the metrics.

The Findings:

• The Smart Matchmaker (AHSIV) was just as good as the best simple method overall.
• Crucially, it was much better at picking the right model for the right specific time horizon. It didn't just get the average right; it got the specific future moments right more often.
• It reduced the "volume error," meaning the total amount of goods predicted matched the total amount of goods actually sold much more closely. This translates to less wasted money and happier customers.

The Bottom Line

This paper argues that in a complex business world, you can't use a static rulebook. You need a dynamic, adaptive system that understands that:

1. Different products need different prediction tools.
2. The future gets "foggier" the further out you look, and your tools need to account for that.

By building a system that adapts to the product's personality and the time horizon's difficulty, businesses can finally stop guessing and start planning with confidence. It's the difference between throwing darts in the dark and using a GPS that updates its route as the road conditions change.

Technical Summary

1. Problem Statement

The paper addresses a critical gap in demand forecasting: the lack of a robust, reproducible mechanism for selecting the best forecasting model in complex, multi-SKU business environments. The authors identify three core challenges:

• No Universal Dominance: Empirical evidence (e.g., M-competitions) shows no single forecasting model dominates across all datasets, demand regimes, or time horizons.
• Metric Inconsistency: Different error metrics (e.g., MAE vs. RMSE) often produce conflicting model rankings, creating ambiguity in decision-making.
• Horizon-Induced Degradation: Forecast error typically increases as the prediction horizon extends. However, most selection frameworks evaluate models at a single, fixed short-term horizon, leading to "optimistic bias" when models are deployed for longer operational planning periods. Furthermore, model rankings often shift as the horizon lengthens, rendering static selection rules ineffective.

2. Methodology

The study proposes a comprehensive experimental framework and a novel selection mechanism, validated across four major datasets (Walmart, M3, M4, and M5) with 12-step forecasting horizons and varying train-test splits (91:9, 80:20, 70:30).

A. Core Components

1. Metric Degradation by Forecast Horizon (MDFH):

   • A novel procedure that projects out-of-sample error metrics from the evaluation horizon to future operational horizons.
   • It models error growth using a power-law function ($E(h) \propto h^\alpha$) but only activates under structurally stable regimes.
   • It diagnoses structural stability using the coefficient of variation of increments and second differences. If the series is biased or explosive, error extrapolation is halted to prevent invalid projections.

2. Adaptive Hybrid Selector for Intermittency and Variability (AHSIV):

   • A regime-conditioned, multi-objective model selector.
   • Structural Classification: Classifies demand series into "Regular-Stable" or "Intermittent/High-Variability" based on demand frequency ($p$) and relative variability ($c$).
   • Decision Logic:
     • For Stable Regimes: Uses Pareto dominance to select models minimizing both scaled error (RMSSE) and absolute error (MAE), adjusted by MDFH. A hierarchical refinement then minimizes systematic bias (BIAS) and sMAPE.
     • For Intermittent/Unstable Regimes: Adopts a conservative strategy, selecting the model with the minimum projected scaled error (RMSSE) to avoid instability in multi-objective optimization.

3. Comparative Baselines:

   • RMSSE$_h$: A monometric baseline selecting the model with the lowest MDFH-adjusted RMSSE at horizon $h$.
   • Equilibrium Ranking Aggregation (ERA): A multivariate selector aggregating ordinal rankings of MAE, RMSE, and $R^2$.

4. Strategic Evaluation Metric (GRA):

   • Global Relative Accuracy (GRA): A volumetric metric measuring the coherence between total forecasted volume and total realized demand. It assesses the strategic alignment of forecasts with inventory planning, rather than just point-wise accuracy.

B. Experimental Setup

• Datasets: Walmart (weekly), M5 (daily), M4 (weekly), M3 (monthly).
• Models: A diverse set of 25+ models including statistical (ARIMA, SES), machine learning (XGBoost, Random Forest), and deep learning (LSTM, GRU, Transformers).
• Validation: Fixed-origin backtesting with multi-step horizons ($h=1$ to $12$).

3. Key Contributions

1. Formalization of MDFH: The paper introduces a mathematically grounded method to adjust error metrics for future horizons without requiring unobserved future data, ensuring comparisons are horizon-consistent.
2. AHSIV Framework: It proposes a selection mechanism that dynamically adapts its decision logic based on the structural properties of the demand series (intermittency vs. stability) and the forecasting horizon.
3. Strategic Alignment: By integrating the GRA metric, the study links statistical model selection directly to operational outcomes (inventory coherence), moving beyond pure statistical accuracy.
4. Empirical Evidence of Horizon Dependence: The study provides robust statistical proof (via Kruskal-Wallis and Dunn tests) that model rankings are not static; they vary significantly across horizons, validating the need for horizon-aware selection.

4. Results

• Performance Equivalence: AHSIV and the monometric RMSSE$_h$ selector achieved statistical equivalence in terms of aggregated Global Relative Accuracy (GRA) across all datasets and partition schemes. Both significantly outperformed the ERA selector.
• Horizon-Specific Superiority: While AHSIV did not strictly dominate in average performance, it demonstrated a higher frequency of selecting the optimal model for specific forecasting horizons compared to static baselines.
• Stability: AHSIV maintained high and stable GRA values across the 12-step horizon, whereas the ERA selector showed greater variability and lower performance, particularly in the lower tail of the distribution.
• Robustness: The results were consistent across different train-test splits (91:9, 80:20, 70:30), indicating that the framework's performance is not dependent on a specific data allocation ratio.

5. Significance and Implications

• Operational Relevance: The findings suggest that treating model selection as a static ranking problem is flawed. Businesses must adopt horizon-aware and structure-adaptive mechanisms to ensure forecasting models remain effective as planning horizons extend.
• Inventory Optimization: By prioritizing volumetric coherence (GRA) and bias minimization, the proposed framework helps organizations reduce the economic costs of overstocking and stockouts, directly supporting strategic inventory planning.
• Methodological Shift: The paper advocates for a shift from single-metric optimization to multi-objective, regime-conditioned selection processes that account for the degradation of predictive power over time.

In conclusion, the study demonstrates that the AHSIV framework, supported by the MDFH procedure, provides a principled, reproducible, and operationally coherent solution for demand forecasting in heterogeneous, multi-SKU environments, effectively addressing the instability caused by horizon-induced degradation.