📊 statistics

End-to-end probabilistic hierarchical forecasting of large hierarchies via probabilistic top-down

The paper introduces e2eTD, a fast and scalable method for generating coherent probabilistic forecasts of large hierarchical time series by directly predicting a small subset of aggregate series and propagating them to the bottom level via a novel probabilistic top-down sampling algorithm, achieving state-of-the-art performance on major retail datasets with minimal computational cost.

Original authors: Lorenzo Zambon, Dario Azzimonti, Giorgio Corani

Published 2026-06-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lorenzo Zambon, Dario Azzimonti, Giorgio Corani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a massive chain of grocery stores. You need to know how much milk, bread, and cereal to order for every single store, every single day.

This is a nightmare of numbers. You have thousands of products (SKUs) in hundreds of stores. If you try to predict the sales for every individual item in every store, the data is messy. Some days a specific item sells 100 units; other days, it sells zero. It's like trying to predict the exact path of a single raindrop in a storm. It's too noisy, too "intermittent," and too hard to get right.

However, if you look at the big picture—say, the total milk sales for the entire country—that number is much smoother and easier to predict. It's like looking at the storm from space; you can see the pattern even if you can't track every drop.

The Problem: The "Conflicting Maps" Dilemma
Retailers usually try to solve this in two ways, but both have flaws:

The "Bottom-Up" approach: Predict every single item first, then add them up. This is slow, expensive, and because the individual items are so messy, the final total is often wrong.
The "Top-Down" approach: Predict the big total first, then guess how to split it up. This is fast, but the "splitting" is often done with rigid, old rules that don't account for uncertainty.

The result? You end up with a plan where the total for the country says "1 million units," but the sum of all your store plans says "900,000 units." This is a coherence problem. It's like having a map where the distance from New York to Boston is 200 miles, but the distance from New York to Philadelphia plus Philadelphia to Boston is 300 miles. You can't make good business decisions with a broken map.

The Solution: e2eTD (The "Smart Middle-Man")
The paper introduces a new method called e2eTD. Think of it as a smart, probabilistic middle-man that bridges the gap between the smooth big picture and the messy small details.

Here is how it works, step-by-step:

Pick the "Easy" Targets: Instead of trying to predict the messy individual items (the bottom), e2eTD only predicts a tiny, manageable slice of the "smooth" middle layers (about 0.3% of the data). It's like a weather forecaster who only predicts the general storm front, not every single raindrop.
Make a "Group Plan": It creates a probabilistic forecast for these smooth layers. This means it doesn't just say "We will sell 1,000 units"; it says "We are 90% sure we will sell between 900 and 1,100 units."
The "Smart Split" (Probabilistic Top-Down): This is the paper's secret sauce. Usually, when you split a big number down to small numbers, you use a fixed ratio (e.g., "Store A always gets 10%"). e2eTD is smarter. It looks at history and says, "When the total was 1,000, Store A usually got between 9% and 11%, and Store B got the rest, and they tend to move together."
- It uses a mathematical tool called a Copula (think of it as a "glue" that holds the relationships between stores together) to simulate thousands of possible scenarios.
- It randomly splits the big forecast into small pieces, but it respects the historical "dance" between the stores. If Store A usually spikes when Store B dips, the simulation knows that.
The "Check" (Reconciliation): Finally, it sums all the small pieces back up. Because it started with a coherent big number and split it carefully, the small pieces automatically add up to the big number. No conflicts. No broken maps.

Why is this a big deal?

Speed: The paper claims this method is incredibly fast. On a standard laptop, it can process a dataset with 300,000 time series (like the Favorita dataset) in about 20 minutes. Other methods that try to do this with complex neural networks might need supercomputers or take hours.
Accuracy: When tested against real-world data (the M5 and Favorita datasets), e2eTD beat almost every other method. If this method had been entered into the famous M5 forecasting competition, it would have ranked 11th out of 892 teams.
Risk Management: Because it produces "probabilistic" forecasts (ranges of possibilities), retailers can make safer decisions. They can say, "We need enough stock to cover the 95% worst-case scenario," rather than just guessing the average.

In a Nutshell
e2eTD is a fast, efficient way to predict demand for massive retail chains. It avoids the messiness of predicting every single item by focusing on the smooth, predictable totals first. Then, it uses a clever, history-based "splitting" algorithm to distribute those totals down to the individual items, ensuring that the small numbers always add up perfectly to the big numbers. It's like building a house by first laying a perfect foundation and then carefully placing the bricks, rather than trying to guess where every brick goes and hoping the roof fits.

Technical Summary: End-to-end Probabilistic Hierarchical Forecasting via Probabilistic Top-Down (e2eTD)

1. Problem Statement

Retail and supply chain operations rely on demand forecasts for decisions ranging from product-level replenishment to store-level capacity planning. These forecasts must satisfy two critical, often conflicting, requirements:

Probabilistic Nature: To enable risk-aware decision-making (e.g., setting safety stocks), forecasts must provide full predictive distributions rather than single point estimates.
Coherence: Forecasts across the aggregation hierarchy (e.g., SKU, Store, Region, Total) must be consistent. The forecast for a parent node must equal the sum of its children's forecasts.

Existing approaches face significant scalability and modeling challenges in large hierarchies (thousands of time series) characterized by intermittent demand (sparse, zero-inflated bottom-level series):

Two-step (Forecast-then-Reconcile): While effective for point forecasts, probabilistic reconciliation is computationally expensive for large hierarchies. It often relies on restrictive assumptions (e.g., Gaussianity) that fail for intermittent count data or requires high-dimensional matrix inversions.
End-to-End Neural Models: These produce coherent forecasts but typically require complex architectures, specialized hardware (GPUs), and significant engineering effort, making them impractical for rapid, large-scale deployment.
Traditional Top-Down: While scalable, standard top-down methods often rely on deterministic historical proportions, failing to capture the uncertainty and cross-series dependence required for probabilistic coherence.

The paper identifies a gap for a method that is fast, scalable, and capable of producing coherent probabilistic forecasts for large hierarchies with intermittent bottom series without requiring specialized hardware.

2. Methodology: e2eTD

The proposed method, e2eTD (end-to-end Top-Down), generates coherent probabilistic forecasts by modeling only a small subset of smooth upper-level series and propagating samples downward via a novel probabilistic top-down algorithm.

Core Workflow

The method proceeds in five steps (illustrated in Fig. 2 of the paper):

Selection of Upper Series: Instead of forecasting all bottom series, e2eTD selects a small subhierarchy of smooth, predictable upper series (approximately 0.3% of the total series in experiments). This subset must form a subhierarchy that covers all bottom series.
Probabilistic Forecasting: Probabilistic forecasts are generated independently for the selected upper series. The authors use Exponential Smoothing (ETS) with automatic model selection, capable of incorporating exogenous regressors (e.g., holidays, promotions).
Reconciliation: To ensure coherence among the selected upper series, a Gaussian probabilistic reconciliation step is applied. This combines the independent forecasts into a joint multivariate normal distribution. Samples are drawn from this reconciled distribution at the lowest level of the selected subhierarchy.
Probabilistic Top-Down Sampling (Novelty): This is the core contribution. Instead of using fixed historical proportions, the algorithm models the disaggregation proportions as joint distributions.
- Modeling: For each pair of series being split, the method estimates the joint distribution of their historical counts. Marginals are fitted as Poisson or Negative Binomial (depending on dispersion), and dependence is modeled using a Plackett copula.
- Sampling: Given a sample $z$ from an upper node, the algorithm draws a pair $(a, b)$ such that $a+b=z$ . The probability of each split is determined by conditioning the estimated joint distribution on the constraint $A+B=z$ .
- Recursion: This process is applied recursively (via binary splitting) from the selected upper nodes down to the bottom level. Crucially, the algorithm preserves the cross-series dependence inherited from the reconciliation step by not reshuffling sample indices.
Aggregation: The resulting joint bottom-level samples are summed using the aggregation matrix to obtain coherent probabilistic forecasts for all levels of the hierarchy.

Computational Strategies

To ensure scalability to hundreds of thousands of series:

Batching: The algorithm computes sampling weights for unique integer values of the upper samples once, rather than looping over every sample.
Parallelization: Upper-level forecasting and top-down sampling for different branches of the hierarchy are parallelized across CPU cores.
Efficient Estimation: The method uses moment matching for marginal parameters and Spearman rank correlation inversion for the copula parameter, avoiding computationally expensive Maximum Likelihood Estimation (MLE).

3. Key Contributions

Scalable Probabilistic Coherence: e2eTD achieves coherent probabilistic forecasts for hierarchies with ~300k series (Favorita dataset) in under 20 minutes on a standard laptop, without specialized hardware.
Probabilistic Top-Down Algorithm: The paper introduces a sampling mechanism that treats disaggregation proportions as joint distributions (via copulas) rather than deterministic ratios, preserving cross-series dependence and ensuring coherence at all levels.
Handling Intermittency: By forecasting only smooth upper series and inducing intermittency naturally during the top-down sampling step, the method avoids the difficulties of modeling sparse, zero-inflated bottom series directly.
Empirical Validation: The method is evaluated on the two largest publicly available retail datasets (M5 and Favorita), demonstrating superior performance over existing scalable baselines.

4. Experimental Results

The authors evaluated e2eTD against five competing methods (Empirical Distribution, Seasonal Empirical, WLS Reconciliation, Long et al. (2025), and HINT) on the M5 and Favorita datasets.

Accuracy (WSPL): e2eTD achieved the lowest Weighted Scaled Pinball Loss (WSPL) across all aggregation levels on both datasets.
- On M5, e2eTD would have ranked 11th out of 892 teams in the M5 Uncertainty competition.
- On Favorita, it significantly outperformed all competitors, including neural network-based approaches.
- The method maintained high accuracy even at the bottom level (SKU-store), where it would have ranked 7th among the top 50 M5 teams.
Computational Efficiency:
- M5 (~30k series): ~4.5 minutes.
- Favorita (~160k series): ~17 minutes.
- In contrast, neural methods (HINT) required GPUs, and reconciliation-based methods (WLS) scaled poorly, taking significantly longer on larger datasets.
Ablation Study:
- Reconciliation: Removing the upper-level reconciliation step degraded performance significantly (+8.8% WSPL on M5), confirming the value of combining information across levels.
- Copula: Using an independent copula (ignoring cross-series dependence) had a negligible effect on M5 but increased error on Favorita, suggesting dependence structures vary by dataset.
- Exogenous Regressors: Including covariates (holidays, promotions) significantly improved M5 performance but had a minimal effect on Favorita.

5. Significance and Claims

The paper positions e2eTD as a practical solution for the "first-order operational concern" of forecasting cost in retail.

Operational Impact: The authors note that large retailers may need to refresh billions of forecasts daily. Switching from heavy methods to e2eTD could save tens of millions of dollars annually and reduce the environmental footprint (CO2e) of forecasting pipelines.
Decision Support: By providing coherent probabilistic forecasts, the method enables risk-aware decisions (e.g., safety stock optimization) that are consistent across strategic, tactical, and operational levels.
Modesty: The authors acknowledge limitations, such as the assumption of approximate stationarity in the bottom-level proportions and the manual selection of the upper subhierarchy. They suggest future work could involve principled criteria for selecting the subhierarchy and more sophisticated modeling of time-varying proportions.

In summary, e2eTD bridges the gap between the need for coherent, probabilistic forecasts and the computational constraints of large-scale retail operations, offering a fast, accurate, and hardware-efficient alternative to existing neural and reconciliation-based approaches.