📊 statistics

Going NUTS with ADVI: Exploring various Bayesian Inference techniques with Facebook Prophet

This paper presents a complete reimplementation of Facebook Prophet in PyMC to overcome the limitations of its default inference methods, enabling a detailed comparison of various Bayesian techniques—including MCMS, MAP, and Variational Inference—on time-series forecasting problems.

Original authors: Jovan Krajevski, Biljana Tojtovska Ribarski

Published 2026-01-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jovan Krajevski, Biljana Tojtovska Ribarski

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Trying to Predict the Future

Imagine you are trying to predict how many people will visit a specific Wikipedia page tomorrow. You have a tool called Facebook Prophet that is very good at this. It looks at the past data and tries to guess the future by breaking the problem down into three parts:

The Trend: Is the page getting more popular over time? (Like a slowly rising hill).
The Seasons: Do people visit more on weekends or during the holidays? (Like the tides going in and out).
The Noise: Random, unpredictable bumps in the data.

The authors of this paper wanted to use this tool to test different ways of "solving" the math behind the prediction. They found that the original tool was a bit too rigid, so they rebuilt it from scratch using a new toolkit called PyMC.

The Three Ways to Solve the Puzzle

The paper compares three different "detectives" (inference methods) trying to solve the mystery of the future data. Think of them as three different ways to find the best route through a foggy mountain.

1. The "Best Guess" Detective (MAP Estimation)

The Analogy: Imagine you are looking for the highest peak in a mountain range. This detective looks at the map, finds the single highest point, and says, "The answer is right here."
What the paper says: This method is very fast. It gives you one specific number for the answer. However, it ignores the fact that there might be other high peaks nearby or that the ground might be shaky. It tells you where the answer is, but not how sure it is.
Result: It was fast and gave good predictions, but it didn't tell the whole story about uncertainty.

2. The "Exhaustive Hiker" (Full MCMC / NUTS)

The Analogy: This detective doesn't just pick one spot. They send out a whole team of hikers to walk every single path in the mountain range, taking thousands of steps to map out the entire terrain. They want to know the shape of the whole mountain, not just the top.
What the paper says: This is the most accurate method. It explores the "posterior distribution" (the full shape of the possible answers). The authors tested three types of hikers:
- Metropolis-Hastings (MH): A hiker who takes random, slow steps. They got lost and didn't explore the mountain well.
- DMZ: A hiker who tries to copy the steps of others. They also struggled to cover the ground efficiently.
- NUTS (No-U-Turn Sampler): A super-smart hiker who uses a map and a compass (gradients) to walk efficiently. This was the winner. It found the best path quickly and mapped the mountain perfectly without wasting time.
Result: NUTS was the most reliable, but it took longer to run than the "Best Guess" detective.

3. The "Fast Approximator" (Variational Inference / VI)

The Analogy: Instead of walking the whole mountain, this detective looks at the mountain from a helicopter and draws a simple, smooth oval shape that looks like the mountain. It's a shortcut.
What the paper says: This method tries to fit a simple shape (like a bell curve) to the complex data. It is much faster than the "Exhaustive Hiker."
- ADVI: Drew a simple oval. It was fast but got a bit too confident (the oval was too narrow).
- FullRank ADVI: Drew a more complex, stretched oval. It was also fast but got a bit too unsure (the oval was too wide).
Result: It was the fastest method, but the "shape" of the answer wasn't perfectly accurate compared to the Exhaustive Hiker.

The Race Results

The authors put these detectives to the test using data about a famous football player's Wikipedia page. Here is how they compared:

Speed: The "Fast Approximator" (VI) was the quickest. The "Best Guess" (MAP) was also very fast. The "Exhaustive Hiker" (NUTS) was slower but still manageable. The other hikers (MH and DMZ) were so slow and inefficient they didn't finish the job well.
Accuracy: The "Exhaustive Hiker" (NUTS) gave the most trustworthy map of the future. The "Fast Approximator" was close but had some flaws in how it measured uncertainty. The "Best Guess" was good for a quick answer but missed the details.
The Winner: For this specific problem, NUTS was the best overall choice. It wasn't the fastest, but it was the only one that successfully mapped the whole mountain without getting lost or making bad guesses.

The New Tool (The Rebuilt API)

The authors also mentioned that the original Facebook Prophet tool was like a pre-packaged meal: you could eat it, but you couldn't easily change the ingredients.
They rebuilt the tool in PyMC to be like a modular Lego set. Now, instead of just following a recipe, you can snap different pieces together (like "Linear Trend" + "Yearly Seasonality") to build exactly the model you want. This new "intuitive" way of building models makes it much easier to experiment with different ideas.

Summary

The paper is essentially a race to see how to best predict the future using a specific mathematical model.

If you need speed and a single answer, use the Best Guess (MAP).
If you need a quick approximation of uncertainty, use the Fast Approximator (VI).
If you need the most accurate, reliable picture of the future and its uncertainties, the Exhaustive Hiker (NUTS) is the best choice, even if it takes a little more time.

The authors concluded that while the fast methods are tempting, the smart, thorough method (NUTS) provided the best balance of accuracy and reliability for their time-series forecasting problem.

Technical Summary: Going NUTS with ADVI: Exploring various Bayesian inference techniques with Facebook Prophet

Problem Statement
Facebook Prophet has gained traction for time-series forecasting due to its decomposable model structure and built-in inference methods: Maximum A Posteriori (MAP) estimation via L-BFGS-B and Markov Chain Monte Carlo (MCMC) sampling via the No-U-Turn Sampler (NUTS). However, the authors identified two primary limitations in the original implementation:

Inflexibility: The fluent API design restricts the ability to incorporate custom modeling components or alternative inference techniques beyond the defaults.
Inference Constraints: Full MCMC sampling is often computationally prohibitive for rapid experimentation, while MAP estimation collapses the posterior distribution into a single point, discarding critical information regarding parameter uncertainty.

To address these issues, the authors sought to evaluate and compare multiple Bayesian inference techniques (Full MCMC, MAP, and Variational Inference) on a specific time-series forecasting scenario.

Methodology
The core of the work involves a complete reimplementation of the Facebook Prophet model using PyMC, a probabilistic programming library.

Model Reimplementation: The authors reconstructed the Prophet model ( $y(t) \sim g(t) + \sum s(t; p_i, c_i) + h(t) + \epsilon(t)$ ) within PyMC. They replaced the original fluent API with a modular, intuitive API that mirrors mathematical notation. This allows users to define components (e.g., LinearTrend, FourierSeasonality) separately and combine them using standard arithmetic operators (e.g., model = trend * (1 + yearly + weekly)).
Inference Techniques Evaluated:
- Full MCMC: Three algorithms were tested: Metropolis-Hastings (MH), NUTS, and Z-score adaptation of DEMetropolis (DMZ).
- MAP Estimation: Performed using the L-BFGS-B algorithm.
- Variational Inference (VI): Two algorithms were investigated: Automatic Differentiation Variational Inference (ADVI) and Full-Rank ADVI (FR ADVI).
Experimental Setup:
- Dataset: Two years of log daily page views for the Wikipedia page of Peyton Manning (forecasting the subsequent year).
- Model Configuration: Default Prophet settings (piecewise linear trend with 25 change points, additive yearly and weekly seasonality).
- Diagnostics: Convergence was assessed using R-hat (Gelman-Rubin), Effective Sample Size (ESS), and autocorrelation plots for MCMC; and Negative Evidence Lower Bound (ELBO) convergence for VI.
- Metrics: Forecasting performance was measured via MSE, RMSE, MAE, and MAPE, alongside execution time and computational efficiency (effective samples per second).

Key Contributions

PyMC-based Prophet Reimplementation: A flexible, modular framework that decouples the model definition from the inference engine, enabling the integration of custom components and diverse inference algorithms.
Comparative Analysis: A detailed empirical comparison of MAP, Full MCMC (MH, DMZ, NUTS), and VI (ADVI, FR ADVI) on a standard time-series problem, analyzing both statistical quality and computational efficiency.
Diagnostic Insights: The paper provides specific diagnostic findings regarding the convergence behavior of different samplers on the Prophet model structure, highlighting that standard MH and DMZ struggle with convergence in this context compared to NUTS.

Results

MCMC Convergence:
- NUTS was the only MCMC algorithm to achieve convergence (R-hat < 1.01, ESS > 400) efficiently, requiring only 550 samples to converge. It demonstrated superior effective sampling rates (169.69 samples/sec) compared to MH and DMZ.
- MH and DMZ failed to converge even after 1,000,000 samples. MH showed high autocorrelation, and DMZ produced low ESS values, indicating inefficient exploration of the posterior.
- Despite non-convergence, DMZ surprisingly produced the best forecasting metrics among MCMC methods, though this is attributed to the specific nature of the point estimates rather than a reliable posterior approximation.
Variational Inference:
- Both ADVI and FR ADVI converged around 80,000 iterations.
- ADVI yielded the highest quality forecasts among VI methods but underestimated posterior uncertainty (overconfident).
- FR ADVI overestimated posterior uncertainty (less certain) and performed worse in forecasting metrics.
- VI methods were significantly faster (approx. 18–23 seconds) than MCMC methods, though they did not match the posterior fidelity of NUTS.
MAP Estimation:
- MAP provided high-quality forecasts with the fastest execution times.
- The authors note that MAP is equivalent to Maximum Likelihood Estimation (MLE) with regularization (L1 for Laplace priors, L2 for Normal priors).
Posterior Approximation:
- Histograms of the slope parameter revealed that NUTS provided a robust posterior distribution.
- ADVI was "overconfident" (narrower distribution), while FR ADVI was "underconfident" (wider distribution) compared to the NUTS ground truth.

Significance and Conclusions
The paper concludes that there is no universally optimal inference method; the choice depends on the trade-off between computational cost and the need for accurate uncertainty quantification.

NUTS is identified as the best choice for this specific problem given the current settings, offering a balance of convergence reliability and reasonable time-to-convergence, despite being slower than VI.
MAP is suitable for scenarios where capturing the full posterior shape is less critical than speed and point-estimate accuracy.
VI (specifically ADVI) offers a viable alternative when computational resources are constrained, providing comparable forecasting metrics to MCMC with significantly reduced execution time, though it suffers from limitations in accurately capturing posterior uncertainty.

The authors emphasize that while VI is faster, the over/underestimation of uncertainty by ADVI and FR ADVI respectively remains an issue requiring further investigation. Future work will focus on GPU acceleration (via JAX/NumPyro) to speed up likelihood computations, though the authors note that the sequential nature of MCMC limits parallelization benefits compared to the more parallelizable VI approaches.