📊 statistics

Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes

This paper introduces Online Bayesian Stacking (OBS), a novel ensemble method that leverages insights from online optimization and portfolio selection to adaptively combine Bayesian models, offering a theoretically grounded alternative to online Bayesian model averaging that addresses its limitations and provides clear guidance for practical application.

Original authors: Daniel Waxman, Fernando Llorente, Petar M. Djurić

Published 2026-01-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Daniel Waxman, Fernando Llorente, Petar M. Djurić

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather. You have a team of five different meteorologists.

Meteorologist A only looks at the barometer.
Meteorologist B only looks at the wind speed.
Meteorologist C only looks at the humidity.
Meteorologist D uses a supercomputer model.
Meteorologist E relies on old-school folklore.

Your goal is to combine their opinions to get the best possible forecast for tomorrow. This is the problem of Bayesian Ensembling.

This paper, titled "Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes," tackles a specific challenge: How do you combine these experts as the days go by, learning from new data in real-time, without making a huge mistake?

Here is the breakdown of their findings using simple analogies.

1. The Old Way: "The Favorite Child" (Bayesian Model Averaging)

For a long time, the standard method was called Bayesian Model Averaging (BMA).

How it works: You start by giving every meteorologist an equal chance. As days pass, you check who was right. If Meteorologist A was right yesterday, you give them a slightly bigger slice of the "prediction pie" for today. If they were wrong, you shrink their slice.
The Problem: The paper argues that this method is too rigid. If Meteorologist A happens to get lucky for a few days, the algorithm might decide, "A is the only one who matters!" and give them 100% of the pie, ignoring everyone else.
The Result: This is called "collapsing." The system bets everything on one expert. If that expert turns out to be wrong later (or if the weather changes in a way they can't predict), the whole system fails because it has forgotten the other four experts. The paper calls this a "pathology" of the old method.

2. The New Way: "The Smart Portfolio Manager" (Online Bayesian Stacking)

The authors propose a new method called Online Bayesian Stacking (OBS).

The Analogy: Instead of just picking a favorite meteorologist, imagine you are a stock market portfolio manager.
- Each meteorologist is a different stock.
- Their prediction accuracy is like the stock price.
- Your goal isn't to find the single "best" stock; it's to build a diversified portfolio that grows the most wealth over time.
How it works: OBS uses math from Online Convex Optimization (a field usually used for trading stocks) to constantly rebalance your portfolio.
- If Meteorologist A does well, you buy a little more of their "stock."
- If they do poorly, you sell a little.
- Crucially, you never bet everything on one person. You keep a mix, ensuring that if the weather changes and a different expert becomes the star, you still have a piece of them in your portfolio.

3. The "Aha!" Moment: Connecting Two Worlds

The paper's biggest insight is realizing that combining prediction models is exactly the same math problem as managing a stock portfolio.

In finance, you want to maximize your wealth.
In prediction, you want to maximize your "log-score" (a fancy way of saying "how confident and correct your predictions are").
Because these problems are mathematically identical, the authors can borrow powerful, proven algorithms from the stock market world (like the Exponentiated Gradient and Online Newton Step) and use them to combine AI models. This gives them a "superpower" of mathematical guarantees that the old method didn't have.

4. Why This Matters (The Experiments)

The authors tested this new "Portfolio Manager" approach against the old "Favorite Child" approach in several scenarios:

Toy Problems: Simple math puzzles where the truth is hidden.
Neural Networks: Combining different AI brain structures.
Stock Market Data: Predicting the S&P 500 using complex financial models.
Robotics: Predicting how a robot arm moves when the environment changes.

The Results:
In almost every test, the new OBS (Portfolio) method beat the old O-BMA (Favorite Child) method.

In stable environments: OBS performed just as well as the best possible static mix.
In changing environments: When the "weather" changed (non-stationary data), the old method often collapsed onto the wrong expert and never recovered. The new OBS method adapted quickly, shifting its portfolio to the experts who were currently doing well.

5. The "Empirical Bayes" Explanation

The paper also uses a concept called Empirical Bayes to explain why the old method fails.

Think of the old method (BMA) as trying to find the one true God of the data. It assumes one of the models is perfect and tries to find it.
The new method (OBS) assumes no single model is perfect. Instead, it treats the "weights" (how much we trust each model) as the thing to be learned. It's like saying, "We don't know who is the best, so let's just find the best combination of all of them."

Summary

The paper says: Stop trying to pick the single best AI model. Instead, treat your collection of models like a diversified investment portfolio. Use the math of stock trading to constantly rebalance your trust in each model. This prevents the system from "collapsing" onto a single wrong answer and allows it to adapt smoothly when the world changes.

Who should use this?
The authors suggest that if you are working in a real-world scenario where the data might change (which is almost always the case in machine learning), you should prefer this new Online Bayesian Stacking method over the traditional Bayesian Model Averaging.

Technical Summary: Bayesian Ensembling via Online Optimization and Empirical Bayes

Problem Statement

The paper addresses the challenge of combining multiple Bayesian models for predictive inference in an online learning setting. While combining models (ensembles) is a well-established practice in statistics and machine learning, existing methods face specific limitations when data arrives sequentially and the true data-generating process is unknown.

The primary methods for Bayesian ensembling are:

Bayesian Model Averaging (BMA): Weights models based on their marginal likelihoods. While optimal in the "M-closed" setting (where the true model is among the candidates), BMA suffers from "weight collapse" in "M-open" settings (where the true model is not present), concentrating all weight on a single model that may not generalize well.
Bayesian Stacking: Optimizes a log-score criterion (predictive likelihood) over a validation set to find optimal convex combinations of models. However, standard stacking is typically a batch process requiring separate datasets, making it ill-suited for strictly online scenarios where weights must be updated as new data arrives without holding out data.

The paper seeks to bridge this gap by formulating Online Bayesian Stacking (OBS): a method to adaptively combine Bayesian models in real-time by optimizing the log-score over predictive distributions, while providing a theoretical framework to understand its relationship with online BMA.

Methodology

1. Connection to Online Portfolio Selection (OPS)

The core methodological insight is the reinterpretation of Bayesian stacking as a Universal Online Portfolio Selection (OPS) problem.

Analogy: The predictive densities of the $K$ models, $p_k(y_t | x_t, D_{t-1})$ , are treated as the "price-relative" vectors (returns) of $K$ assets in a financial market.
Objective: The goal of maximizing the cumulative log-predictive likelihood (log-score) is mathematically equivalent to maximizing the log-wealth of a portfolio.
Algorithms: This connection allows the authors to leverage efficient, theoretically grounded algorithms from the Online Convex Optimization (OCO) literature, specifically:
- Exponentiated Gradients (EG): A multiplicative update rule.
- Online Newton Step (ONS): Utilizes second-order information for tighter regret bounds.
- Soft-Bayes: A variant that provides regret bounds without assuming bounded market variability (gradients).
- Discounted ONS (D-ONS): An extension designed for non-stationary environments by incorporating a forgetting factor on second-order information.

2. Theoretical Analysis of Regret and Loss Functions

The paper establishes distinct theoretical properties for OBS versus Online BMA (O-BMA):

O-BMA as Hedge: The authors demonstrate that O-BMA updates are equivalent to the Hedge algorithm with a learning rate of $\eta=1$ and a specific loss function. O-BMA minimizes regret with respect to the best single expert (model selection).
OBS as Mixture Optimization: OBS minimizes regret with respect to the best convex combination of experts (model combination).
Regret Bounds: By mapping OBS to OPS, the paper derives regret bounds. Under mild assumptions, OBS achieves $O(\sqrt{T})$ regret (e.g., via Soft-Bayes) or $O(\log T)$ regret (via ONS with bounded gradients). In contrast, O-BMA achieves constant regret only with respect to the best single model, which can be arbitrarily poor if the best single model is a bad predictor compared to a mixture.

3. Empirical Bayes Perspective

The authors provide an Empirical Bayes (EB) interpretation to explain the pathologies of BMA:

BMA as EB on Indicators: BMA performs EB over an indicator variable, effectively selecting a single model with the highest marginal likelihood. This leads to the "collapse" phenomenon where the ensemble degenerates to a single model, even if a mixture would yield better predictive accuracy.
OBS as EB on Weights: OBS performs EB directly on the mixture weights themselves, treating them as hyperparameters to be optimized. This avoids the collapse by seeking the optimal weighted combination rather than a single winner.

Key Contributions

Novel Connection: The paper establishes a rigorous equivalence between Online Bayesian Stacking and Universal Online Portfolio Selection, enabling the use of efficient OCO algorithms for Bayesian ensembling.
Theoretical Clarification: It clarifies the distinction between O-BMA and OBS, showing they optimize different cost functions (model selection vs. model combination) and providing regret bounds for OBS based on the OPS literature.
Empirical Bayes Explanation: It offers a compelling EB argument explaining why BMA collapses in M-open settings and how stacking naturally avoids this issue.
Hybrid Approach: The authors propose a hybrid method that maintains both O-BMA and OBS mixtures, averaged via a secondary layer, to achieve constant regret with respect to the best single model while retaining the expressiveness of OBS.

Experimental Results

The authors evaluate OBS across four scenarios using ensembles of state-of-the-art models (Gaussian Processes, Variational Bayesian Neural Networks, and Stochastic Volatility models estimated via Sequential Monte Carlo):

Toy Linear Regression: In an "Open" setting (true model not in the ensemble), OBS variants (EG, ONS) significantly outperformed O-BMA, which collapsed to a single poor model. In a "Closed" setting, OBS performed nearly as well as the theoretically optimal O-BMA.
Online Variational Inference (MNIST): Using Bayesian Neural Networks, OBS consistently outperformed O-BMA and Dynamic Model Averaging (DMA). BMA collapsed to a single network, while OBS converged to balanced weights.
Financial Forecasting (S&P 500): Ensembling diverse GARCH models, OBS methods (EG, ONS, Soft-Bayes) outperformed O-BMA and DMA. In this non-stationary environment, D-ONS (Discounted ONS) achieved the best performance.
Non-Stationary Regression: On datasets with covariate shift (e.g., robotics data), OBS methods, particularly D-ONS and Soft-Bayes, were robust and consistently outperformed O-BMA and standard DMA.

Key Finding: In all tested cases, OBS significantly outperformed online BMA. The computational cost of OBS was found to be negligible compared to the cost of online training and prediction of the underlying ML models.

Significance and Claims

The paper claims that Online Bayesian Stacking (OBS) provides a principled, theoretically grounded, and empirically superior alternative to Online BMA, particularly in realistic "M-open" scenarios where the true data-generating process is not contained within the candidate model set.

Practical Guidance: The authors suggest that practitioners should prefer OBS in most machine learning applications (which are typically M-open). They recommend specific OCO algorithms based on the environment: D-ONS for anticipated non-stationarity, ONS for general scenarios with sufficient compute, and Soft-Bayes for robustness without gradient bounds.
Theoretical Bridge: The work bridges the gap between Bayesian ensemble learning and the rich theoretical framework of online convex optimization and portfolio selection, offering new tools for analyzing and improving Bayesian predictive performance.
Modesty: The authors acknowledge that while OBS is generally superior, O-BMA remains optimal in strictly M-closed settings. They also note that the choice of OCO algorithm and hyperparameters (like learning rates) can influence performance, though the proposed methods are generally robust to these choices within reasonable ranges.

The paper concludes that the connection between OBS and portfolio selection opens new avenues for research in continual learning, adaptive regret, and the initialization of Bayesian ensembles.