📊 statistics

Bayesian inference for the learning rate in Generalised Bayesian inference

This paper proposes a framework for estimating learning rates and hyperparameters in Generalised Bayesian Inference using held-out data to derive hyperparameter posteriors, thereby enabling joint uncertainty quantification and demonstrating improved performance over standard Bayesian inference in both simulated and real-world text analysis tasks.

Original authors: Jeong Eun Lee, Sitong Liu, Geoff K. Nicholls

Published 2026-05-18

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jeong Eun Lee, Sitong Liu, Geoff K. Nicholls

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Tuning the "Trust Dial"

Imagine you are a detective trying to solve a case. You have a notebook (your prior belief) about how the world works. Then, you start gathering evidence (your data).

In standard detective work (standard statistics), you assume your notebook is perfect and your evidence is 100% reliable. You simply combine them to get a final conclusion.

But in the real world, things are messy.

Your notebook might be slightly outdated.
Your evidence might be noisy, contain outliers (fake clues), or come from a source you don't fully trust.

Generalised Bayesian Inference (GBI) is a flexible framework that lets you say, "I'm not 100% sure my evidence is perfect, so I'll be a little cautious." It introduces a "Learning Rate" (let's call it $\eta$ or the Trust Dial).

If you turn the dial to 1, you trust the data completely (standard statistics).
If you turn it down to 0.5, you trust the data, but you keep your skepticism high, giving more weight to your original notebook.
If you turn it to 0, you ignore the data entirely and stick to your notebook.

The Problem: How do you know what setting to put the Trust Dial on? If you guess wrong, your conclusion could be terrible.

The Paper's Solution: The "Test Drive"

This paper proposes a clever way to automatically find the perfect setting for the Trust Dial (and other related settings) using held-out data.

Think of it like buying a car. You don't just drive it once and decide it's perfect. You take it for a test drive on a different road to see how it handles.

Training Data ( $x$ ): The car you are buying.
Calibration Data ( $y$ ): The test drive road.
The Goal: Adjust the suspension (the Trust Dial) so the car handles the test road perfectly.

The authors create a system where they use the "test drive" data to calculate a score for every possible setting of the dial. They then use Bayesian math to create a "map" (a posterior distribution) showing which settings are most likely to be the best. This allows them to not just pick one number, but to understand the uncertainty around that number.

Two Ways to Score the Drive

The paper highlights two different ways to judge the car, leading to two different "maps":

The "Cover the Truth" Strategy (Pooled Loss):
- Analogy: Imagine you want the car's suspension to be set so that the car physically sits exactly where the "true" road surface is, even if the road is bumpy.
- Goal: You want your final conclusion to cover the "pseudo-true" answer (the best possible answer given your imperfect model).
- Result: The dial setting here doesn't settle on a single sharp number; it stays somewhat spread out, acknowledging that many settings might get you close enough to the truth.
The "Predict the Future" Strategy (Product Loss):
- Analogy: Imagine you want the car to handle every single bump on the test road perfectly so you can predict how it will handle a new road tomorrow.
- Goal: You want your predictions to be as accurate as possible for new, unseen data.
- Result: As you drive more and more test miles (more data), the dial setting locks onto one specific, precise number. It becomes very confident about the best setting.

Handling "Broken" Modules (Semi-Modular Inference)

Sometimes, you have a complex machine made of two parts (Modules).

Module A works perfectly.
Module B is broken or has weird outliers.

In standard statistics, the broken part (Module B) can ruin the whole machine. In this paper's framework (called Semi-Modular Inference or SMI), you can use the Trust Dial to "cut" the connection between the broken part and the rest of the machine.

You can turn the dial down to 0 to completely ignore the broken part (Cut Model).
You can turn it to 1 to let it influence everything (Standard Bayes).
You can find a middle ground (e.g., 0.3) where you let the broken part whisper a little bit of information without shouting.

The paper shows that their method can automatically find this "Goldilocks" setting (not too much, not too little) even when the broken part is tricky.

Real-World Tests

The authors tested their idea in three scenarios:

The "Outlier" Test (Normal Mixture):
- They simulated data where most points were normal, but a few were wild outliers.
- Result: Their method automatically dialed down the trust in the data to ignore the outliers, outperforming standard methods that got confused by them.
The "Hidden State" Test (State-Space Model):
- They simulated a system where some data was missing or measured with different errors.
- Result: Their method found the right balance between trusting the good data and ignoring the bad data, predicting future outcomes better than standard methods.
The "Word Meaning" Test (Text Analysis):
- They analyzed a huge collection of English text from 1810 to 2010 to see how word meanings changed (e.g., how the word "bug" shifted from meaning an insect to a software error).
- Result: The model is very complex and likely imperfect. By using their method to tune the Trust Dial, they got much better at grouping words by meaning compared to using a fixed setting. They found that a lower trust setting (around 0.3) worked best, effectively filtering out the "noise" in the text.

The Takeaway

This paper provides a universal remote control for statistical models. Instead of guessing how much to trust your data or how to handle broken parts of a model, you can use a small piece of extra data to "test drive" the model. The system then tells you exactly how to tune the settings to get the most accurate predictions, while also telling you how confident it is in those settings.

It works best when:

Your data is messy or contains outliers.
Your model is a complex mix of reliable and unreliable parts.
You want to predict future outcomes accurately.

Technical Summary: Bayesian Inference for the Learning Rate in Generalised Bayesian Inference

Problem Statement
Generalised Bayesian Inference (GBI) updates beliefs using a Gibbs posterior of the form $\pi_s(\phi|x) \propto \exp(-\eta \ell_\beta(\phi; x))\pi(\phi)$ , where $\eta$ is a learning rate and $\beta$ represents hyperparameters of the loss function $\ell_\beta$ . While GBI is effective for handling model misspecification and combining heterogeneous data sets, the inference hyperparameters $s = (\eta, \beta)$ cannot be estimated jointly with model parameters $\phi$ by simply assigning them a prior and conditioning on the data. Doing so distorts the posterior for $\phi$ due to the normalising constant $c(\phi, s)$ . Consequently, existing methods often rely on ad-hoc selection or cross-validation. This paper addresses the need for a coherent framework to estimate $s$ and quantify uncertainty about it, particularly in settings involving multiple data modules where some may be misspecified.

Methodology
The authors propose a framework for Bayesian inference on the inference hyperparameters $s$ using held-out calibration data $y$ . The core idea is to treat the posterior predictive distribution of the calibration data, $p_s(y|x)$ , as a likelihood-like quantity for $s$ . Two distinct posterior distributions for $s$ are defined based on different loss functions:

Pooled Loss: Defined as $l^{(1,J)}(s; y, x) = -\log p_s(y|x)$ , where $p_s(y|x) = \int p(y|\phi)\pi_s(\phi|x)d\phi$ . This approach targets the coverage of the "pseudo-true" parameter $\tilde{\phi}$ (the parameter minimizing the Kullback-Leibler divergence between the true data generating process and the model). The resulting posterior $\rho^{(1,J)}(s|y; x)$ does not concentrate as the size of calibration data $J \to \infty$ but converges to a diffuse distribution proportional to $\pi_s(\tilde{\phi}|x)\rho(s)$ .
Product Loss: Defined as $l^{(J,1)}(s; y, x) = -\sum_{j=1}^J \log p_s(y_j|x)$ , corresponding to the negative Expected Log Pointwise Predictive Density (ELPPD). This approach targets the minimization of predictive risk. Under regularity conditions, the resulting posterior $\rho^{(J,1)}(s|y; x)$ concentrates at the optimal hyperparameter $s^*$ at a rate of $J^{-1/2}$ , with an asymptotic normal distribution.

The paper extends this framework to Semi-Modular Inference (SMI), a setting where a model consists of multiple modules, some of which may be misspecified. The authors identify SMI as a specific instance of a Sequential-Gibbs posterior (Winter et al., 2023). They introduce a new variant, the $(\eta, \beta)$ -SMI posterior, which jointly controls the flow of information from a suspect module (via $\eta$ ) and the robustness of the loss function (via $\beta$ ).

Key Contributions

Joint Hyperparameter Estimation: The paper provides a unified framework for the joint estimation of the learning rate $\eta$ and loss hyperparameters (e.g., $\beta$ in the $\beta$ -loss) using held-out data, moving beyond single-parameter tuning.
Theoretical Characterization: The authors establish asymptotic properties for both pooled and product posteriors. They prove that the product posterior concentrates on the optimal hyperparameters for prediction, while the pooled posterior targets the pseudo-true parameter. They also provide results for non-regular cases where the optimal $s$ lies on the boundary of the parameter space (e.g., $\eta=0$ in Cut models), showing convergence to a Gamma distribution.
SMI Integration: By framing SMI as a Sequential-Gibbs posterior, the authors import existing consistency and Bernstein-von Mises (BvM) theorems to the modular setting, validating the asymptotic behavior of the new $(\eta, \beta)$ -SMI update.
Estimators: Several estimators for $s$ are proposed and compared, including the posterior mean, the loss minimiser (MAP), the harmonic mean, and a KL-divergence minimiser.

Results
The proposed methods were evaluated through simulation studies and a real-world text analysis application:

Normal Mixture and State-Space Models: In synthetic experiments with misspecified models, the learned hyperparameters (particularly via the product loss) consistently outperformed fixed Bayesian inference ( $\eta=1$ ) and Cut models ( $\eta=0$ ) in terms of expected risk ratios on test data. The $(\eta, \beta)$ -SMI approach demonstrated performance comparable to $\eta$ -SMI but with the added flexibility of robust loss functions, often achieving lower risk than standard Bayes or Cut updates.
Text Analysis (Sense Change): Applied to a corpus of English text spanning 1810–2010 (the EDiSC model), the method was used to cluster word senses. The learned learning rates $\hat{\eta}$ yielded higher posterior predictive probabilities on held-out test data compared to $\eta=1$ or values selected via posterior predictive checks. The resulting clusters showed improved semantic coherence (measured by Brier scores) compared to standard Bayesian inference, effectively handling the model's misspecification and sparse data.
Asymptotic Validation: Simulations confirmed the theoretical predictions: product posteriors concentrated around optimal values as $J$ increased, while pooled posteriors remained diffuse, aligning with the distinct inferential goals of prediction versus parameter coverage.

Significance and Claims
The paper claims that treating inference hyperparameters as random variables with a posterior derived from held-out data provides a coherent route for uncertainty quantification and optimal selection in Generalised Bayesian Inference. The authors emphasize that the choice between pooled and product calibration corresponds to different inferential objectives (covering the pseudo-true parameter vs. maximizing predictive density), and thus, the resulting optimal hyperparameters need not coincide.

The work is particularly significant for multi-modular inference, where it offers a principled way to balance information flow between well-specified and misspecified modules without resorting to arbitrary tuning. The authors note that while the approach is computationally intensive for high-dimensional hyperparameter spaces, it is practical for low-dimensional cases (e.g., $\eta$ and $\beta$ ) and offers a robust alternative to methods like WAIC, which can be unreliable in misspecified settings. The framework supports the use of Generalised Bayes in complex, real-world problems involving heterogeneous data and model misspecification.