📊 statistics

Generalised Robust Bayes for Joint Inference of Model and Contamination

This paper introduces Hölder-Bayes, a Generalised Bayesian inference framework that enables joint inference of model parameters and contamination proportions using Hölder divergence, thereby providing robust parameter estimation, theoretical guarantees on bias and risk, and a self-contained probabilistic mechanism for uncertainty-aware outlier detection without external thresholds.

Original authors: Masahiro Fujisawa, Masaki Adachi, Takuo Matsubara

Published 2026-07-29

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Masahiro Fujisawa, Masaki Adachi, Takuo Matsubara

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Detective's Dilemma: When Data Lies

Imagine you are a detective trying to solve a mystery by looking at a pile of clues. In the world of statistics, this "mystery" is figuring out the true rules of how the world works, and the "clues" are data points collected from experiments or observations. For a long time, detectives relied on a method called Bayesian inference. Think of this as a super-smart detective who updates their theory about the crime every time they find a new clue. If the clues are honest and fit the pattern, this detective is brilliant. But there's a catch: if even a few clues are fake—like a planted fingerprint or a forged note—the whole theory can collapse. The detective gets confused, and the final conclusion is wrong. This is what happens when data is "contaminated" by outliers or errors.

To fix this, scientists developed a tougher version called Generalised Bayesian Inference (GBI). Instead of getting upset by a weird clue, this detective learns to ignore the parts of the clue that don't make sense, focusing only on the core pattern. It's like wearing noise-canceling headphones while listening to a song; you hear the music clearly even if someone is shouting in the background. However, there was still a problem with this approach. While the detective could ignore the noise, they couldn't tell how much noise was there, or exactly which clues were fake. They could solve the mystery, but they couldn't count the liars in the room. This paper introduces a new tool that does both: it solves the mystery and counts the liars, all at the same time.

The New Detective: Hölder-Bayes

The paper introduces a new framework called Hölder-Bayes. Imagine you are trying to bake the perfect cake based on a recipe, but you suspect someone has sneaked a handful of salt into your flour. A standard baker (standard Bayesian inference) would taste the batter, get confused by the salt, and ruin the cake. A "robust" baker (existing GBI methods) would ignore the salty taste and bake a good cake anyway, but they wouldn't know how much salt was in the bag.

Hölder-Bayes is like a baker who not only bakes a perfect cake despite the salt but also figures out exactly how much salt was added and which specific grains of flour were the culprits. It does this by treating the "amount of clean data" as a mystery to be solved alongside the recipe itself.

How It Works: The Scaled Model

The secret sauce of Hölder-Bayes is a clever trick involving a "scaled model." Usually, a statistical model assumes that all the data belongs to the same group (like assuming every person in a room is a fan of the same band). But in the real world, some people are just there to cause trouble (outliers).

Hölder-Bayes introduces a new character to the story: a variable called $\alpha$ (alpha). Think of $\alpha$ as a "clean data dial."

If $\alpha = 1$ , it means 100% of the data is honest.
If $\alpha = 0.8$ , it means the model assumes only 80% of the data is honest, and the other 20% is noise.

Instead of forcing the model to fit every single data point, Hölder-Bayes shrinks the model's expectations to match only the "clean" portion. It asks: "If I assume only 80% of this data is real, what does the recipe look like?" By adjusting this dial, the method can find the true recipe and the true percentage of noise simultaneously. It doesn't need to guess which data points are bad; it lets the math figure out the proportion of bad data naturally.

The Magic of the "Frequency-of-Detection" Score

Once the model has figured out the recipe and the noise level, it can do something amazing: it can point a finger at the liars. The paper calls this the Frequency-of-Detection (FoD) score.

Here is how it works in practice:

The computer runs the model thousands of times, each time drawing a slightly different guess for the recipe and the noise level (like rolling dice to see different possible worlds).
In each "world," it ranks the data points from "most likely to be real" to "most likely to be fake."
It counts how many times a specific data point was ranked as "fake."
If a data point is ranked as fake in 90% of the worlds, it gets a high FoD score. If it's ranked as fake only 10% of the time, it's probably safe.

This is a huge leap forward because it doesn't require a human to say, "Hey, anything with a score above 5 is bad." The model itself tells you how uncertain it is. If the model is unsure whether a point is bad, the score will be in the middle (like 50%), warning you to be careful. If it's sure, the score is near 0 or 100.

What the Paper Found

The authors tested this new detective on both fake data (simulations) and real-world data (like housing prices and machine performance logs).

In Simulations: They created scenarios where up to 40% of the data was fake noise. Standard methods failed miserably, getting the recipe completely wrong. Existing robust methods kept the recipe right but couldn't tell you how much noise was there. Hölder-Bayes, however, got the recipe right and correctly estimated the noise level (e.g., "It's 20% noise"). It also successfully identified the fake data points with high accuracy.
In Real Data: They applied it to real datasets where they secretly injected errors. Hölder-Bayes was able to clean the data effectively. When they removed the points the model flagged as "fake," the remaining data predicted future outcomes much better than if they had just used the standard method.

The paper also showed that the method is mathematically "safe." They proved that even if the noise is extreme, the model won't go crazy; the influence of a single bad data point is capped, so it can't break the whole system. They also showed that the method works well even when the noise is heavy and the data is messy, provided the noise doesn't look exactly like the real signal (a condition they call "small tail overlap").

Why It Matters

The most exciting part of this paper is that it unifies two things that usually require separate tools: robust inference (getting the right answer despite noise) and outlier detection (finding the noise). Before this, you might have used one tool to get a safe answer and a completely different tool to find the bad data. Hölder-Bayes does it all in one go, with a built-in measure of confidence.

The authors suggest that this approach is particularly useful when you don't know exactly what the "bad" data looks like. You don't need to build a complex model of the noise; the method just assumes the noise is different enough from the signal to be detected. While the paper focuses on independent data points (like a list of numbers), the authors hint that this could be a powerful foundation for more complex data in the future, like time-series or spatial data.

In short, Hölder-Bayes gives us a detective that doesn't just solve the case; it also counts the liars, points them out, and tells you how sure it is about its accusations. It turns a messy, noisy pile of clues into a clear, trustworthy story.

Technical Summary: Generalised Robust Bayes for Joint Inference of Model and Contamination

1. Problem Statement

Standard Bayesian inference is highly sensitive to data contamination; even a small proportion of outliers can severely distort posterior distributions. While Generalised Bayesian Inference (GBI) offers a robust alternative by replacing the log-likelihood with robust divergences (e.g., $\alpha$ -divergence, $\gamma$ -divergence), existing robust GBI frameworks typically provide only qualitative robustness. They mitigate sensitivity to contamination but lack an intrinsic mechanism to:

Quantify the contamination proportion.
Identify specific anomalous observations.
Propagate uncertainty regarding the contamination level to downstream tasks.

Furthermore, existing frequentist robust methods that jointly estimate model parameters and contamination ratios have not been fully integrated into a unified Bayesian framework that avoids explicit outlier modeling (e.g., Dirichlet process mixtures) or external anomaly-score thresholds.

2. Methodology: Hölder-Bayes

The paper introduces Hölder-Bayes, a GBI framework designed for the joint inference of the model parameter $\theta$ and the clean data ratio $\alpha$ (where the contamination ratio is $\epsilon = 1 - \alpha$ ).

Core Mechanism

The framework constructs a generalised joint posterior over $(\theta, \alpha)$ by applying the Hölder divergence to a scaled model density $\alpha P_\theta$ .

Scaled Model: Instead of fitting $P_\theta$ to the entire dataset, the model density is scaled by $\alpha \in (0, 1]$ . This scaling parameter acts as a formal inferential object representing the proportion of data generated by the target signal.
Hölder Divergence: The loss function is defined as the Hölder divergence between the empirical distribution and the scaled model $\alpha P_\theta$ . The divergence is parameterized by a power constant $\gamma > 0$ and a function $\phi$ .
Joint Posterior: The Hölder posterior is defined as:
$\pi_n^H(\theta, \alpha) \propto \exp\left\{ -\beta n H_n^{(\gamma)}(\theta, \alpha) \right\} \pi(\theta, \alpha)$
where $H_n^{(\gamma)}$ is the empirical Hölder risk and $\beta$ is a temperature parameter.

Temperature Calibration

A novel contribution is the interpretation of the temperature parameter $\beta$ . The authors show that for the Hölder posterior, temperature calibration corresponds to an affine volume scaling of the data space. Specifically, setting $\beta = |\det \hat{\Omega}_n|^{\gamma/2}$ (where $\hat{\Omega}_n$ is the sample covariance) is equivalent to performing inference on standardised data. This provides a direct, parameter-free modelling interpretation, distinct from standard GBI approaches that rely on iterative calibration to match frequentist confidence regions.

Outlier Detection (Frequency-of-Detection)

The framework enables uncertainty-aware outlier detection without explicit outlier models:

Draw samples $(\theta^{(s)}, \alpha^{(s)})$ from the joint Hölder posterior.
For each draw, rank observations by their model likelihood (clean data likelihood).
Flag the top $K^{(s)} = \text{round}(n \epsilon^{(s)})$ lowest-likelihood points as outliers.
Compute the Frequency-of-Detection (FoD) score for each observation by averaging the flagging indicator across all posterior draws.
This propagates uncertainty in both the model parameters and the contamination level to the observation-level scores, eliminating the need for user-defined thresholds.

3. Theoretical Contributions

The paper establishes comprehensive theoretical guarantees for the Hölder posterior under a heavy-contamination regime (where the contamination proportion $\epsilon_0$ is fixed and non-negligible):

Global Bias-Robustness: The authors prove that the Hölder posterior is globally bias-robust. The Posterior Influence Function (PIF) is uniformly bounded, meaning extreme outliers cannot arbitrarily perturb the posterior density. The influence of outliers saturates for standard tail-decaying model densities.
Finite-Sample Excess-Risk Bound: A bound is derived showing the posterior concentrates around the minimiser of the population Hölder risk at a rate of $O(n^{-1/2})$ .
Bias Control via Tail Overlap: The bias between the population minimiser $(\theta^*, \alpha^*)$ and the true parameter $(\theta_0, \alpha_0)$ is strictly bounded by the tail-overlap quantity $\rho_\gamma(\theta^*)$ . This quantity measures the overlap between the outlier distribution and the model's low-density tails. If the overlap is small, the bias is controlled regardless of the contamination magnitude.
Bernstein–von Mises (BvM) Theorem: The paper proves the asymptotic normality of the Hölder posterior around the population minimiser. It further establishes that the distributional bias of this limiting normal distribution is controlled by the tail-overlap quantity, ensuring the posterior remains tightly bound to the oracle distribution under small overlap conditions.

4. Empirical Results

The framework was evaluated on synthetic data and real-world regression benchmarks (cal-housing, kin8nm, pumadyn-32fh).

Parameter Inference: Hölder-Bayes successfully safeguarded parameter estimates against contamination, maintaining accuracy where standard Bayesian inference and heavy-tailed alternatives (e.g., Student-t) failed or degraded.
Contamination Recovery: The posterior for $\alpha$ (and thus $\epsilon$ ) accurately recovered the true contamination proportion across various contamination levels and types (separated clusters, response shifts).
Outlier Detection: The FoD scores effectively identified anomalous observations. The method outperformed standard unsupervised anomaly detectors (Isolation Forest, LOF, One-Class SVM) in terms of ranking accuracy (AUPRC) and predictive performance after data cleaning.
Hyperparameter Sensitivity: The tuning parameter $\gamma$ governs the robustness-efficiency trade-off. Intermediate values (e.g., $\gamma = 0.5$ ) provided an optimal balance, while extreme values led to either under-estimation of contamination or excessive conservatism.

5. Significance and Claims

The paper claims that Hölder-Bayes offers a unified framework for robustification and outlier detection with principled uncertainty propagation. Its primary significance lies in:

Joint Inference: Moving beyond mere parameter protection to explicitly infer the contamination level and identify outliers within a single probabilistic model.
Model-Based Detection: Providing a mechanism for outlier detection that relies on the robustified model likelihood rather than external, heuristic scoring rules or explicit outlier mixture models.
Theoretical Rigor: Providing the first rigorous theoretical guarantees for joint inference of model and contamination parameters in a GBI setting under heavy contamination, linking bias control directly to tail-overlap properties.
Interpretability: Offering a geometric interpretation of the temperature parameter as affine volume scaling, simplifying calibration.

The authors conclude that robust Bayesian inference need not merely protect a scientific model from anomalies but can simultaneously quantify and propagate the inherent uncertainty surrounding the contamination itself.