🤖 machine learning

On the QUEST for Uncertainty Quantification via Highest Density Regions

This paper introduces QUEST, a novel uncertainty quantification framework that characterizes uncertainty via the volume of highest density regions, offering a theoretically grounded alternative to proper scoring rules that satisfies key axioms like monotonicity and invariance while outperforming standard measures in selective prediction benchmarks.

Original authors: Sam Goring, Tom Kuipers, Nicola Paoletti, David S. Watson

Published 2026-06-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sam Goring, Tom Kuipers, Nicola Paoletti, David S. Watson

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather. You have a model that says, "It will rain tomorrow." But how sure is the model? Is it 99% sure, or is it just guessing? In the world of machine learning, this "how sure" part is called Uncertainty Quantification (UQ).

For a long time, scientists have used standard tools to measure this uncertainty, kind of like using a ruler to measure everything. But the authors of this paper, Sam Goring and his team, argue that sometimes a ruler is the wrong tool. If you try to measure the "squishiness" of a jellyfish with a ruler, you get a weird number that doesn't really tell you what the jellyfish is like.

Here is a simple breakdown of their new idea, QUEST, using everyday analogies.

The Problem: The "Ruler" Doesn't Always Work

Most current methods measure uncertainty by looking at the average (the mean) and how far things spread out from that average (variance).

The Analogy: Imagine you are looking at a crowd of people.
- Scenario A (Normal): Everyone is standing in a tight, neat circle. The average is right in the middle, and everyone is close to it. The "uncertainty" is low.
- Scenario B (Skewed): Most people are in a tight circle, but one giant person is standing 10 miles away. The "average" position of the crowd is now pulled way over toward that giant.
- Scenario C (Bimodal): Half the crowd is in a circle on the left, and half is in a circle on the right, with a huge empty gap in the middle.

The old methods (like variance) get confused here. In Scenario B, the giant person makes the "spread" look huge, so the computer thinks the uncertainty is massive, even though 99% of the people are actually very predictable. In Scenario C, the old methods might say the uncertainty is high because the crowd is split, but they might also accidentally assign high probability to the empty gap in the middle, which is impossible.

The paper argues that when we want to find the most likely outcome (the "mode" or the peak of the crowd), these old tools give us misleading answers.

The Solution: QUEST (The "Flashlight" Method)

The authors propose a new framework called QUEST (Quantifying Uncertainty via highest dEnSiTy regions).

Instead of measuring the whole spread from the average, QUEST asks a different question: "How much space do we need to cover to catch the most probable 90% of the crowd?"

The Analogy: Imagine you have a flashlight that can only shine on the brightest parts of a room (the areas with the most people).
- Low Uncertainty: If the crowd is a tight huddle, your flashlight only needs to be a tiny circle to catch 90% of them. The "volume" of the light is small.
- High Uncertainty: If the crowd is spread out over a huge field, your flashlight has to be massive to catch 90% of them. The "volume" of the light is huge.

QUEST measures the size (volume) of that flashlight beam.

Small Volume = High confidence (the crowd is bunched up).
Large Volume = Low confidence (the crowd is scattered).

This works perfectly for weird shapes. If the crowd is split into two groups (Scenario C), the flashlight just shines on both groups separately. It doesn't waste light on the empty gap in the middle. This gives a much more honest answer about how uncertain the prediction really is.

The Three Types of Uncertainty

The paper breaks uncertainty down into three buckets, and QUEST measures all of them:

Aleatoric Uncertainty (The "Noise"): This is the natural randomness of the world. Even if you knew everything about the model, the weather might still be unpredictable. QUEST measures this by looking at the "tightness" of the true crowd.
Epistemic Uncertainty (The "Ignorance"): This is the uncertainty because the model doesn't know enough yet. It's like a student who hasn't studied. QUEST measures this by looking at how spread out the model's beliefs are. If the model is guessing wildly, the "flashlight" is huge. If the model is confident, the flashlight is small.
Total Uncertainty: This is the combination of both. QUEST combines the natural noise and the model's ignorance into one score.

Why This Matters (The "Axioms")

The authors didn't just make up a cool idea; they proved it follows a set of logical rules (axioms) that any good uncertainty measure should follow.

Rule 1: If you spread the crowd out more, the uncertainty should go up. (QUEST passes this; old methods sometimes fail).
Rule 2: If you just move the crowd to a different location without spreading them out, the uncertainty should stay the same. (QUEST passes this; old methods sometimes get confused).
Rule 3: The measure should never be negative. (QUEST passes this; some old math tools can give negative numbers, which makes no sense for "uncertainty").

The Results

The team tested QUEST against the old methods (variance and entropy) using computer simulations with tricky data shapes (skewed crowds, split crowds).

The Result: QUEST was better at picking out the most reliable predictions. When the researchers told the computer to only make predictions on the "most certain" data points, QUEST helped them avoid mistakes much better than the old methods, especially when the data was weird or had outliers.

Summary

In short, the paper says: Stop measuring uncertainty by how far the average is from the edges. Start measuring it by how much space you need to cover the "most likely" stuff.

It's like switching from measuring the distance between the two ends of a messy pile of laundry (which changes wildly if you add one sock) to measuring the size of the basket needed to hold the cleanest, most folded 90% of the clothes. It's a more robust, logical, and "honest" way to know how much you can trust your AI's predictions.

Technical Summary: QUEST for Uncertainty Quantification via Highest Density Regions

1. Problem Statement

Uncertainty quantification (UQ) is critical for reliable decision-making in safety-critical probabilistic machine learning. While entropy-based measures are standard for classification, regression UQ has largely relied on Proper Scoring Rules (PSRs), which operationalize uncertainty as the pointwise predictive risk of a model relative to the true distribution.

The authors identify a fundamental flaw in PSR-based approaches (such as variance and differential entropy) when the target statistic is not the conditional expectation (mean). In many practical settings—such as reinforcement learning or Bayesian modeling—the target is the conditional mode (Maximum A Posteriori, or MAP). In the presence of skewness, multimodality, or outliers, the mean and mode can diverge significantly. Standard measures like variance often overstate uncertainty in these cases because they are sensitive to tail mass rather than the concentration of probability around the mode. Furthermore, existing axiomatic frameworks for regression UQ often fail to satisfy intuitive properties like monotonicity under distributional spread or invariance to location shifts when applied to non-Gaussian or asymmetric distributions.

2. Methodology: The QUEST Framework

The paper introduces QUEST (Quantifying Uncertainty via highest dEnSiTy regions), a framework that characterizes uncertainty by the volume of the most probable subset of a distribution's support, rather than by pointwise risk.

Core Concepts

Highest Density Regions (HDRs): For a probability density function (PDF) $p$ and a robustness parameter $\alpha \in (0,1)$ , the $(1-\alpha)$ -HDR ( $C_\alpha$ ) is the smallest set containing $1-\alpha$ of the probability mass. It is defined as $C_\alpha = \{x : p(x) \ge t_\alpha(p)\}$ , where $t_\alpha$ is the threshold ensuring the mass constraint.
$\alpha$ -Volume (Local Measure): Defined as $V_\alpha(p) := \lambda(C_\alpha)$ , where $\lambda$ is the Lebesgue measure. This measures the "size" of the region containing the bulk of the probability mass. Small volumes indicate high concentration (low uncertainty); large volumes indicate dispersion (high uncertainty).
Integrated Volume (Global Measure): Defined as $IV(p) = \int_0^1 V_\alpha(p) d\alpha$ . This summarizes the entire coverage curve, providing a global assessment of concentration. The authors establish a geometric equivalence between the coverage curve and the Lorenz curve from economics, and between integrated volume and the Gini coefficient.

Uncertainty Typology in QUEST

The framework defines three types of uncertainty based on a second-order distribution $q$ (representing beliefs over parameters $\theta$ ) and the true ground-truth distribution $p_{\theta^*}$ :

Aleatoric Uncertainty (AU): Quantified by the volume of the HDR of the true distribution $p_{\theta^*}$ $p_{θ^{*}}$ . This captures the irreducible uncertainty inherent in the data-generating process.
- Local: $V_\alpha(p_{\theta^*})$
- Global: $IV(p_{\theta^*})$
Epistemic Uncertainty (EU): Quantified by the volume of the HDR of the second-order distribution $q$ $q$ . Unlike standard approaches that treat EU as a residual (Total - Aleatoric), QUEST treats EU as a purely subjective measure of the agent's belief concentration, independent of the truth.
- Local: $V_\alpha(q)$
- Global: $IV(q)$
Total Uncertainty (TU): Defined via an interaction between AU and EU. Since AU and EU exist on different spaces, they are not simply added. Instead, TU is modeled as the AU volume scaled by a penalty factor derived from the divergence between the true conditional density and the predictive conditional density within their respective HDRs.
- Formula: $TU = V_\alpha(p_{\theta^*}) \cdot g_f(D_f(p^*_\alpha || \hat{p}_\alpha))$
- Here, $D_f$ is an admissible $f$ -divergence (e.g., Total Variation), and $g_f$ is a strictly increasing function that penalizes misalignment between the agent's belief and the truth.

3. Key Contributions

The paper makes four primary contributions:

Conceptual: Proposes a volume-centric interpretation of uncertainty based on density concentration and a user-supplied robustness parameter $\alpha$ , offering an alternative to risk-based PSR measures.
Methodological: Introduces novel local ( $\alpha$ -volume) and global (integrated volume) measures for aleatoric and epistemic uncertainty. It establishes theoretical connections between these measures and classical statistics from information theory (differential entropy) and economics (Lorenz curves, Gini coefficients).
Theoretical: Demonstrates that QUEST measures for AU and EU satisfy a set of axioms adapted from the UQ literature (A0–A5). These include non-negativity, minimization at Dirac masses, monotonicity under "levelling" (distributional spread), and invariance to location shifts. Notably, the paper argues that Total Uncertainty (TU) should not satisfy the monotonicity axiom (A4) in all cases, as improved calibration (reducing TU) can occur even if epistemic uncertainty increases.
Empirical: Validates the framework through selective prediction benchmarks. Results show that QUEST (particularly the global measure) outperforms standard variance and differential entropy in skewed and multimodal settings, leading to more accurate rankings of prediction reliability.

4. Experimental Results

The authors evaluated QUEST against variance, differential entropy, and random baselines using deep mixture-of-experts ensembles on three synthetic data generating processes (DGPs):

Unimodal Gaussian: All methods performed similarly, as expected.
Unimodal Skewed: Variance and entropy struggled due to sensitivity to tail mass. QUEST provided more reliable uncertainty rankings.
Bimodal Mixture: Standard measures often mis-rank uncertainty in multimodal settings. QUEST significantly outperformed variance and entropy, correctly identifying regions of high density concentration.

In selective prediction tasks (where predictions are made only on the most certain samples), QUEST (global) achieved lower loss across all trials compared to competitors. A supplementary active learning experiment showed QUEST performing on par with variance and entropy, though the authors note that QUEST's performance is sensitive to the quality of the model's estimate of the true density in non-oracle settings.

5. Significance and Claims

The paper claims that QUEST addresses specific pathologies of PSR-based UQ, particularly when the target is the mode rather than the mean. By shifting the focus from pointwise risk to the concentration of measure, QUEST provides a more intuitive and robust characterization of uncertainty in non-Gaussian, skewed, or multimodal distributions.

The authors emphasize that their approach:

Avoids the counterintuitive results of variance-based measures in the presence of outliers or skew.
Provides a sound axiomatic foundation for AU and EU that respects the geometry of probability densities.
Offers a flexible framework where the robustness parameter $\alpha$ allows users to control the sensitivity to tail fluctuations.

The paper concludes modestly, acknowledging limitations such as the assumption of absolute continuity with respect to the Lebesgue measure and the computational cost of estimating HDRs in high dimensions. It suggests future work in extending the framework to statistical inference and handling distributions with atoms or lower-dimensional supports.