Technical Summary: Integral Imprecise Probability Metrics (IIPM)

1. Problem Statement

Quantifying differences between probability distributions is fundamental to statistics and machine learning, primarily for comparing statistical (aleatoric) uncertainty. However, classical probability theory, which relies on a single additive measure, struggles to represent epistemic uncertainty (EU)—uncertainty arising from incomplete knowledge, ambiguity, or "unknown unknowns." Classical models often fail to distinguish between genuine ignorance and indifference, leading to issues such as overconfidence and poor generalization in machine learning systems.

While Imprecise Probabilistic Machine Learning (IPML) offers richer representations (e.g., credal sets, belief functions, lower/upper probabilities) to model EU, there is a lack of robust statistical metrics to compare these models. Existing divergence measures often lack a unified theoretical framework or fail to metrize convergence in the space of imprecise models. Furthermore, quantifying EU within a single imprecise model remains challenging, with many existing measures lacking rigorous axiomatic support or computational scalability.

2. Methodology

2.1 Theoretical Framework: IIPM

The authors introduce the Integral Imprecise Probability Metric (IIPM), a generalization of classical Integral Probability Metrics (IPMs) to the setting of capacities (non-additive set functions).

• Mathematical Foundation: The framework leverages Choquet integration to define expectations over capacities, generalizing the Lebesgue integral used in standard IPMs.
• Definition: For a function class $\mathcal{F} \subseteq C_b(X)$ and two capacities $\nu, \mu$, the IIPM is defined as:
  $$\text{IIPM}_{\mathcal{F}}(\nu, \mu) := \sup_{f \in \mathcal{F}} \left| \int f d\nu - \int f d\mu \right|$$
  where the integrals are Choquet integrals.
• Properties: The authors establish that IIPM is a pseudometric on the space of capacities. Under specific conditions (e.g., when $\mathcal{F}$ is dense in $C_b(X)$ with respect to the $\|\cdot\|_\infty$ norm), IIPM serves as a valid metric that metrises Choquet weak convergence. This ensures that convergence in IIPM implies the "true" convergence of the underlying capacities.
• Generality: The framework encompasses various IP models, including lower probabilities, belief functions, and probability intervals, as special cases of capacities.

2.2 Epistemic Uncertainty Quantification: MMI

To address the quantification of EU within a single model, the authors propose Maximum Mean Imprecision (MMI). This measure quantifies the discrepancy between a capacity (e.g., a lower probability $\underline{P}$) and its conjugate (the upper probability $\overline{P}$).

• Definition:
  $$\text{MMI}_{\mathcal{F}}(\underline{P}) := \text{IIPM}_{\mathcal{F}}(\underline{P}, \overline{P}) = \sup_{f \in \mathcal{F}} \left( \int f d\overline{P} - \int f d\underline{P} \right)$$
  (Note: The absolute value is omitted because $\overline{P}$ setwise dominates $\underline{P}$, ensuring non-negativity).
• Axiomatic Properties: The authors prove that MMI satisfies key axioms for credal uncertainty measures proposed in the literature (A1–A6), including non-negativity, continuity, monotonicity, and probability consistency (zero EU for precise probabilities).
• Computational Efficiency: For the specific case of Total Variation (TV) distance, computing MMI exactly requires evaluating $2^K$ subsets, which is intractable for large $K$. The authors derive a linear-time upper bound (MMI-Lin) by approximating the lower probability with the closest $\epsilon$-contamination model. This bound is computationally efficient ($O(K)$) and theoretically justified via the monotonicity axiom.

3. Key Contributions

1. Unified Metric Framework: The introduction of IIPM, the first family of statistical distances specifically designed for imprecise probabilities using Choquet integration. This framework generalizes classical IPMs (like Wasserstein distance and MMD) to the imprecise setting.
2. Theoretical Guarantees: Proof that IIPM metrises Choquet weak convergence, providing a rigorous foundation for comparing imprecise models. The paper also demonstrates how IIPM can recover existing results, such as those in optimal transport under $\epsilon$-contamination.
3. New EU Measure (MMI): The proposal of MMI as a principled measure of epistemic uncertainty that satisfies established axiomatic properties.
4. Scalable Approximation: The derivation of a linear-time upper bound (MMI-Lin) for MMI, enabling its application in high-dimensional classification tasks where exact computation is infeasible.

4. Empirical Results

The authors validated MMI and MMI-Lin through selective classification experiments on datasets with varying numbers of classes ($K=7, 10, 100$), including UCI datasets (Obesity, Digits) and image datasets (CIFAR-10, CIFAR-100).

• Performance Metric: The study used Accuracy-Rejection (AR) curves, where a model abstains from predicting on uncertain inputs. An informative EU measure should yield a monotonic AR curve approaching the top-left corner.
• Comparisons: MMI and MMI-Lin were compared against the Entropy Difference (E-Diff) and the Generalised Hartley (GH) measure.
• Findings:
  • MMI and MMI-Lin consistently outperformed E-Diff across all datasets.
  • MMI and MMI-Lin performed comparably to GH, which is often considered the gold standard for credal UQ.
  • Scalability: On large-scale problems ($K=100$), where exact computation of MMI and GH becomes intractable ($O(2^K)$), the linear-time bound MMI-Lin remained efficient and continued to outperform E-Diff, demonstrating practical utility in high-dimensional settings.

5. Significance and Claims

The paper claims to advance both the theory and practice of IPML:

• Theoretical Advancement: It provides a principled, Choquet-integral-based framework for comparing capacities, filling a gap in the literature regarding statistical distances for imprecise models. It establishes the conditions under which these metrics are valid and metrize convergence.
• Practical Utility: By introducing MMI, the work offers a new, axiomatically sound tool for quantifying epistemic uncertainty. The linear-time upper bound addresses the computational bottleneck that has limited the application of credal UQ measures in large-scale machine learning.
• Broader Impact: The framework enables the comparison of diverse IP models (e.g., belief functions, random sets) under a unified metric, potentially facilitating better decision-making under ambiguity in safety-critical applications and robust statistics.

The authors acknowledge limitations, specifically the current lack of a sampling-based estimation method for general IIPMs (unlike classical IPMs) due to the subjective nature of imprecise models, and identify the development of non-parametric estimators as a key direction for future work.