Technical Summary: Maxitive Donsker-Varadhan Formulation for Possibilistic Variational Inference

1. Problem Statement

Variational Inference (VI) is a cornerstone of modern Bayesian learning, enabling approximate inference in complex models by maximizing the Evidence Lower Bound (ELBO). However, standard VI relies on expectations and divergences (like the Kullback-Leibler divergence) defined through high-dimensional integrals, which presuppose additive probability measures. This reliance makes analytical treatment difficult and necessitates heavy approximations.

Conversely, Possibility Theory offers an imprecise probability framework for modeling epistemic uncertainty without requiring additivity. Instead of sums, it utilizes max/min operators, allowing for the direct modeling of uncertainty under sparse or imprecise information. While Possibility Theory provides robustness and interpretability, adapting VI to this setting is non-trivial. Standard objectives like the ELBO and divergences like KL do not have direct analogues in the possibilistic setting because they fundamentally rely on additive measures. Existing attempts, such as those by Cella and Martin (2025), have used Monte Carlo strategies to approximate possibilistic inferential models, but a principled, general variational framework for possibilistic VI has been lacking.

2. Methodology

The Maxitive Donsker-Varadhan Analogue

The core theoretical contribution of this work is the establishment of a maxitive analogue of the classical Donsker-Varadhan (DV) variational formula.

In standard probabilistic VI, the DV formula characterizes the log-evidence as the supremum of an objective involving the KL divergence. The authors derive a dual variational characterization for the possibilistic setting. Let $\pi$ be a prior possibility function and $\ell$ a loss function. The maxitive posterior $g^*_{\max}$ is defined as:
$$g^*_{\max}(\theta) = \frac{\exp(-\ell(\theta))\pi(\theta)}{\sup_{\theta'} \exp(-\ell(\theta'))\pi(\theta')}$$

The authors prove that the log-consistency (the log of the maxitive marginal likelihood) can be expressed via two dual variational problems:

1. Lower Bound (Maximization): $\log Z_{\max} = \sup_{g \in \mathcal{F}(\Theta)} \inf_{\theta} \{ -\ell(\theta) - \log(g(\theta)/\pi(\theta)) \}$
2. Upper Bound (Minimization): $\log Z_{\max} = \inf_{g \in \mathcal{F}(\Theta)} \sup_{\theta} \{ -\ell(\theta) - \log(g(\theta)/\pi(\theta)) \}$

Here, $\mathcal{F}(\Theta)$ is the set of possibility functions. The quantity optimized on the right-hand side is termed the Consistency Bound (CBO).

• Maximizing the lower CBO yields possibility functions that lower-bound the true posterior ($g \preceq g^*_{\max}$).
• Minimizing the upper CBO yields possibility functions that upper-bound the true posterior ($g^*_{\max} \preceq g$).

The standard possibilistic posterior $g^*_{\max}$ is the unique element that simultaneously satisfies both bounds, acting as the canonical representative of the solution set. The optimization involves a max-relative entropy ($D_{\max}$), defined as $D_{\max}(g \| f) = \sup_{\theta} \log(g(\theta)/f(\theta))$, which plays a role analogous to the KL divergence in standard VI.

Exponential Family Treatment and Learning Rules

To make this framework practical for neural networks, the authors develop a treatment for exponential-family possibility functions. They define a class of candidates $g_\lambda(\theta) = \exp(\lambda^\top T(\theta) - A^\dagger(\lambda) - A(\theta))$, where $A^\dagger$ is the Legendre transform of the log-partition function $A$.

Key properties derived include:

• Conjugacy: Under an uninformative prior, the posterior remains within the same exponential family, with hyperparameters updated as a running mean of sufficient statistics.
• Newton-like Updates: By applying Danskin's theorem and second-order Taylor approximations, the authors derive a learning rule for optimizing the CBO. This leads to a natural gradient update that resembles a Newton step:
  $$\eta_{t+1} = \eta_t - \rho_t H_s(\eta_t)^{-1} \nabla \hat{\ell}(\eta_t)$$
  where $H_s$ is a signed curvature matrix dependent on the Hessian of the loss and the candidate's precision.

The CBOpt Optimizers

Based on these theoretical derivations, the authors propose a family of optimizers called CBOpt.

• uCBOpt: Optimizes the upper CBO (minimizing the upper bound).
• lCBOpt: Optimizes the lower CBO (maximizing the lower bound).
• Adaptive Variants (uCBOpt-adapt, lCBOpt-adapt): These variants dynamically adjust the curvature parameter $\vartheta$ during training to satisfy the necessary convexity/concavity conditions of the inner optimization problem.

The practical implementation uses a diagonal approximation of the Hessian (similar to Adam or Vprop) and exponential moving averages (EMAs) for gradient and squared-gradient estimates, making it computationally efficient and scalable to large neural networks.

3. Key Contributions

1. Theoretical Formulation: The paper establishes the first principled maxitive analogue of the Donsker-Varadhan formulation, providing a rigorous variational foundation for possibilistic inference.
2. Dual Characterization: It introduces the Consistency Bound (CBO) with dual lower and upper formulations, demonstrating that the true possibilistic posterior is the unique intersection of these bounds.
3. Exponential Family Framework: The authors develop a conjugate prior framework for possibilistic exponential families, revealing structural parallels with the probabilistic case while accommodating the maxitive nature of the theory.
4. CBOpt Optimizers: A new family of optimizers is derived and implemented, offering a practical method for training neural networks with possibilistic variational inference.
5. Empirical Validation: The methods are evaluated on in-domain and out-of-domain (OOD) image classification tasks, demonstrating competitive predictive performance and reliable uncertainty estimates.

4. Experimental Results

The authors evaluated CBOpt variants (uCBOpt, uCBOpt-adapt, lCBOpt-adapt) against baselines including SGD, AdamW, IVON, MC Dropout, Laplace approximation, and SWAG on three datasets: Fashion-MNIST, CIFAR-10, and CIFAR-100.

• In-Domain Performance: On Fashion-MNIST, the CBOpt variants achieved the strongest overall performance, with lCBOpt-adapt obtaining the best accuracy (0.913) and Negative Log-Likelihood (NLL). On CIFAR-100, all CBOpt variants outperformed baselines on most in-domain metrics, suggesting scalability to larger models.
• Out-of-Domain (OOD) Detection: The methods demonstrated competitive OOD detection capabilities. uCBOpt showed consistent behavior across datasets, matching or approaching the performance of MC Dropout and SWAG in terms of FPR@95 and AUROC.
• Efficiency: The CBOpt variants maintained computational efficiency comparable to first-order optimizers like SGD and Adam, requiring only per-parameter gradient and scalar curvature estimates, without the need for multiple stochastic forward passes at inference time (unlike MC Dropout).
• Combination with MC Dropout: An additional experiment showed that combining CBOpt with MC Dropout preserved the strong in-domain performance of MC Dropout while further improving OOD detection metrics.

5. Significance and Claims

The paper claims that possibilistic variational inference is not merely a reformulation of existing methods but offers a distinct perspective on uncertainty representation that leads to practically useful algorithms.

• Conceptual Shift: By moving from additive probability to maxitive possibility, the framework naturally handles epistemic uncertainty and allows for fully uninformative priors without the issue of improper integrals common in probability theory.
• Practical Utility: The development of CBOpt demonstrates that the theoretical framework can be instantiated into scalable optimizers that compete with state-of-the-art uncertainty-aware learning methods (like IVON and MC Dropout) while avoiding the computational overhead of sampling at test time.
• Foundation for Future Work: The authors state that their work provides a foundation for extending variational reasoning to possibility theory. They note that components of this framework (specifically the maxitive pseudo-divergence and posterior under uninformative priors) have already been leveraged by Ni et al. (2026) to derive second-order predictors, validating the utility of the underlying theory.

The authors remain modest regarding limitations, acknowledging that their Newton-like updates rely on second-order approximations and specific curvature conditions, and that their current empirical validation is restricted to isotropic Gaussian candidates with diagonal Fisher approximations. However, they assert that the framework successfully bridges the gap between theoretical possibilistic inference and scalable deep learning applications.