Technical Summary: Structural Separation of Epistemic and Aleatoric Uncertainty

1. Problem Statement

Predictive uncertainty is traditionally decomposed into epistemic uncertainty (EU) (reducible model ignorance) and aleatoric uncertainty (AU) (persistent data ambiguity). While conceptually distinct, standard methods for estimating these quantities often fail to separate them empirically. Techniques such as Monte Carlo dropout, deep ensembles, and evidential networks derive both estimates from the same predictive distribution $p(y|x)$.

This creates an "algebraic trap": because both EU and AU are functions of the same distribution, they inherit a shared algebraic dependency. Recent benchmarks indicate that these components are often strongly correlated ($\rho \ge 0.78$), making it impossible to distinguish between a model that is confused (high EU) and data that is inherently ambiguous (high AU). Formal results (e.g., Tomov et al., 2026) suggest that no post-hoc function of $p(y|x)$ can uniquely distinguish between reducible error and intrinsic ambiguity, as different underlying uncertainty states can induce the same optimal predictive distribution.

2. Methodology: Structural Separation

The paper proposes structural separation as a design principle to break this algebraic coupling. Instead of deriving both uncertainties from a single predictive object, the authors assign EU and AU to disjoint parameter paths within Supervised Latent Variable Models (SLVMs).

Core Architecture: Credal Concept Bottleneck Model (CBM)

The primary instantiation is a Credal CBM built upon a frozen encoder (e.g., DistilBERT). The architecture introduces three orthogonal parameter paths:

1. Mean Path ($\mu$): Predicts the point estimate of concepts, feeding the task head.
2. Epistemic Head ($\sigma_{epi}$): Trained to track reducible prediction error. It is parameterized by a disjoint set $\phi_{epi}$ and supervised by a loss combining prediction error (with stop-gradients to prevent gaming) and a Hausdorff KL regularizer.
3. Aleatoric Head ($\sigma_{ale}$): Trained to track persistent label ambiguity. It is parameterized by a disjoint set $\phi_{ale}$ and supervised directly by the entropy of the ground-truth label distribution ($H[\hat{p}^*]$), derived from multi-annotator disagreement or corpus-derived answer distributions.

Key Mechanisms

• Disjoint Parameterization: The epistemic and aleatoric heads use non-overlapping parameter sets ($\phi_{epi} \cap \phi_{ale} = \emptyset$).
• Orthogonal Projections: The encoder output $h$ is projected into orthogonal subspaces ($W_{epi}h$, $W_{ale}h$) to ensure the heads receive non-overlapping feature inputs.
• Gradient Isolation: The training loss is decomposed such that the epistemic loss ($L_{epi}$) depends only on $\phi_{epi}$ and the aleatoric loss ($L_{ale}$) depends only on $\phi_{ale}$. The authors prove (Theorem 3.3) that under these conditions, the cross-gradients are identically zero ($\nabla_{\phi_{epi}} L_{ale} = 0$ and $\nabla_{\phi_{ale}} L_{epi} = 0$).
• Credal Sets: The model represents concepts as credal sets (convex sets of probability distributions) defined by a mean $\mu$ and an epistemic covariance $\Sigma_{epi}$, while the aleatoric variance $\sigma_{ale}$ is modeled separately outside the credal set.

3. Key Contributions

1. Structural Separation Principle: The paper introduces a framework for SLVMs where EU and AU are computed from distinct supervised parameter paths rather than post-hoc decomposition of $p(y|x)$.
2. Gradient-Isolation Theorem: A formal proof demonstrating that with disjoint parameters and separate loss terms, the EU and AU heads remain uncoupled during training, avoiding the algebraic trap identified in prior work.
3. Credal CBM Instantiation: A specific implementation using a Credal Concept Bottleneck Model that utilizes orthogonal projections and hybrid loss functions to achieve this separation.
4. Comprehensive Diagnostics: The authors introduce and validate metrics to test not just decorrelation, but semantic validity:
   • Gradient Isolation: Verifying zero cross-gradients.
   • Correlation: Measuring $\rho(U_{epi}, U_{ale})$.
   • Data-Scaling Reducibility: Checking if EU decreases as data increases.
   • Ambiguity Tracking: Verifying if AU correlates with ground-truth ambiguity ($H[\hat{p}^*]$).

4. Experimental Results

The method was evaluated across five benchmarks: three concept-bottleneck datasets (CEBaB, HateXplain, GoEmotions) with annotator-derived ambiguity, and two QA datasets (MAQA*, AmbigQA*) with corpus-derived ambiguity.

• Decorrelation: Structural separation reduced the Spearman correlation between EU and AU from the baseline range of $\rho \in [0.75, 0.84]$ (observed in standard methods like Deep Ensembles and MC Dropout) to $\rho \approx 0.05$. Even without an explicit decorrelation penalty ($\lambda_d = 0$), the correlation remained low ($\rho \in [0.12, 0.23]$), significantly below the "post-hoc floor."
• Semantic Validity:
  • Aleatoric Tracking: The aleatoric head showed strong correlation with ground-truth ambiguity ($\rho(\sigma_{ale}, H) \in [0.42, 0.74]$), whereas baselines showed much weaker tracking.
  • Epistemic Sensitivity: The epistemic head remained sensitive to prediction error and showed the expected reducibility signature (decreasing gradient norms as training data increased).
• Downstream Utility: The method enabled quadrant-based routing, successfully distinguishing between:
  • DATA (High EU, Low AU): Cases needing more data (e.g., obscure facts).
  • REVIEW (Low EU, High AU): Cases needing human review due to inherent ambiguity (e.g., subjective opinions).
    Standard baselines collapsed these distinct cases onto a diagonal, failing to support these operational decisions.
• Robustness: The results held across different encoder architectures (DistilBERT, BERT, RoBERTa) and were corroborated on a Self-Explaining Neural Network (SENN) instantiation.

5. Significance and Claims

The paper claims that supervised latent-variable architectures provide a practical route toward uncertainty estimates that are not merely decorrelated but operationally distinguishable.

• Overcoming Impossibility: By moving estimation from the output space (functions of $p(y|x)$) to the parameter space (separate heads with separate gradients), the approach circumvents the impossibility results that apply to post-hoc decompositions.
• Operational Distinction: The primary significance is not just statistical decorrelation, but the ability to support different decisions: high EU signals a need for more data or model improvement, while high AU signals a need for abstention or human review.
• Limitations: The authors modestly note that the framework relies on the availability of supervised latent structures (concepts) and ambiguity-related supervision signals (e.g., multi-annotator data). They acknowledge that scaling to end-to-end fine-tuning of large LLMs may require careful gradient routing to maintain separation, and that the current benchmarks are purpose-built and relatively small.

In summary, the paper argues that architectural separation, combined with distinct supervision targets, is a necessary and effective condition for producing uncertainty estimates that are semantically valid and useful for decision-making.