💻 computer science

Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?

This paper reveals that deep ensembles fail to effectively capture uncertainty in graph neural networks because independently trained models converge to functionally identical predictions, causing epistemic collapse and limiting the ensemble's benefits to merely stabilizing point predictions rather than improving uncertainty estimates.

Original authors: Pedro C. Vieira, Pedro Ribeiro, Viacheslav Borovitskiy

Published 2026-05-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pedro C. Vieira, Pedro Ribeiro, Viacheslav Borovitskiy

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Do "Crowds" Work for Graph AI?

Imagine you are trying to solve a difficult puzzle. In the world of Artificial Intelligence (AI), a popular trick to get a better answer is to ask a group of experts (an "ensemble") instead of just one. You train five different AI models on the same data, starting them with different random seeds (like giving them different initial guesses), and then you let them vote on the final answer.

In standard AI tasks (like recognizing cats in photos), this "crowd wisdom" works beautifully. The experts often disagree on why they think something is a cat, and that disagreement helps the group realize, "Hey, we aren't 100% sure about this one!" This is called capturing uncertainty.

This paper asks: Does this same "crowd wisdom" trick work when the AI is looking at graphs?

What is a graph? Think of a graph not as a chart, but as a map of connections. It could be a social network (people connected by friendships), a road map (cities connected by streets), or a molecule (atoms connected by bonds). The AI used for these is called a Graph Neural Network (GNN).

The Short Answer: No, the Crowd is Just a Mirror

The researchers found that for Graph Neural Networks, the "crowd" trick fails.

When they trained five different GNNs to work as a team, the team didn't act like a diverse group of experts. Instead, they all ended up thinking and acting exactly the same way.

The Analogy: Imagine asking five different chefs to cook a specific soup. In a normal kitchen (standard AI), they might use different spices or techniques, and if they all agree the soup is too salty, you know for sure it's salty.
The Graph Reality: In this study, the five chefs all ended up using the exact same recipe, the exact same amount of salt, and the exact same technique. They didn't just agree; they were clones. Because they were so identical, the group couldn't tell you if they were unsure about the taste. They just gave you a single, confident answer, even if they were wrong.

What Did They Discover?

The paper breaks down three main findings:

1. The "Epistemic Collapse" (The Group Think)
The authors coined a term for this phenomenon: Epistemic Collapse.

Epistemic means "knowledge."
Collapse means "falling apart."
Basically, the "knowledge" part of the group's uncertainty vanished. Even though the five models were trained separately, they all converged on the exact same solution. Because they didn't disagree with each other, the system couldn't measure how unsure it was. The "crowd" provided no extra safety net.

2. Why Did They Improve at All? (The Noise Filter)
You might ask: "If they are all the same, why did the paper say the group sometimes got slightly better scores?"
The answer is optimization noise.

The Analogy: Imagine trying to hit a bullseye on a dartboard while standing on a shaky boat. Sometimes you miss because the boat shook, not because you are a bad aim.
If you take five throws from five different people on that boat, and you average their positions, the "shakiness" (noise) cancels out, and the average point is closer to the bullseye.
The paper found that the GNN ensembles only got slightly better at predicting the exact number (the point prediction) because averaging five similar models smoothed out the random errors of training. It did not make them better at knowing when they were unsure.

3. The "Convex" Mystery (Why are they so similar?)
Why did five different models end up thinking the same thing?

In standard AI, the "landscape" of possible solutions is like a rugged mountain range with many different valleys. Different models get stuck in different valleys, leading to diverse opinions.
The authors suspect that for Graph Neural Networks, the landscape is actually more like a smooth, single bowl. No matter where you start rolling a ball (training a model), it always rolls to the exact same spot at the bottom.
They tested this by mixing the "brains" (weights) of the models together. If the models were in different valleys, mixing them should have made a worse model. But since they were all heading to the same functional spot, mixing them didn't help much. This suggests the "message-passing" nature of these networks forces them into a single, narrow way of thinking.

The Conclusion

The paper concludes that Deep Ensembles are not a magic bullet for Graph AI.

If you are building a system that needs to know when it is unsure (like a self-driving car or a medical diagnosis tool), simply training five Graph Neural Networks and averaging them will not work. They will just give you five copies of the same confident (but potentially wrong) answer.

Instead of wasting money and computer power training five identical models, the paper suggests you are better off training one model, picking the one that happened to work best by chance, and using that. The "crowd" adds no real value because the crowd isn't actually diverse.

Technical Summary: Do Deep Ensembles Actually Capture Uncertainty in Graph Neural Networks?

Problem Statement

Deep ensembles (DEs) are the standard method for uncertainty quantification (UQ) in deep learning, particularly in computer vision, where they rely on the disagreement among independently trained models to capture epistemic uncertainty. However, their effectiveness for graph-structured data, specifically message-passing Graph Neural Networks (GNNs), has been largely assumed rather than rigorously tested. While anecdotal evidence and limited studies suggest DEs may fail on graphs (e.g., the PEMS road network benchmark), a systematic evaluation across diverse graph tasks is missing. The core problem is whether the structural properties of GNNs and the nature of message-passing mechanisms prevent DEs from generating the necessary model diversity to effectively quantify uncertainty.

Methodology

The authors conduct a comprehensive empirical evaluation of standard deep ensembles applied to GNNs. Their methodology involves:

Datasets: A benchmark suite of seven diverse graph datasets covering node classification (Cora, Citeseer, Tolokers2), node regression (Artnetviews, Chameleon), and graph regression (QM9-5%).
Models: Standard message-passing architectures (GCNs and GATv2) trained as deep ensembles (typically $M=5$ models) with different random weight initializations. For regression tasks, models include a dedicated variance head to capture aleatoric uncertainty.
Baselines: Comparisons are made against single GNNs, a "Trivial" baseline (predicting a constant Gaussian distribution derived from training labels), and in specific cases, Bayesian GNNs or Graph Gaussian Processes.
Metrics:
- Primary: Negative Log-Likelihood (NLL) to evaluate joint predictive performance and uncertainty.
- Secondary: Point estimation metrics (RMSE for regression, Accuracy for classification) and calibration metrics (Expected Calibration Error - ECE).
- Decomposition: Explicit separation of aleatoric and epistemic uncertainty components using variance decomposition for regression and entropy decomposition for classification.
- Diversity Analysis: Investigation of "epistemic collapse" by analyzing the disagreement between ensemble members and testing weight-space vs. functional-space convexity via "model soups" (averaging weights).

Key Contributions

Systematic Benchmarking: The paper provides the first large-scale, systematic benchmark of standard deep ensembles for GNNs across multiple tasks and complexities, moving beyond isolated case studies.
Identification of Epistemic Collapse: The authors empirically demonstrate a phenomenon termed "epistemic collapse," where independently trained GNNs converge to highly similar predictive functions regardless of initialization. This lack of functional diversity neutralizes the primary mechanism (disagreement) by which ensembles capture epistemic uncertainty.
Deconstruction of NLL Gains: The study reveals that the marginal improvements in NLL observed with GNN ensembles are driven almost entirely by the stabilization of point predictions (reducing optimization noise) rather than by genuine improvements in uncertainty calibration or the capture of epistemic uncertainty.
Convexity Analysis: Through weight-space averaging experiments, the authors show that while the weight space of GNNs remains highly non-convex (models reside in different weight basins), the functional space appears to exhibit a form of convexity. Distinct weight solutions induce nearly identical behaviors, suggesting the message-passing architecture itself restricts the hypothesis space.

Results

Limited Improvement: Across most datasets, deep ensembles provide only marginal improvements over single GNNs in terms of NLL (e.g., 0.1% to 1% likelihood gain for classification, compared to ~20% in computer vision).
Calibration Degradation: In several cases (e.g., Chameleon, QM9-5%), ensembles actually worsen calibration (ECE) while slightly improving point accuracy, indicating that the NLL gain is not due to better uncertainty estimates.
Epistemic Collapse: The decomposition of uncertainty shows that epistemic uncertainty in GNN ensembles is negligible (often near zero) compared to aleatoric uncertainty. The ratio of epistemic to aleatoric uncertainty is consistently low, contrasting sharply with computer vision benchmarks where epistemic uncertainty is dominant.
Model Soup Failure: Averaging the weights of ensemble members ("model soup") results in degraded performance compared to single models. This confirms that the models do not share a simple convex basin in weight space; rather, they find different weights that map to the same function.
PEMS Benchmark: The study revisits the PEMS road network benchmark, confirming that GNN ensembles can perform catastrophically worse than a trivial baseline in uncertainty quantification, a failure attributed to the interplay between message passing and the ensemble mechanism.

Significance and Claims

The paper concludes that the success of deep ensembles in computer vision does not seamlessly transfer to graph machine learning. The authors claim that standard deep ensembles are inefficient for message-passing GNNs because they fail to capture meaningful epistemic uncertainty.

Inefficiency: Training and deploying an ensemble incurs significant computational and memory costs, yet the ensemble merely acts as a variance-reducing smoother for optimization noise rather than providing diverse uncertainty estimates.
Practical Implication: A single "lucky" model (selected via validation) can often match or exceed the performance of a full ensemble. Therefore, the multiplicative cost of ensembling is unjustified for standard GNNs.
Theoretical Insight: The findings suggest that the message-passing mechanism imposes a fundamental architectural bias that homogenizes model behavior, leading to functional convexity even in non-convex weight spaces. This challenges the assumption that random initialization alone is sufficient to generate the diversity required for ensemble-based UQ in graph domains.

The paper posits that achieving high-quality uncertainty quantification in graph learning requires moving beyond naive ensembling of standard message-passing architectures, potentially necessitating new methods that explicitly address the lack of functional diversity.