💻 computer science

GraphNetz: Statistical Benchmarking of Graph Neural Networks with Paired Tests and Rank Aggregation

This paper introduces GraphNetz, a benchmarking framework that replaces raw accuracy tables with structured statistical reports—including confidence intervals, corrected paired tests, and rank aggregation—to provide reproducible and principled comparisons of Graph Neural Networks that account for performance variability across seeds and datasets.

Original authors: Kleyton da Costa, Bernardo Modenesi

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kleyton da Costa, Bernardo Modenesi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a judge at a cooking competition. In the past, when people compared different Graph Neural Networks (GNNs)—which are like special recipes for teaching computers to understand connections in data—they would usually say, "Recipe A made a dish that was 92% delicious, while Recipe B made one that was 91% delicious. Therefore, Recipe A is the winner!"

But there was a problem: they were only tasting the dish once. If you taste a soup once, you might get lucky with a perfect spoonful, or unlucky with a salty one. The paper argues that judging a recipe based on a single taste is unfair and misleading.

GRAPHNETZ is a new tool designed to fix this. Instead of just giving a single score, it acts like a rigorous food critic who tastes every dish 10 times (using different random "seeds" or ingredients) and then uses strict statistics to decide who actually won.

Here is how the paper breaks it down, using simple metaphors:

1. The Problem: The "Single Taste" Trap

The authors point out that many previous studies claimed one AI model was "better" than another based on tiny differences in performance. However, these differences were often just noise—like the difference between a spoonful of soup that happened to have a bit more salt versus one that didn't. When you account for the randomness of how the experiment was run, those "winners" often turn out to be tied.

2. The Solution: GRAPHNETZ (The Rigorous Judge)

The authors built a framework called GRAPHNETZ. Think of it as an automated judge that doesn't just spit out a scorecard; it spits out a statistical report.

The Catalog: It has a massive pantry with 63 different datasets (ingredients) covering 10 different areas like biology, finance, social networks, and even physics.
The Contestants: It tests 5 famous AI models (GCN, GAT, GraphSAGE, Graph Transformer, and GIN) against these ingredients.
The Process: Instead of running the test once, it runs every single combination 10 times with different random seeds. This is like cooking the same dish 10 times to see if the chef is consistently good or just getting lucky.

3. The Tools: How It Decides a Winner

GRAPHNETZ uses three specific statistical tools to ensure fairness:

Confidence Intervals (The Safety Net): Instead of saying "Model A got 92%," it says "Model A got 92%, give or take 1%." This shows the range of likely outcomes, so you know if the difference is real or just a fluke.
Paired Tests with Corrections (The Fair Comparison): It compares models head-to-head on the same dataset and corrects for the fact that it's making many comparisons at once. This prevents the judge from accidentally declaring a winner just because they looked at enough data to find a tiny, meaningless difference.
The Critical Difference Diagram (The Tie-Breaker): This is a visual chart that groups models. If two models are connected by a line on this chart, it means statistically, they are tied. You cannot say one is better than the other with confidence.

4. The Big Discovery: The Great Tie

When the authors ran this rigorous test across 10 different types of tasks (from predicting molecular properties to analyzing social networks), they found something surprising.

Even though the raw numbers looked like one model was slightly ahead of the others, the statistical report showed they were all tied.

The Metaphor: Imagine four runners in a race. One finishes in 10.01 seconds, another in 10.02, another in 10.03, and the last in 10.04. Without a stopwatch that measures thousandths of a second and accounts for wind conditions, you might say the first one won. But with GRAPHNETZ's "statistical stopwatch," the judge says, "These four runners are effectively tied; the difference is too small to call a winner."

In the paper's terms, all four main AI models fell into a single "Nemenyi clique," meaning none of them was significantly better than the others at the standard level of confidence.

5. Why This Matters

The paper argues that the field of AI has been "cherry-picking" results—highlighting the tiny wins and ignoring the ties. GRAPHNETZ forces researchers to be honest. It provides a standard, reproducible way to compare new AI models against old ones, ensuring that when someone claims a new model is "better," they have the statistical evidence to prove it's not just a lucky guess.

In short: GRAPHNETZ is a tool that stops AI researchers from shouting "I won!" based on a fluke. It forces them to run the race 10 times, measure the wind, and only declare a winner if the difference is real. In their big test, they found that the current top models are actually all running neck-and-neck.

Technical Summary: GRAPHNETZ

Problem Statement

Current benchmarks for Graph Neural Networks (GNNs) frequently suffer from a lack of statistical rigor. Typical studies report single-point accuracy estimates (often with standard deviations) derived from limited random seeds and specific train/test splits. This approach often leads to the conclusion that one model "outperforms" another based on headline differences that are smaller than the noise introduced by random seed variance, dataset splits, or evaluation artifacts.

As noted in prior re-evaluations (Shchur et al., 2018; Errica et al., 2020), many reported architectural gains collapse when seeds and splits are randomized. While standard statistical tools exist—such as confidence intervals (CIs), paired comparisons with multiple-comparison correction, and rank-based aggregation—they are rarely the default output of graph-learning benchmark suites. Existing frameworks like PyTorch Geometric, Open Graph Benchmark (OGB), and Benchmarking-GNNs focus on architecture and dataset breadth but leave statistical reporting, multi-seed orchestration, and rigorous hypothesis testing to the user.

Methodology: The GRAPHNETZ Framework

The authors introduce GRAPHNETZ, a benchmarking framework designed to make a structured statistical report the default output of a benchmark run, rather than a raw accuracy table. The framework is organized around four conceptual layers:

Categorized Dataset Catalog:
- The framework includes 63 dataset loaders mapped to 10 research categories (Combinatorial Optimization, Biology, Social, Knowledge, Infrastructure, Finance, Computing, Vision, Physics, and Security).
- It supports 4 task types: Node Classification, Graph Classification, Graph Regression, and Link Prediction.
- Loaders include canonical public benchmarks (e.g., Planetoid, TUDataset, QM9, ZINC) and synthetic instances (e.g., TSP, VRP, Ising lattice) where canonical benchmarks are absent.
Uniform Evaluation Protocol:
- The framework evaluates 5 canonical GNN architectures: GCN, GAT, GIN, GraphSAGE, and Graph Transformer.
- It employs task adapters (e.g., graph-level pooling heads, dot-product link-prediction heads) to ensure node-level encoders can be tested across all task types.
- Algorithm 1 defines the protocol: For a given category, models are trained across $S$ seeds (default $S=10$ ) and $E$ epochs. The process is deterministic given a fixed software stack, with RNGs reset before each run.
Statistical Reporting Layer:
- Per-Cell Analysis: Computes the mean $\pm$ 95% Student's t confidence interval for every (task, model) pair.
- Within-Task Pairwise Tests: Performs paired t-tests (parametric) and Wilcoxon signed-rank tests (non-parametric) on seed-aligned final metrics. These are corrected for multiple comparisons using the Holm-Bonferroni method.
- Across-Task Aggregation: Ranks models per task, computes the Friedman omnibus test, and generates Nemenyi critical-difference (CD) diagrams to visualize significant differences across tasks.
Implementation:
- The framework is implemented in Python, with a single entry point run_benchmark.
- It supports custom datasets and models via a one-line API extension.
- All random number generators are seeded to ensure reproducibility.

Key Contributions

The paper claims four primary contributions:

A Categorical Taxonomy: A mapping of 63 loaders to 10 research areas, explicitly noting subtopics that currently lack canonical public benchmarks.
A Uniform Protocol: A standardized evaluation setup for five major GNN architectures across four task types, ensuring performance differences reflect modeling choices rather than evaluation idiosyncrasies.
A Principled Statistical Layer: The default return type of a benchmark run is a structured report containing per-cell CIs, Holm-corrected pairwise p-values, and Friedman-Nemenyi CD diagrams.
Empirical Validation: A cross-category benchmark demonstrating that under rigorous statistical scrutiny, apparent rank differences among canonical encoders often fall within a single Nemenyi clique, indicating no significant difference.

Experimental Results

The authors conducted a cross-category experiment across 10 heterogeneous tasks (one representative dataset per category) using 10 seeds and 5 architectures.

Per-Category Results: For each task, mean performance and 95% CIs were calculated. In many cases, the confidence intervals of different architectures overlapped significantly.
Pairwise Test Agreement: On the social/Planetoid datasets (Cora, CiteSeer, PubMed), the parametric paired t-test and the non-parametric Wilcoxon signed-rank test agreed on all 18 comparisons after Holm-Bonferroni correction at $\alpha = 0.05$ .
Across-Category Ranking:
- The four node-level encoders (GCN, GAT, GraphSAGE, Graph Transformer) were ranked across the 10 tasks.
- The Friedman omnibus test yielded $\chi^2_3 = 2.04$ ( $p = 0.56$ ), failing to reject the global null hypothesis of equal mean ranks.
- The Nemenyi critical difference was calculated as $CD_{0.05} = 1.48$ rank units.
- The maximum rank gap observed was between GraphSAGE (mean rank 2.10) and GAT (mean rank 2.90), a difference of 0.80.
- Conclusion: Since $0.80 < 1.48$ , none of the four architectures was significantly better than any other at $\alpha = 0.05$ . All four fell within a single Nemenyi clique.

Significance and Claims

The paper positions GRAPHNETZ as a tool to address the reproducibility crisis in graph learning by embedding statistical rigor into the default pipeline.

Honest Comparison: The framework provides a "reproducible and honest model comparison" that accounts for seed uncertainty and prevents cherry-picking of results.
Calibrated Evidence: The authors argue that the field's typical sample sizes (e.g., $N=10$ tasks, $S=10$ seeds) often lack the power to distinguish between canonical architectures. The framework's statistical outputs (CIs, CD diagrams) are "precisely calibrated" to reflect the actual evidence available, showing that headline differences often dissolve when noise is properly accounted for.
Community Standard: By making the full chain of statistical reporting (from multi-seed evaluation to CD diagrams) the default output, GRAPHNETZ aims to serve the graph-learning community with a pipeline ready to be adopted in research papers, moving the field away from reporting raw accuracy tables toward principled statistical evidence.

The authors remain modest regarding future implications, noting that increasing the number of datasets ( $N$ ) would tighten the critical difference threshold, and that extending the framework to include hyperparameter search or foundation models are natural next steps, but not part of the current release.