Technical Summary: GraphLand: Evaluating Graph Machine Learning Models on Diverse Industrial Data

Problem Statement

Despite the prevalence of graph-structured data in real-world industrial applications, the evaluation of Graph Machine Learning (GML) models, particularly Graph Neural Networks (GNNs) and Graph Foundation Models (GFMs), remains constrained by a narrow set of benchmarks. Current popular benchmarks predominantly rely on academic citation networks (e.g., Cora, Citeseer, Pubmed) and a few e-commerce co-purchasing networks. These datasets suffer from several critical limitations:

• Domain Narrowness: They cover only a single or very limited application domains, failing to represent the diversity of real-world graphs such as social networks, web graphs, and road networks.
• Feature Homogeneity: Most existing datasets provide only textual node attributes, overlooking the complex mix of numerical and categorical features common in industrial settings.
• Structural Bias: They are largely homophilous (edges connect similar nodes), whereas many industrial graphs are non-homophilous.
• Evaluation Gaps: Standard benchmarks often lack realistic temporal data splits and inductive settings, failing to assess model performance under temporal distributional shifts or when new nodes appear without prior training data.
• Foundation Model Limitations: Proposed GFMs are frequently evaluated only on text-attributed graphs, ignoring the challenge of generalizing to graphs with arbitrary, non-textual node feature sets.

Consequently, there is a lack of reliable, diverse, and realistic benchmarks to evaluate the true generalization capabilities of GML models in industrial contexts.

Methodology

To address these gaps, the authors introduce GraphLand, a benchmark comprising 14 diverse graph datasets derived from industrial applications and underutilized open data sources.

Dataset Construction

The benchmark includes 14 datasets covering domains such as web fraud detection, art creator social networks, road traffic prediction, ad click-through rate (CTR) prediction, and crowdsourcing worker fraud detection. Key characteristics include:

• Diversity: Graphs range from ~11K to ~3M nodes, with varying densities, diameters, and clustering coefficients.
• Feature Richness: Nodes possess diverse attributes, including numerical and categorical features (e.g., user interests, road segment metadata, product prices), moving beyond purely textual descriptions.
• Task Variety: Tasks include binary classification (fraud detection), multiclass classification (topic prediction, region prediction), and regression (traffic speed, view counts, CTR).
• Data Splits: The authors provide three distinct splitting strategies for most datasets:
  1. RL (Random Low): 10% training, 10% validation, 80% testing.
  2. RH (Random High): 50% training, 25% validation, 25% testing.
  3. TH (Temporal High): 50% training, 25% validation, 25% testing based on temporal evolution, simulating realistic distributional shifts.
  4. THI (Temporal High Inductive): Similar to TH but with a strict inductive setting where test nodes and their incident edges are unavailable during training.

Experimental Setup

The authors conducted extensive experiments comparing various model classes:

• Graph-Agnostic Baselines: ResMLP (a ResNet-style MLP) and Gradient-Boosted Decision Trees (GBDTs: XGBoost, LightGBM, CatBoost).
• Graph-Augmented Baselines: The above models augmented with Neighborhood Feature Aggregation (NFA), which appends aggregated neighbor statistics (mean, max, min) to node features.
• Graph Neural Networks (GNNs): Classic architectures (GCN, GraphSAGE) and attention-based models (GAT, Graph Transformer).
• Graph Foundation Models (GFMs): Evaluation of OpenGraph, AnyGraph, TS-GNN, and GCOPE, focusing on their ability to handle non-textual features and transfer across domains.

All models underwent rigorous hyperparameter tuning. Experiments were run in both transductive and inductive settings to isolate the impact of graph structure and temporal shifts.

Key Results

1. Utility of Graph Structure

The graph structure in GraphLand datasets is highly beneficial. GNNs consistently outperformed graph-agnostic ResMLP models. Furthermore, augmenting GBDTs with NFA significantly improved their performance, often allowing them to match or exceed standard GNNs, particularly in regression tasks. This confirms that GBDTs with graph-derived features serve as strong baselines in industrial settings.

2. Performance of GNNs

• Attention Mechanisms: Attention-based GNNs (GAT, Graph Transformer) generally outperformed classic GNNs (GCN, GraphSAGE) across most datasets, suggesting that weighting neighbor messages based on content is crucial for diverse industrial graphs.
• Temporal Shifts: Models performed significantly worse under temporal (TH) splits compared to random (RH) splits, indicating that realistic distributional shifts pose a substantial challenge.
• Inductive Settings: Performance dropped further in the inductive (THI) setting compared to the transductive setting, highlighting the difficulty of generalizing to unseen nodes in evolving graphs.

3. Limitations of Current Foundation Models

Current GFMs (OpenGraph, AnyGraph, TS-GNN, GCOPE) failed to produce competitive results on GraphLand datasets.

• They struggled with non-textual node features and large-scale graphs.
• Their performance was often inferior to classic GNNs and even GBDT baselines.
• Many models encountered runtime errors or memory limits on larger datasets, and none supported node regression tasks effectively in the provided implementations.

Significance and Claims

The authors position GraphLand as a necessary step toward more realistic and diverse evaluation of GML methods. The paper claims the following significance:

1. Benchmarking Realism: GraphLand provides the first unified benchmark for node property prediction that spans diverse industrial domains, feature types, and structural properties, moving beyond the dominance of citation networks.
2. Highlighting Research Gaps: The benchmark reveals that current GML methods, including GFMs, are not yet robust to temporal distributional shifts or capable of handling arbitrary node feature sets. It identifies the design of models resilient to these shifts as a critical research direction.
3. Baseline Re-evaluation: The results demonstrate that GBDTs with graph-based features are formidable competitors to GNNs in industrial settings, challenging the assumption that GNNs are always superior.
4. Foundation Model Assessment: The poor performance of current GFMs on these datasets suggests that the problem of creating truly generalizable graph foundation models is far from solved, particularly regarding feature generalization beyond text.

The authors conclude that GraphLand serves as a reliable testbed for future research, noting that early follow-up work has already utilized the benchmark to develop GFMs that achieve strong results on these datasets. The datasets and code are made publicly available via GitHub and PyTorch Geometric.