Technical Summary: A Fair Evaluation of Graph Foundation Models for Node Property Prediction

Problem Statement

The development of Graph Foundation Models (GFMs) has gained significant traction due to the ubiquity of graph-structured data in industry and science. Specifically, GFMs designed for node property prediction (predicting unknown node labels from features, structure, and known labels) are of high interest, with applications ranging from fraud detection to recommendation systems. However, the field lacks a unified evaluation setting. Existing works evaluate models using widely different protocols, often employing narrow, unrepresentative datasets and weak baselines. This inconsistency prevents reliable comparison between different GFMs and between GFMs and traditional Graph Neural Networks (GNNs). Furthermore, the term "Foundation Model" is often applied loosely to models targeting vastly different graph domains (e.g., molecules vs. social networks) and tasks, obscuring their actual scope and capabilities.

Methodology

To address these issues, the authors conducted a rigorous, large-scale empirical study reevaluating 9 recent GFMs for node property prediction against strong, well-tuned GNN baselines.

1. Datasets

The study utilizes 10 datasets from the GraphLand benchmark (Bazhenov et al., 2025). This benchmark was specifically designed to overcome the limitations of prior datasets (e.g., bugs, narrow domain coverage, questionable practical relevance) by representing real-world industrial applications across diverse domains such as e-commerce, transportation, and social networks. The authors use standard RL (Random Label) data splits.

2. Baselines (GNNs)

To ensure a fair comparison, the authors avoid under-tuned baselines. They employ two versions of improved GNN architectures known to be strong baselines:

• Platonov et al. (2023b): Variants of GCN, GraphSAGE, GAT, and Local Graph Transformer (LGT) featuring architectural improvements like skip-connections and normalization.
• Luo et al. (2024): Similar improved variants with additional architectural options like ego-neighbor embedding separation.
• Hyperparameter Tuning: Both baseline sets undergo extensive hyperparameter search (100 trials using the TPE algorithm) to ensure optimal performance, countering the tendency in literature to under-tune baselines.

3. Models Evaluated

The study evaluates 9 GFMs, categorized by their underlying mechanisms:

• Non-PFN-based GFMs: OpenGraph, AnyGraph, GCOPE, SAMGPT, MDGFM, and TS-GNN. These models typically rely on mechanisms like feature projection, dimensionality reduction, or converting graphs to tables without native graph-aware pretraining.
• PFN-based GFMs: G2T-FM, TAG, and GraphPFN. These are based on the Prior-data Fitted Networks (PFN) paradigm. They are trained on synthetic datasets to perform In-Context Learning (ICL), making predictions in a single forward pass without per-dataset training.
  • G2T-FM and TAG augment node features with graph information and feed them into pre-trained tabular foundation models (LimiX-16M or TabPFNv2).
  • GraphPFN uses a custom graph-native architecture that incorporates message passing directly into the PFN structure.

4. Evaluation Protocols

Models are evaluated under all supported regimes:

• In-Context Learning (ICL): Zero-shot prediction using the pre-trained model.
• Fine-Tuning (FT): Dataset-specific training of the model parameters.
• Metrics: Accuracy (multiclass), Average Precision (binary), and R² (regression), aggregated via average rank and average normalized score across datasets.
• Efficiency: Computational costs for hyperparameter tuning, training, and inference are measured.

Key Results

1. Predictive Performance

The results clearly bifurcate GFMs into two distinct groups:

• Non-PFN-based GFMs: These models consistently underperform well-tuned classic GNNs, often by a substantial margin, across all datasets.
• PFN-based GFMs: These models generally outperform GNNs.
  • In the ICL regime, PFN-based models (specifically GraphPFN, G2T-FM, and TAG) achieve top results on the majority of datasets, surpassing all GNNs. GraphPFN leads in average rank.
  • In the Fine-Tuning regime, performance improves significantly for PFN-based models. GraphPFN (FT) emerges as the overall best model, achieving the best results on all 10 datasets, often by a large margin.
  • Architecture Comparison: Among PFN-based models, those using the LimiX-16M backbone (G2T-FM, TAG) perform better than those using TabPFNv2. However, GraphPFN, which natively handles graph structure via message passing within the PFN architecture, outperforms the tabular-backbone approaches (G2T-FM and TAG) in both ICL and FT regimes.

2. Computational Efficiency

• Inference: GNNs are significantly more efficient at inference, typically requiring <1 second. In contrast, PFN-based GFMs require seconds to minutes (with some TAG models taking up to 30 minutes) due to the computational cost of processing the entire context in a single forward pass.
• Training/Development: GNNs require extensive hyperparameter tuning and multiple training runs (hundreds of forward/backward passes) for each new dataset. PFN-based models, once pre-trained, require far fewer hyperparameters to tune (e.g., only learning rate for fine-tuning) and fewer training runs. In scenarios where development time is critical and inference latency is less constrained, PFNs may be preferable.

Key Contributions

1. Unified Evaluation Framework: The paper establishes a fair, rigorous evaluation protocol for node property prediction GFMs, utilizing the GraphLand benchmark and strong, well-tuned GNN baselines.
2. Performance Disparity Identification: It demonstrates that the majority of current GFMs (non-PFN based) do not outperform properly tuned GNNs, challenging the assumption that "foundation models" inherently yield superior performance in graph tasks.
3. Validation of PFN Paradigm: The study identifies that only GFMs based on the Prior-data Fitted Networks (PFN) paradigm achieve state-of-the-art performance, surpassing GNNs in both ICL and fine-tuning settings.
4. Architecture Insights: It highlights that while adapting tabular foundation models to graphs (via feature augmentation) works, native graph-aware architectures (like GraphPFN) yield superior results compared to treating graphs as tables.
5. Cost-Benefit Analysis: The paper provides a detailed breakdown of the trade-off between the high inference costs of PFNs and their reduced development/tuning costs compared to GNNs.

Significance and Claims

The authors claim that this work provides a "fair and rigorous reevaluation" that clarifies the current state of the field. The primary significance lies in:

• Correcting Misconceptions: Demonstrating that not all models labeled "GFMs" are effective; only those leveraging the PFN paradigm currently offer a performance advantage over GNNs.
• Defining Scope: Arguing that the term "GFM" is ill-defined and urging researchers to explicitly clarify the scope of their models (e.g., specific data domains or tasks).
• Practical Guidance: Offering a clear trade-off analysis for practitioners: if inference speed is paramount, well-tuned GNNs remain the dominant choice; if rapid adaptation to new datasets with minimal tuning is required, PFN-based GFMs are the superior option despite higher inference latency.

The paper concludes that while the field has not yet converged to a unified solution, the PFN paradigm represents the most promising direction for Graph Foundation Models in node property prediction, provided the computational cost of inference is acceptable for the specific application.