Multi-Domain Riemannian Graph Gluing for Building Graph Foundation Models

This paper introduces GraphGlue, a framework that leverages Riemannian geometry and neural manifold gluing to unify diverse graph datasets into a smooth manifold, thereby providing a theoretical foundation for systematic knowledge integration and transfer in building graph foundation models.

The Gist

The Big Picture: Building a Universal Translator for Graphs

Imagine you have a collection of maps from completely different worlds:

• World A: A map of a social network (like Facebook), where people are dots and friendships are lines.
• World B: A map of a chemical lab, where atoms are dots and bonds are lines.
• World C: A map of a city's subway system.

Currently, AI models are like cartographers who only know how to read one type of map. If you teach a model to read subway maps, it gets confused when you show it a chemical structure. It doesn't know that the "lines" in chemistry mean something different than the "lines" in a subway.

Graph Foundation Models are the dream of creating a "Universal Cartographer" that can learn from all these different worlds at once and then instantly understand a new, unseen map.

The problem is: How do you teach a computer that a social network and a molecule are fundamentally similar, even though they look totally different?

This paper, GRAPHGLUE, solves this by treating all these different maps not as separate pieces of paper, but as pieces of a single, giant, stretchy fabric.

---

The Core Idea: The "Stretchy Fabric" Analogy

The authors use a branch of math called Riemannian Geometry. To understand this, imagine a giant, smooth, stretchy rubber sheet (a Manifold).

1. The Problem with Current Models:
   Most AI models try to flatten these complex maps onto a flat table (Euclidean space). But if you try to flatten a globe onto a table, you get tears and distortions. Similarly, forcing a social network and a molecule into the same flat space causes "tears" in the data, making it hard for the AI to transfer knowledge from one to the other.

2. The GRAPHGLUE Solution:
   Instead of flattening the maps, GRAPHGLUE keeps them on a 3D stretchy fabric.

   • Local Geometry (The Patch): First, the AI looks at a small patch of a graph (like a neighborhood in a city). It figures out the "shape" of that specific patch.
   • The Glue (Neural Manifold Gluing): This is the paper's magic trick. It takes the "patches" from the social network, the chemical lab, and the subway, and glues them together onto one giant, smooth rubber sheet.
   • The Result: The social network might be a bumpy hill on the sheet, and the chemical molecule might be a valley. But because they are on the same smooth sheet, the AI can "roll" knowledge from the hill to the valley without tearing the fabric.

How It Works: The Three Steps

The paper proposes a three-step process to build this universal fabric:

1. Measuring the Shape (Adaptive Orthogonal Frame)

Imagine you are standing on a patch of grass. To understand the shape of the ground, you stick a few sticks into the ground in different directions to see how the ground tilts.

• In the paper: The AI uses a "sparse perturbation" (a tiny, strategic poke) to generate "tangent vectors" (the sticks). This tells the AI the local shape and orientation of that specific graph piece.

2. Gluing the Patches (Metric Compatibility & Holonomy)

Now you have many patches. How do you stick them together without creating wrinkles or holes?

• Edge Gluing: When two patches share a border (an edge), the AI ensures the "texture" matches perfectly. It's like sewing two pieces of fabric so the pattern lines up.
• Triangle Gluing (Holonomy): Imagine walking in a triangle around a patch. If you start facing North and end up facing East, the fabric is twisted. The AI checks for these twists. If there are twists, it "smooths" them out until walking a triangle brings you back to exactly where you started. This ensures the fabric is seamless.

3. Smoothing the Surface (Ricci Curvature)

Even if the patches are glued, the fabric might still be bumpy.

• The Fix: The AI measures the "curvature" (how much the fabric bends). It applies a loss function (a penalty) to force the fabric to be as smooth as possible.
• Why it matters: A smooth fabric allows information to flow easily. If the fabric is bumpy, knowledge gets stuck. A smooth fabric means the AI can learn from a massive dataset and apply it to a tiny new dataset effortlessly.

The "Geometric Scaling Law"

The paper discovered a cool rule: The more data you feed the AI, the smoother the fabric becomes.

Think of it like sanding a piece of wood.

• With a little sandpaper (small dataset), the wood is still rough.
• With a lot of sandpaper (massive dataset), the wood becomes perfectly smooth.
• The Result: The smoother the fabric, the better the AI can transfer knowledge. This explains why "bigger is better" in AI, but specifically why it works for graphs: more data creates a smoother, more universal geometric space.

The "Transfer Metric" (GTM)

One of the coolest features of GRAPHGLUE is that it can tell you how hard it will be to apply what it learned to a new task.

• Imagine you have a glove (the pre-trained model).
• You want to put it on a hand (the new task).
• GTM (Geometric Transfer Metric): This measures how much the glove needs to stretch or twist to fit the hand.
  • Low GTM: The glove fits perfectly. The new task is very similar to what the AI already knows.
  • High GTM: The glove is too tight or twisted. The new task is very different, and the AI will struggle.
• This gives scientists a "ruler" to predict if a model will work before they even try it.

Summary: Why This Matters

• For AI: It solves the "silo" problem. Instead of training a separate brain for social networks, molecules, and traffic, we can train one brain on a unified "fabric" of all graphs.
• For Science: It provides a mathematical proof (using geometry) of why and how knowledge transfers between different types of data.
• For the Future: It paves the way for Graph Foundation Models—super-intelligent AIs that can understand any network structure, from the internet to the human brain, by seeing them all as different shapes on the same smooth, universal sheet.

In short: GRAPHGLUE is the ultimate "sewing kit" that stitches together all the different maps of our world into one seamless, smooth, and intelligent fabric.

Technical Summary

1. Problem Statement

The paper addresses the challenge of Multi-Domain Graph Pre-training for building Graph Foundation Models (GFMs). While existing methods attempt to integrate knowledge from diverse domains (e.g., social networks, molecular graphs, knowledge graphs), they suffer from a fundamental theoretical gap:

• Lack of Theoretical Framework: Current approaches do not rigorously explain how knowledge is integrated or transferred across domains with significant semantic heterogeneity.
• Inconsistent Transferability: Existing similarity measures fail to frame pre-training and domain adaptation within a consistent framework, making it difficult to assess transfer difficulty, especially for unseen graphs.
• Limitations of Text-Based Approaches: Many methods rely on Large Language Models (LLMs) to generate textual attributes for graphs, which is labor-intensive, hallucination-prone, and inapplicable to text-free graphs.

The core problem is to establish a principled, geometric framework that unifies disparate graph datasets into a coherent structure to facilitate systematic knowledge transfer.

2. Methodology: GRAPHGLUE

The authors propose GRAPHGLUE, a framework grounded in Riemannian Geometry and Differential Geometry. The core intuition is to treat the embedding space of diverse graph datasets as a unified, smooth Riemannian manifold.

A. Theoretical Foundation: Neural Manifold Gluing

The paper introduces a novel theory called Neural Manifold Gluing, which constructs a global manifold from local graph pieces through three steps:

1. Learning Local Geometry (Adaptive Orthogonal Frame):

   • Instead of assuming a fixed manifold, the model learns the local geometry at each point (graph representation) using a $(k, M)$-sparse perturbation.
   • This perturbation mimics directional derivatives to generate tangent vectors.
   • An Adaptive Orthogonal Frame (AOF) is constructed via QR-decomposition to form a basis for the tangent space, capturing local metric properties (stretching and twisting).

2. Gluing Local Pieces (Metric Compatibility & Holonomy):

   • Edge Tangent Translation: To connect local manifolds along edges, the method defines a linear map (isometry) that translates tangent spaces between connected nodes, ensuring metric compatibility.
   • Holonomy Triviality: To ensure the manifold is continuous and not "twisted" when traversing cycles (e.g., triangles), the method enforces trivial holonomy. If the composition of transport maps around a cycle is the identity, the manifold is coherent. This is enforced via a Holonomy Loss ($L_{holo}$).

3. Smoothing the Manifold (Ricci Curvature):

   • To achieve $C^2$ smoothness (preventing "folds" that hinder knowledge transport), the method controls the Ricci curvature.
   • Since calculating Ricci curvature is expensive, the authors approximate it using the ratio of volume elements (metric determinants) between connected nodes.
   • A Curvature Loss ($L_{curv}$) is introduced to enforce log-determinant smoothness, ensuring the manifold is smooth enough for effective knowledge flow.

B. The GRAPHGLUE Framework

The practical implementation consists of two phases:

1. Pre-training with EMA Prototyping:

   • Batched Processing: Uses Exponential Moving Average (EMA) to update Riemannian Prototypes (global location and metric) for each domain. This allows efficient handling of large-scale graphs in batches.
   • Contrastive Learning: A prototype-level contrastive loss ensures that different domains are distinguished by their locations on the manifold while maintaining internal coherence.

2. Consistent Adaptation:

   • Prompting & Riemannian MoE: For a target domain, the model uses learnable prompts to adapt global coordinates. A Riemannian Mixture-of-Experts (MoE) aggregates knowledge from pre-trained prototypes based on geometric proximity.
   • Geometric Gluing: The target domain is "glued" to the pre-trained manifold by minimizing the Holonomy and Curvature losses relative to the nearest prototypes.
   • Geometric Transfer Metric (GTM): A new metric is defined to quantify transfer difficulty as the sum of Holonomy Disagreement ($\Delta H$) (twisting) and Curvature Disagreement ($\Delta C$) (bending). Low GTM implies high transferability.

3. Key Contributions

1. Theoretical Innovation: The first theoretical framework to interpret multi-domain graph pre-training through Neural Manifold Gluing, unifying disparate graphs into a single smooth Riemannian manifold.
2. Methodological Advancement: The GRAPHGLUE framework, which supports batched pre-training, introduces Riemannian prototypes, and utilizes a Riemannian MoE for adaptation.
3. Quantifiable Transferability: The proposal of the Geometric Transfer Metric (GTM), which provides an interpretable, geometry-based measure of how difficult it is to transfer knowledge to a new domain.
4. Geometric Scaling Law: Empirical validation showing that increasing the quantity of pre-training datasets leads to a smoother manifold, thereby improving model transferability (scaling law).

4. Experimental Results

The authors evaluated GRAPHGLUE on 6 diverse domains (Arxiv, Computers, Reddit, FB15k-237, PROTEINS, HIV) across node, link, and graph classification tasks.

• Performance: GRAPHGLUE significantly outperforms state-of-the-art baselines (including GFT, SAMGPT, GCOPE, and MDGFM) in few-shot settings (1-shot and 5-shot).
  • Example: On the Reddit dataset (1-shot), GRAPHGLUE achieved 67.12% ACC, outperforming the runner-up (GCOPE) by ~4.6%.
  • Example: On Computers (1-shot), it achieved 59.50% ACC, surpassing the best baseline by 4.9%.
• Ablation Studies: Removing $L_{holo}$ or $L_{curv}$ significantly degrades performance, confirming the necessity of both topological consistency (gluing) and geometric smoothness.
• Scaling Law: Experiments demonstrated a logarithmic scaling law: as the number of pre-training datasets increases, the transfer loss decreases and accuracy increases, validating the "smoother manifold" hypothesis.
• Negative Transfer Resistance: Unlike some baselines (e.g., GCOPE) that suffer from negative transfer when adding semantically distinct domains (e.g., adding molecular graphs to social networks), GRAPHGLUE showed steady performance improvements.
• Visualization: t-SNE visualizations confirmed that semantically similar domains cluster closely on the manifold, while distinct domains remain separated but connected via smooth transitions.

5. Significance

• Bridging Geometry and Deep Learning: This work successfully bridges differential geometry (specifically Riemannian manifolds and holonomy) with deep learning, providing a rigorous mathematical foundation for graph foundation models.
• Solving the "Black Box" of Transfer: By defining transferability via geometric consistency (GTM), the paper moves beyond empirical trial-and-error to a principled understanding of why and how knowledge transfers between domains.
• Scalability and Generality: The framework is designed for text-free graphs and scales effectively with data volume, offering a robust path toward building universal Graph Foundation Models capable of handling highly heterogeneous real-world data.
• Efficiency: The use of sparse perturbations and EMA prototyping allows the model to handle large-scale graphs efficiently, avoiding the computational bottlenecks of dense attention mechanisms used in some LLM-based graph approaches.

In summary, GRAPHGLUE represents a paradigm shift in graph pre-training, moving from heuristic alignment to geometric unification, enabling the construction of foundation models that are theoretically grounded, geometrically consistent, and empirically superior in cross-domain transfer.