🤖 machine learning

Are Common Substructures Transferable? Riemannian Graph Foundation Model with Neural Vector Bundles

This paper introduces GAUGE, a Riemannian graph foundation model based on Neural Vector Bundles, which theoretically links transferable substructures to intrinsic geometry and empirically demonstrates superior performance in zero-shot link prediction and graph isomorphism tasks.

Original authors: Li Sun, Zhenhao Huang, Yiding Wang, Qin Chen, Pietro Lio, Philip S. Yu

Published 2026-06-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Li Sun, Zhenhao Huang, Yiding Wang, Qin Chen, Pietro Lio, Philip S. Yu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Can We Reuse Graph "Lego" Pieces?

Imagine you are building with Lego. You have built a castle, a spaceship, and a car. Now, you want to build a new, different spaceship. You might think, "I can just reuse the same 'engine' or 'wing' pieces I used in the first spaceship, right?"

In the world of computer science, graphs are like these Lego structures (nodes are the bricks, edges are the connections). Scientists have been trying to build "Foundation Models"—super-smart AI brains that learn from many different graphs so they can easily understand new ones without starting from scratch.

The big question this paper asks is: "Are the common patterns (substructures) we learn in one graph actually transferable to another?"

Previous researchers tried to find these patterns by looking at the discrete shape of the Lego pieces (e.g., "Is this a triangle? Is this a star?"). But the authors of this paper say: "Wait a minute. It's not just about the shape; it's about how the piece behaves in its neighborhood."

The Core Idea: Behavior is Geometry

The authors propose a new way of thinking:

Old Way: "This piece is a triangle, so it works everywhere."
New Way: "This piece behaves in a specific, stable way regardless of what is around it. If it behaves the same way, it is transferable."

To understand this "behavior," they use a branch of math called Riemannian Geometry. Think of this as a way to measure the "shape" of the space where the data lives.

The Analogy: The Fabric of Space

Imagine the graph is a piece of fabric.

If you stretch the fabric unevenly, it gets wrinkled (curved).
If the fabric is smooth and flat, it is flat.

The authors discovered that the "transferable" parts of a graph are the parts where the fabric is flat. If a part of the graph is "flat" (geometrically), it means the rules governing that part are consistent and don't change no matter where you move them. If it's "wrinkled" (curved), the rules are messy and specific to that exact spot, making it hard to reuse.

The Solution: The "Neural Vector Bundle"

To find these "flat" parts, the authors invented a new tool called a Neural Vector Bundle.

The Metaphor: The Backpack and the Map
Imagine every node (brick) in the graph has a backpack (a vector space) attached to it.

The Base: The graph itself is the ground you are walking on.
The Backpacks: Each node carries a local map (a set of coordinates) inside its backpack.
The Problem: When you walk from one node to a neighbor, the maps in the backpacks might be rotated or twisted differently. It's like trying to compare a map of New York with a map of London without knowing how they are rotated relative to each other.
The Fix (GAUGE): The authors built a system called GAUGE that acts like a magical translator. It aligns all these backpacks so they point in the same direction. It "flattens" the wrinkles in the fabric.

By aligning these local maps, the AI can see which parts of the graph are "flat" (stable and transferable) and which are "curved" (messy and specific).

How GAUGE Works (The "Flattening" Process)

The paper introduces a new training method called GAUGE (Graph Architecture with Universal Geometry Embedding).

Learning the Shape: Instead of just memorizing connections, GAUGE learns the "intrinsic geometry" (the natural shape) of the graph.
The Dirichlet Loss (The "Smoothness" Test): They created a new math test called Dirichlet Loss. Think of this as a "smoothness meter."
- If the AI tries to predict a node's value based on its neighbors, and the prediction is perfect, the "smoothness meter" reads zero. This means the area is flat and transferable.
- If the prediction is messy, the meter reads high. This means the area is curved and specific to that graph.
The Goal: The AI is trained to minimize this "smoothness meter." It tries to find and lock onto the parts of the graph that are perfectly flat, because those are the parts it can take to a new graph and use immediately.

What Did They Prove?

The authors tested this on some very tough challenges:

Zero-Shot Link Prediction: Imagine giving the AI a map of a city it has never seen, with no prior training on that specific city, and asking it to guess where the roads connect.
- Result: GAUGE was much better at guessing these connections than previous models because it learned the "flat," universal rules of how roads connect, rather than memorizing specific city layouts.
Graph Isomorphism: This is like asking, "Are these two different-looking Lego structures actually built from the exact same blueprint?"
- Result: GAUGE was incredibly good at spotting that two different-looking graphs were actually the same underlying structure, proving it understands the deep geometry, not just the surface shape.

Summary

The Problem: We don't know which parts of a graph are reusable (transferable) to new graphs.
The Insight: Reusable parts are those that behave consistently, which mathematically means they exist in a "flat" geometric space.
The Tool: They built GAUGE, a system that uses Neural Vector Bundles to align local maps and a Dirichlet Loss to measure "smoothness."
The Result: By focusing on geometric flatness, GAUGE learns universal graph rules that work better on new, unseen data than previous methods.

In short, they stopped looking at the shape of the Lego bricks and started measuring the smoothness of the table they sit on, finding that the smoothest tables hold the most reusable pieces.

Technical Summary: Are Common Substructures Transferable? Riemannian Graph Foundation Model with Neural Vector Bundles

1. Problem Statement

Foundation models have revolutionized natural language processing and computer vision, yet their application to graphs remains challenging due to the unique nature of graph data. Unlike text or images, graphs possess rich structural patterns where the function of a substructure is derived from its surrounding structural neighborhood rather than isolated discrete patterns.

Current approaches to graph foundation models often rely on identifying common discrete substructures (e.g., motifs, graphons, or structural vocabularies) or depend heavily on text-attributed graphs via Large Language Models (LLMs). However, these methods face limitations:

Discrete vs. Continuous: Prior work focuses on the discrete realm, failing to characterize the intrinsic geometry underlying representations generated by standard graph models.
Transferability Gap: There is a lack of a principled framework to understand which structural behaviors learned during pretraining are truly transferable to unseen target graphs.
Geometric Limitations: Existing Riemannian graph learning methods typically rely on predefined, extrinsic geometric priors (e.g., hyperbolic or spherical spaces) rather than learning the intrinsic geometry induced by the data itself.

The core question motivating this work is: Are common substructures truly transferable? The authors argue that transferability should be understood through the lens of functional behavior invariance rather than discrete pattern matching.

2. Methodology

The paper proposes a framework grounded in Riemannian geometry to model structural transferability. The methodology consists of three main components:

A. Theoretical Foundation: Connecting Transferability to Intrinsic Geometry

The authors establish a theoretical link between behavioral invariance and geometric flatness in the representation space.

Behavioral Invariance: A structural behavior is considered transferable if it requires little adaptation when applied to new graphs. This implies the behavior is invariant to the structural neighborhood.
Vector Bundle Representation: The graph structure is modeled as an abstract base manifold, where each node is equipped with a fiber (a vector space) representing its local structure. This avoids the need for predefined geometric priors.
Flatness and Holonomy: Using the concept of holonomy in Riemannian geometry, the authors prove that an invariant substructure corresponds to a flat intrinsic geometry. Specifically, if the parallel transport around any closed loop (2-simplex) in a substructure results in the identity mapping, the substructure is geometrically flat and thus invariant.

B. Neural Vector Bundle Framework

To implement this theory, the authors introduce the Neural Vector Bundle, a data-driven deep learning framework that parses intrinsic geometry using local coordinates.

Local Trivialization: The framework learns parametric local coordinates for each fiber. It decomposes node representations into disjoint subspaces and uses multi-head attention to derive residual vectors. These are then orthonormalized (via QR decomposition) to form a local basis matrix $Q^{(i)}$ for each node.
Pseudo Parallel Transport: The relationship between fibers of adjacent nodes is modeled via a pseudo parallel transport map $P^{(i,j)} = (Q^{(i)})^\top Q^{(j)}$ . This map captures the geometric "torsion" when moving a vector from one node's fiber to another's.

C. GAUGE Architecture and Dirichlet Loss

Based on the Neural Vector Bundle, the authors design GAUGE (Graph Architecture with Universal Geometry Embedding), a pretrainable architecture.

Gated Flattening: GAUGE employs a gated aggregation mechanism that selectively flattens geometrically compatible fibers. Neighboring fibers with high compatibility (low torsion in parallel transport) are aggregated to refine local coordinates, progressively flattening the bundle.
Dirichlet Loss: A novel pretraining loss is formulated based on Dirichlet energy.
$\mathcal{L}(G) = \sum_{i=1}^N \left\| (Q^{(i)})^\top \hat{z}_i^{(0)} - \frac{1}{|N_i|} \sum_{j \in N_i} (Q^{(j)})^\top z_j^L \right\|_2^2$
This loss measures the discrepancy between a node's local representation and the average of its neighbors' representations in the local coordinate space. Minimizing this loss is proven to be sufficient for learning behavioral invariant substructures. Theoretically, the Dirichlet energy serves as an upper bound for invariance and decreases monotonically during training, ensuring the progressive flattening of the vector bundle.

D. Adaptation Strategy

During downstream adaptation (fine-tuning), the pretrained parameters of the neural vector bundle are kept frozen. Only the input encoding and task-specific linear layers are fine-tuned. The authors prove that for $\tau$ -invariant substructures (those with low prediction error), the gradient updates during fine-tuning are constrained, ensuring that the learned invariant behaviors are preserved.

3. Key Contributions

Principled Geometric Understanding: The paper bridges the gap between structural transferability and intrinsic graph geometry. Instead of focusing on discrete substructures, it connects transferable structures to geometric flatness via behavioral invariance.
Neural Vector Bundle: A new framework for intrinsic geometry learning that characterizes local regions of the underlying manifold using parametric local coordinates, without relying on predefined geometric priors.
GAUGE Architecture: A novel pretraining architecture that constructs the vector bundle by flattening compatible fibers and utilizes a Dirichlet loss to measure transfer effort.
Theoretical Guarantees: The work provides theoretical proofs linking Dirichlet energy to behavioral invariance, demonstrating monotonic flattening of the bundle, and establishing parameter stability during fine-tuning for invariant substructures.

4. Experimental Results

The authors evaluated GAUGE against 16 strong baselines, including supervised GNNs, self-supervised GNNs, and existing Graph Foundation Models (GFMs).

Cross-Domain Transfer Learning: In few-shot settings (1-shot and 5-shot) across diverse domains (biomedical, social, e-commerce), GAUGE consistently outperformed all baselines. For example, on the PubMed dataset with 1-shot, GAUGE achieved 61.26% accuracy compared to the next best (GraphAny at 54.18%).
Zero-Shot Link Prediction: Without any fine-tuning, GAUGE achieved state-of-the-art performance on link prediction tasks across four datasets, demonstrating its ability to capture transferable structural regularities directly from pretraining.
Graph Isomorphism: GAUGE showed superior expressiveness in distinguishing graph isomorphisms (CSL, MUTAG, ZINC12K datasets), achieving 92.56% accuracy on the CSL dataset, significantly outperforming conventional GNNs and other foundation models.
Visualization: The model successfully identified invariant substructures in toy examples and real-world graphs, aligning closely with actual structural patterns (e.g., trees, grids, stars), validating its ability to learn behavioral mechanisms of diverse topologies.

5. Significance and Claims

The paper claims to establish a new theoretical cornerstone for the development of graph foundation models by bridging two previously distinct research areas: graph knowledge transfer and intrinsic geometry.

Theoretical Impact: By introducing the concept of vector bundles (rooted in the mathematics of Élie Cartan and Shiing-Shen Chern) into graph learning, the work provides a rigorous framework for characterizing intrinsic graph geometry. It shifts the paradigm from discrete substructure matching to continuous geometric behavior invariance.
Practical Impact: The authors argue that a profound understanding of graph knowledge transfer can reduce the need for extensive model retraining in real-world applications. This efficiency can conserve computational power and lower carbon emissions, addressing the sustainability concerns of large-scale model training.
Modesty: The authors acknowledge that while their approach is superior in the tested scenarios, the field of graph foundation models is still evolving. They emphasize that their work offers a principled direction rather than a final solution, particularly noting that the intrinsic geometry of graphs is complex and its characterization tools are still under development.

The paper concludes that common substructures are transferable if and only if they correspond to geometrically flat regions in the representation space, a property that GAUGE successfully learns and leverages.