🤖 machine learning

TopoFormer: Topology Meets Attention for Graph Learning

TopoFormer is a lightweight, scalable framework that integrates topological structure into attention-based models via a novel Topo-Scan module, which decomposes graphs into parallelizable sequences to achieve state-of-the-art performance in graph learning without the computational costs of traditional persistent homology.

Original authors: Md Joshem Uddin, Astrit Tola, Cuneyt Gurcan Akcora, Baris Coskunuzer

Published 2026-07-31

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Md Joshem Uddin, Astrit Tola, Cuneyt Gurcan Akcora, Baris Coskunuzer

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to understand the shape of things. In the world of data, we often use "graphs" to represent connections—like friends in a social network, atoms in a molecule, or stops on a subway map. These aren't neat grids like a spreadsheet; they are messy, irregular webs. For a long time, computers struggled to see the big picture in these webs because they were too busy looking at individual dots (nodes) and lines (edges), missing the forest for the trees.

To help computers see the "shape" of these webs, scientists use a branch of math called Topological Data Analysis (TDA). Think of TDA as a way to count the holes, loops, and islands in a shape without caring about the exact size or position of the dots. It's like looking at a donut and knowing it has one hole, regardless of whether the donut is giant or tiny. Another key idea is the "Transformer," a type of AI famous for reading language and looking at images. Transformers are great at spotting patterns in long lists of words or pixels, but they usually struggle with messy, irregular graphs. The big question researchers have been asking is: How can we turn these messy, irregular webs into a neat, ordered list that a Transformer can read, while keeping the important "shape" information intact?

This is where a new paper called TOPOFORMER comes in. The authors, a team of researchers from universities in the US, have built a clever tool that acts like a high-tech scanner for graphs. Instead of trying to force a messy graph into a rigid grid, TOPOFORMER slices the graph into a series of thin, ordered layers, much like slicing a loaf of bread or taking a series of CT scans of a body.

Here is how their magic trick works: They use a method they call Topo-Scan. Imagine you have a complex knot of string. Traditional methods might try to measure the whole knot at once or build a complicated 3D map of every twist, which takes a huge amount of time and computing power. TOPOFORMER, however, takes a different approach. It slowly "slices" through the graph using a sliding window. As it moves through the graph, it counts simple things at each slice: how many separate pieces of string are there? How many loops have formed? How many knots are visible right now?

These counts create a short, neat list of numbers—a "topological token sequence." This list is perfectly shaped for a Transformer to read. The Transformer then looks at this list and learns to recognize the overall structure of the graph, from small local loops to the big global shape. The authors found that this method is not only faster and lighter on computer resources than older techniques (which often get bogged down by expensive calculations), but it also works incredibly well.

In their experiments, TOPOFORMER was tested on two main tasks: classifying different types of graphs (like telling if a graph represents a chemical molecule or a social network) and predicting properties of molecules (like whether a drug will work or be toxic). The results were impressive. On eight different graph classification benchmarks, TOPOFORMer achieved the best or second-best accuracy, beating many complex, heavy-duty models that have been the standard for years. When it came to predicting molecular properties, it performed just as well as, or better than, the top models currently in use, often ranking in the top three.

What makes this particularly exciting is that TOPOFORMER doesn't need to learn a unique "embedding" (a complex internal code) for every single node in the graph first. Instead, it treats the whole graph as a single, structured object and scans it directly. The authors suggest that this approach offers a new, efficient path forward for creating "foundation models" for graphs—super-smart AI systems that can understand any kind of network, from biology to social media, without needing to be retrained from scratch every time. They also proved mathematically that their method is stable, meaning if you slightly change the input graph, the output doesn't go haywire, which is crucial for real-world reliability.

In short, TOPOFORMER takes the messy, irregular world of graphs, slices it into a clean, readable story, and lets a powerful AI read that story to understand the shape of the data. It's a lighter, faster, and surprisingly effective way to teach machines to see the world in terms of connections and shapes.

Technical Summary: TOPOFORMER

Problem Statement

Graph representation learning faces significant challenges in achieving the generalization capabilities seen in foundation models for vision and language. The irregular, non-Euclidean nature of graphs complicates the design of transferable inductive biases. While Topological Data Analysis (TDA), specifically Persistent Homology (PH), offers a principled way to encode stable, multi-scale structural information (global connectivity and cycles), its adoption in deep learning has been hindered by two primary bottlenecks:

Computational Cost: Standard PH pipelines require expensive global reductions to compute persistence diagrams, which do not parallelize well.
Vectorization Friction: Converting persistence diagrams into vectors (e.g., persistence images, landscapes) is often ad hoc, can destroy sequential structural information, and significantly impacts downstream performance. Furthermore, traditional sublevel filtrations on graphs often suffer from "early saturation," where high-valued nodes activate early, filling the complex and suppressing late-emerging topological features.

Existing graph transformers (e.g., Graphormer, GPS) typically operate directly on node tokens, injecting structure via positional encodings or attention biases, which can be computationally heavy and less focused on global topological invariants.

Methodology: TOPOFORMER and Topo-Scan

The authors propose TOPOFORMER, a lightweight and scalable framework that encodes topological structure into attention-friendly sequences, bypassing the need for full persistence diagrams and complex vectorization steps.

Core Component: Topo-Scan

The central innovation is Topo-Scan, a module that decomposes a graph into a short, ordered sequence of topological tokens. Unlike standard PH which builds a single nested filtration, Topo-Scan employs a sliding window approach:

Filtration: A scalar function $f: V \to \mathbb{R}$ is defined on vertices (e.g., degree, curvature, heat kernel signature).
Slicing: Instead of a cumulative threshold, the graph is sliced into overlapping subgraphs $G_i$ based on intervals $[\alpha_i, \alpha_i + m]$ of the filtration function.
Token Generation: For each slice, the clique complex is formed, and four invariants are computed:
1. $\beta_0$ : Number of connected components (Betti-0).
2. $\beta_1$ : Number of cycles/holes (Betti-1).
3. $|V_i|$ : Node count.
4. $|E_i|$ : Edge count.
Sequence Formation: These invariants form a compact, ordered sequence $\Gamma(G)$ of length $4N$ (where $N$ is the number of slices). This sequence preserves the evolution of topological features across scales without the early saturation issues of standard sublevel filtrations.

Transformer Architecture

The generated topological sequences are fed directly into a Transformer encoder.

Input: A sequence of tokens where each token represents the topological state of a specific slice.
Processing: The Transformer utilizes self-attention mechanisms to capture dependencies across the sequence, effectively modeling multi-resolution structural patterns.
Dual-Transformer Framework: The model can process multiple filtration functions (e.g., Ollivier-Ricci curvature, Heat Kernel Signature) in parallel. Outputs from independent transformers are fused via a learnable weighted sum and an MLP classifier to produce graph-level embeddings.
Molecular Property Prediction (MPP): For molecular tasks, the framework integrates topological sequences with standard molecular fingerprints (e.g., ECFPs) processed by an MLP, combining them via attention-based concatenation.

Theoretical Guarantees

The paper provides theoretical stability guarantees for Topo-Scan. It establishes a discrete $\ell_1$ stability bound, proving that small perturbations in the filtration function result in bounded changes in the output sequence. This relies on the relationship between Topo-Scan tokens and interlevel persistence modules, showing that the method is robust to noise in the input signal.

Key Contributions

Scalable Topological Encoding: A method to convert topological structure into attention-ready sequences, enabling Transformers to process graphs without relying on node embeddings or heavy preprocessing.
Bridging TDA and Deep Learning: A unified framework that captures both local and global graph structure through a sequential attention mechanism, avoiding the computational overhead of persistence diagrams.
Comprehensive Evaluation: Extensive experiments demonstrating state-of-the-art (SOTA) performance on graph classification and molecular property prediction benchmarks.
Efficiency and Stability: Theoretical guarantees on representation robustness and empirical evidence of predictable, efficient compute suitable for large-scale applications.

Experimental Results

The authors evaluated TOPOFORMER on diverse datasets:

Graph Classification: Tested on 9 benchmarks (including BZR, MUTAG, IMDB-B/M, REDDIT-B/5K, and OGBG-MOLHIV).
- TOPOFORMER achieved the best or second-best accuracy on 7 out of 8 benchmarks.
- It established new SOTA on BZR, MUTAG, PROTEINS, IMDB-M, REDDIT-B, and REDDIT-5K.
- It outperformed 20 baselines, including GNNs, topological methods (PersLay, DMP), and contrastive learning models.
- On the large-scale OGBG-MOLHIV dataset, it achieved an AUC within ~2 points of the strong Graphormer baseline, demonstrating scalability.
Molecular Property Prediction (MPP): Tested on 7 datasets from MoleculeNet (BBBP, Tox21, ToxCast, ClinTox, SIDER, BACE, HIV).
- TOPOFORMER* (combined with fingerprints) achieved the best ROC AUC on ToxCast, ClinTox, and BACE.
- It was the runner-up on Tox21 and highly competitive on HIV.
- It achieved the second-lowest average deviation from the best model and the second-lowest average rank across all datasets.
Ablation Studies:
- vs. PH: TOPOFORMER outperformed both standard PH-MLP and PH-Transformer baselines, confirming that Topo-Scan captures more expressive structure than standard PH filtrations and that sequential encoding is superior to static vectorization.
- Filtration Sensitivity: The model showed robustness across different filtration functions (Degree, Ollivier-Ricci, HKS) and window widths.
- Feature Combination: Combining topological features with molecular fingerprints consistently outperformed using either modality alone.

Significance and Claims

The paper positions TOPOFORMER not as a full foundation model itself, but as a step toward topology-aware graph foundation models. Its significance lies in:

Unifying Approach: It offers a parallelizable, unifying approach to graph representation learning that integrates topological inductive biases directly into attention frameworks.
Efficiency: By bypassing global persistence diagram computations, it offers predictable and efficient compute, making it practical for large-scale applications where traditional TDA is too slow.
Generalization: The fixed-size, transferable representations produced by TOPOFORMER are well-suited for tasks requiring consistent embeddings across graphs of varying sizes and domains, addressing a central challenge in graph learning.
Modest Scope: The authors explicitly state that the work focuses on a streamlined, graph-level instantiation. They leave node/edge-level tasks, heterogeneous settings (temporal/citation graphs), and large-scale pretraining on heterogeneous corpora to future work. They view Topo-Scan as a lightweight summary that complements rather than replaces full persistent homology.