🔢 mathematics

Have Graph -- Will Lift? The Case for Higher-Order Benchmarks

This opinion paper advocates for the development of new, native higher-order datasets to address the current scarcity of benchmarks in topological deep learning, arguing that relying solely on lifted graph datasets is insufficient for advancing the field.

Original authors: Bastian Rieck

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Bastian Rieck

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Building a Better Map

Imagine you are trying to teach a robot to understand the world. For a long time, researchers have taught robots using graphs. Think of a graph as a simple map of dots (people, cities, atoms) connected by lines (friendships, roads, chemical bonds). This has worked well, but the world is more complex than just dots and lines. Sometimes, things interact in groups of three, four, or more at once.

To handle this, researchers invented Topological Deep Learning (TDL). This is like upgrading from a 2D map to a 3D model that captures shapes, holes, and complex structures. The paper argues that while our tools (the models) are getting fancy, our test questions (the datasets) are still stuck in the past.

The Problem: The "Lifting" Trap

Currently, most researchers don't have new, complex 3D data. So, they take their old 2D graph data and try to "lift" it into 3D.

The Analogy:
Imagine you have a flat, black-and-white photograph of a house. You want to test a robot's ability to understand 3D architecture. Since you don't have a real 3D model, you take that flat photo and try to "guess" the 3D shape.

Strategy A: You assume every group of three people in the photo who are standing close together form a triangle in 3D space.
Strategy B: You assume every group of three people who share a common friend form a triangle.

Both strategies turn the flat photo into a 3D shape, but they create different shapes. The paper calls this "lifting."

The Issue:
The author points out that researchers often pick one of these guessing strategies, run their model, and say, "Look! My model works better!" But they rarely ask: "Did the model actually get smarter, or did we just get lucky because our specific guessing strategy happened to fit the data?"

It's like testing a chef by giving them a pre-chopped salad. If they make a great salad, did they cook well, or did they just get lucky because the ingredients were already perfect? The paper argues that "lifting" is often just a pre-chopped ingredient that hides whether the model is truly learning.

The Solution: Stop Guessing, Start Collecting

The author suggests we stop trying to turn flat photos into 3D models. Instead, we need to go out and collect real 3D data.

The Analogy:
Instead of guessing what a house looks like from a photo, we need to go to a construction site and measure the actual bricks, beams, and rooms. We need datasets where the complex, group-based structure is real and crucial to the task, not just an artificial addition.

The paper lists four rules for these new datasets:

Meaningful: The complex structure must actually matter for the task.
Known Difficulty: We need to know if the task is easy or hard (by comparing it to simple baselines).
Standardized: Everyone must use the same test splits so results can be compared fairly.
Maintained: The data must be stored safely so it doesn't disappear (like a file on a defunct website).

The New Project: MANTRA

To prove this is possible, the author and their team created a new dataset called MANTRA.

The Analogy:
Think of MANTRA as a library of abstract Lego structures.

Most other datasets give you the Lego bricks with instructions on how to paint them (features/coordinates).
MANTRA gives you only the connections: "Brick A connects to Brick B, which connects to Brick C." It doesn't tell you where they are in space or what color they are.

The task for the AI is to look at these abstract connections and answer questions like:

"Is this shape a sphere or a donut?"
"Does this shape have a hole in it?"

Why is this hard?
Because the AI can't just look at the "shape" (since there are no coordinates). It has to understand the logic of the connections. The paper found that even the smartest current AI models struggle with this. They are good at counting local connections (like "how many neighbors does this brick have?") but bad at understanding the global shape (like "is this a sphere?").

The Conclusion: We Are Just Starting

The paper concludes that we are at the very beginning of this journey.

Current State: Our models are like students who are good at memorizing facts but bad at understanding the underlying principles.
The Goal: We need better "textbooks" (datasets) that force models to learn the deep, structural rules of the universe, not just the surface-level patterns.

The author urges the community to stop relying on "lifted" data (guessing 3D from 2D) and start building and sharing these new, "intrinsically higher-order" datasets. Only then can we be sure our AI is actually learning to see the world in 3D, rather than just guessing based on a flat photo.

Technical Summary: "Have Graph — Will Lift? The Case for Higher-Order Benchmarks"

Problem Statement
The paper identifies a critical bottleneck in the advancement of Geometric Deep Learning (GDL) and Topological Deep Learning (TDL): the scarcity of suitable, native higher-order benchmark datasets. Currently, the field relies heavily on "lifting" existing graph datasets into higher-order representations (such as simplicial or cellular complexes) to evaluate higher-order methods. The author argues that this practice is flawed for two primary reasons:

Arbitrariness of Lifting: There is no "one-size-fits-all" strategy for lifting a graph to a complex (e.g., clique complex vs. neighborhood complex). The choice of lifting strategy acts as a hand-crafted feature that significantly influences model performance but is often treated as a fixed preprocessing step rather than a tunable hyperparameter.
Obscured Relevance: Recent evidence suggests that for many standard graph datasets, the graph structure itself is largely irrelevant to task performance. Consequently, lifting these datasets may hide the true utility of higher-order information or conflate performance gains with the specific lifting strategy rather than the model's architectural capabilities. Furthermore, the lack of standardized data provenance and maintenance in current repositories risks a future replication crisis.

Methodology and Proposed Framework
Rather than proposing a new specific model architecture, the paper advocates for a shift in research culture and benchmarking practices:

Multiverse Analysis for Lifting: The author proposes treating the choice of lifting strategy as part of a "multiverse analysis." Researchers should explicitly report and ablate different lifting strategies alongside model architectures and hyperparameters to ensure transparency and reproducibility.
Learnable Lifting: While acknowledging recent work on learnable lifting strategies, the paper insists these must be critically assessed like any other architectural choice.
Data Stewardship Standards: The paper outlines four criteria for new datasets:
1. Meaningful construction where higher-order structure is crucial for the task.
2. Known task complexity (including simple graph baselines).
3. Standardized splits and sufficient size for statistical reasoning.
4. Version control and maintenance plans to prevent data loss.

Key Contributions: The MANTRA Dataset
To demonstrate the viability of native higher-order benchmarks, the paper introduces MANTRA ("Manifold Triangulations Assemblage").

Source: Based on the combinatorial enumeration of minimal triangulations of 2-manifolds and 3-manifolds by Frank H. Lutz.
Characteristics:
- Intrinsically Higher-Order: The data consists of abstract simplicial complexes without any prior lifting from graphs.
- Coordinate-Free: The dataset contains only combinatorial information (vertex/simplicial connectivity). While random realizations exist, they are non-canonical and do not aid in recognizing the underlying manifold, thereby increasing task difficulty.
- Scale: Contains over 43,000 2-manifolds and 250,000 3-manifolds.
Tasks: The dataset supports regression (predicting Betti numbers, torsion coefficients) and classification (predicting orientability, vertex transitivity, or canonical names like $S^2$ or $\#_k T^2$ ).
Theoretical Baseline: For 2-manifolds, the classification problem is theoretically solvable (via Euler characteristic and orientability), serving as a "smoke test" for model expressivity.

Results and Preliminary Findings
The paper reports preliminary experiments (referenced from Ballester et al., 2025) evaluating contemporary GDL and TDL models on the MANTRA dataset:

Performance: Both GDL (restricted to 1-skeletons) and TDL models (using higher-order message passing) struggle to solve the tasks. Even for 2-manifolds, where the problem is theoretically simple, models fail to achieve perfect predictive performance.
Failure Mode: The models appear to rely on local aggregations of combinatorial structures rather than capturing the inherent topological equivalence relations (e.g., homeomorphism).
Sensitivity: Performance degrades when triangulations undergo topological-preserving transformations (e.g., barycentric subdivision), indicating that current models lack robustness to topological invariants.

Significance and Claims
The paper positions itself as a call to action for the community to move beyond "lifting" existing data and toward the curation of native higher-order datasets.

Diagnostic Value: The inability of current TDL models to solve the MANTRA tasks suggests fundamental challenges in model design, specifically a misalignment between the combinatorial priors used in message passing and the topological equivalence relations required for the tasks.
Community Resource: MANTRA is presented not as a final solution but as a starting point for a repository of diverse triangulations, intended to serve as a higher-order companion to graph benchmarks like GraphBench.
Future Direction: The author emphasizes that for TDL to mature, methods must not only be inspired by topological concepts but must possess strong topological guarantees and be evaluated on data where the higher-order structure is the primary signal, not an artifact of lifting.