🤖 machine learning

RelBench v2: A Large-Scale Benchmark and Repository for Relational Data

This paper introduces RelBench v2, a significantly expanded benchmark and repository for relational deep learning that incorporates four new large-scale datasets, novel autocomplete tasks, and integrations with external frameworks to facilitate systematic evaluation and demonstrate the superiority of relational models over single-table baselines.

Original authors: Justin Gu, Rishabh Ranjan, Charilaos Kanatsoulis, Haiming Tang, Martin Jurkovic, Valter Hudovernik, Mark Znidar, Pranshu Chaturvedi, Parth Shroff, Fengyu Li, Jure Leskovec

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Justin Gu, Rishabh Ranjan, Charilaos Kanatsoulis, Haiming Tang, Martin Jurkovic, Valter Hudovernik, Mark Znidar, Pranshu Chaturvedi, Parth Shroff, Fengyu Li, Jure Leskovec

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive library of information. In the old days, if you wanted to find a pattern, you'd have to take every single book, rip out the pages, and paste them all onto one giant, messy sheet of paper. You'd lose the context of which book the page came from, and you'd have to manually draw lines connecting related ideas. This is how traditional machine learning used to handle complex data.

Relational Deep Learning (RDL) is like a new kind of librarian who doesn't rip the pages out. Instead, they look at the whole library as a living map. They understand that a "Book" is connected to an "Author," who is connected to a "University," which is connected to a "City." They see the relationships, not just the isolated facts.

This paper introduces RELBENCH v2, a massive upgrade to a "training gym" for these smart librarians. Here is what they built:

1. The New Training Grounds (The Datasets)

The original gym (RELBENCH v1) had a few practice fields. RELBENCH v2 has added four huge new stadiums, bringing the total to 11. These aren't just small practice fields; they are massive, real-world data worlds:

The Scholar's Library (rel-arxiv): A map of millions of research papers, authors, and who cited whom.
The Corporate Office (rel-salt): A simulation of a giant company's sales orders, shipping routes, and payment terms.
The Beer Lover's Forum (rel-ratebeer): A decade of users rating beers, breweries, and places.
The Hospital Records (rel-mimic): Anonymized patient records from a major medical center (with dates shifted into the future to protect privacy).

Together, these contain over 22 million rows of data. It's like giving the AI a library with 22 million books to learn from, all interconnected.

2. The New Games (The Tasks)

The paper introduces a new type of game called "Autocomplete."

The Old Game (Forecasting): Imagine a crystal ball. You give the AI data up to today, and it predicts something that hasn't happened yet (like "Will this customer leave next month?").
The New Game (Autocomplete): Imagine a partially filled-out form. The AI sees some fields (like "Customer Name" and "Order Date") but the "Payment Method" box is blank. The AI has to guess the missing box right now, using only the clues from the other tables it can see.

Why is this hard? It's like trying to guess a person's favorite food just by knowing their address and the time of day, without them telling you. The AI has to look at the "neighborhood" (related tables) to figure it out. The paper shows that the AI is much better at this when it understands the relationships between the tables than when it just looks at the single row.

3. The "Universal Adapter" (Integrating Other Benchmarks)

The authors didn't just build new rooms; they built a universal adapter that lets the gym connect to other popular training centers:

The Time-Stream Gym (TGB): They translated datasets that usually look like a flowing river of events into their library format, so the AI can practice on them too.
The 70-Database Challenge (ReDeLEx): They connected to a collection of 70+ real-world databases, giving the AI a chance to practice on almost anything.
The 4D Toolbox (4DBInfer): They added a set of tools that test the AI from four different angles (data, tasks, graph structure, and models) to make sure it's truly robust.

4. The Results: The Power of Connection

The paper ran a series of tests comparing the "Smart Librarian" (RDL models) against the "Old School Worker" (traditional models that flatten the data).

The Verdict: The Smart Librarian won almost every time.

Whether guessing missing values (Autocomplete), predicting the future (Forecasting), or recommending the next item (Recommendation), the model that understood the relationships between tables performed significantly better.
Even when the data was messy or the categories were unbalanced, the relational model used the connections to find signals that the old models missed.

Summary

Think of RELBENCH v2 as the ultimate "Driver's Ed" for Artificial Intelligence trying to understand complex databases. It provides:

More Cars: 11 massive, diverse datasets (Academic, Business, Consumer, Medical).
New Drills: A new "Autocomplete" challenge to test how well the AI fills in missing blanks using context.
A Universal Garage: Tools to connect with other testing grounds so the AI can learn from everything.

The main takeaway is simple: When data is connected, you have to treat it as a connected web, not a flat list. The models that do this are winning the race.

Technical Summary: RELBENCH V2

Problem Statement

Relational databases are the primary storage abstraction for structured data across enterprise, scientific, and healthcare systems, organizing information via primary–foreign key relationships that capture rich structural and temporal dependencies. While Relational Deep Learning (RDL) has emerged as a paradigm for learning directly on these databases—treating them as heterogeneous graphs to avoid manual feature engineering and flattening—there remains a critical need for scalable, realistic benchmarks to evaluate progress, particularly as the field shifts toward relational foundation models. Existing benchmarks often lack scale, diversity in domains, or the ability to evaluate tasks beyond traditional forecasting constructed via SQL queries.

Methodology and Framework

RELBENCH v2 expands upon the original RELBENCH framework (Robinson et al., 2024) by introducing new datasets, novel task types, and a unified integration of external benchmarks. The core methodology relies on:

Temporal Constraints and Leakage Prevention: All datasets utilize fixed VAL_TIMESTAMP and TEST_TIMESTAMP cutoffs. Models are trained on data up to VAL_TIMESTAMP, validated on rows between the cutoffs, and tested on rows after TEST_TIMESTAMP. Data beyond the test cutoff is hidden during inference to prevent test-time leakage, utilizing temporal neighbor sampling strategies (Fey et al., 2024).
RDL Architecture: The benchmark utilizes a standard RDL pipeline:
- Encoding: Raw row-level data is encoded into initial node embeddings using PyTorch Frame with a ResNet tabular model.
- Sampling: Temporal-aware subgraph sampling is performed around each entity node at a given seed time.
- Aggregation: Embeddings are passed through a heterogeneous GraphSAGE model with sum-based neighbor aggregation to iteratively update node embeddings.
- Prediction: Task-specific heads generate final predictions.
Baseline Comparisons: Performance is benchmarked against strong baselines, primarily LightGBM (applied to raw entity table features or joined features) and various aggregation strategies (global mean/median, entity mean/median).

Key Contributions

1. Expansion of Datasets

RELBENCH v2 introduces four new large-scale relational datasets, bringing the total to 11 datasets spanning diverse domains:

rel-arxiv: A scholarly publication database with 222,000 papers, 1.5 million citation links, and author-category relationships.
rel-salt: An enterprise resource planning (ERP) dataset from SAP AI Research modeling sales orders, workflows, and business transactions.
rel-ratebeer: A consumer platform dataset with over two decades of user interactions, beer ratings, and reviews.
rel-mimic: A clinical dataset derived from MIMIC-IV electronic health records, featuring date shifting for privacy.
Scale: The new datasets contribute over 22 million rows across 29 tables.

2. Introduction of Autocomplete Tasks

A significant methodological contribution is the introduction of autocomplete tasks. Unlike traditional forecasting tasks that predict future outcomes via SQL queries, autocomplete tasks require models to infer missing attribute values within existing relational tables at a given timestamp.

Mechanism: Models receive observed values for a row at a seed time and must predict target columns using only relational and temporal context available up to that time.
Leakage Control: To prevent information leakage, columns highly correlated with the target (e.g., review_text for review-rating) are manually identified and removed based on domain knowledge.
Types: Includes both classification (binary and multiclass) and regression tasks.

3. Integration of External Benchmarks

RELBENCH v2 unifies evaluation by integrating external frameworks:

Temporal Graph Benchmark (TGB): Event streams from TGB are translated into relational schemas (normalizing nodes and edges into tables) to enable direct comparison between temporal GNNs and relational deep learning models.
ReDeLEx: Integration provides uniform access to 70+ real-world databases suitable for pretraining.
4DBInfer: Incorporation of 4DBInfer datasets and tasks to broaden multi-table evaluation coverage and explore the 4D design space (datasets, tasks, graph construction, models).

Experimental Results

Experimental results across the new datasets and tasks demonstrate that RDL models consistently outperform single-table baselines (specifically LightGBM and aggregation baselines):

Autocomplete Tasks: RDL models significantly outperformed LightGBM in both binary and multiclass classification (e.g., achieving ~98% accuracy on item-shippoint in rel-salt vs. ~5% for LightGBM) and regression tasks (higher $R^2$ values). The results highlight that relational context provides informative signals for predicting missing attributes, even in the presence of class imbalance.
Forecasting Tasks: RDL models achieved superior performance in entity classification (e.g., author-category prediction in rel-arxiv) and regression. The multiclass author-category task showed a massive gap, with RDL achieving ~50% accuracy compared to ~2% for LightGBM, suggesting relational signals (coauthorship, citations) are crucial for complex classification.
Recommendation Tasks: GNN-based models (GraphSAGE and ID-GNN) generally outperformed baselines like Past Visit and Global Popularity. ID-GNN, which explicitly incorporates node identity, excelled in entity-specific settings, while deeper GraphSAGE models (4 layers) showed improved performance in multi-hop reasoning tasks, though limited by GPU memory constraints.
TGB Integration: In translated TGB datasets, relational "event-as-node" GraphSAGE models performed competitively against graph-native projected-edge GraphSAGE and temporal baselines (TGN), with performance varying based on the richness of event semantics and schema structure.

Significance

The paper claims that RELBENCH v2 provides a scalable and realistic benchmark essential for the development and evaluation of RDL systems and relational foundation models. By expanding the scale to over 22 million rows, introducing the novel autocomplete paradigm, and unifying external benchmarks, it addresses the limitations of previous single-table or small-scale evaluations. The consistent outperformance of RDL models over baselines across diverse domains (academic, enterprise, consumer, clinical) underscores the importance of explicitly modeling relational structure and temporal dependencies for predictive tasks. The benchmark serves as a foundational tool for advancing the field toward foundation models capable of reasoning over complex, multi-table data.