🤖 AI

Towards Foundation Models for Relational Databases with Language Models and Graph Neural Networks

This paper proposes a lightweight hybrid architecture that combines a fine-tuned BART encoder for intra-row semantics with a GraphSAGE-based GNN for relational context, demonstrating that this approach significantly enhances performance on relational database tasks and offers a resource-efficient path toward foundation models for structured data.

Original authors: Jingcheng Wu, Ratan Bahadur Thapa, Mojtaba Nayyeri, Lucas Etteldorf, Max Finkenbeiner, Fabian Leeske, Steffen Staab

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jingcheng Wu, Ratan Bahadur Thapa, Mojtaba Nayyeri, Lucas Etteldorf, Max Finkenbeiner, Fabian Leeske, Steffen Staab

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a massive library where every book is a database table. In a traditional library, if you want to find a specific story about a driver who didn't finish a race, a librarian (the computer) has to manually pull books from the "Drivers" shelf, the "Races" shelf, and the "Results" shelf, copy the relevant pages, and glue them together into one giant, messy document. This process is slow, prone to human error, and often loses the subtle connections between the books.

This paper proposes a new way to read these libraries using a "hybrid brain" that combines two powerful tools: a Language Expert and a Social Network Detective.

The Problem: Flattening the Library

Currently, to use deep learning (AI) on these databases, we usually "flatten" them. We take all the separate tables and smash them into one big, flat spreadsheet.

The Analogy: Imagine taking a 3D puzzle, breaking it apart, and gluing all the pieces onto a single flat piece of cardboard. You can see the colors, but you've lost the 3D structure that tells you how the pieces actually fit together. This destroys the "relational context"—the fact that a driver is linked to a specific race, which is linked to a specific team.

The Solution: The Hybrid Brain

The authors built a system with two parts that work together:

1. The Language Expert (BART)
First, they use a pre-trained language model (BART) to read the rows of the database.

What it does: Think of this as a very smart reader who looks at a single row of data (e.g., "Driver: Alice, Team: Red, Date: Monday") and understands the meaning of that specific sentence. It knows that "Alice" is a name and "Monday" is a time.
The Limitation: This expert is great at understanding one row in isolation, but it doesn't know that Alice is also linked to a specific race happening on that Monday. It's like a reader who knows the plot of one chapter but hasn't read the rest of the book.

2. The Social Network Detective (Graph Neural Network)
Next, they use a Graph Neural Network (GNN), specifically a version called GraphSAGE.

What it does: This acts like a detective who looks at the "connections" (the primary and foreign keys). It sees that the "Driver" row is connected to the "Race" row, which is connected to the "Team" row.
The Magic: It takes the "understanding" from the Language Expert and passes it along these connections. It tells the Detective, "Hey, this driver is linked to a team that usually wins on Mondays." The Detective then updates the driver's profile with this new, relational context.

How They Tested It

They tested this "Hybrid Brain" on a specific challenge from the RelBench benchmark: predicting whether a Formula 1 driver would fail to finish a race (DNF) in the next month.

The Setup: They trained the Language Expert on 6 different databases (like Amazon, event logs, etc.) and then used the Detective to learn the connections.
The Result: When they tested it on a new database (F1 racing data) it had never seen before:
- The Hybrid Brain Score: 67.40 (on a scale where higher is better).
- The "Human Expert" Score: 69.80 (Data scientists manually building features).
- The "Old AI" Score: 68.86 (LightGBM, a standard machine learning tool).
- The "Super AI" Score: 82.63 (KumoRFM, a massive, expensive commercial model).

What This Means

The paper claims that this lightweight hybrid approach is a major step forward.

It bridges the gap: It performs almost as well as the best manual methods and standard AI tools, but without needing a human to manually glue the tables together.
It proves the connection matters: When they turned off the "Detective" (the GNN) and only used the "Language Expert," the score crashed to 43.90. This proves that understanding the relationships between data points is just as important as understanding the data itself.
It's efficient: Unlike the massive commercial models that require huge computing power, this approach uses a "lightweight" architecture that is much more accessible.

The Catch (Limitations)

The authors are honest about where they still have work to do:

Scale: They trained on a relatively small amount of data compared to the "foundation models" that power things like ChatGPT.
Two-Step Process: They train the Language Expert first, then the Detective. They haven't yet figured out how to train them simultaneously to learn even better together.
Not Perfect Yet: While they are close to the "human expert" level, they are still about 15 points behind the massive, proprietary commercial models.

The Bottom Line

This paper suggests that we don't need to choose between "understanding the words" (Language Models) and "understanding the connections" (Graph Networks). By combining them, we can create a system that reads relational databases much more naturally, moving us closer to a future where AI can understand complex, multi-table data without needing a human to flatten it first.

Technical Summary: Towards Foundation Models for Relational Databases with Language Models and Graph Neural Networks

Problem Statement

Relational databases (RDBs) store a vast amount of the world's structured information, yet deep learning progress on this data type remains limited. Conventional approaches typically flatten multi-table databases into single tables via manual feature engineering (joins and aggregations). This process is labor-intensive, error-prone, and critically discards the relational context encoded by primary-foreign key links.

While Relational Deep Learning (RDL) addresses this by modeling databases as Relational Entity Graphs (REGs) for Graph Neural Networks (GNNs), existing RDL models are generally task- and database-specific, lacking transferable representations. Conversely, recent relational foundation models (e.g., Griffin, KumoRFM) show promise but are often computationally intensive and complex, hindering widespread adoption. There is a need for a lightweight, accessible architecture that combines the semantic understanding of Language Models (LMs) with the structural reasoning of GNNs to serve as a foundation for general-purpose relational learning.

Methodology

The authors propose a hybrid architecture that integrates a fine-tuned Language Model (BART) with a Graph Neural Network (GraphSAGE) operating over REGs. The framework follows a two-stage pipeline:

1. Architecture

Row-Level Semantics (BART Encoder): Database rows are linearized into schema-aware sequences (e.g., <table> Table Name <attr> Attr Name <value> Value). A pre-trained BART model is fine-tuned on these sequences to capture intra-row semantics. The output is a 1024-dimensional embedding per row, serving as the initial node features for the graph.
Relational Context (GraphSAGE GNN): The database is represented as a heterogeneous REG where nodes are rows and edges represent primary-foreign key relationships. A HeteroGraphSAGE model (implemented via PyTorch Geometric) processes these nodes. It employs node-type-specific linear layers to project BART embeddings into a hidden space (256 dimensions) and shared SAGEConv layers to perform message passing across edge types.
Output: The GNN enriches the initial embeddings with relational context, outputting a 1024-dimensional vector for reconstruction during pre-training or a 256-dimensional vector for downstream tasks.

2. Pre-Training Strategy

The model is pre-trained using self-supervised objectives on six RelBench databases (excluding the test set):

BART Stage: Fine-tuned using a schema-aware masking strategy. Table names, attribute names, and cell values are masked with specific probabilities (0.30, 0.20, and 0.40, respectively) to force the model to reconstruct the original sequence.
GNN Stage: Trained with a masked value reconstruction objective. Random feature dimensions are masked, and the model must infer them via message passing from neighbors. The loss function is a weighted combination of Scaled Cosine Error and Mean Squared Error (MSE), prioritizing directional alignment ( $\alpha=0.7$ ).

3. Downstream Adaptation

For downstream tasks (e.g., node classification), the framework constructs a new REG for the target database. Rows are encoded by the pre-trained BART model, and the shared SAGEConv layers are transferred. Two adaptation modes are tested:

Frozen: Only new node-type-specific linear layers and the prediction head are trained.
Fine-tuned: The transferred SAGEConv layers are updated jointly with the prediction head.

Key Contributions

Novel Hybrid Architecture: The paper introduces a seamless integration of a pre-trained BART encoder and a GraphSAGE-based GNN. This design captures deep intra-row semantics while enriching them with structural relational context via message passing.
Systematic Empirical Evaluation: The authors evaluate the architecture on the RelBench benchmark, specifically the driver-dnf node classification task on the rel-f1 dataset.
Ablation and Analysis: Extensive ablation studies demonstrate that the GNN is critical for performance. The study highlights that while BART provides strong semantic features, it fails without relational context, and random embeddings fail without semantic pre-training.
Strategic Roadmap: The paper outlines future directions for relational foundation models, including joint LM-GNN training and scaling to diverse databases.

Results

The experiments were conducted on the driver-dnf task (predicting if a driver will fail to finish a race) using the rel-f1 dataset.

Performance: The fine-tuned hybrid model achieved a ROC-AUC of 67.40.
Comparison to Baselines:
- Supervised Methods: The result is competitive with LightGBM (68.86) and manual feature engineering by data scientists (69.80).
- RDL: It narrows the gap to the fully supervised, per-database RDL baseline (72.62) to within 5.22 points.
- Foundation Models:
  - It outperforms the pre-trained Griffin variant (59.64) in the frozen setting (61.40), demonstrating the value of pre-training on relational structures rather than just single tables.
  - It remains below the state-of-the-art KumoRFM (82.63), which the authors attribute to KumoRFM's larger scale, advanced graph transformers, and commercial resources.
Ablation Insights:
- BART + GNN (Fine-tuned): 67.40 (Best)
- BART + GNN (Frozen): 61.40
- BART Only (No GNN): 43.90 (Significant drop, indicating the MLP head cannot compensate for missing relational context).
- Random Embeddings + GNN (Fine-tuned): 58.00 (Confirms BART's semantic value).

Significance and Claims

The paper posits that lightweight hybrid LM-GNN architectures offer a promising and resource-efficient path toward foundation models for relational databases.

Bridging the Gap: The results suggest that such hybrid models can effectively bridge the gap between traditional supervised methods and complex, database-specific RDL models without requiring the massive computational resources of current commercial foundation models.
Relational Context is Critical: The study emphasizes that structural message passing is essential; semantic embeddings alone (BART) are insufficient for relational tasks, and the GNN's ability to propagate latent semantic structure is what drives performance gains.
Modest Claims: The authors acknowledge that their framework is not yet a "true" relational foundation model. Limitations include the relatively small pre-training scale (100k rows for BART, 6 databases for GNN), the sequential (non-joint) training pipeline, and evaluation on a single downstream task. They frame their work as a proof-of-concept that validates the hybrid paradigm and identifies key research directions (e.g., joint training, scaling) necessary to transform the framework into a robust foundation model.