🤖 AI

GRATE: Temporal Extensions for Inductive KG Foundation Models via Gated Rotary Attention

The paper introduces GRATE, a parameter-free temporal encoding method using gated rotary attention that enables Knowledge Graph foundation models to achieve inductive transfer across disjoint temporal datasets, validated by newly constructed benchmarks with non-overlapping entities, relations, and timestamps.

Original authors: Jiaxin Pan, Osama Mohammed, Daniel Hernández, Steffen Staab

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiaxin Pan, Osama Mohammed, Daniel Hernández, Steffen Staab

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart detective who can solve mysteries by looking at a giant map of connections between people, places, and things. This detective, called a "Knowledge Graph Foundation Model," is amazing because it can figure out new connections it has never seen before, even if the names of the people or the types of relationships are totally different from what it learned in school. It's like a detective who knows how to spot a "friendship" or a "rivalry" just by looking at the pattern of interactions, without needing to memorize every single name in the world.

But here's the catch: real life happens in time. A meeting between two presidents in 1974 is very different from a meeting between them in 2024, even if the map looks the same. The old detectives (existing models) were stuck in the past; they treated a fact from 1974 and a fact from 2024 as if they were equally important, or they needed to be retrained from scratch every time a new year rolled around. They couldn't handle the flow of time.

Enter GRATE (Gated Rotary Attention for Temporal Encoding). Think of GRATE as a magical, invisible time-spectacles upgrade for our detective. It doesn't add any new heavy brain cells (learnable parameters) to the detective's head; instead, it changes how the detective looks at the clues.

How GRATE Works: The Time-Spin and The Filter

GRATE uses two clever tricks to make the detective time-aware:

The Time-Spin (Rotary Encoding): Imagine every clue the detective finds is a spinning top. If a clue happened a long time ago, GRATE spins that top a lot. If it happened just yesterday, it spins it just a tiny bit. This "spin" is based on the difference in time between the clue and the question, not the specific date on the calendar. This means the detective can understand that a meeting from 2024 is "closer" to a 2024 question than a meeting from 1974, even if the detective has never seen the year 2024 before. It turns time into a smooth, continuous rotation rather than a rigid list of dates.
The Smart Filter (Query-Conditioned Gating): Just because a clue is close in time doesn't mean it's the right one. If you ask, "Who did the US President meet in 2024?", a meeting with a sports star might be close in time but irrelevant, while a meeting with a foreign leader is crucial. GRATE acts like a bouncer at a club. It looks at the question (the query) and the spinning clue, and decides: "Yes, this clue is relevant, let it in," or "No, this clue is off-topic, block it." This filter is super smart because it doesn't need to be taught what "relevant" means; it figures it out on the fly based on the question itself.

The Big Test: Can It Handle the Unknown?

The authors didn't just build this; they put it through a grueling obstacle course. They created special test sets called GDELTINDT and WIKIINDT. These are like "final exams" where the detective is given a mystery involving people, relationships, and dates it has never seen before. It's the ultimate "zero-shot" test: no cheating, no retraining, just pure logic.

The results were impressive. When the detective used GRATE:

On standard tests (like ICEWS14), it improved its accuracy (MRR) by about 25% to 38% compared to the detective without the time-spectacles.
On the "never-seen-before" tests, it consistently beat the old methods. For example, on the GDELT dataset, adding GRATE boosted the score by 0.16 to 0.21 points in interpolation (guessing within known time ranges) and 0.12 to 0.18 points in extrapolation (guessing into the future).
Even when compared to giant language models (LLMs) that use text to guess, GRATE held its own, sometimes beating them on specific metrics like Hits@10 on ICEWS18.

What GRATE Is NOT

It's important to know what this paper doesn't claim. The authors are very clear:

GRATE is not a magic bullet that solves every time problem. It works best when there are lots of clues to choose from. On very sparse datasets (like WIKI, where facts are isolated and rare), the "bouncer" doesn't have enough evidence to make good choices, so the improvement is smaller.
It does not explicitly model complex calendar quirks like "leap years" or irregular time gaps. It treats time as a simple difference in numbers. If the time steps are too coarse or messy, the effect weakens.
The paper does not claim this is a "solved" problem for all future AI. They suggest that combining this with faster, more scalable ways to process data is a job for future researchers.

The Bottom Line

The paper suggests that by adding this "time-spin" and "smart filter" to existing AI detectives, we can make them understand the flow of time without needing to memorize every single date in history. It's a lightweight, parameter-free upgrade that lets these models transfer their knowledge to new, unseen worlds of time and events. The authors measured this across multiple datasets and found that in most settings, especially where old patterns don't repeat, GRATE helps the model see the future a little more clearly.

Technical Summary: GRATE: Temporal Extensions for Inductive KG Foundation Models via Gated Rotary Attention

1. Problem Definition

The paper addresses the challenge of extending inductive Knowledge Graph (KG) foundation models (such as ULTRA and TRIX) to Temporal Knowledge Graphs (TKGs). While existing foundation models achieve strong inductive transfer by learning vocabulary-agnostic representations that generalize to unseen entities and relations, they lack mechanisms to handle temporal dynamics.

Current TKG models typically rely on transductive settings, learning dataset-specific embeddings for entities, relations, and timestamps. This makes them unsuitable for scenarios where new vocabulary (entities, relations, or time points) arrives continuously, such as emerging political actors or rapidly evolving event streams. Furthermore, existing inductive KG models treat facts from all timestamps as equally aligned with a query, failing to account for the fact that the relevance of a supporting fact often depends on its relative temporal displacement from the query time. The authors identify a gap: there is no method that enables inductive temporal reasoning (generalizing to disjoint vocabularies and timestamps) without sacrificing the parameter-free transferability of foundation models.

2. Methodology: GRATE

The authors propose GRATE (Gated Rotary Attention for Temporal Encoding), a parameter-free message function designed to augment NBFNet-style KG foundation models. GRATE integrates into the entity-side message passing mechanism without introducing any learnable parameters, thereby preserving the model's ability to transfer across graphs with disjoint vocabularies.

GRATE operates through two complementary mechanisms:

A. Rotary Relative-Time Encoding
Instead of using learned timestamp embeddings, GRATE encodes time through relative time differences. For a query at time $\tau$ and an observed edge $(u, r, v)$ at time $t_{uv}$ , the relative time gap is defined as $\Delta t_{uv} = \tau - t_{uv}$ .

The method adapts Rotary Position Embeddings (RoPE), originally from Transformers, to rotate the structural message $m_{uv}$ based on $\Delta t_{uv}$ .
The rotation matrix $R(\Delta t_{uv})$ applies a frequency-dependent rotation to the message vector. This transforms the structural message into a continuous function of its temporal displacement, allowing the model to distinguish between facts that are temporally close versus distant relative to the query, independent of absolute timestamp identities.

B. Query-Conditioned Gating
Rotation alone does not determine which temporally shifted facts are relevant to a specific query. GRATE introduces a gating mechanism to weight messages based on their alignment with the query.

A query state $Q_q$ is constructed by summing the representation of the query head entity and the query relation embedding.
A scalar gate $g_{uv}$ is computed using a sigmoid function: $g_{uv} = \sigma(Q_q^\top m_{uv}^{rot} / \sqrt{d})$ , where $m_{uv}^{rot}$ is the rotated message.
This gate independently evaluates each edge, emphasizing signals that are both temporally and semantically relevant to the query while suppressing irrelevant ones. Unlike softmax attention, this independent gating is stable under inductive transfer where neighborhood sizes may vary significantly.

C. Multi-TKG Pretraining
To train on multiple TKGs simultaneously, the authors remap timestamps of each source dataset onto disjoint integer ranges before joint training. Since GRATE relies only on relative gaps ( $\Delta t$ ), this remapping preserves per-source temporal relationships while preventing ID collisions across datasets.

3. Key Contributions

GRATE Mechanism: A novel, parameter-free temporal message function that enables inductive KG foundation models to perform temporal reasoning via relative-time rotation and query-conditioned gating.
Inductive Transfer Benchmarks: The construction of GDELTINDT and WIKIINDT, benchmark suites specifically designed to test cross-dataset temporal transfer. These benchmarks feature disjoint entities, relations, and timestamps between training and inference graphs, covering both interpolation and extrapolation regimes.
Zero-Shot Temporal Reasoning: Demonstration that a single jointly pretrained GRATE checkpoint can improve temporal reasoning performance on unseen datasets without fine-tuning, outperforming static base models and existing inductive baselines.

4. Experimental Results

The authors evaluate GRATE across three regimes using a single jointly pretrained checkpoint:

Transductive In-Distribution (ICEWS14, ICEWS05-15): Adding GRATE to ULTRA and TRIX significantly improves performance. Relative gains range from +25% to +38% in MRR and +38% to +58% in Hits@1, indicating that relative-time alignment primarily improves top-rank precision.
Inductive Transfer (GDELTINDT, WIKIINDT):
- On GDELT (a dense event graph), TRIX + GRATE outperforms a recurrence-only baseline by +0.16 to +0.21 MRR in interpolation and +0.12 to +0.18 MRR in extrapolation. The gating mechanism effectively discriminates among abundant neighboring facts.
- On WIKI (a sparse graph of isolated assertions), gains are more variable. When the "literal recurrence" (DRec) is high (facts repeating exactly one year prior), GRATE anchors on these matches, achieving significant gains. However, in low-recurrence settings, the static base model sometimes outperforms GRATE, though GRATE still generally beats the recurrence baseline.
Zero-Shot Transfer (ICEWS18, YAGO): GRATE outperforms LLM-based forecasting methods (ICL, GenTKG) on Hits@10 for ICEWS18 and achieves higher Hits@1 and MRR on YAGO, despite being purely structure-driven without textual priors.
Ablation Studies: Removing the rotation or the query-conditioned gate degrades performance. The ablation confirms that relative-time alignment captures the majority of the temporal signal, while the gate provides additional gains, particularly in zero-shot settings where structural cues are weaker.

5. Significance and Claims

The paper claims that temporal reasoning can be added to inductive KG foundation models without sacrificing parameter-free transferability. GRATE demonstrates that models can generalize to TKGs with completely unseen entities, relations, and timestamps by relying on relative time differences and query-conditioned relevance rather than dataset-specific embeddings.

The authors note that the impact of GRATE scales with the richness of the base model's relation graph (e.g., TRIX's iterative refinement benefits more than ULTRA's single-pass aggregation). They also acknowledge limitations: GRATE currently encodes time purely through relative index differences and does not explicitly model temporal granularity or irregular intervals, which may weaken performance on coarse-grained datasets. Future work is suggested to explore mixed-granularity pretraining and more scalable propagation strategies.