DyMRL: Dynamic Multispace Representation Learning for Multimodal Event Forecasting in Knowledge Graph

The paper proposes DyMRL, a novel framework that integrates time-sensitive structural features across Euclidean, hyperbolic, and complex spaces with dual fusion-evolution attention mechanisms to effectively address the challenges of dynamic multimodal knowledge acquisition and fusion for accurate event forecasting.

The Gist

Imagine you are trying to predict the future of a complex story, like a political drama or a celebrity's life. You have a massive library of information about this story, but it's not just text; it's a mix of facts (who did what to whom), photos (what they looked like at the time), and news articles (what people were saying).

The problem is that this story is constantly changing. A photo from 1990 tells a different story than a photo from 2025. A news headline from yesterday might be irrelevant today.

Most computer programs trying to predict the future are like static librarians. They take a snapshot of the library, glue all the books together, and try to guess what happens next based on that single, frozen picture. They miss the fact that the story is a living, breathing movie, not a still photo.

This paper introduces DyMRL (Dynamic Multispace Representation Learning). Think of DyMRL as a super-intelligent, time-traveling detective who doesn't just read the books; they understand how the story evolves over time.

Here is how DyMRL works, broken down into three simple superpowers:

1. The "Three-Dimensional Brain" (Multispace Learning)

Human brains don't just think in straight lines. We think in chains of association, we see hierarchies (like a family tree), and we understand complex logic (like "if A happens, B might happen, but only if C is true").

Existing computer models usually try to fit all this information into a single, flat grid (like a spreadsheet). It's like trying to fit a 3D sculpture into a 2D drawing; you lose a lot of detail.

DyMRL is different. It uses three different "mental rooms" to process information simultaneously:

• The Chain Room (Euclidean Space): Good for linking things in a straight line (e.g., "Trump was born in New York, then went to school, then started a business").
• The Pyramid Room (Hyperbolic Space): Good for understanding hierarchies and big groups (e.g., "Trump is a President, who is a type of Leader, who is a type of Human").
• The Logic Room (Complex Space): Good for understanding tricky relationships like opposites, mirrors, or combinations (e.g., "If A is the father of B, then B is the son of A").

By using all three rooms at once, DyMRL builds a much deeper, richer understanding of the story than models that only use one room.

2. The "Time-Sensitive Camera" (Dynamic Acquisition)

Imagine you are watching a movie. If you only look at the first frame, you don't know the plot. If you only look at the last frame, you don't know the backstory.

DyMRL acts like a camera that records the entire movie, not just a still.

• For Facts: It watches how the relationships between people change over time.
• For Photos & Text: It uses pre-trained AI (like a super-smart photographer and a super-smart journalist) to look at the images and articles specifically for that moment in time. It knows that a photo of Trump in 1983 looks different and means something different than a photo of him in 2025.

It updates its memory constantly, ensuring it never confuses "then" with "now."

3. The "Smart Spotlight" (Dual Fusion-Evolution Attention)

This is the most human-like part. When you try to predict what will happen next, you don't treat every piece of evidence equally.

• Sometimes, a photo is the most important clue.
• Sometimes, a text article is the key.
• Sometimes, the relationship between two people is what matters most.
• And sometimes, what happened yesterday is more important than what happened ten years ago.

Old models use a "static spotlight" that shines the same amount of light on everything, everywhere.

DyMRL uses a dynamic spotlight. It has a "Dual Fusion-Evolution" mechanism:

• Fusion: It decides which type of information (photo, text, or fact) is most important right now.
• Evolution: It decides which time period is most important. It realizes that recent events usually have a bigger impact on the immediate future than ancient history.

It's like a detective who knows: "For this specific prediction, I need to focus heavily on the text from last week, but ignore the photos from 2010."

The Result

The researchers tested DyMRL on four huge datasets involving real-world events (like global news and political crises). They compared it against the best existing methods.

The verdict? DyMRL crushed the competition. It predicted future events much more accurately because it didn't just memorize the data; it understood the geometry of the relationships, the evolution of the timeline, and the changing importance of different clues.

In short: If other models are like students trying to guess the ending of a movie by reading a single page of the script, DyMRL is like a director who has watched the whole movie, understands the characters' histories, and knows exactly how the plot twists will unfold.

Technical Summary

1. Problem Definition

The paper addresses the challenge of Multimodal Event Forecasting within Dynamic Temporal Knowledge Graphs (TKGs).

• Context: Real-world scenarios involve events that evolve over time, containing not only structural data (subject, relation, object, timestamp) but also rich auxiliary modalities (images and text).
• Limitations of Existing Work:
  • Static Focus: Most existing methods treat knowledge graphs as static, failing to capture the dynamic evolution of multimodal information.
  • Shallow Geometric Learning: Dynamic learning methods are often limited to single geometric spaces (e.g., only Euclidean) or shallow pairwise translations, failing to capture deep, relation-aware geometric features (hierarchical, chain-like, or logical structures).
  • Ineffective Fusion: Static co-attention mechanisms cannot adaptively assign weights to different modalities at different timestamps, ignoring the varying historical contributions of visual, linguistic, and structural data to future events.
• Goal: To develop a model that can effectively acquire deep structural representations from multiple geometric spaces and fuse evolving multimodal features to predict future events (e.g., predicting the object or subject of a future event).

2. Methodology: The DyMRL Framework

The proposed DyMRL (Dynamic Multispace Representation Learning) model consists of three core modules designed to mimic human cognitive processes (associative thinking, high-order abstracting, and logical reasoning).

A. Dynamic Structural Modality Acquisition

This module learns deep structural representations by integrating information from three distinct geometric spaces, each corresponding to a specific cognitive capability:

1. Euclidean Space (Associative Thinking): Captures local, chain-like neighborhood interactions using standard vector addition ($s + r$).
2. Hyperbolic Space (High-Order Abstracting): Captures global, hierarchical patterns and high-order inter-neighborhood relationships. It utilizes the superlinear property of negative curvature and learnable curvatures to distinguish event groups across different manifolds.
3. Complex Space (Logical Reasoning): Captures relational logics inherent in KGs (symmetry, asymmetry, inversion, and composition) using the complex Hadamard product.

• Deep Propagation: These shallow geometric messages are integrated via an additive attention mechanism and propagated through Multilayer Graph Neural Networks (GNNs) to form deep structural representations.
• Temporal Update: A Recurrent Neural Network (RNN) update module processes these structural embeddings across a sequence of $k$ historical time windows to capture chronological shifts.

B. Dynamic Auxiliary Modality Acquisition

This module encodes time-sensitive visual and linguistic data:

• Visual Modality: Uses pretrained VGG models to extract image features for each entity at specific timestamps, followed by mean pooling and dimensionality projection.
• Linguistic Modality: Uses pretrained BERT models to extract time-sensitive text descriptions.
• Temporal Evolution: Similar to the structural module, RNN-based update modules model the chronological evolution of these auxiliary features.

C. Dual Fusion-Evolution Attention

To effectively fuse the acquired modalities and capture temporal dependencies, DyMRL introduces a novel Dual Fusion-Evolution Attention mechanism:

1. Fusion Attention (Intra-timestamp): Operates at each specific timestamp to fuse structural, visual, and linguistic features. It uses a third-party initialized matrix ($E_{init}$) as an "attention assigner" (Query) and the modality embeddings as "learners" (Key/Value). This allows the model to assign dynamic weights to different modalities at a specific time.
2. Evolution Attention (Inter-timestamp): Operates across the sequence of historical timestamps to capture how the importance of fused features changes over time. It further refines the temporal dependencies between historical data and future events.

D. Decoding and Training

• Curvature-Adaptive Decoder: The final unified embeddings are decoded using a curvature-adaptive hyperbolic distance function to generate forecasting scores.
• Loss Function: The model is trained using a multilabel learning framework with cross-entropy loss to predict missing entities in future time steps.

3. Key Contributions

1. First Dynamic Multimodal Approach: DyMRL is the first framework designed specifically for dynamic multimodal event forecasting in Knowledge Graphs, bridging the gap between static multimodal methods and dynamic unimodal methods.
2. Multispace Deep Representation: It proposes a novel mechanism to integrate Euclidean, Hyperbolic, and Complex spaces into a deep message-passing framework. This aligns with human cognitive paradigms (associative, abstract, and logical reasoning) to capture deep structural topologies.
3. Dual Fusion-Evolution Attention: It introduces a symmetric attention mechanism that dynamically assigns adaptive weights to different modalities at different timestamps, effectively modeling temporal dependencies rather than static inter-modal correlations.
4. New Benchmarks: The authors constructed four new multimodal temporal KG datasets (GDELT-IMG-TXT, ICE14-IMG-TXT, ICE0515-IMG-TXT, ICE18-IMG-TXT) by enriching existing temporal KGs with time-sensitive images and text, providing a standard for future research.

4. Experimental Results

The authors evaluated DyMRL against state-of-the-art (SOTA) baselines, including static multimodal methods (e.g., TransAE, MoSE, IMF) and dynamic unimodal methods (e.g., xERTE, RE-GCN, ReTIN).

• Performance: DyMRL significantly outperformed all baselines across all four datasets.
  • On the GDELT-IMG-TXT dataset, it achieved an MRR of 79.34%, surpassing the second-best dynamic unimodal baseline (ReTIN at 67.56%) by a large margin.
  • Improvements ranged from 5.35% to 29.5% in MRR and Hits@1 metrics compared to the best baselines.
• Ablation Studies:
  • Removing any of the three geometric spaces (Euclidean, Hyperbolic, Complex) caused performance drops, confirming the necessity of multispace integration.
  • Removing the Multilayer message propagation resulted in a drastic drop, proving that deep structural learning is superior to shallow pairwise translations.
  • Removing the Attention Assigner (degrading to standard co-attention) significantly hurt performance, validating the need for dynamic weight assignment.
• Analysis:
  • Modalities: Structural modality was found to be the most impactful, followed by linguistic, then visual.
  • Time Sensitivity: Short-term historical timestamps closer to the prediction time provided higher forecasting value.
  • Window Size: Performance was sensitive to the historical window length ($k$), with optimal results at $k=3$ and $k=5$ depending on the dataset.

5. Significance

• Theoretical Advancement: The paper advances the field of Knowledge Graph representation learning by successfully unifying multimodal data, temporal dynamics, and multi-geometry spaces (Euclidean, Hyperbolic, Complex) into a single coherent framework.
• Cognitive Alignment: By explicitly modeling human cognitive traits (associative thinking, abstraction, reasoning) through specific geometric spaces, the model offers a more biologically plausible approach to event forecasting.
• Practical Impact: The proposed method and the newly constructed datasets provide a robust foundation for real-world applications requiring dynamic reasoning, such as crisis early warning systems, political event prediction, and dynamic recommendation systems.
• Resource Availability: The authors have open-sourced the code and the four new datasets, fostering reproducibility and further research in dynamic multimodal knowledge graphs.