Bridging Discrete Marks and Continuous Dynamics: Dual-Path Cross-Interaction for Marked Temporal Point Processes

The paper introduces NEXTPP, a dual-channel framework that unifies discrete event marks and continuous-time dynamics through cross-interaction mechanisms to significantly improve the prediction of irregularly spaced event sequences with discrete marks.

The Gist

Imagine you are trying to predict the future of a chaotic, bustling city. You have two types of information:

1. The "What" (Discrete Marks): Specific events happen, like a car crash, a concert starting, or a tweet going viral. These are distinct, separate moments with clear labels.
2. The "When" (Continuous Dynamics): Time doesn't tick in a uniform grid; it flows like a river. The time between events matters just as much as the events themselves. Sometimes things happen in rapid-fire bursts; other times, there are long, quiet gaps.

The Problem:
For a long time, computer scientists tried to predict these events using two separate tools, and neither worked perfectly on its own:

• The "Discrete" Tool (like a standard AI): It's great at remembering the sequence of events ("First a crash, then a concert"). But it treats time like a checklist, ignoring the smooth, flowing nature of the gaps between them. It's like trying to describe a river by only looking at the rocks on the bank.
• The "Continuous" Tool (like a Neural ODE): It's amazing at modeling the smooth flow of time and how things evolve gradually. But it often forgets what actually happened. It sees the river flowing but doesn't notice the specific rocks (events) that changed its course.

The Solution: NEXTPP
The authors of this paper built a new system called NEXTPP. Think of it as a two-lane highway with a smart bridge connecting them.

The Two Lanes (Dual-Path)

1. Lane A (The Discrete Lane): This lane uses a "Self-Attention" mechanism. Imagine a super-intelligent librarian who looks at a list of past events and says, "Ah, that earthquake was big, so it probably caused these smaller aftershocks." It focuses on the meaning and type of the events.
2. Lane B (The Continuous Lane): This lane uses a "Neural ODE" (Ordinary Differential Equation). Imagine a smooth, flowing animation of time. It doesn't just jump from event to event; it simulates the invisible, continuous evolution of the system between the events. It asks, "How is the tension building up right now, even though nothing has happened yet?"

The Bridge (Cross-Interaction)

Here is the magic part. Usually, these two lanes run parallel and never talk. NEXTPP builds a bridge between them using "Cross-Attention."

• How it works: The "Continuous" lane tells the "Discrete" lane, "Hey, the tension has been building up for a long time, so the next event is likely to be huge."
• The Reverse: The "Discrete" lane tells the "Continuous" lane, "That last event was a massive earthquake, so reset the clock and expect a rapid series of smaller ones."

They constantly chat and adjust each other. The type of event changes the timing, and the timing changes the prediction of the type.

A Real-World Analogy: The Seismic Sequence

The paper uses an earthquake example to explain this.

• Old Models: Might predict the next earthquake based only on the size of the last one (Discrete), or based only on how much time has passed (Continuous).
• NEXTPP: Realizes that a massive "Mainshock" (Discrete) creates a specific, high-stress environment (Continuous) that makes "Aftershocks" happen very quickly. It understands that the story of the earthquake (the marks) and the flow of time are inseparable.

Why is this better?

The researchers tested NEXTPP on five real-world datasets:

• Taxi pickups in New York.
• Amazon product reviews.
• Earthquake records.
• Twitter retweets.
• Stack Overflow badges.

In every case, NEXTPP was more accurate at predicting when the next event would happen and what kind of event it would be. It was like having a weather forecaster who not only knows the history of storms but also understands the invisible pressure systems building up in the atmosphere.

The Bottom Line

NEXTPP is a new way of teaching computers to understand time. Instead of choosing between "what happened" and "when it happened," it forces the computer to learn how they influence each other. It bridges the gap between the discrete "ticks" of events and the continuous "flow" of time, leading to much smarter predictions for everything from traffic jams to earthquakes.

Technical Summary

1. Problem Statement

The paper addresses the challenge of modeling Marked Temporal Point Processes (MTPPs), which involve sequences of events occurring at irregular time intervals, where each event has a discrete categorical "mark" (e.g., earthquake magnitude, user action type).

Existing approaches suffer from a fundamental dichotomy:

• Discrete-time models (e.g., RNNs, Transformers): excel at capturing dependencies between event tokens (marks) but treat time as discrete steps, ignoring the continuous evolution between events.
• Continuous-time models (e.g., Neural ODEs): model the smooth dynamics of time effectively but often fail to integrate the influence of observed event marks on future timing, missing the bidirectional relationship between discrete events and continuous time.

The core problem is the lack of a unified framework that simultaneously captures discrete event dependencies and continuous temporal dynamics while modeling their mutual influence (i.e., how past marks affect future timing and how time context affects mark prediction).

2. Methodology: NEXTPP

The authors propose NEXTPP, a dual-channel framework that unifies discrete and continuous representations through Event-granular Neural Evolution with Cross-Interaction. The architecture consists of three main stages:

A. Dual-Stream Encoding

1. Discrete Stream (Self-Attention):
   • Event marks ($m_i$) and timestamps ($t_i$) are embedded into dense vectors.
   • A Self-Attention mechanism processes the sequence to capture intrinsic dependencies among event tokens, producing an intermediate representation $A_i$.
2. Continuous Stream (Neural Evolution):
   • Events are mapped to a low-dimensional latent space via an encoder.
   • A Neural Ordinary Differential Equation (Neural ODE) evolves the latent state $z(t)$ continuously between events: $\frac{dz}{dt} = f_\theta(z, t)$.
   • This captures the fine-grained, smooth temporal dynamics between discrete event occurrences. The state is updated using an ODE solver with adjoint sensitivity for memory efficiency.

B. Cross-Interaction Fusion (X-Interaction)

• A Cross-Attention module fuses the two streams.
• The continuous representation (output of the Neural ODE) serves as the Query, while the discrete representation (Self-Attention output) serves as the Key/Value.
• Mechanism: This allows explicit bidirectional information flow:
  • Historical event marks influence the continuous temporal dynamics.
  • Temporal context refines the prediction of future event marks.

C. Intensity Function and Generation

• The fused representation drives the conditional intensity function of a Neural Hawkes Process.
• The intensity function $\lambda(t, m)$ determines the probability of an event with mark $m$ occurring at time $t$.
• Future events are generated using an Iterative Thinning Sampler, which simulates the marked event sequence adhering to the learned intensity.

D. Training Objective

The model is trained using a composite loss function:

1. Negative Log-Likelihood (NLL): Standard MTPP loss to fit the intensity function.
2. KL Divergence: Regularizes the variational posterior of the latent states to match a prior distribution.
3. Continuity Loss ($L_{cont}$): Penalizes the discrepancy between the evolved state of the current event and the initial state of the next event, ensuring smooth latent trajectories across the sequence.

3. Key Contributions

1. Dual-Channel Architecture: Introduction of a framework that simultaneously models discrete token dependencies (via Self-Attention) and continuous time evolution (via Neural ODEs), bridging the gap between discrete and continuous approaches.
2. Cross-Interaction Mechanism: A novel Cross-Attention module that enables explicit bidirectional interaction, allowing event marks to adjust timing predictions and temporal context to refine mark forecasts.
3. Event-Granularity Strategy: A sequential evolution strategy that strictly preserves the global structural consistency of the Hawkes process while modeling complex dependencies at the granularity of individual events.
4. State-of-the-Art Performance: Demonstrated superior performance across five diverse real-world datasets compared to existing discrete and continuous baselines.

4. Experimental Results

The model was evaluated on five datasets: Taxi (NYC pickups), Amazon (reviews), StackOverflow (badges), Earthquake (seismic events), and Retweet (social media).

• Performance Metrics: NEXTPP achieved the best results in Root Mean Square Error (RMSE) for timestamp prediction and Error Rate for mark classification across all datasets. It also achieved the highest Log-Likelihood, indicating a better fit to the underlying data distribution.
• Comparison: It outperformed discrete models (RMTPP, THP, SAHP) and continuous models (ODETPP, IFTPP, DLTPP). Notably, it showed robustness in low-data regimes where Transformer-based models struggled.
• Ablation Studies:
  • Removing the Neural Evolution (ODE) module significantly degraded log-likelihood, proving the necessity of continuous dynamics.
  • Removing Cross-Attention increased error rates by >4%, confirming the importance of bidirectional interaction.
  • Replacing Neural ODEs with GRU/LSTM resulted in higher errors, highlighting the advantage of true continuous-time modeling over discrete approximations.
• Efficiency: NEXTPP demonstrated faster training times (less than half of ODETPP) due to its low-dimensional latent representation and efficient ODE solver usage.

5. Significance

This work is significant because it resolves the long-standing trade-off in MTPP modeling between capturing discrete event semantics and continuous temporal dynamics.

• Theoretical Impact: It provides a unified mathematical framework where discrete marks and continuous time are not treated as separate domains but as interdependent variables influencing each other.
• Practical Application: The model's ability to accurately predict both when an event will happen and what type it will be is crucial for applications like seismic forecasting (predicting aftershocks), epidemic modeling, and real-time user behavior prediction.
• Interpretability: The cross-attention weights offer insights into how specific historical events influence future timing, providing a level of interpretability often missing in black-box deep learning models.

In summary, NEXTPP represents a paradigm shift in temporal point process modeling by effectively fusing the strengths of Transformer-based sequence modeling and Neural ODEs through a novel cross-interaction mechanism.