📊 statistics

Advances in Temporal Point Processes: Bayesian, Neural, and LLM Approaches

This survey provides a comprehensive review of temporal point processes by examining their fundamental concepts, model designs, and estimation techniques across Bayesian, deep learning, and large language model frameworks, while also highlighting classic applications and outlining future research challenges.

Original authors: Feng Zhou, Quyu Kong, Jie Qiao, Cheng Wan, Yixuan Zhang, Ruichu Cai

Published 2026-06-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Feng Zhou, Quyu Kong, Jie Qiao, Cheng Wan, Yixuan Zhang, Ruichu Cai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are watching a busy city street. Cars honk, pedestrians cross, and buses arrive. These aren't happening on a strict schedule like a train timetable; they are random, messy, and often triggered by what just happened (a bus arriving might cause a crowd to gather, which causes more honking).

This paper is a massive "map" of how scientists try to understand and predict these kinds of random, time-based events. It calls these events Temporal Point Processes (TPPs).

Here is a simple breakdown of the three main "schools of thought" the paper reviews, using everyday analogies:

1. The Old School: Bayesian Methods (The "Gardeners")

Think of traditional statistical models as gardeners who know exactly how a specific plant grows. They have a fixed formula: "If it rains, the plant grows 2 inches."

The Problem: Real life isn't that simple. Sometimes the plant grows 5 inches, sometimes 0.5, and sometimes it grows sideways.
The Bayesian Solution: Instead of guessing one exact formula, Bayesian methods act like a gardener who keeps a notebook of possibilities. They don't just say "It will rain"; they say, "There's a 70% chance of rain, a 20% chance of drizzle, and a 10% chance of a storm."
What the paper says: This approach is great for knowing how sure you are about your prediction (uncertainty). The paper highlights that while these methods are flexible, they can be mathematically heavy and slow to calculate.

2. The New School: Neural Networks (The "Super-Recognizers")

If the gardeners are the old school, Neural Networks are like a super-smart detective who has watched a million movies and can spot patterns humans miss.

How they work: Instead of a fixed formula, these models use deep learning (like the tech behind voice assistants) to look at the history of events and guess what happens next.
The Three Types of Detectives:
- Recurrent (The Note-Taker): Reads the story one word at a time. It's fast at predicting the next word but gets tired and forgets the beginning of a very long story.
- Autoregressive/Transformer (The Big Picture Viewer): Reads the whole story at once. It's amazing at seeing long-term connections (like how a plot twist in chapter 1 affects chapter 10) but takes a lot of energy and time to process.
- Differential Equation (The Smooth Flow): Instead of checking the story at specific moments, this model imagines the story flowing like a river. It can describe exactly what happens between the events, not just at the events.
The Paper's Insight: Neural networks are very powerful and flexible, but they can be "black boxes" (hard to explain why they made a guess) and sometimes struggle to predict far into the future without making mistakes that pile up.

3. The Future School: LLMs (The "Storytellers")

This is the newest trend. Large Language Models (LLMs) are the AI chatbots you might use to write emails or summarize articles.

The Twist: Usually, LLMs just read text. But this paper looks at how we can teach them to read time and events as if they were words in a sentence.
How it works: Imagine turning a timestamp (like "3:00 PM") into a word that the AI understands. Now, the AI can look at a sequence of events (like "User clicked," "User bought," "User left") and not just predict the next click, but also explain why it happened in plain English.
The Paper's Insight: This is a game-changer because it lets us mix time data with text, images, and other messy real-world info. It moves beyond just "predicting the next number" to "understanding the story of the events."

Why Does This Matter? (The Applications)

The paper lists where these tools are actually used:

Social Media: Predicting when a tweet will go viral.
Finance: Guessing when a stock order will be placed.
Earthquakes: Figuring out where the next aftershock might hit.
Healthcare: Tracking when a patient might have a seizure or when a disease might spread.
Causal Discovery: Instead of just guessing the future, these models try to figure out cause and effect. (e.g., "Did the earthquake cause the power outage, or did the power outage cause the earthquake?")

The Big Challenges (The "But...")

Even with all this progress, the paper points out three big hurdles:

The "Black Box" Problem: We can get great predictions from Neural Networks, but we often can't explain why they made that prediction. In fields like medicine or law, knowing the "why" is just as important as the "what."
The Speed Problem: Some of these models are so complex that they take forever to train or to run on real-time data.
The "Messy Data" Problem: Real-world data is messy. Sometimes events happen every second; sometimes every year. Sometimes the data is missing. Making one model that handles all these different "flavors" of data is still very hard.

In a nutshell: This paper is a guidebook showing how we are moving from simple, rigid formulas to smart, flexible AI that can understand the chaotic, continuous flow of time-based events in our world. It celebrates the new tools (especially LLMs) while reminding us that we still have work to do to make them faster, clearer, and more reliable.

Technical Summary: Advances in Temporal Point Processes: Bayesian, Neural, and LLM Approaches

Problem Definition

Temporal Point Processes (TPPs) are stochastic models designed to characterize sequences of events occurring in continuous time. Unlike synchronous time series, event sequences (e.g., neural spikes, financial orders, social media posts) are asynchronous, often exhibit complex self-exciting or inhibiting dynamics, and may depend on the entire history of past events. The core challenge in TPP research lies in modeling the conditional intensity function $\lambda^*(t)$ , which defines the probability of an event occurring at time $t$ given the history $H_{t^-}$ .

While traditional statistical TPPs (e.g., Poisson, Hawkes processes) offer interpretability, they often rely on rigid parametric assumptions that limit their expressiveness on complex datasets. Conversely, recent advancements in deep learning and large language models (LLMs) offer greater flexibility but introduce challenges regarding uncertainty quantification, computational efficiency in continuous time, and the integration of semantic context. This survey addresses the need for a comprehensive review of these evolving paradigms, specifically focusing on the period from 2020 to 2025.

Methodology and Taxonomy

The paper organizes recent advances in TPPs into three primary methodological frameworks: Bayesian, Neural, and LLM-based approaches.

1. Bayesian Nonparametric TPPs

This framework moves beyond fixed parametric forms by treating the intensity function itself as a random variable with infinite dimensions, typically modeled using Gaussian Processes (GPs).

Poisson Processes: The intensity $\lambda(t)$ is modeled as $l \circ g(t)$ , where $g(t)$ follows a GP prior and $l(\cdot)$ is a link function (e.g., exponential, sigmoid). Inference is challenging due to the "doubly intractable" nature of the posterior (integrals over time and function space). Solutions include Markov Chain Monte Carlo (MCMC) with latent thinned points, Laplace approximations (using Kronecker or spectral methods), and Variational Inference (VI).
Hawkes Processes: The conditional intensity consists of a baseline $\mu(t)$ and a triggering function $\phi(\tau)$ . Bayesian nonparametric approaches place GPs on both components. A key technique involves branching latent variables (indicating whether an event is a background event or triggered by a previous one) to decouple the likelihood into independent Poisson processes, enabling the use of standard GP inference methods (e.g., Gibbs sampling, EM algorithms, VI). Recent works also utilize Pólya-Gamma data augmentation to achieve conditional conjugacy.

2. Neural TPPs

Neural TPPs leverage deep learning architectures to model the history embedding and the conditional distribution of the next event. The survey categorizes these into three architectural families:

Recurrent Neural TPPs: Utilize RNNs, LSTMs, or GRUs to update a hidden state $h_n$ sequentially upon each event. While computationally efficient for prediction ( $O(1)$ ), they suffer from sequential training bottlenecks ( $O(N)$ ) and difficulties in capturing long-range dependencies. Recent integrations with State Space Models (e.g., S4, Mamba) aim to address these limitations.
Autoregressive Neural TPPs: Employ Transformer architectures with self-attention mechanisms to capture long-range dependencies and enable parallel training. These models encode event times and marks into embeddings and use attention scores to generate history representations. However, they typically incur quadratic training complexity ( $O(N^2)$ ) and linear prediction complexity ( $O(N)$ ) unless optimized with KV caching.
Differential Equation-based Neural TPPs: These models (e.g., Neural ODEs/SDEs) define the hidden state evolution continuously between events. The state evolves smoothly via a neural ODE and jumps at event times via a neural SDE. This allows for flexible modeling of the intensity function over continuous intervals, overcoming the discrete-time limitations of RNNs and Transformers, though at the cost of expensive numerical integration during training.
Diffusion-based TPPs: A newer paradigm that treats event sequence generation as a denoising process. Unlike autoregressive models, diffusion models generate the entire sequence (or multiple future events) simultaneously, potentially mitigating error accumulation in long-horizon forecasting, though they face challenges in enforcing temporal consistency and computational cost.

Parameterization Strategies: The paper highlights that neural TPPs can parameterize the next event distribution via:

Conditional Intensity Function ( $\lambda^*$ ): Requires numerical integration for Maximum Likelihood Estimation (MLE).
Conditional Density Function ( $f$ ): "Intensity-free" modeling; avoids integration, enabling efficient sampling.
Cumulative Intensity ( $\Lambda^*$ ) or Distribution ( $F$ ): Reformulates the log-likelihood to use derivatives instead of integrals, improving training efficiency via automatic differentiation.

3. LLM-based TPPs

This emerging field integrates Large Language Models to handle the semantic and multimodal aspects of event sequences.

LLM-inspired TPPs: Use LLM mechanisms (e.g., prompt learning, abductive reasoning) to enhance conventional neural TPPs. Examples include PromptTPP for continual learning and LAMP for causal reasoning.
Direct LLM-TPP Integration: Treats the LLM as the core sequence model. Events are tokenized (time, type, text) and fed directly into the LLM. TPP-LLM uses textual descriptions for events, while Language-TPP encodes continuous time intervals as byte-level tokens, allowing the LLM to model time and semantics jointly.
Extensions: Recent work extends TPPs to multimodal benchmarks (e.g., DanmakuTPPBench) involving text, video, and timestamps, and to retrieval-augmented generation tasks.

Key Contributions

Comprehensive Taxonomy: The paper provides a structured review of TPPs across Bayesian, Neural, and LLM paradigms, updating the field beyond 2020. It specifically highlights the under-reviewed area of Bayesian nonparametric TPPs and the nascent field of LLM-based TPPs.
Methodological Synthesis: It details the trade-offs between different modeling approaches (e.g., expressiveness vs. computational cost, discrete vs. continuous time) and parameterization strategies (intensity-based vs. density-based).
Training and Evaluation Analysis: The survey compares training objectives (MLE, Wasserstein distance, Noise Contrastive Estimation, Fisher Divergence) and discusses the lack of standardized benchmarks, highlighting recent efforts like EasyTPP and DanmakuTPPBench.
Application Review: It revisits classic applications (event prediction, causal discovery) across domains like finance, epidemiology, and neuroscience, noting how modern methods improve upon traditional baselines.

Results and Empirical Insights

The paper synthesizes empirical findings from existing literature rather than presenting new experimental results:

Performance: Neural TPPs, particularly Transformer-based models, generally outperform classical parametric models (e.g., Hawkes) on large-scale, complex datasets for next-event prediction.
Efficiency: Models parameterizing the conditional density or cumulative intensity function often achieve better training efficiency and comparable accuracy to intensity-based models by avoiding numerical integration.
Limitations: Long-horizon prediction remains a challenge for all classes; autoregressive methods suffer from error accumulation, while non-autoregressive (diffusion) methods struggle with temporal consistency.
LLM Impact: LLM-based approaches show promise in tasks requiring semantic understanding or multimodal reasoning but do not necessarily dominate purely temporal prediction benchmarks.

Significance and Future Directions

The paper claims significance as a timely resource that bridges the gap between statistical rigor (Bayesian methods), deep learning flexibility (Neural methods), and semantic understanding (LLMs). It emphasizes that the field is shifting from purely modeling event occurrence probabilities to understanding temporally indexed event data with rich context.

Identified Challenges:

Data and Benchmarks: The lack of standardized, curated datasets and evaluation protocols hinders fair comparison.
Interpretability: Neural TPPs often act as black boxes, making causal discovery and decision support difficult.
Scalability: Continuous-time modeling combined with long sequences creates computational bottlenecks (e.g., $O(N^2)$ for attention, numerical integration costs).
Sampling Efficiency: Generating valid event sequences remains computationally expensive compared to likelihood evaluation.
Multimodal Integration: Effectively aligning continuous time representations with discrete, high-dimensional modalities (text, images) remains an open problem.

The authors conclude that future progress depends on integrating statistical rigor with neural flexibility and language capabilities, addressing scalability and interpretability to enable broader application in real-world, multimodal settings.