Byte-token Enhanced Language Models for Temporal Point Processes Analysis

1. Problem Statement

Temporal Point Processes (TPPs) are widely used to model event sequences on the Web (e.g., user reviews, social media posts, transactions). However, existing approaches face two primary limitations:

• Textual Under-utilization: Traditional TPP models focus heavily on temporal dynamics and event types but struggle to effectively incorporate rich textual descriptions (multi-modal information) accompanying events.
• Temporal Gap in LLMs: While Large Language Models (LLMs) excel at processing text, they lack native mechanisms to handle the continuous temporal dynamics inherent in event sequences.
• Integration Challenges: Previous attempts to combine LLMs with TPPs often rely on TPP-specific encoding strategies (e.g., random embeddings for event types, specific temporal positional encodings) that do not seamlessly integrate with standard LLM architectures, limiting their generative capabilities for event descriptions.

The paper aims to bridge this gap by creating a unified framework that jointly models temporal dynamics and textual information, enabling both accurate prediction and high-quality generation of event descriptions.

2. Methodology: Language-TPP

The authors propose Language-TPP, a framework that unifies TPPs with LLMs (specifically based on Qwen2.5) through a novel tokenization and prompting strategy.

A. Novel Temporal Encoding (Byte-Tokens)

The core innovation is converting continuous time intervals into discrete byte-tokens to allow direct processing by standard LLM tokenizers without architectural modifications.

• Mechanism: Instead of treating timestamps as raw text strings (which is token-inefficient), the model represents a 32-bit floating-point time interval as 4 byte tokens.
• Vocabulary Expansion: The LLM vocabulary is augmented with 256 special tokens representing all unique byte values (e.g., <|byte_61|>, <|byte_155|>).
• Efficiency: This reduces the token count required to represent a timestamp significantly (e.g., from ~11 tokens for "0.075999237" down to 4 byte tokens), improving processing efficiency and precision.

B. Unified Input Representation

Events are converted into a standardized textual template that the LLM can process autoregressively. An event sequence is formatted as:
[Sequence Prompt] + [Event Template] + [Task Token]
Where the Event Template includes:

• Event Type: Textual label (e.g., "Women Accessories").
• Event Description: Natural language text (e.g., review content).
• Temporal Byte Tokens: The 4-byte representation of the time interval since the previous event.
• Special Delimiters: Tokens like <|start_of_event|> and <|end_of_event|> structure the sequence.

C. Training Protocol

The model undergoes a two-stage training process:

1. Continued Pre-training (Stage 1): The LLM backbone is trained on the tokenized event sequences using next-token prediction loss. This allows the model to learn the joint distribution of time, type, and text.
2. Next-Event Fine-tuning (Stage 2): The model is fine-tuned on prompt-response pairs. The prompt consists of a history of events, and the response is the next event (time, type, and description). This enables specific downstream tasks.

D. Inference and Tasks

The framework supports three main tasks via task-specific tokens:

• Event Time Prediction: Decoding the generated byte tokens back into a float value to predict the next time interval.
• Event Type Prediction: Generating the next event type token.
• Event Description Generation: Autoregressively generating natural language text for the next event.

3. Key Contributions

1. Unified Framework (Language-TPP): A novel architecture that seamlessly integrates TPPs and LLMs, enabling the modeling of multi-modal event sequences (time, type, text) within a standard decoder-only Transformer.
2. Byte-Token Temporal Encoding: A specialized, token-efficient method to discretize continuous time intervals into byte-level tokens, eliminating the need for custom temporal layers and allowing direct use of standard LLM tokenizers.
3. Generative Capabilities: The framework extends TPP research beyond prediction to event description generation, a previously unexplored direction in TPP literature.
4. Comprehensive Evaluation: Demonstrated state-of-the-art (SOTA) performance across multiple real-world Web datasets (Amazon, Twitter, StackOverflow, Taobao, Taxi) on both conventional TPP tasks and new generative tasks.

4. Experimental Results

The model was evaluated on five datasets (Retweet, StackOverflow, Taobao, Taxi, Amazon Review) against strong baselines including Neural Hawkes Process (NHP), Transformer Hawkes Process (THP), and LLM-based methods like TPP-LLM and LAMP.

• Temporal & Type Prediction: Language-TPP achieved SOTA performance in Root Mean Squared Error (RMSE) for time prediction and Accuracy (ACC) for type prediction across most datasets. For instance, on the Retweet dataset, it achieved an RMSE of 18.1 (vs. 21.3 for TPP-LLM) and 59.7% ACC.
• Description Generation: On the Amazon Review dataset, Language-TPP outperformed a fine-tuned Qwen2.5-0.5B (without temporal info) in generating review summaries, achieving a ROUGE-L score of 24.78 vs. 22.60.
• Sentiment Alignment: Analysis showed that incorporating temporal dynamics improved the sentiment distribution of generated text, making it more aligned with ground truth (better preservation of neutral/positive peaks) compared to standard LLMs.
• Distribution Learning: Qualitative analysis confirmed that the model accurately captures the complex, multi-modal distributions of inter-event times and event types, matching ground-truth patterns in StackOverflow data.
• Ablation Studies:
  • Removing byte-tokens (using standard string tokenization) significantly degraded performance (RMSE increased from 18.1 to 21.8 on Retweet).
  • Stage 1 pre-training was crucial for temporal understanding.
  • Surprisingly, smaller models (0.5B) often outperformed larger ones (1.5B), suggesting better parameter efficiency for the specific scale of TPP datasets.

5. Significance and Impact

• Bridging Modalities: Language-TPP successfully demonstrates that temporal dynamics and textual semantics are mutually beneficial. Temporal context improves text generation quality, while text context improves temporal prediction accuracy.
• New Research Direction: It opens a new avenue for TPP research focused on generative tasks (creating event descriptions) rather than just predictive tasks, which is vital for content generation and user behavior simulation.
• Scalability: The byte-token approach is compatible with existing LLM inference optimizations (e.g., FlashAttention, vLLM), allowing the model to handle long sequences (up to 3,000 events) efficiently.
• Applications: The framework has broad implications for Web applications, including personalized recommendation systems (timing and content), content moderation, and deep user behavior analysis.

In conclusion, Language-TPP represents a significant step forward in modeling Web event sequences by treating time as a language-like token, thereby unlocking the full generative and predictive potential of Large Language Models for temporal data.