Technical Summary: Revisiting Predictive Process Monitoring in the Age of Foundation Models

Problem Statement

Predictive Process Monitoring (PPM) leverages historical event logs to forecast the future of running process instances, addressing tasks such as predicting the next activity (NA), the remaining time until case completion (RT), and the time until the next event (NT). While recent PPM research has been dominated by deep sequence models (e.g., LSTMs and Transformers) trained from scratch, the emergence of foundation models presents a new landscape. Specifically, Large Language Models (LLMs) adapted via parameter-efficient fine-tuning (PEFT) and tabular foundation models with in-context learning capabilities offer alternative paradigms. However, it remains unclear whether classical sequence models remain competitive against these foundation-based approaches, particularly given that tabular foundation models have not yet been systematically benchmarked for PPM.

Methodology

The authors conduct a controlled empirical benchmark comparing three distinct modeling paradigms across five real-world event logs (BPI12, BPI17, and three BPI20 variants) and three prediction tasks (NA, RT, NT).

Modeling Paradigms

1. Classical Sequence Models: These models are trained from scratch on event logs. They encode prefixes as ordered sequences of event embeddings (combining categorical and numerical attributes) using recurrent (LSTM) or attention-based (Transformer) backbones. All parameters are learned via supervised optimization.
2. LLM-Based Approaches: These approaches utilize pre-trained decoder-only LLMs (Llama-3.2-1B and Gemma-2-2b) adapted to process data. They preserve the sequential prefix representation but employ a structured event-log encoding where activity labels are treated as process-native tokens. Adaptation is achieved via Low-Rank Adaptation (LoRA), optimizing only a small set of parameters while keeping the pre-trained backbone frozen.
3. Tabular Foundation Models: These models (TabPFN-3 and ConTextTab) treat PPM as a tabular prediction problem. They map prefixes to fixed-length feature vectors (summarizing properties like current activity, elapsed time, and temporal indicators) rather than preserving the explicit event sequence. They utilize in-context learning, requiring no gradient-based training at inference time, relying instead on contextual conditioning on observed rows.

Experimental Setup

• Data: Five public event logs with varying sizes, activity alphabets, and trace lengths.
• Splitting: A leakage-avoiding temporal split (80/20 case-level) is applied, ordering cases by their first-event timestamp.
• Features: Inputs include activity labels and numerical features (accumulated time, calendar attributes).
• Evaluation Metrics: Accuracy for NA; Mean Squared Error (MSE) for RT and NT; wall-clock runtime for computational efficiency.
• Baselines: The study compares the three paradigms against each other and against a majority baseline.

Key Contributions

1. Conceptual Comparison: The paper provides a structured comparison of the three paradigms along dimensions of prefix representation, backbone parameterization, and adaptation regimes.
2. Systematic Benchmark: It presents the first controlled empirical evaluation of tabular foundation models for PPM, alongside a comparison of fine-tuned LLMs and classical sequence models.
3. Failure Pattern Analysis: The study investigates performance differences, specifically analyzing how "branching" (the ambiguity of the next step) affects model performance and identifying specific failure modes, such as premature end-of-sequence predictions in LLMs.

Results

The benchmark yields distinct performance patterns across tasks and datasets:

• Next Activity (NA) Prediction: Classical sequence models (LSTM and Transformer) consistently achieve the highest accuracy across all five datasets. Tabular foundation models generally underperform, with the gap widening as branching complexity increases. LLMs perform better than tabular models on NA but generally lag behind the best sequence models.
• Temporal Tasks (RT and NT): The results are more mixed. Tabular foundation models frequently achieve the best performance on Remaining Time (RT) prediction. For Next Event Time (NT), both sequence and tabular models secure the best results on different logs. LLM-based models generally lag behind on temporal tasks despite their higher computational cost.
• Runtime: Sequence models and ConTextTab exhibit substantially lower runtimes compared to LoRA-fine-tuned LLMs. The runtime for LLMs increases markedly with log size. Notably, the absence of gradient-based training in tabular models does not automatically guarantee lower runtime; TabPFN-3 is often slower than sequence baselines.
• Branching Sensitivity: A slice-level analysis reveals that the performance gap between sequence and tabular models for NA prediction increases with branching complexity. Sequence models retain event order, allowing them to handle high-branching scenarios better, whereas tabular models, which encode prefixes as fixed-length state vectors, struggle with high ambiguity.
• LLM Specific Issues: LLMs exhibit specific failure patterns, such as predicting the end-of-sequence token prematurely. In the BPI12 dataset, this accounted for 23–25% of NA errors for LLMs, compared to 0% for sequence baselines.

Significance and Claims

The paper claims that while foundation models offer promising alternatives, they do not universally outperform classical sequence models in PPM. The study concludes that:

• Sequence models remain the state-of-the-art for next activity prediction, particularly in complex, high-branching processes.
• Tabular foundation models are competitive for temporal prediction tasks (RT/NT), offering a viable alternative that does not require training, though they struggle with NA in complex logs.
• LLMs incur high costs (runtime and computational resources) without consistently delivering superior accuracy, often suffering from dataset-dependent adaptation issues and specific failure modes like premature termination.

The authors emphasize that the choice of paradigm depends heavily on the specific prediction task and the structural complexity (branching) of the process log. They note that their evaluation used relatively small LLMs and did not explore alternative feature representations for tabular models, suggesting these as areas for future investigation.