🤖 machine learning

David vs. Goliath in Next Activity Prediction: Argmax vs. LSTM, Transformer, and LLM

This paper presents a systematic benchmark across seven real-life event logs demonstrating that a simple counting-based argmax baseline often matches or approaches the performance of complex deep learning models and billion-parameter LLMs in next activity prediction, challenging the assumption that increased model complexity and pretraining consistently yield superior results.

Original authors: Hans Weytjens, Ingo Weber

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hans Weytjens, Ingo Weber

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Do We Need a Super-Computer to Predict the Next Step?

Imagine you are watching a movie. You've seen the first half, and you want to guess what happens next.

The "Goliath" approach: You hire a team of 1,000 expert film critics who have watched every movie ever made. They use their massive knowledge of history, culture, and storytelling to guess the ending.
The "David" approach: You just look at the last scene. If the hero just picked up a sword, you guess the next scene involves fighting. If the hero just sat down for dinner, you guess the next scene involves eating.

This paper asks: Do we really need the 1,000 experts (Goliath), or is looking at the last scene (David) enough?

The Setting: Predicting Business Processes

The researchers are working in a field called Predictive Process Monitoring. Think of a business process like a recipe or a factory assembly line.

The Goal: Given a list of steps already taken (e.g., "Order received," "Payment checked," "Item packed"), predict the very next step (e.g., "Ship item").
The Contenders:
1. Argmax (David): A super-simple rule. It just counts how often a specific step follows another in the past. If "Shipping" followed "Packing" 90% of the time, it guesses "Shipping."
2. LSTMs: Older, smart computers that remember sequences well.
3. Transformers & LLMs (Goliath): The massive, billion-parameter AI models (like the ones powering chatbots) that have been "pre-trained" on the entire internet. The idea is that because they know so much about language and patterns, they should be amazing at predicting business steps.

The Experiment: The "Vocabulary" Problem

The researchers tried to use these giant AI models (LLMs) for business tasks. But business data isn't natural language. It's not "The cat sat on the mat"; it's "Event ID 405: Invoice Approved."

To make the AI understand, they had to translate the business data into the AI's language. They did this by:

Taking a giant pre-trained model (Goliath).
Swapping out its dictionary so it understands business terms instead of English words.
Teaching it the specific business rules (Fine-tuning).

They then compared these "translated" giants against:

Smaller models trained from scratch (learning only on the business data).
The simple "Argmax" counter.
Older LSTM models.

The Results: The Giant Stumbles

The paper's findings are surprising and can be summarized as "David vs. Goliath," where David wins or ties.

1. The "Pre-training" Myth
The researchers found that the "knowledge" the giant AI learned from the internet (pre-training) did not help.

Analogy: Imagine a chef who has cooked every dish in the world (the pre-trained AI). You ask them to make a very specific, simple sandwich using only three ingredients. The chef tries to use their vast knowledge of French cuisine to make the sandwich, but it turns out a local deli worker who only makes that specific sandwich (the model trained from scratch) does a better job. The chef's massive knowledge was actually a distraction.

2. Bigger is Not Better
They tested models ranging from tiny to massive (billions of parameters).

Result: Making the model 1,000 times bigger did not make it 1,000 times smarter. In fact, the smaller, simpler models often performed just as well, or better, than the giants.

3. The Simple Counter Wins (Mostly)
On most of the business datasets they tested, the simple "Argmax" counter (David) performed just as well as the billion-parameter AI (Goliath).

Why? Business processes are often repetitive. If 90% of the time Step B follows Step A, you don't need a super-computer to figure that out. You just need a calculator. The complex AI was overkill.

4. When Does the Giant Win?
The giant AI (and the models trained from scratch) only showed a clear advantage on two specific datasets (BPI12 and BPI17). These were the only cases where the business process was complex enough, or the data varied enough, to require a more sophisticated brain than a simple counter.

The Takeaway

The paper concludes that the field of business process prediction has been trying to solve problems with "sledgehammers" (massive AI models) when a "sling" (a simple counter or a small model) would do.

Don't bring a giant to a fight where a sling will do.
Before spending millions on training massive AI models, practitioners should try a simple frequency counter first.
If a more complex model is needed, it's better to train a small model from scratch on the specific data than to try to force a giant, pre-trained internet model to do the job.

In short: The "smart" AI didn't act smarter than a simple calculator in this specific job. The researchers suggest we stop over-complicating things and start using simpler, more efficient tools.

Technical Summary: David vs. Goliath in Next Activity Prediction

Problem Statement
Next Activity Prediction (NAP) is a fundamental task in Predictive Process Monitoring (PPM), aiming to predict the subsequent activity in a business process trace given its prefix. The field has evolved from classical machine learning to deep learning architectures like LSTMs, Transformers, and recently, Large Language Models (LLMs). While recent literature suggests that vocabulary-adapted, fine-tuned LLMs offer superior performance, there is a lack of systematic benchmarks that directly compare these complex models against simpler baselines and architectures trained from scratch. Specifically, it remains unclear whether the massive computational cost and complexity of pretraining and scaling LLMs yield consistent performance gains over smaller, task-specific models or simple statistical baselines in the context of direct sequence modeling for NAP.

Methodology
The authors conducted a systematic benchmark across seven real-life event logs from the BPI Challenge (BPI12, BPI15, BPI17, BPI19, BPI20_PTC, BPI20_TPD, BPI20_RFP). The study employed a "direct sequence modeling" approach, where event attributes are represented as learned embeddings without converting data into natural language text.

The experimental setup compared four categories of models:

Pretrained LLMs: Vocabulary-adapted versions of GPT-2 (Small, Medium, Large) and Qwen-3 (0.6B, 1.7B, 4B) models. These involved replacing input/output layers to match the dataset vocabulary while keeping the backbone fixed or minimally adapted via Parameter-Efficient Fine-Tuning (PEFT).
Transformers Trained from Scratch: Custom "nanoGPT" models of varying sizes (ranging from ~0.1M to ~85M parameters) trained directly on the NAP datasets.
Distilled Models: Smaller "student" models trained via logits-only knowledge distillation from the fine-tuned pretrained "teacher" models.
Baselines:
- LSTMs: Four LSTM models of varying sizes.
- Argmax Baseline: A simple first-order empirical transition model predicting the next activity based solely on the relative transition frequencies of the current activity ( $a_{k+1} = \arg\max_a P(a | a_k)$ ).

The evaluation utilized two metrics: Accuracy and Macro-F1, computed at the prefix level. The study controlled for hyperparameters, data leakage, and preprocessing (including lifecycle augmentation and handling of unique prefixes).

Key Contributions

Systematic Benchmarking: The paper provides the first joint comparison of vocabulary-adapted LLMs, scratch-trained Transformers, distilled models, LSTMs, and a counting-based argmax baseline across multiple datasets.
Distillation Methodology: It introduces a method to compress fine-tuned LLMs into smaller architectures for which no pretrained versions exist, allowing for a controlled study of knowledge transfer.
Baseline Inclusion: By reintroducing a simple count-based argmax baseline (absent in many recent DL-focused NAP papers), the authors provide a critical reality check on reported performance gains.
Research Questions: The study explicitly addresses five questions regarding the value of pretraining, distillation, model scaling, architectural modernity, and the necessity of complex models over baselines.

Results
The experimental results present a "David vs. Goliath" narrative, challenging the assumption that larger, pretrained models are inherently superior:

Pretraining (RQ1): There is no consistent added value from generic LLM pretraining. Fine-tuned GPT-2 and Qwen-3 models did not systematically outperform their counterparts trained from scratch (nanoGPT) of similar size. In some cases, scratch-trained models performed better.
Distillation (RQ2): Logits-only distillation from fine-tuned teachers did not improve the performance of smaller student architectures compared to training them directly from scratch. In fact, distilled models often underperformed.
Model Capacity and Modernity (RQ3 & RQ4): Increasing model size (from millions to billions of parameters) yielded no consistent performance benefit. Similarly, newer architectures (Qwen-3) did not consistently outperform older ones (GPT-2).
Complexity vs. Baseline (RQ5):
- LSTMs vs. LLMs: LSTMs performed comparably to LLMs, particularly on Macro-F1, where LLMs only showed significant gains on the BPI12 dataset.
- LLMs vs. Argmax: On most datasets (BPI15, BPI19, BPI20 variants), the simple argmax baseline matched or approached the performance of billion-parameter LLMs. Significant improvements over the argmax baseline were observed only on BPI12 and BPI17.
- Optimal Strategy: For datasets with high prefix variance, few training samples, or high class imbalance, the argmax baseline was often the most effective option. However, comparatively small Transformer models trained from scratch frequently achieved the best overall predictive performance, outperforming both the baseline and the fine-tuned LLMs.

Significance and Claims
The paper claims a "sobering reality check" for the PPM community. The authors argue that the field's tendency to benchmark advanced models against other deep learning models, rather than simple baselines, has overstated the practical relevance of recent progress.

Pretraining is not a silver bullet: Generic language priors learned during pretraining do not effectively transfer to NAP via vocabulary-adapted direct sequence modeling. The "mismatched priors" of large models may even hinder performance on specific process distributions.
Simplicity is competitive: A simple frequency-based model is a highly competitive option for many process logs.
Training from scratch is preferable: When complex models are necessary, training small Transformer models from scratch is more beneficial than increasing model size or relying on pretrained/distilled architectures.
Future Direction: The findings suggest that future research should move beyond architectural scaling and focus on richer process-aware representations, transfer learning, and multi-task objectives, using the established benchmarks and simple baselines as a clear starting point for measuring genuine progress.

The authors conclude that practitioners should "not bring a giant to a fight where a sling will do," emphasizing that the complexity of billion-parameter models is often unjustified for standard NAP tasks.