🤖 AI

Learning optimal policies from event logs through reinforcement learning: a comparison of deep and MDP-based approaches

This paper proposes a Reinforcement Learning framework for Prescriptive Process Monitoring that learns optimal behavioral policies directly from historical event logs to recommend actions for optimizing Key Performance Indicators, comparing a model-based Markov Decision Process approach with a model-free Deep RL method and demonstrating that both effectively improve KPIs while the model-based approach offers superior computational efficiency.

Original authors: Stefano Branchi, Andrei Buliga, Chiara Di Francescomarino, Chiara Ghidini, Riccardo Graziosi, Francesca Meneghello, Massimiliano Ronzani

Published 2026-06-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Stefano Branchi, Andrei Buliga, Chiara Di Francescomarino, Chiara Ghidini, Riccardo Graziosi, Francesca Meneghello, Massimiliano Ronzani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the manager of a busy bank. Every day, customers apply for loans. Sometimes the loan gets approved, sometimes it gets rejected, and sometimes the customer just walks away. Your goal is simple: get as many customers to say "Yes" to your loan offer as possible.

However, you don't control everything. You control the bank's actions (like sending an offer or asking for more info), but you don't control the customer's actions (like deciding to accept, reject, or cancel). It's like playing a game of chess where you can only move your own pieces, but your opponent (the customer) moves theirs randomly based on their mood.

This paper is about teaching a computer to be the best possible bank manager by looking at old records of how the bank played the game in the past. The goal is to figure out the perfect strategy to win more often.

The Two "Coaches"

The researchers tested two different ways (or "coaches") to teach the computer this strategy using Reinforcement Learning (a type of AI that learns by trial and error, but in this case, it learns from a history book instead of playing live).

1. The "Map Maker" (MDP-based Approach)

This coach looks at the thousands of past loan applications and tries to build a detailed map of the game.

How it works: It groups similar situations together (like "Customer asked for $10k and we sent an offer") and calculates the odds of what happens next. It creates a clear rulebook: "If you are in Situation A, do Action B."
The Analogy: Imagine a GPS that has mapped every single road in a city. It knows exactly which turn leads to traffic and which leads to a shortcut. It builds a complete model of the city before telling you where to go.
The Catch: Building this map takes some work, and if the map is too detailed, it might get confused by rare, weird events. To fix this, the researchers added a "filter" to ignore rare, unreliable paths in the history book.

2. The "Deep Learner" (Offline Deep RL)

This coach doesn't try to build a map. Instead, it uses a super-smart neural network (like a brain with many layers) to look at the history book and guess the best move directly.

How it works: It reads the past records and tries to find hidden patterns that a human or a simple map might miss. It learns by looking at the data over and over again.
The Analogy: Imagine a grandmaster chess player who has watched millions of games. They don't need to draw a map of the board; they just "feel" the right move based on intuition built from experience.
The Catch: This "brain" is very heavy. It takes a long time to train and requires a lot of computing power, like trying to solve a giant puzzle with a supercomputer.

The Experiment: The Simulation Arena

Since you can't just start giving bad advice to real customers to see what happens (that would cost the bank money), the researchers built a virtual simulation.

They created a "video game" version of the loan process.
They let the computer play the game thousands of times using the strategies it learned.
They measured the score: How much profit did the bank make? (Profit = Interest earned minus the time spent working on the loan).

What They Found

The results were interesting:

Both Coaches Won: Both the "Map Maker" and the "Deep Learner" figured out strategies that were much better than the bank's old, standard way of doing things. They both helped the bank get more "Yes" answers from customers.
The Map Maker was Slightly Better: The "Map Maker" (MDP) consistently found slightly better strategies, especially in the tricky, rare situations. It was more reliable at knowing exactly what to do.
The Map Maker was Much Faster: This was the biggest difference. The "Map Maker" learned its strategy in a matter of minutes. The "Deep Learner" took hours (or even days, depending on the data size) to train.
- Analogy: The Map Maker was like a student who studied a textbook and passed the test in 10 minutes. The Deep Learner was like a student who had to re-read the entire encyclopedia 100 times to get the same grade.

The Bottom Line

The paper concludes that if you want to teach a computer how to run a business process (like loan approvals) using past data:

You don't always need the most complex, heavy-duty AI (Deep Learning).
A simpler approach that builds a clear model of the process (MDP) works just as well, if not better, and is much faster and cheaper to run.

It's a reminder that sometimes, a good, clear map is better than a super-complex brain, especially when you just want to get the job done efficiently.

Technical Summary: Learning Optimal Policies from Event Logs through Reinforcement Learning

Problem Definition

The paper addresses the challenge of Prescriptive Process Monitoring (PPM), specifically focusing on learning optimal behavioral policies directly from historical event logs. Unlike existing PPM approaches that recommend pre-defined interventions or specific next activities based on expert rules, this work aims to discover general strategies that determine the best sequence of actions to maximize a specific Key Performance Indicator (KPI).

A critical constraint of the problem is the multi-actor environment. In many real-world processes (e.g., loan applications), the process owner (the agent) controls only a subset of activities, while others are driven by external actors (e.g., customers) or exogenous factors. The goal is to learn a policy for the agent that optimizes the KPI despite having partial control over the process execution.

The authors identify two primary challenges in applying Reinforcement Learning (RL) to this domain:

Data-Driven Learning: Avoiding reliance on extensive domain knowledge to manually define state spaces or intervention sets.
Evaluation Safety: The impracticality of testing policies on live systems due to the risk of disruption. Therefore, a robust evaluation mechanism is required before deployment.

Methodology

The paper proposes and compares two distinct AI-based approaches to learn optimal policies from event data, both operating within a multi-actor setting where the agent controls specific activities.

1. Model-Based Approach (MDP-based RL)

This approach constructs an explicit Markov Decision Process (MDP) from the event log and solves it using Dynamic Programming (DP).

MDP Construction:
- Actions: Defined as activities controlled by the agent.
- States: Constructed by encoding the trace history. The authors use k-means clustering on encoded prefixes (activity frequencies, positions, and normalized prefix rewards) to manage state space dimensionality and prevent explosion. A state is defined as the pair of the last performed activity and the assigned cluster.
- Transitions & Rewards: Built by replaying traces to estimate transition probabilities and average rewards.
Policy Learning: The optimal policy is derived using Value Iteration.
Handling Data Sparsity: To prevent the agent from exploiting rare, high-reward transitions (overfitting), the authors introduce a scaling function $h(s, a)$ applied to the Q-value. They test step functions and sigmoid functions parameterized by the frequency of state-action pairs (using quartiles of occurrence counts) to penalize low-frequency transitions.

2. Model-Free Approach (Offline Deep RL)

This approach employs Conservative Q-Learning (CQL), a state-of-the-art Offline Deep RL technique, which learns directly from historical data without interaction with the environment.

Architecture: A Recurrent Neural Network (RNN) with Long Short-Term Memory (LSTM) cells is used to approximate the state-action value function, handling the sequential nature of event logs.
Data Preparation: Transitions are extracted from traces. Crucially, transitions involving environment-controlled actions are removed, and their rewards are transferred to the preceding valid agent transition to align the dataset with the agent's perspective.
Training: The model is trained offline using the CQL algorithm to minimize overestimation bias common in offline RL.

Evaluation Framework

To evaluate the learned policies without risking live system performance, the authors utilize Business Process Simulation (BPS).

Synthetic Logs: Generated from a BPMN model to test robustness across varying log sizes and success rates.
Real-World Logs: BPIC2012 and BPIC2017 datasets are used.
Simulation Models: Two types of simulation models are employed:
1. Fixed Probability ( $M_\pi$ ): Uses standard probabilities for decision points.
2. Decision Tree ( $M^\pi_{DT}$ ): Uses decision trees trained on historical data to model environment responses more sensitively to process history.
Baselines: Comparisons are made against the customary policy (the behavior inherent in the original logs) and an informed heuristic (a strong baseline using privileged knowledge of simulation logic, available only for synthetic logs).

Key Contributions

Direct Policy Learning: The paper demonstrates a method to learn general behavioral policies directly from event logs with minimal domain knowledge, distinguishing between controllable and uncontrollable activities.
MDP Construction from Logs: It proposes a specific pipeline to construct an MDP from event logs using clustering to handle state abstraction, overcoming the difficulty of manually defining MDP states for process mining.
Scaling Function for Robustness: The introduction of frequency-based scaling functions ( $h$ ) for the Q-value in the MDP approach effectively mitigates overfitting to rare transitions, significantly improving policy stability.
Comprehensive Comparison: It provides a systematic comparison between a classical model-based DP approach and modern model-free Offline Deep RL (CQL) in the context of process mining.
Simulation-Based Evaluation: The work adapts BPS to evaluate next-activity selection policies in multi-actor settings, offering a controlled environment for A/B testing of learned strategies.

Results

The evaluation was conducted across eight synthetic logs (varying in size and success rates) and two real-world logs (BPIC2012 and BPIC2017).

Performance: Both methods successfully learned policies that consistently improved the target KPI (profit) compared to the customary policy.
- In synthetic logs, the MDP-based approach (specifically with scaling functions like $h_{sig}^{Q1}$ and $h_{step}^{Q1}$ ) generally outperformed the Offline Deep RL approach, achieving KPIs comparable to the informed heuristic in most cases.
- In real-world logs, both methods showed solid performance, with the MDP-based approach showing a slight or comparable advantage over CQL.
Robustness: The MDP-based approach demonstrated greater stability across different log characteristics. The naive MDP approach (without scaling) was unstable, while the CQL approach never appeared among the top-performing policies in the synthetic evaluation.
Computational Efficiency: The MDP-based approach was significantly more efficient. Training times for MDP-based methods ranged from seconds to a few minutes, whereas Offline Deep RL required approximately 30 minutes per epoch (totaling hours for effective policies). The MDP approach's time scaled with log size due to clustering, while Deep RL's per-epoch time remained constant but high.

Significance and Claims

The paper claims that learning optimal behavioral policies directly from event data is feasible and effective, even in complex multi-actor scenarios where the agent has partial control.

Practicality: The results suggest that the MDP-based approach offers a favorable balance between simplicity, interpretability, computational efficiency, and performance. The authors argue that for process mining scenarios, where data is structured and decision points are well-defined, complex deep learning models may not be strictly necessary to achieve high-quality policies.
Data-Driven Discovery: The work validates that it is possible to discover effective interventions and strategies solely from historical data, without pre-defining the set of possible actions or relying on extensive expert knowledge of the domain dynamics.
Future Potential: The authors position these learned policies as strong initializations for broader optimization frameworks that could later incorporate online interaction or feedback. They acknowledge limitations regarding the reliance on control-flow information and the need for richer state representations (e.g., data attributes) in future work.

The paper concludes that while both paradigms are viable, the classical model-based approach, when augmented with appropriate data-driven state abstraction and regularization, currently provides a more efficient and robust solution for prescriptive process monitoring in the contexts studied.