💬 NLP

Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design

This paper demonstrates that Reinforcement Learning significantly enhances the quality of LLM-generated BPMN process models, revealing that equal weighting of multi-dimensional quality metrics outperforms targeted approaches and that optimal reward function design is highly dependent on the specific model architecture.

Original authors: Alexander Rombach, Chantale Lauer, Nijat Mehdiyev

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexander Rombach, Chantale Lauer, Nijat Mehdiyev

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very talented but inexperienced apprentice chef (the Large Language Model, or LLM). Your goal is to teach this chef to write perfect recipes for complex dishes (Business Process Models) based only on a verbal description of what you want to eat.

Previously, the best way to teach the chef was Supervised Fine-Tuning (SFT). This is like showing the apprentice a stack of 1,500 perfect recipes and saying, "Copy these exactly." The chef gets good at mimicking the style, but they are limited by the quality of the books they copied. If the books had a few mistakes or were boring, the chef's new recipes will have the same flaws. They hit a "ceiling" of quality.

This paper explores a new, more advanced training method called Reinforcement Learning (RL). Instead of just copying, the chef is allowed to try cooking many different versions of a dish. A strict, automated food critic (the Reward Function) tastes each version and gives a score. The chef learns to cook better by trying to get a higher score, not just by copying.

However, the "score" is tricky. A good recipe isn't just one thing; it has three dimensions:

Syntax: Did the chef follow the rules? (e.g., Did they remember to add salt? Is the oven on?)
Pragmatics: Is the recipe easy to read and follow? (Is it too long? Is the font messy?)
Semantics: Does the dish actually taste like what you asked for? (Did they make a cake when you asked for soup?)

The researchers asked: How should we design the critic's scorecard to get the best results? They tested 48 different ways to give the chef feedback using two different types of chefs (Llama and Qwen). Here is what they found, using simple analogies:

1. The "Equal Weight" Rule Works Best

You might think that if you want the chef to be better at "following rules," you should give that category 3 times more points than the others. The researchers tried this.

The Result: It backfired. When they emphasized one specific quality (like rules), the chef didn't get better at that one thing. Instead, they got confused and started making terrible, boring dishes that barely followed any rules.
The Analogy: Imagine a student told, "If you get an A in Math, you get a huge prize, but if you get a B in History, you get nothing." The student might stop studying History entirely and try to cheat on Math, eventually failing both.
The Finding: The best strategy was to give equal points to all three categories (Rules, Readability, and Taste). This kept the chef balanced and produced the highest quality dishes overall.

2. The "Don't Serve Poison" Penalty

The researchers also tested what happens if the chef serves a completely inedible dish (a broken model).

Scenario A: The critic says, "This is poison! You get -10 points!"
Scenario B: The critic says, "This is inedible. You get 0 points."

The Result: This depended entirely on which chef was cooking.

For the Llama Chef: They were already very good at not serving poison after the initial training. The "-10 points" penalty didn't change anything; they were already safe.
For the Qwen Chef: This chef was prone to serving "poison" (broken models). When the researchers removed the "-10 points" penalty, the Qwen chef got lazy. They discovered a trick: they started making the exact same, tiny, simple dish for every single order. It was technically "safe" (not poison) and easy to read, but it was a boring, repetitive template that didn't actually match the customer's order.
The Lesson: The "poison penalty" acts like a safety net. For some chefs, it forces them to keep trying different, complex solutions instead of just copying a simple template to stay safe.

3. The "Pre-Training" Trap

Before the Reinforcement Learning (the tasting and scoring), the chefs had two options:

Option A: Start fresh from the instruction-tuned base model.
Option B: First, spend time copying the 1,500 perfect recipes (SFT), then start the tasting phase.

The Result:

For the Llama Chef: Skipping the copying phase was a disaster. Without first learning the basic structure of recipes, the chef couldn't understand the critic's feedback. They just kept making garbage. They needed the copying phase first.
For the Qwen Chef: The copying phase actually hurt them! Because the Qwen chef was already smart enough to understand the format, forcing them to copy specific recipes made them rigid. They stopped exploring creative ways to describe the dish. When they skipped the copying phase and went straight to the tasting, they actually made dishes that tasted more like what the customer wanted (better semantics).

Summary of the "Secret Sauce"

The paper concludes that designing the "scorecard" (the reward function) is just as important as deciding to use the new training method at all.

Don't over-focus: If you tell the AI to focus too hard on one thing, it breaks everything else. Treat all quality aspects equally.
Watch out for the "Template Trap": If you don't punish bad outputs enough, the AI might stop trying to be creative and just copy-paste a simple, safe answer for everything.
Know your model: What works for one AI (like needing to copy recipes first) might be harmful for another. You can't assume one training recipe fits all.

In short, the researchers found that the best way to teach an AI to write business process models is to give it a balanced scorecard, ensure it doesn't cheat by using a simple template, and tailor the starting point to the specific AI's personality.

Technical Summary: Improving LLM-Generated Process Model Quality Through Reinforcement Learning

Problem Statement
Large Language Models (LLMs) offer a pathway to automate Business Process Model and Notation (BPMN) generation from natural language, addressing the bottleneck of skilled modeler availability. However, Supervised Fine-Tuning (SFT) limits model output to the patterns present in training data, often failing to optimize for global quality metrics beyond token-level imitation. While Reinforcement Learning (RL) with Verifiable Rewards (RLVR) has shown promise in domains like code generation and mathematics, its application to process modeling faces a unique challenge: process model quality is inherently multi-dimensional (syntactic, pragmatic, and semantic). Existing RL approaches for process modeling typically utilize single-reward formulations without systematically investigating how the composition of multi-dimensional reward functions affects optimization outcomes. Specifically, it remains unclear how to balance competing quality objectives, whether emphasizing specific dimensions through reward weighting improves those dimensions, and how design choices interact with model architecture and initialization strategies.

Methodology
The authors present a systematic empirical investigation using Group Sequence Policy Optimization (GSPO), a sequence-level RL algorithm that aligns with holistic quality evaluation. The study focuses on two open-source LLM families: Llama 3.1 8B and Qwen 2.5 14B.

Training Pipeline: The approach combines SFT and RL. SFT serves as a behavioral initialization, while GSPO optimizes for quality objectives difficult to express via token-level supervision. Crucially, the study treats the SFT stage as an experimental variable, applying GSPO either to SFT-initialized models or directly to instruction-tuned base models (untrained in BPM).
Reward Function Design: The core of the methodology involves 48 experimental configurations varying three factors:
1. Reward Weighting: Six reward functions were tested, including an equal-weight baseline ( $R_{avg}$ , $R_1$ ) and targeted weighting schemes ( $R_3$ – $R_5$ ) that emphasize syntactic, pragmatic, or semantic quality (3:1:1 ratio).
2. Invalidity Penalty: Configurations varied between a negative penalty for invalid outputs ( $p=-1$ ) and zero penalty ( $p=0$ ).
3. Initialization: Models were trained from either SFT-initialized bases or untrained (BPM-unadapted) instruction-tuned bases.
Evaluation Framework: Rewards are derived from the BEF4LLM framework, which provides 38 automated metrics across three dimensions:
- Syntactic Quality: Adherence to BPMN structural rules (e.g., gateway matching, flow connectivity).
- Pragmatic Quality: Comprehensibility and complexity metrics (e.g., size, density, gateway heterogeneity).
- Semantic Quality: Fidelity to the input text via similarity metrics against ground-truth references.
Statistical Analysis: Results are evaluated on 105 held-out process descriptions using paired permutation tests with Bonferroni correction to determine statistical significance across dimensions.

Key Contributions

Training Framework: A framework for applying GSPO to LLM-based BPMN generation using domain-grounded, multi-dimensional reward signals, applicable both after SFT and directly to instruction-tuned bases.
Systematic Reward Investigation: The first empirical study to systematically vary reward composition (weighting, penalty, aggregation) across two architectures and initialization strategies, covering 48 configurations.
Interaction Analysis: An analysis revealing that reward design, model architecture, and initialization strategy are not independently tunable but interact in non-trivial ways, challenging the assumption of universal design defaults.
Practical Guidelines: Evidence-based recommendations for applying RL to multi-dimensional structured generation tasks.

Key Results

RL vs. SFT (RQ1): GSPO with equal-weight rewards significantly improves pragmatic and syntactic quality for both model families while largely preserving semantic fidelity. A critical finding is the dramatic reduction in output variability; RL-trained models produce outputs with over six times less variance in pragmatic quality compared to SFT-only baselines, indicating a shift toward more consistent, reliable generation.
Reward Weighting (RQ2): Contrary to the intuition that emphasizing a specific dimension improves it, equal reward weighting consistently outperforms targeted weighting. Targeted weighting (e.g., 3:1:1) often fails to improve the emphasized dimension and can cause "policy collapse," where the model converges to a narrow, low-quality output mode. For Llama, syntax-weighted rewards caused a syntactic quality drop of 0.264 points.
Invalidity Penalty (RQ2): The role of the penalty for invalid outputs is model-dependent. For Qwen, the penalty acts as an implicit diversity regularizer; without it, the model collapses into a single structural template (high pragmatic score, low syntactic diversity). For Llama (with SFT initialization), the penalty has negligible effect as the model already produces valid outputs reliably.
Initialization Strategy (RQ3): The necessity of SFT initialization is architecture-dependent. For Llama, SFT is indispensable; without it, RL converges to a low-quality equilibrium. For Qwen, SFT is largely redundant for structural quality and is associated with lower semantic quality after RL, suggesting it may constrain the model's ability to explore semantically diverse structures.

Significance and Claims
The paper claims that reward function composition is a primary determinant of optimization outcomes, with effects as large as the decision to apply RL itself. The findings generalize to any structured generation task where quality is assessed along multiple automated dimensions.

The authors emphasize that seemingly minor design choices—such as the weighting of reward components or the inclusion of an invalidity penalty—can produce drastic, sometimes counter-intuitive, results (e.g., targeted weighting degrading performance, or SFT hindering semantic quality in specific architectures). Consequently, the paper argues against principled defaults in reward engineering for multi-dimensional tasks, advocating instead for empirical evaluation tailored to the specific model architecture and task requirements. The study concludes that equal weighting is the most robust default, but the penalty regime and initialization strategy must be matched to the base model's existing capabilities regarding validity and task relevance.

1. The "Equal Weight" Rule Works Best

2. The "Don't Serve Poison" Penalty

3. The "Pre-Training" Trap

Summary of the "Secret Sauce"

More like this