Technical Summary: Assessing the Business Process Modeling Competences of Large Language Models

Problem Statement

The creation of Business Process Model and Notation (BPMN) models is a cognitively demanding task requiring expertise in both domain knowledge and modeling conventions. While Large Language Models (LLMs) have demonstrated potential in generating structured artifacts from natural language, there is a lack of systematic evaluation regarding their ability to produce high-quality BPMN models. Existing evaluations often rely on "LLM-as-a-judge" approaches or fail to consider established dimensions of process model quality, such as syntactic, pragmatic, and semantic correctness. Furthermore, the validity of generated XML files—a critical requirement for practical deployment—has not been rigorously benchmarked across open-source LLMs.

Methodology

The BEF4LLM Framework

The authors introduce BEF4LLM (BPMN Evaluation Framework for LLMs), a novel evaluation framework extending the SIQ (Syntactic, Pragmatic, and Semantic Quality) framework. Unlike previous approaches, BEF4LLM incorporates a fourth dimension: Validity. The framework comprises 39 metrics across four dimensions:

1. Validity ($Q_{val}$): A binary measure (0 or 1) checking if the generated BPMN-XML file is parsable and adheres to the BPMN 2.0 XSD schema.
2. Syntactic Quality ($Q_{syn}$): 16 metrics evaluating adherence to BPMN modeling rules (e.g., existence of start/end events, correct gateway matching, sequence flow rules).
3. Pragmatic Quality ($Q_{prag}$): 15 metrics assessing understandability and complexity, including size, density, connector interplay, partitionability, and concurrency. These are normalized using empirically validated thresholds.
4. Semantic Quality ($Q_{sem}$): 7 metrics measuring the similarity between the generated model and a ground-truth model across natural language (label similarity), graph structure (edit distance), and behavior (causal footprints).

Scores for each dimension are aggregated into a total process quality score ($Q_{qual}$) and a total score ($Q_{total}$) which includes validity.

Experimental Setup

• Dataset: A curated dataset of 105 text-BPMN pairs derived from five public datasets (Camunda, LRE, etc.), ensuring high-quality ground truths.
• Models: 17 open-source LLMs from families including Llama 3, Qwen 2.5/3, Deepseek R1, Phi 4, and Falcon 3. Models were categorized by size (Small $\le$7B, Medium 7B–14B, Large >14B).
• Procedure:
  1. Prompting: A system prompt (one-shot example) and a modeling prompt were used to instruct the LLM to generate BPMN-XML.
  2. Refinement: If the initial output was invalid, a refinement prompt was sent to correct specific XML errors.
  3. Execution: Five runs per sample were conducted with a temperature of 0.1 to allow minor variations while maintaining instruction adherence.
  4. Filtering: LLMs generating fewer than 30 valid BPMN-XMLs on average were excluded from the final analysis to ensure statistical robustness.
• Statistical Analysis: The study employed non-parametric tests (Skillings-Mack for global differences, Wilcoxon signed-rank for pairwise comparisons) with Bonferroni correction to handle missing data and ordinal scores.
• Human Benchmark: A subset of 9 text descriptions was modeled by 5 human experts to establish a baseline for comparison.

Key Contributions

1. BEF4LLM Framework: The first framework specifically designed to evaluate LLMs in BPMN modeling, integrating validity as a critical dimension alongside syntactic, pragmatic, and semantic quality.
2. Large-Scale Benchmark: An extensive evaluation of 17 open-source LLMs across 105 text-BPMN pairs, providing the first comprehensive view of the current LLM landscape in process modeling.
3. Statistical Insights: A detailed analysis of performance differences between LLM families and sizes, revealing that parameter count does not linearly correlate with overall modeling quality.
4. Human Comparison: A direct benchmarking of top-performing LLMs against human modeling experts, highlighting areas of convergence and divergence.

Results

Comparative Performance

• Syntactic Quality: LLMs generally performed well, with scores consistently above 0.75. Falcon 3 10b achieved the highest syntactic score (0.9082), though Llama 3.3 70b was the most consistent top performer across dimensions.
• Pragmatic Quality: Scores were high (mostly >0.8), indicating generated models are often easy to understand. Qwen 2.5 14b scored highest (0.8907).
• Semantic Quality: This remained the weakest dimension for all models (scores ~0.5–0.57). Qwen 2.5 32b and Llama 3.3 70b led this category.
• Validity: A major bottleneck. Only Llama 3.3 70b achieved a validity rate above 90% (97.33%). Many smaller models failed to generate valid XMLs frequently.
• Overall: When excluding validity, Llama 3.3 70b achieved the highest total process quality ($Q_{qual}$ = 0.8143). When including validity ($Q_{total}$), it remained the leader (0.8143), while others dropped significantly.

Influence of Model Size

The study found that "larger is not always better."

• Scaling Effects: Increasing parameter counts generally improved syntactic and semantic quality (by 3–5 percentage points) but often degraded pragmatic quality (by ~2 points). Larger models tended to generate more complex, less organized processes.
• Architecture Dependence: The Qwen 3 family showed oscillating results where increasing size from 14B to 30B hurt semantic quality, while a jump to 235B restored it but further hurt pragmatic quality.
• Conclusion: Parameter count is a poor proxy for overall BPMN quality; smaller models (e.g., Llama 3 8b, Qwen 2.5 14b) often offer a better balance of quality and computational cost.

Comparison with Human Experts

• Comparable Scores: LLMs and human experts achieved scores in similar ranges.
• Strengths/Weaknesses: LLMs outperformed humans in syntactic and pragmatic quality (fewer structural errors, better formatting). However, humans significantly outperformed LLMs in semantic quality (0.5152 vs. ~0.50 for top LLMs), indicating better alignment with the intended meaning of the text.
• Consistency: Human models exhibited lower variance in quality scores compared to the high variance seen in LLM outputs.

Significance and Claims

The paper claims that BEF4LLM provides a standardized procedure for evaluating LLMs in BPMN modeling, moving beyond simple "LLM-as-a-judge" metrics. The findings suggest that while current open-source LLMs have reached a level of competency comparable to human experts in structural and readability aspects, they still struggle with deep semantic alignment and, crucially, with generating valid, parsable XML files.

The authors emphasize that validity is the primary hurdle for practical deployment. They argue that instruction-tuned variants are essential for this task and that future development should focus on fine-tuning for semantic accuracy and reducing token usage through more compact representations. The study concludes that AI-driven process design can lower barriers to entry for process modeling, but significant challenges remain before fully automated, context-aware BPM driven by LLMs becomes a reality.