MermaidSeqBench: An Evaluation Benchmark for NL-to-Mermaid Sequence Diagram Generation

Technical Summary: MermaidSeqBench

MermaidSeqBench is a research paper from IBM Research that introduces a specialized evaluation benchmark designed to assess the ability of Large Language Models (LLMs) to generate Mermaid sequence diagrams from natural language (NL) descriptions.

---

1. The Problem: Lack of Reliable Evaluation for Structured Diagrams

While LLMs have shown proficiency in code generation, their ability to generate structured visual models—specifically Mermaid sequence diagrams used in software engineering—remains largely unvalidated.

• Criticality: In DevOps and system architecture, a single syntax error or a logical flaw in a sequence diagram can lead to catastrophic misunderstandings of system interactions.
• Benchmark Gap: Existing benchmarks focus on general code (like HumanEval) or other diagram formats (like PlantUML), but there is a lack of public, fine-grained, and scalable datasets specifically for Mermaid sequence diagrams.
• Deployment Risk: Without a way to measure correctness in syntax, logic, and usability, it is difficult to trust LLMs for mission-critical software engineering documentation.

2. Methodology: A Hybrid, Multi-Stage Approach

The authors developed a robust pipeline to create a benchmark that is both high-quality and scalable.

A. Dataset Construction (132 Samples):
The benchmark was built using a three-stage hybrid methodology:

1. Initial Seed Set: A small core of 10 high-quality diagrams was manually crafted and verified by Subject Matter Experts (SMEs) to ensure semantic and syntactic perfection.
2. Synthetic Expansion: Using the Scalable Synthetic Data Generation (SDG) framework and the Mistral-Large-Instruct model, the authors expanded the dataset. These synthetic samples were then manually verified via the Mermaid Live Editor and SME review.
3. Rule-Based Augmentation: To increase diversity without changing the underlying logic, the authors applied deterministic rules to reorder conditional constructs (alt, else, end) and normalize participant identifiers.

B. Evaluation Framework (LLM-as-a-Judge):
Instead of simple string matching, the paper proposes a sophisticated LLM-as-a-Judge system. Two massive models (DeepSeek-V3 and GPT-OSS) act as judges to score generated outputs across six fine-grained dimensions:

• Syntax: Structural correctness of the Mermaid code.
• Mermaid Only: Ensuring the output is pure code without conversational "chatter."
• Logic: Correctness of the interaction flow and request/response pairs.
• Completeness: Whether all described components and paths are present.
• Activation Handling: Proper use of activate and deactivate commands.
• Error & Status Tracking: Effectiveness in representing success/failure paths and state changes.

3. Key Contributions

• MermaidSeqBench Dataset: The first public, human-verified, and synthetically extended dataset for NL-to-Mermaid sequence diagram generation.
• Fine-Grained Evaluation Metric: A multi-dimensional scoring rubric that moves beyond "pass/fail" to assess the nuance of diagrammatic representation.
• Open Source Contribution: The authors have open-sourced the evaluation code and the dataset to encourage standardized research in structured diagram generation.

4. Results and Discussion

The authors conducted extensive experiments across different model families (Llama, Qwen, Granite) and scales.

• Scaling Effects: There is a clear correlation between model size and performance. Larger models (e.g., Qwen-2.5-72B) significantly outperform smaller ones, particularly in complex areas like logic and error tracking.
• Capability Gaps: The benchmark revealed that while many models are good at basic Syntax, they struggle significantly with Activation Handling and Error/Status Tracking.
• Model Divergence: Different model families show unique strengths; for example, Granite models showed strong syntactic adherence but lagged in logical completeness compared to Llama and Qwen.
• Judge Variability: The study highlighted that different LLM judges (DeepSeek vs. GPT-OSS) provide different perspectives, with GPT-OSS generally being stricter, emphasizing the need for multi-judge validation.

5. Significance

MermaidSeqBench establishes a new standard for evaluating the intersection of natural language processing and software engineering visualization. By providing a way to quantify the "correctness" of structured diagrams, it paves the way for the reliable deployment of LLM-based coding assistants in professional software development workflows, ensuring that the diagrams they produce are both syntactically valid and logically sound.