Structure- and Event-Driven Frameworks for State Machine Modeling with Large Language Models

1. Problem Statement

The paper addresses the challenge of fully automating the generation of UML state machines from non-structured Natural Language (NL) requirements.

• Current State: Traditionally, state machines are manually crafted by experienced engineers, a process that is time-consuming and error-prone. Existing automated approaches often rely on structured NL inputs (e.g., specific templates or Given-When-Then formats), which limits their applicability to real-world, unstructured documentation.
• The Gap: While Large Language Models (LLMs) have shown promise in generating various software models, their ability to autonomously derive complex, hierarchical state machines (including guards, actions, parallel regions, and history states) from raw text remains unproven. Furthermore, it is unclear whether reasoning-focused LLMs (which generate step-by-step logic) outperform non-reasoning LLMs in this specific domain, and whether multi-step prompting strategies (decomposing the task) improve results over single-step generation.

2. Methodology

The authors propose a framework that converts non-structured NL system descriptions into UML state machines (specifically in Umple syntax) using two distinct types of LLMs and four generation strategies.

A. Models Evaluated

1. Non-Reasoning LLM: GPT-4o (OpenAI). Provides direct answers without explicit internal reasoning traces.
2. Reasoning LLM: Claude 3.5 Sonnet (Anthropic). Designed to produce step-by-step logical reasoning (Chain-of-Thought) as part of its output.

B. Generation Strategies

The study compares four distinct approaches:

1. Single-Prompt Baseline: A single-step approach where the LLM is prompted to generate the complete state machine (states, transitions, guards, actions, etc.) in one go.
2. Structure-Driven SMF (State Machine Framework): A multi-step, linear strategy inspired by human modeling. It breaks the generation into sequential stages (e.g., Step 1: Identify states; Step 2: Identify parallel regions; Step 3: Define transitions; Step 4: Define actions). Intermediate outputs are formatted as HTML tables to guide the next step.
3. Event-Driven SMF: An iterative, multi-step strategy where the LLM first identifies the system states, then processes each identified event individually to determine state transitions and actions. This mimics an event-centric human approach.
4. Hybrid Approach: Combines the Single-Prompt Baseline with the Structure-Driven SMF. The LLM first generates a complete draft via a single prompt. This draft is then appended to the prompts of the Structure-Driven SMF steps, serving as a "baseline" for the LLM to refine and expand upon iteratively.

C. Evaluation Setup

• Dataset: 8 non-structured NL system descriptions (e.g., Dishwasher, Chess Clock, Printer) from an undergraduate course, paired with expert-created ground-truth UML state machines.
• Metrics: Precision, Recall, and F1-scores calculated for seven components: States, Transitions, Guards, Actions, Hierarchical States, Parallel Regions, and History States.
• Post-Processing: A strict rule-based module merges and refines LLM outputs (especially for multi-step strategies) to ensure syntactic correctness.

3. Key Contributions

1. First Fully Automated Pipeline: The first study to attempt fully automated state machine generation from non-structured NL requirements using LLMs without human intervention.
2. Novel Frameworks: Introduction of Structure-Driven and Event-Driven SMFs, which decompose the modeling task into manageable sub-tasks inspired by human engineering practices.
3. Hybrid Approach: A novel strategy that leverages a single-prompt draft to bootstrap a multi-step refinement process.
4. Reasoning vs. Non-Reasoning Comparison: A systematic comparison of GPT-4o and Claude 3.5 Sonnet, revealing that the optimal generation strategy depends heavily on the model's architecture (reasoning vs. non-reasoning).
5. Benchmarking: Establishment of a baseline dataset and evaluation metrics for future research in automated model generation.

4. Key Results

A. Single-Prompt Performance (RQ1)

• Claude 3.5 Sonnet outperformed GPT-4o significantly.
  • Claude: Overall F1-score of 0.70.
  • GPT-4o: Overall F1-score of 0.54.
• Strengths: Both models performed well on identifying States (F1 ~0.80–0.90) and Transitions (F1 ~0.57–0.75).
• Weaknesses: Both struggled with Actions (GPT-4o F1 = 0.00; Claude F1 = 0.16), Guards, Parallel Regions, and History States.
• Observation: Models exhibited high precision but low recall, meaning they generated correct elements but missed many required components.

B. Impact of Multi-Step Strategies on Non-Reasoning LLMs (RQ2 - GPT-4o)

• Multi-step strategies significantly improved GPT-4o's performance.
• Hybrid Approach achieved the best overall F1-score (0.66), followed by Structure-Driven SMF (0.63).
• Improvement: The Hybrid approach boosted the F1-score for Actions from 0.00 to 0.34 and improved recall substantially (+0.16 to +0.17) compared to the baseline.
• Conclusion: Decomposing the task helps non-reasoning models handle complexity better.

C. Impact of Multi-Step Strategies on Reasoning LLMs (RQ3 - Claude 3.5 Sonnet)

• Counter-intuitive Finding: Multi-step strategies degraded the performance of the reasoning LLM (Claude 3.5 Sonnet).
• Single-Prompt Baseline remained the best performer for Claude (F1 = 0.70).
• Multi-step results: Hybrid (0.63), Structure-Driven (0.50), and Event-Driven (0.31) all performed worse than the baseline.
• Reasoning: The authors suggest that forcing a reasoning LLM into a rigid, external multi-step framework interferes with its internal step-by-step reasoning capabilities. It may be more effective to let the reasoning model handle the complexity in a single, well-structured prompt.

5. Significance and Conclusion

• Potential: LLMs show strong potential for automating the "skeleton" of state machines (states and transitions) but currently lack the precision to fully automate complex behaviors (actions, guards, parallelism) without human oversight.
• Strategic Insight: The choice of generation strategy must be tailored to the LLM type.
  • Non-Reasoning Models: Benefit from multi-step, structured frameworks (Structure-Driven or Hybrid).
  • Reasoning Models: Perform best with single-step, high-quality prompts that leverage their internal reasoning, rather than external decomposition.
• Future Work: The paper identifies the need for better strategies to handle "Actions" and "Parallel Regions," the development of reasoning-specific generation frameworks, and the creation of larger benchmark datasets.

In summary, this paper provides a critical baseline for the field of AI-driven software modeling, demonstrating that while LLMs are powerful, the "one-size-fits-all" prompting approach is insufficient; the generation strategy must be aligned with the specific cognitive architecture of the model being used.