💻 computer science

From Text to DSL: Evaluating Grammar-Based Model Generation Using Open LLMs

This paper demonstrates that compact, open-source large language models can effectively generate syntactically valid and semantically complete Domain-Specific Language (DSL) models from natural language using few-shot prompting, offering a cost-effective alternative to proprietary models for automating model-driven software engineering tasks.

Original authors: Junaid Baber, Nicolas Hili, Didier Schwab, Léo Challier, Cécilia Satrin

Published 2026-05-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Junaid Baber, Nicolas Hili, Didier Schwab, Léo Challier, Cécilia Satrin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a custom house, but instead of hiring a human architect, you ask a robot to draw the blueprints based on your spoken description.

This paper is essentially a report card on how well different "robot architects" (called Large Language Models, or LLMs) can do this job. Specifically, the researchers wanted to see if these robots could follow a very strict, rule-based language (called a DSL) to draw the blueprints for software, rather than just writing messy, free-form text.

Here is the breakdown of their experiment using simple analogies:

The Challenge: The "Strict Grammar" Test

In the world of software, a DSL (Domain Specific Language) is like a very rigid dialect. It's not like normal English where you can say things loosely; it's like a legal contract or a musical score where if you miss a single comma or play the wrong note, the whole thing breaks.

The researchers wanted to know: Can a robot listen to a human say, "I want an ice cream shop website," and instantly draw the perfect, rule-following blueprint without needing to go to school (fine-tuning) first?

The Experiment: The "Taste Test"

The researchers set up a massive taste test involving 39 different robots (AI models).

The Big Robots: Some were massive, expensive, and powerful (like a giant supercomputer).
The Small Robots: Others were tiny, cheap, and could run on a regular laptop (ranging from very small to medium-sized).

The Rules of the Game:

No Schooling: They didn't teach the robots anything new. They just gave them a "cheat sheet" (a prompt) with examples of how to speak the strict language.
The Task: The robots had to generate two things from scratch:
- The Data Model: The "ingredients list" (e.g., Ice Cream, Customer, Price).
- The UI Model: The "store layout" (e.g., where the menu goes, how customers order).
- Note: In previous studies, the robots were only asked to draw the layout based on a pre-made ingredients list. This time, they had to invent the ingredients list and the layout.

The Process: The "Grammar Police" and the "Human Judge"

After the robots tried to draw their blueprints, the researchers checked them in two ways:

The Grammar Police (Automatic Check): A computer program acted as a strict editor. It scanned the blueprints to see if they followed the exact rules. If a robot forgot a semicolon or used the wrong symbol, the program said, "Fail!" and asked the robot to try again.
The Human Judges (Expert Check): If the blueprint passed the Grammar Police, three human experts (who are like master architects) looked at it. They asked: "Does this actually make sense? Did the robot remember to include a 'Discount' section? Did it connect the 'Customer' to the 'Order' correctly?"

The Results: The "David vs. Goliath" Surprise

The big news is that size didn't matter as much as everyone thought.

The Big Robots: As expected, the giant, expensive models did a good job.
The Small Robots: Surprisingly, several of the smaller, open-source robots (like gemma3:12b and mistral:7b) performed just as well as the giants. They managed to follow the strict rules and create complete blueprints without needing to be retrained.

Key Findings:

Prompting is Key: Just giving the robot a clear set of instructions (the "cheat sheet") was enough to make even small robots work well.
Retry Mechanism: If a robot made a small mistake, letting it try again with a slightly different setting helped it fix the error.
Cost-Effective: You don't need to rent a supercomputer to build these software blueprints; a smaller, cheaper model can do the job if you guide it correctly.

The Bottom Line

This paper proves that you don't need the most expensive, massive AI to generate complex software designs. Smaller, open-source models can be just as effective if you give them clear instructions and a way to check their work. It's like discovering that a well-trained local carpenter can build a perfect house just as well as a famous architect, as long as you give them the right blueprint and let them double-check their measurements.

What the paper does NOT say:

It does not claim these robots are ready to build entire apps for you to download today.
It does not say they are perfect for medical or legal advice.
It strictly focuses on the ability to generate syntax-correct and semantically complete models for software design, not on deploying them into real-world clinical or business systems yet.

Technical Summary: From Text to DSL

Problem Statement

The paper addresses the challenge of automating Model-Driven Engineering (MDE) tasks, specifically the generation of models conforming to Domain-Specific Languages (DSLs) from natural language specifications. While Large Language Models (LLMs) show promise in this area, existing approaches rely heavily on large, proprietary, high-parameter models (e.g., GPT-4), which incur high costs and limit deployability. Furthermore, current methods often assume fixed, pre-defined data schemas, failing to test the model's ability to infer domain concepts and maintain consistency across multiple interrelated models (e.g., both UI and data models).

The core difficulty lies in the strict syntactic and semantic rules of DSLs. LLMs are not inherently structure-aware, and DSLs are often underrepresented in training corpora. Consequently, there is a need to evaluate whether smaller, open-source LLMs can generate syntactically valid, semantically complete, and structurally consistent DSL models using only few-shot prompting, without the need for fine-tuning or retraining.

Methodology

The authors propose a structured evaluation pipeline involving 39 open-source LLMs ranging from 0.5B to 32B parameters (including families like LLaMA, Qwen, Gemma, Phi, and Granite). The methodology is characterized by the following components:

Task Scope Expansion: Unlike prior work that generated UI models over fixed data schemas, this study requires LLMs to generate both the data model and the UI model entirely from scratch based on natural language descriptions (e.g., "create a website for an online ice cream parlor"). This tests the model's ability to infer domain concepts, attributes, and relationships.
Prompt Engineering: A unified, strict prompt template was used for all models. The prompt explicitly defines the role ("DSL generator"), enforces specific grammar rules (e.g., cardinality keywords like one, some, lone, and reference syntax like -->), and provides few-shot examples. No fine-tuning was applied.
Two-Phase Evaluation Pipeline:
- Syntactic Validation: An automated LARK parser (bottom-up LALR) checks outputs for grammar conformance. The system employs a retry mechanism: if a model fails to parse, it is re-issued a prompt with a reduced temperature (0.1) to encourage deterministic behavior. This loop runs up to $N$ times. Models failing to produce valid syntax after retries are excluded from further analysis.
- Semantic Evaluation: For models passing syntactic validation, a human-centered study was conducted with three domain experts. They evaluated outputs on a 5-point Likert scale across four criteria: Semantic Correctness, Concept Identification, Completeness, and the inclusion of Advanced Features (e.g., enums, subset relations, cross-entity references).

Key Contributions

The paper makes three primary contributions:

Structured Evaluation of Open-Source LLMs: It provides a comprehensive assessment of lightweight (0.5B) to large (32B) open-source models for DSL generation, demonstrating that few-shot prompting alone is a viable strategy without fine-tuning.
Generalization of Model Generation: By shifting from fixed-schema UI generation to the simultaneous generation of data and UI models, the study evaluates the LLM's capacity to infer application-specific domain concepts and maintain structural consistency across multiple interconnected models.
Hybrid Validation Pipeline: The authors designed a robust evaluation framework combining automated syntax parsing (with retry logic) and expert human feedback to benchmark performance on criteria that structural validation alone cannot capture.

Results

The evaluation of 39 models yielded the following findings:

Syntactic Success: Only 26 of the 39 models produced at least one syntactically correct output within the retry limit.
Performance vs. Size: There is no strict linear correlation between parameter count and DSL generation quality. Several compact models (e.g., gemma3:12b, mistral:7b-instruct, notus:latest) achieved performance comparable to or exceeding that of much larger models (e.g., phi4:latest, codellama:13b).
Impact of Prompting: Instruction-tuned models generally outperformed base models. The use of structured prompts and retry mechanisms with reduced temperature significantly improved the ability of smaller models to generate conformant outputs.
Semantic Quality: Top-performing models demonstrated consistent quality in concept identification and completeness. However, the inclusion of advanced semantic features (like complex relationships or enums) varied significantly, with some models generating only essential entity definitions.

Significance and Claims

The paper claims to demonstrate the feasibility of using smaller, open-source LLMs for grammar-conformant DSL generation in MDE workflows. The findings suggest that:

High-parameter, commercial models are not strictly necessary for generating valid DSLs if the prompt engineering is rigorous and validation loops are implemented.
Compact, open-source models offer a cost-effective and deployable alternative for building intelligent modeling assistants, particularly in contexts constrained by privacy, cost, or deployment requirements (e.g., low-code/no-code platforms).
The ability of small models to infer domain relationships and maintain consistency across multiple models without fine-tuning challenges the assumption that only massive models can handle complex, multi-model MDE tasks.

The authors conclude that while large models remain reliable, the gap in performance is narrowing, making resource-efficient models a practical option for specific application domains. Future work is suggested to explore fine-tuning small models and integrating tighter verification workflows.