Towards Standardized Evaluation in Automated Domain Modeling: Introducing a Benchmark
This paper introduces a standardized benchmark for evaluating automated domain modeling approaches by combining existing datasets to enable comparative assessment across varying levels of complexity and scale, while providing the resource as a reusable research artifact in accordance with FAIR4RS recommendations.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of software engineering, before a single line of code is written, teams must agree on what the system is supposed to be. This agreement takes the form of a map, a visual blueprint that outlines the key players in a story and how they interact. In technical terms, this is called a domain model, often drawn as a class diagram. It is a precise language that developers use to ensure everyone understands the rules of the game, whether the project is a banking app or a hospital scheduling system. For decades, creating these maps has been a manual task, requiring human experts to translate vague descriptions of business needs into strict diagrams. Recently, powerful computer programs known as large language models have begun to attempt this translation automatically, promising to speed up the process and reduce errors. However, without a common way to measure how well these programs are doing, it has been difficult to know if they are truly helpful or just guessing.
To solve this problem, researcher Vasiliy Seibert has introduced a standardized test designed to evaluate these automated tools fairly. The work addresses a critical gap in the field: while many researchers have built their own methods for turning text into diagrams, they have tested them on their own private sets of examples, making it impossible to compare results across different studies. Seibert's solution is a benchmark, a shared yardstick that combines two existing collections of data into a single, robust set of fifty-three examples. Each example consists of a natural language description of a system, such as a detailed explanation of how a laboratory manages patient test requests, paired with a perfect, human-created diagram that serves as the correct answer. The goal is simple: feed the text description into an automated tool and see how closely the resulting diagram matches the human-made reference.
The study puts this benchmark to work by testing three different approaches to the task. The first is a traditional method that relies on fixed rules and linguistic patterns to extract information, a technique that has been used for years. The other two approaches use modern large language models, which are the same type of technology behind advanced chatbots, to read the text and generate the diagram directly. To ensure the results are reliable, the researchers ran the tests multiple times, using two different large language models and checking the consistency of the output. They measured the success of each attempt using a specific scoring system that compares the generated diagram against the correct one, counting how many of the right elements were found and how many mistakes were made.
The findings reveal a clear shift in capability. On the larger set of forty-five examples, the tools powered by large language models significantly outperformed the traditional rule-based method. The best-performing combination of a language model and a specific prompting strategy achieved high marks, correctly identifying the majority of the classes, attributes, and relationships described in the text. The traditional method, while consistent, struggled to capture the full complexity of the descriptions, scoring much lower across the board. However, the study also uncovered a hidden challenge. When the researchers tested the tools on a smaller set of eight more complex examples, the results became less stable. The scores for the language models varied more from one run to the next, and the rate of errors increased. This suggests that while these tools are powerful, their performance can be sensitive to the size and complexity of the task at hand.
Perhaps the most important contribution of this work is not just the ranking of the tools, but the establishment of a standard way to measure them. By providing a public dataset and a clear method for scoring, the researchers have created a foundation for future competition and improvement. This allows other scientists to test their own ideas against the same baseline, ensuring that progress in the field is real and measurable. The benchmark is released as an open resource, inviting the community to use it to refine their methods and push the boundaries of what automated software design can achieve. In doing so, the paper moves the field from a collection of isolated experiments toward a unified science where tools can be compared, understood, and trusted.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.