Technical Summary: On the Limitations of Large Language Models for Conceptual Database Modeling

Problem Statement

The design of relational databases relies heavily on the conceptual modeling phase, where system requirements are translated into Entity–Relationship (ER) diagrams. This process is traditionally challenging due to ambiguities, omissions, and inconsistencies in natural language requirements, often requiring iterative validation by domain specialists. While Large Language Models (LLMs) have emerged as potential tools to automate the extraction of entities, relationships, and attributes from textual requirements, their reliability in complex, real-world scenarios remains unproven. Specifically, there is a lack of understanding regarding how different prompting strategies and model architectures perform when tasked with generating structurally and semantically consistent ER diagrams, particularly as the complexity of business constraints increases.

Methodology

The study employs an experimental framework to evaluate the performance of LLMs in generating ER diagrams from natural language requirements.

• Dataset Construction: Three requirement documents were developed for a single factual scenario (hospital access management), featuring progressively increasing complexity through the cumulative addition of business constraints. The documents deliberately avoided technical jargon to simulate non-expert client inputs. A reference ER diagram for the most complex scenario was created to serve as a gold standard.
• Models Evaluated: Two high-performance LLMs were tested:
  • Qwen-3: Representing the open-weights category, selected for its strong performance in coding and reasoning benchmarks.
  • GPT-5.1: Representing the state-of-the-art proprietary models, selected for its adaptive reasoning capabilities.
• Prompting Techniques: The models were subjected to three distinct prompting strategies:
  1. Zero-Shot (One-shot): Direct instructions without explicit reasoning steps.
  2. Chain-of-Thought (CoT): Instructions requiring step-by-step reasoning to identify implicit elements.
  3. Chain-of-Thought + Verifier: CoT followed by an automatic validation step to simulate a critical review.
• Implementation: A Python-based framework automated the pipeline: processing text via the LLM, normalizing the JSON output to ensure structural consistency, and rendering the final ER diagrams using Graphviz.
• Evaluation Criteria: A qualitative, subjective evaluation was conducted using an ordinal ranking system (L1–L4) based on four dimensions:
  • Completeness: Identification of all entities, relationships, and attributes.
  • Correctness: Accuracy of relationship types, cardinalities, and constraints.
  • Clarity: Unambiguous naming and readable layout.
  • Simplicity/Flexibility: Avoidance of unnecessary complexity and support for future evolution.
  • Note: The evaluation did not operationalize these criteria into quantitative metrics (e.g., exact counts of errors), relying instead on expert qualitative judgment.

Key Results

The experimental results indicate that current LLMs are not yet sufficiently mature for reliable, autonomous conceptual database modeling, particularly as scenario complexity increases.

• Performance Ceiling: None of the generated diagrams achieved L3 (Structural Robustness and Data Integrity). The Qwen model failed to even reach L1 (Basic Scope Recognition) in several instances.
• Model-Specific Biases:
  • GPT-5.1 (Over-specification): Exhibited a tendency toward "undue creativity," introducing redundant entities and relationships not present in the requirements. For example, it created direct relationships between high-level entities and access records that were already transitively linked, compromising schema simplicity and data integrity. It struggled to distinguish between fundamental and derived attributes.
  • Qwen (Under-specification): Demonstrated a bias toward omission, failing to identify essential entities, attributes, and relationships. It often collapsed distinct roles (e.g., nurses vs. physicians) into a single generic entity, losing critical semantic distinctions and failing to model specific constraints (e.g., supervising physician relationships).
• Impact of Prompting: While the Chain-of-Thought (CoT) strategy showed a significant improvement over the Zero-Shot baseline in terms of reasoning clarity, the addition of a Verifier step proved limited. The verifier failed to detect critical flaws that compromised the integrity of the relational database, such as incorrect cardinalities or missing constraints.

Significance and Contributions

The paper makes the following contributions to the field of database modeling and AI:

1. Synthesized Literature Review: It provides a review of LLM applications in conceptual modeling, contrasting them with classical NLP methods which are often rigid and language-specific.
2. Novel Dataset: The authors constructed a new dataset containing textual requirements and a reference ER diagram, specifically designed for the systematic evaluation of LLMs in this domain.
3. Empirical Evidence of Limitations: The study provides empirical data demonstrating that LLMs, in their current state, introduce significant risks (redundancy, omission, semantic inconsistency) when applied to complex database modeling tasks.
4. Prompt Engineering Analysis: It offers a comparative analysis showing that while advanced prompting techniques (CoT) improve reasoning, they do not currently guarantee the structural correctness required for production-grade database schemas.

Conclusion and Future Directions

The authors conclude that while LLMs can generate rapid textual artifacts, their uncritical adoption for conceptual database modeling may compromise the quality of the final product. The effort required to validate and correct LLM-generated diagrams often offsets apparent productivity gains, especially in complex scenarios.

The paper suggests that future work should not focus on replacing the analyst but on using LLMs as support tools for specific, intermediate stages of the modeling process. Proposed directions include configuring specialized agents to identify and propose resolutions for ambiguities in requirements documents as a preliminary semantic validation step, rather than expecting models to perform end-to-end conceptual and logical modeling directly from text.