💻 computer science

Benchmarking Requirement-to-Architecture Generation with Hybrid Evaluation

This paper introduces R2ABench, a novel benchmark featuring real-world projects with PRDs and expert PlantUML diagrams, alongside a hybrid evaluation framework that reveals while LLMs excel at syntax and entity extraction, they fundamentally struggle with relational reasoning in software architecture generation.

Original authors: Minxiao Li, Shuying Yan, Li Zhang, Yang Liu, Fang Liu

Published 2026-04-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Minxiao Li, Shuying Yan, Li Zhang, Yang Liu, Fang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are the CEO of a startup. You have a massive, messy notebook filled with your wildest ideas for a new app: "It needs to talk to users," "It should store photos," "Make it fast," and "It must work on phones." This is your Requirement Document.

Now, imagine you need to hire an architect to draw the blueprints for this building. In the old days, you'd hire a human architect who would read your messy notebook, think hard, and draw a perfect, detailed blueprint. But this takes time, costs money, and if you change your mind later, the architect has to redraw everything.

Enter AI (Large Language Models). You think, "Hey, AI is great at writing code and stories. Can't it just read my notebook and instantly draw the perfect blueprint?"

This paper, R2ABench, is like a rigorous "driver's test" for AI to see if it can actually do that job. Here is the breakdown in simple terms:

1. The Problem: The "Blank Canvas" Gap

The researchers noticed that while AI is amazing at writing code (like building a single brick), it hasn't been tested well on designing the whole building (the architecture) from messy, real-world notes.

The Gap: Existing tests used perfect, short sentences like "Draw a house with a door." Real life is messy: "We need a house that feels like a castle but is built with cardboard, has a secret tunnel, and the door must be blue but only on Tuesdays."
The Solution: They built R2ABench. Think of this as a giant, standardized "exam room" containing 17 real-world software projects. For each project, they have the messy "CEO's notebook" (the Requirements) and the Gold Standard Blueprint (the perfect diagram drawn by human experts).

2. The Test: How Do We Grade the AI?

You can't just ask the AI, "Did I do good?" because AI can lie (hallucinate). You can't just compare the words, because a blueprint is a structure, not a story.

So, they created a Three-Layer Grading System:

Layer 1: The Structural Check (The "Did it fall over?" test)
- They turn the AI's drawing into a graph (dots and lines).
- Did it compile? (Is the drawing valid?)
- Did it get the dots right? (Did it identify the "Database" and "User"?)
- Did it get the lines right? (This is the hard part. Did it connect the User to the Database correctly, or did it connect the User to the Moon?)
- Analogy: It's like checking if a LEGO castle has the right number of towers and if the walls are actually connected to the base.
Layer 2: The Human-like Judge (The "Does it make sense?" test)
- They use a super-smart AI (another AI) to act as a teacher. It reads the messy notes and the AI's drawing and gives it a score on:
  - Completeness: Did it forget the kitchen?
  - Accuracy: Did it invent a swimming pool that wasn't asked for?
  - Rationality: Is the kitchen next to the bathroom? (Maybe not a good idea).
  - Readability: Is the drawing messy or organized?
Layer 3: The "Anti-Pattern" Detector (The "Bad Design" alarm)
- This checks for common architectural mistakes.
- Orphaned Components: Did it draw a room with no door? (Useless).
- The "God Component": Did it draw one giant room that everyone has to walk through to get to the bathroom? (This is a bottleneck; a bad design).

3. The Results: The AI Can Draw, But Can't Connect

They tested the smartest AIs (like GPT-5, Claude, and specialized coding AIs) and some "Agent" systems (AI teams that try to work together). Here is what they found:

The Good News: The AI is great at naming things. If you ask for a "Database," it almost always draws a box labeled "Database." It understands the vocabulary.
The Bad News: The AI is terrible at connecting things. It often draws the boxes but forgets to draw the lines between them, or draws the lines in the wrong direction.
- Analogy: It's like a child who can name every piece of a puzzle perfectly but can't figure out how they fit together. The result is a pile of puzzle pieces, not a picture.
The "Agent" Surprise: They tried using "Agent Frameworks" (where one AI writes the plan, another checks it, a third draws it). They expected this to be like a team of architects working together.
- Result: Instead of a better team, it was like a chaotic meeting where everyone talked over each other. The "Agents" actually made the drawings worse and more unstable.

4. The "Missing Info" Experiment

They tested what happens if you give the AI less information.

Scenario: You tell the AI, "Build a house," but you don't tell it how many rooms or what the roof looks like.
Result: The AI gets really good at guessing the rooms (it still draws a kitchen and a bedroom), but the connections fall apart completely. The rooms become isolated islands with no roads between them.
Lesson: AI needs explicit instructions on how things connect. It cannot "guess" the complex relationships of a system just by reading a vague description.

5. The Conclusion: What's Next?

The paper concludes that while AI is a powerful tool, it is not yet ready to replace the human architect for complex systems.

It can draft the list of rooms (components) quickly.
But it struggles to design the plumbing and wiring (relationships and data flow).
Specialized "Coding" AIs are slightly better at this than general chatbots, but they still make mistakes.

The Takeaway: Think of AI as a very fast, very knowledgeable draftsman who can draw the walls instantly. But you still need a human architect to look at the drawing and say, "Wait, the kitchen is on the roof, and the door leads to a void. Fix the connections."

The R2ABench is now available for anyone to use to test their own AI architects, ensuring we don't build software on shaky foundations.

1. Problem Statement

The automatic generation of software architecture diagrams from unstructured Product Requirements Documents (PRDs) is a critical but challenging task in software engineering. While Large Language Models (LLMs) have shown promise in code generation, their ability to perform high-level architectural reasoning remains underexplored.

Data Gap: Existing benchmarks focus on code implementation or use artificially curated, concise natural language inputs that do not reflect the complexity, ambiguity, and incompleteness of real-world PRDs.
Evaluation Gap: Traditional text-based metrics (e.g., BLEU, ROUGE) fail to capture structural validity. Relying solely on "LLM-as-a-Judge" introduces bias and hallucination risks, making it difficult to rigorously assess complex architectural topologies.
Core Challenge: LLMs struggle to translate static requirement text into dynamic, structurally coherent architectural representations (e.g., PlantUML), particularly regarding relational reasoning between components.

2. Methodology

The authors propose R2ABench (Requirement-To-Architecture Benchmark) and a Hybrid Evaluation Framework to address these gaps.

A. Dataset Construction (R2ABench)

Source: 17 real-world software projects (9 Python, 8 Java) from a university master's course, ensuring high-quality, verified requirements.
Inputs: Comprehensive PRDs structured into six sections: System Introduction, Core Objectives, Functional Features, Technical Constraints, Non-functional Requirements, and System Architecture Description.
Ground Truth: Expert-curated PlantUML code representing the correct architecture, verified by professionals with 5+ years of experience.
Ablation Settings: To test reasoning capabilities, inputs are manipulated into three levels:
- Full: Complete PRD.
- -Arch: Architecture description removed (testing inference from functional constraints).
- Min: Only Core Objectives and Functional Features (testing extreme sparsity).

B. Hybrid Evaluation Framework

The framework evaluates generated diagrams ( $G_p$ ) against reference diagrams ( $G_t$ ) across three complementary layers:

Structural Graph Metrics:
- Syntactic Validity (SV): Percentage of code that compiles successfully.
- Semantic Validity: Node F1 (component recognition), Edge F1 (relationship accuracy), and Layer Accuracy.
- Topological Similarity: Normalized Graph Edit Distance (GED-Accuracy) to measure holistic structural similarity.
Multi-dimensional Scoring (LLM-as-a-Judge):
- Uses a specialized prompt to score diagrams on Completeness, Accuracy (absence of hallucinations), Rationality (adherence to design principles), and Readability.
Architecture Anti-pattern Detection:
- Orphan Ratio: Measures isolated components with no connections.
- God Component Ratio: Detects over-centralized nodes exceeding statistical thresholds ( $\mu + 2\sigma$ ).

C. Experimental Setup

Models Evaluated: State-of-the-art LLMs (GPT-5, Claude-3.5 Sonnet, Gemini-2.5-Pro, DeepSeek-V3.2, Qwen3-Coder) and Agent Frameworks (MetaGPT, OpenHands).
Strategy: Greedy decoding ($temperature=0$) to ensure deterministic outputs.

3. Key Contributions

R2ABench Benchmark: The first benchmark specifically designed for generating architecture diagrams from unstructured, real-world PRDs, featuring diverse projects and expert-verified ground truth.
Hybrid Evaluation Framework: A novel, multi-layered approach combining static graph analysis, semantic scoring, and anti-pattern detection to overcome the limitations of text-based metrics and pure LLM judging.
Comprehensive Empirical Study: An extensive analysis of SOTA models and agent workflows, including ablation studies on PRD completeness and detailed error pattern taxonomy.

4. Key Results & Findings

Performance on Full PRDs (RQ1)

Entity vs. Relationship Imbalance: Models exhibit strong performance in Node Extraction (F1 $\approx$ 0.55) but struggle significantly with Edge/Relationship Generation (Edge F1 < 0.20).
Code-Specialized Models: Models like Qwen3-Coder outperform general-purpose models (e.g., GPT-5) in relationship generation, suggesting that training on structured code aids in modeling dependencies.
Agent Frameworks: Integrating agents (MetaGPT, OpenHands) generally degrades performance. They introduce instability, often reducing syntactic validity and node accuracy while failing to improve structural coherence.

Impact of PRD Completeness (RQ2)

Robust Extraction, Fragile Reasoning: Removing architectural descriptions (the -Arch and Min settings) had a negligible negative impact on entity extraction (Node F1 remained stable or improved).
Structural Fragmentation: However, information sparsity caused a severe drop in architectural cohesion. Models failed to deduce data flows without explicit constraints, leading to high "Orphan Ratios" (isolated components).
Agent Limitations: Even with agents, models could not recover global architectural logic from sparse inputs, often over-fitting to local text fragments.

Error Pattern Analysis (RQ3)

The study identified five primary error categories:

E1 (Omission): Over-abstraction leading to missing fine-grained sub-nodes.
E2 (Hallucination): Injecting unauthorized components based on pre-training priors (e.g., adding a "Log Aggregator" or "Social Graph DB" not in the PRD).
E3 (Boundary Misplacement): Assigning components to incorrect architectural layers (e.g., moving app-layer services to support layers).
E4 (Relational Failure): Inability to infer correct data flows, resulting in collapsed or reversed connections.
E5 (Syntactic Failure): Syntax errors in complex, deeply nested diagrams.

5. Significance

Standardization: R2ABench provides a standardized, reproducible foundation for evaluating LLMs in the domain of software architecture, moving beyond simple code generation benchmarks.
Diagnostic Insight: The study reveals that while LLMs are becoming proficient at "what" (identifying components), they are fundamentally weak at "how" (structuring relationships and flows). This highlights a specific bottleneck in current LLM architectures for complex reasoning tasks.
Practical Guidance: The findings suggest that code-specialized models are currently the best baseline for this task, while agent frameworks may introduce unnecessary complexity and instability for architecture generation. It also emphasizes the critical need for explicit architectural constraints in PRDs to guide LLMs effectively.

Conclusion

The paper concludes that while LLMs can generate syntactically valid and semantically complete architectural diagrams, they fundamentally struggle with relational reasoning and maintaining global structural coherence. The proposed R2ABench and hybrid evaluation framework offer a robust path forward for advancing AI-driven software engineering, particularly in automating the transition from requirements to architecture.