ArchBench: Benchmarking Generative-AI for Software Architecture Tasks

1. Problem Statement

Despite the rapid advancement of Large Language Models (LLMs) and Generative AI (GenAI) in software engineering, current benchmarking efforts suffer from a critical gap:

• Implementation Bias: Existing benchmarks (e.g., HumanEval, SWE-bench) overwhelmingly focus on implementation correctness (snippet-level function generation or repository-level issue resolution).
• Lack of Architectural Evaluation: Software architecture tasks (e.g., modularization, dependency management, system evolution) are under-specified and difficult to compare across models. Architectural flaws often compound silently, yet there are no standardized frameworks to evaluate an LLM's ability to reason about these high-level decisions.
• Fragmented Research: A recent literature review indicates that rigorous testing of GenAI outputs in architecture is missing. Studies rely on isolated datasets, custom evaluation setups, and non-standardized metrics, making cross-study comparison and reproducibility nearly impossible.

2. Methodology: The ArchBench Platform

The authors present ArchBench, the first unified platform designed to benchmark LLM capabilities specifically for software architecture tasks. The system is built on a plugin architecture to ensure extensibility and modularity.

Technical Architecture

The platform consists of two main components:

1. Command-Line Interface (CLI) Tool: A Python-based tool that implements a standardized three-stage pipeline:

   • Stage I: Download: Automatically fetches datasets from external sources (GitHub, Zenodo, HuggingFace) and caches them locally. Each task plugin defines its own data loader for format-specific parsing.
   • Stage II: Inference: Assembles prompts using task-specific templates and dispatches them to various LLM providers (OpenAI, Anthropic, etc.) via a uniform interface. Crucially, it logs full interaction trajectories (prompts, raw responses, token usage, latency) to ensure reproducibility.
   • Stage III: Evaluation: Validates predictions and computes task-appropriate metrics. It supports diverse evaluation strategies:
     • NLP Similarity: ROUGE, BLEU, METEOR, BERTScore (for text generation).
     • Set-Based Metrics: Precision, Recall, F1 (for structured outputs like traceability links).
     • Code Metrics: CodeBLEU, test pass rates, complexity analysis (for code generation).
     • LLM-as-Judge: For qualitative assessment.

2. Web Interface: A React-based public dashboard (available at sabench.com) that hosts an interactive leaderboard. It allows users to filter results by task, model, and metrics, view source papers, and access contribution guidelines.

Supported Tasks

ArchBench currently integrates five tasks drawn from recent research, covering a taxonomy of GenAI applications in software architecture:

1. ADR Generation: Generating Architecture Decision Records from design contexts (Metrics: ROUGE, BLEU, BERTScore).
2. Serverless Component Generation: Generating FaaS functions from specifications (Metrics: Test pass rate, CodeBLEU, Complexity).
3. Dynamic Service Generation: Generating IoT service implementations (Metrics: CodeBERTScore).
4. Traceability Link Recovery: Identifying code files that implement specific architectural elements (Metrics: Precision, Recall, F1).
5. Microservice Generation: Generating complete microservice implementations (controllers, services, data models) from requirements (Metrics: Test pass rate, SLOC, Complexity).

3. Key Contributions

The paper makes two primary contributions to the software engineering community:

• A Centralized Benchmarking Platform: ArchBench aggregates datasets from published research into a single, unified infrastructure. It is the first platform to specifically target software architecture tasks, currently supporting five distinct tasks with a design that allows the community to easily contribute new tasks as self-contained plugins.
• A Standardized Evaluation Pipeline: The authors provide a CLI tool that automates the entire workflow: dataset download, LLM inference with full trajectory logging, and automated metric computation. This produces standardized output formats (JSONL, JSON reports) enabling consistent, reproducible comparisons across different models.

4. Results and Current Status

• Initial Deployment: The platform is live with results from multiple LLMs on the five supported tasks.
• Automation Level: Two tasks (ADR Generation and Traceability Link Recovery) currently feature fully automated evaluation pipelines within the CLI, allowing researchers to go from dataset download to a scored report in a single command.
• Community Integration: The platform follows a "pull request" submission model similar to SWE-bench. Researchers submit evaluation reports via GitHub PRs, which are vetted by maintainers before appearing on the public leaderboard.
• Open Source: Both the CLI tool and the web platform are open-source (GitHub) with a persistent DOI on Zenodo, licensed under CC BY 4.0.

5. Significance and Future Directions

• Bridging Research and Practice: ArchBench addresses the lack of shared infrastructure in the software architecture community. It enables researchers to compare approaches directly and helps practitioners identify which LLMs or agents perform best for specific architectural tasks.
• Enabling New Research: The platform facilitates investigations into how architectural reasoning varies across model families, correlations between task performances, and the impact of prompting strategies. The trajectory logging feature allows for fine-grained analysis of failure points in multi-step reasoning.
• Future Roadmap:
  • Integration of additional tasks covering the full taxonomy of architectural GenAI (e.g., architectural smell detection, dependency analysis).
  • Support for agent-based evaluation environments (sandboxed tools for autonomous agents).
  • Incorporation of qualitative evaluation via "LLM-as-judge."
  • Establishment of a regular evaluation cycle to keep the leaderboard current as new models are released.

In summary, ArchBench represents a foundational step toward maturing the evaluation of GenAI in software architecture, moving the field from ad-hoc, isolated studies to a rigorous, community-driven, and reproducible benchmarking ecosystem.