Technical Summary: QUARCH: A Benchmark for Evaluating LLM Reasoning in Computer Architecture

Problem Statement

The field of computer architecture, which bridges high-level software abstractions and low-level hardware implementations, has remained absent from current Large Language Model (LLM) evaluations. While existing benchmarks in computing systems target technical implementation tasks—such as code generation, Register-Transfer Level (RTL) synthesis, System-on-Chip (SoC) integration, and chip verification—they primarily assess a model's ability to produce programmatic artifacts. They fail to evaluate whether a model can reason about the underlying principles that guide design decisions, such as trade-offs in power, performance, and area (PPA) within a multi-objective optimization space.

Current benchmarks like GSM8K, AIME, and SWE-bench measure reasoning in math, software engineering, and general science, but no dedicated benchmark exists to assess the conceptual and analytical reasoning skills required for computer architecture. This gap hinders the development of AI agents capable of accelerating innovation in computing systems design.

Methodology

Benchmark Construction (QUARCH v1.0)

The authors introduce QUARCH (pronounced 'quark'), the first benchmark designed to facilitate the development and evaluation of LLM knowledge and reasoning capabilities specifically in computer architecture. The benchmark comprises 2,671 expert-validated question-answer (QA) pairs. The construction process employed a three-pronged strategy to ensure high quality and relevance:

1. Synthetic Data Generation: LLMs generated cloze-style multiple-choice questions (MCQs) grounded in a corpus of technical manuals, academic publications, and online resources. These underwent two-stage validation: initial filtering by an LLM-as-a-judge followed by independent review by three human experts.
2. Expert Crowdsourcing & Competitions: A web-based portal was developed to solicit advanced analysis, design, and implementation questions from individuals with technical backgrounds. Submissions were reviewed for ambiguity and correctness.
3. Academic Exams: Questions were curated from university computer architecture exams via manual web scraping and a custom pipeline that extracted text and diagrams, converting them into standalone QA pairs.

Skills Framework

QUARCH evaluates LLMs across four foundational competencies, aligned with how architects reason in practice:

• Recall: Retrieving domain knowledge, definitions, and facts (e.g., identifying components in a diagram).
• Analyze: Deducing, inferring, or interpreting data to reason about workload implications and system behavior (e.g., identifying bottlenecks).
• Design: Proposing or improving architectural features while satisfying constraints and balancing trade-offs (e.g., suggesting a dynamic branch prediction system).
• Implement: Translating a design into executable artifacts (code/RTL/simulation scripts) to validate solutions.

The dataset includes 1,547 MCQs and 1,124 free-response questions (FRQs), with 548 multimodal questions incorporating images (diagrams, schematics, plots) alongside text.

Evaluation Protocol

The authors evaluated 10 frontier models (from Anthropic, DeepSeek, Google, Meta, Mistral, and OpenAI) in a zero-shot setting.

• Grading: For MCQs, correctness is determined by the final choice. For FRQs, an LLM-as-a-judge (specifically CLAUDE-3.7-SONNET-THINKING) was employed to assess correctness against ground truth.
• Validation: The fidelity of the LLM-as-a-judge was validated against human domain experts. The agreement rate between the LLM judge and human experts was 85.48%, comparable to the human-to-human agreement rate of 90.75%.
• Metrics: Performance is reported as "per-generation accuracy" (pass@k=1 estimated over 3 generations).

Key Contributions

1. First Specialized Benchmark: QUARCH v1.0 is the first benchmark explicitly designed to evaluate advanced computer architecture knowledge and reasoning in LLMs, comprising 2,671 expert-validated QAs.
2. Formalized Skills Framework: The paper establishes a framework to systematically assess LLMs on four distinct skills (Recall, Analyze, Design, Implement), revealing that while models may possess domain knowledge, they struggle with higher-order reasoning.
3. Comprehensive Analysis: The authors provide an in-depth analysis of model trends and failure points, including struggles with the architecture-semantics of code execution, assumptions of unconventional architectural properties, difficulties in modeling system state, and heterogeneity in topic expertise.
4. Scalable Evaluation Methodology: The paper demonstrates a trustworthy methodology for evaluating free-form responses using LLM-as-a-judge, showing high agreement with human experts.
5. Knowledge Transfer Demonstration: The authors show that fine-tuning open-source models on QUARCH QAs translates to improved performance on a realistic memory hierarchy design task, resulting in more area-efficient and viable solutions.

Results

Model Performance

• Recall vs. Reasoning: Frontier models demonstrate strong performance on Recall questions (84%–91% accuracy), indicating they possess foundational domain knowledge. However, performance drops significantly on Reasoning skills (Analyze, Design, Implement).
• Performance Gaps: The gap between Recall and Reasoning is substantial. For example, GPT-5.2 achieves 88.4% on Recall but drops to 73.2% on Reasoning. Other models show even wider gaps, with some falling below 35% on higher-order skills despite high recall scores.
• Skill Variance: Implement skills exhibit the widest performance gaps (24%–72%), suggesting that translating design concepts into executable artifacts remains a significant challenge.
• Modality Sensitivity: Multimodal models perform approximately 5% worse on questions containing images compared to text-only questions, indicating difficulties in interpreting architectural diagrams and schematics.

Failure Modes

The analysis identified specific failure modes:

• Code Execution Semantics: Models struggle to understand how high-level code interacts with hardware (e.g., memory access patterns, instruction scheduling).
• Unconventional Assumptions: Models often default to unconventional architectural properties (e.g., word-level addressing) when prompts do not explicitly state standard conventions.
• System State Tracking: Models frequently fail to maintain consistent system state, leading to incorrect inferences about latency and throughput in evolving scenarios.
• Topic Heterogeneity: Expertise varies significantly by topic; a model strong in "Emerging Architectures" may struggle with "IP Design."

Knowledge Transfer Case Study

In a case study involving the design of a memory hierarchy for an ML hardware accelerator:

• Fine-tuning open-source models (GEMMA-3-27B-IT and LLAMA-3.3-70B-INSTRUCT) on QUARCH QAs led to a significant increase in the percentage of trials where models proposed designs satisfying strict area and energy budgets (e.g., increasing success rates from 30% to 70% for Gemma).
• Fine-tuned models discovered designs that were up to 1.99× more area-efficient than those proposed by their base model counterparts.

Significance and Claims

The paper claims that QUARCH provides a foundation for building and measuring LLM capabilities that can accelerate innovation in computing systems. By holistically assessing fundamental skills, QUARCH addresses the critical gap between "knowing" architectural facts and "reasoning" about architectural trade-offs.

The authors emphasize that while frontier models possess domain-specific knowledge, they struggle with the higher-order thinking required for architectural reasoning. The benchmark serves as a diagnostic tool to identify these gaps and a training ground to improve model performance on real-world design tasks. The demonstrated ability of fine-tuned models to generate more efficient hardware designs suggests that QUARCH can effectively translate LLM progress into agentic methodologies for systems design.

The work is modest in its scope, focusing on conceptual reasoning and QA paradigms rather than full agentic toolchains, but argues that isolating and measuring these reasoning capabilities is a necessary precursor to developing robust AI agents for computer architecture. The benchmark and leaderboard are publicly available to foster further research and development in this domain.