Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench

Technical Summary: Is Agentic AI Ready for Real-World Hardware Engineering? A Deep Dive with Phoenix-bench

Problem Statement
While agentic AI systems have demonstrated significant success in resolving software engineering issues (e.g., on SWE-bench), their applicability to hardware engineering remains an open question. Hardware design differs fundamentally from software: it is a parallel network of instantiated modules connected by signal flow, rather than a sequential program executed along a call graph. Existing hardware benchmarks for Large Language Models (LLMs) typically isolate sub-tasks such as standalone module generation, syntax repair, or localized debugging. None currently require the full hardware design workflow: navigating a repository, performing hierarchy-aware localization, verifying fixes with executable Electronic Design Automation (EDA) tools, and applying maintenance-style patches. Consequently, it is unclear whether agents tuned for the "call-graph world" of software can effectively resolve real-world hardware issues expressed in high-level languages (Verilog/SystemVerilog) and pass EDA tests.

Methodology: The Phoenix-bench Benchmark
To address this gap, the authors introduce Phoenix-bench, a synchronized corpus designed to evaluate agentic AI on repository-level hardware issue resolution.

• Dataset Construction: The benchmark comprises 511 verified instances derived from 114 GitHub repositories, selected from a crawl of 18,010 issues across 786 repositories. Each instance includes a real GitHub issue, the full repository snapshot, the developer's patch, design-flow labels, and a Docker-pinned EDA environment.
• Verification Protocol: Correctness is determined by execution-grounded verification rather than LLM judgment. Each instance is verified using a three-layer Docker environment containing open-source EDA tools (Verilator, Icarus Verilog, Surelog, Yosys). A task is considered "resolved" only if the agent's patch passes two specific checks:
  1. Fail-to-Pass: The targeted testbench fails on the buggy snapshot but passes after applying the developer patch.
  2. Pass-to-Pass: The agent's patch ensures all previously passing testbenches continue to pass, preventing regressions.
• Evaluation Scope: The authors conducted a uniform evaluation of:
  • Four commercial coding agents (Claude Code, OpenAI Codex, Gemini CLI, GitHub Copilot).
  • Eight open-source agentic structures (including SWE-agent, OpenHands, Agentless, and retrieval-augmented systems) across four LLM backbones (GPT-5.2, Gemini-3-Pro, DeepSeek-V3.2, Qwen3-Coder-480B).
  • Two diagnostic interventions: providing a "file-level oracle" (the exact list of files to edit) and providing one round of testbench-log feedback.

Key Contributions

1. Phoenix-bench: A hardware-issue-resolution benchmark featuring 511 verified Verilator instances paired with developer patches, design-flow labels, and executable EDA verification. This ensures that resolution rates reflect agent behavior rather than toolchain availability.
2. Comprehensive Agent Evaluation: A systematic comparison of state-of-the-art commercial and open-source agents on hardware tasks, revealing significant performance gaps compared to software benchmarks.
3. Actionable Insights: An analysis identifying specific failure modes in hardware engineering and demonstrating that localization granularity and feedback mechanisms are more critical than raw model strength.

Key Results and Findings
The study yields three primary findings regarding the transferability of software agents to hardware domains:

1. Fundamental Domain Mismatch: Software and hardware are fundamentally different engineering tasks. Agents tuned for software suffer a performance drop of 37% to 58% when moving from SWE-bench Verified to Phoenix-bench.

   • Cause: Hardware bugs propagate across parallel instantiated modules through signal flow, not along a sequential call graph. Software-tuned agents tend to stop at the "symptom file" (where the error manifests) rather than tracing back through the instantiation chain to the root cause.
   • Evidence: An agent achieving 69.6% on SWE-bench dropped to 32.3% on Phoenix-bench.

2. Concentration of Failures: Failures are not uniform but concentrate in specific hardware-centric categories:

   • Design Control-Flow/FSM Bugs: Issues involving finite state machines and control logic.
   • Verification Testbench Bugs: Errors in the testbenches themselves.
   • Cross-Hierarchy Complexity: Cases requiring signal-flow tracking across multiple files and modules (T3 difficulty).
   • Patch Complexity: The median Phoenix-bench patch touches 4 files and 65 lines (73.1% multi-file), compared to the SWE-bench median of 1 file and 7 lines.

3. Localization Granularity vs. Feedback:

   • File-Level Oracle: Providing agents with the exact list of files to edit yields a negligible improvement of only +1.4%. Agents often break files that did not need editing because they lack the context of what to change (e.g., specific signal widths, operators, or directions).
   • Testbench Feedback: Providing a single round of Verilator error logs (indicating the offending line, operator, and module) increases the resolved rate by 42% to 45%. The logs provide the necessary context for the fix ("where the bug is and what the fix has to look like"), effectively guiding the agent's repair strategy.

Significance and Claims
The paper claims that the current generation of software-focused agentic AI is not yet ready for real-world hardware engineering without significant architectural adjustments. The primary barrier is not the underlying LLM's capability but the agent's inability to navigate the parallel, signal-flow nature of hardware designs.

The authors argue that future hardware AI agents must prioritize:

• Hierarchy-Aware Localization: The ability to trace signal propagation across module instantiation chains rather than relying on lexical file matching.
• Feedback-Driven Repair: Leveraging executable EDA logs to guide edits, as this provides more actionable information than static file lists.
• Cross-File Coordination: Handling complex, multi-file patches that maintain signal integrity across the design hierarchy.

The work establishes that while software agents can partially transfer to hardware, they require specific adaptations to handle the unique constraints of circuit design, verification, and signal flow.