Technical Summary: DUET – Agentic Design Understanding via Experimentation and Testing

Problem Statement

While AI agents powered by Large Language Models (LLMs) have achieved human-level performance in software engineering tasks, they struggle significantly with hardware design challenges, particularly at the Register Transfer Level (RTL). The core difficulty lies in the nature of RTL code (e.g., SystemVerilog), which encodes complex, dynamic, and time-evolving behaviors using low-level, heavily implicit syntax. Unlike imperative software languages where sequential lines often correspond to temporal execution or where function names explicitly denote control flow, RTL often disperses state machine logic across hundreds of lines with non-explicit control flow. Consequently, LLMs given only static RTL code fail to infer deep behavioral semantics, leading to poor performance in downstream tasks such as code completion, documentation, and, most critically, formal verification and debugging.

Current hardware designers do not rely solely on reading RTL; they dynamically build understanding through iterative experimentation using simulation, waveform inspection, and formal tools. Existing AI workflows, however, typically treat LLMs as one-shot generators or static analyzers, lacking the mechanism to replicate this dynamic, tool-driven learning process.

Methodology: DUET

The authors propose DUET (Design Understanding via Experimentation and Testing), a general methodology that equips AI agents with the ability to develop design understanding through iterative experimentation, mimicking the workflow of human hardware experts.

Core Mechanism

DUET operates via a subroutine, DOEXPERIMENTATION, which creates a closed-loop experimentation cycle:

1. Hypothesis Generation: The agent generates a hypothesis about the design's behavior based on the current context.
2. Tool Execution: The agent selects and executes specific Electronic Design Automation (EDA) tools to test the hypothesis.
3. Result Analysis: The agent analyzes the output (logs, waveforms, counterexamples) to confirm, refute, or refine the hypothesis.
4. Iteration: This loop continues until the agent synthesizes a robust "experiment report" that integrates findings into a deeper understanding of the design.

Tool Integration

DUET is designed to be tool-agnostic but has been implemented with the following specific EDA tools:

• Simulation: Agents generate SystemVerilog testbenches and run them (e.g., via Verilator) to observe logs. The methodology encourages the use of print statements over assertions to avoid premature testbench termination, allowing for richer data collection.
• Formal Verification: Tools like Jasper are used to prove properties or check reachability.
• Waveform Viewers: Agents use utilities (e.g., vcdcat) to inspect specific portions of simulation waveforms.
• Counterexample Replication: A specialized agent tool that takes a formal counterexample and converts it into a simulation testbench to replicate the failure. This tool proved particularly effective, often requiring multiple simulation iterations to successfully reproduce a counterexample, thereby deepening the agent's understanding of timing and state transitions.

Implementation Details

The framework allows the agent full control over the experimentation loop, deciding when to run tests and when to conclude. The system is implemented using modern agentic programming paradigms (e.g., OpenAI Agents SDK), where the loop is managed internally based on prompts and available tools. The authors note that careful prompting is required to prevent agents from "cheating" (e.g., forcing internal signals to trivially replicate counterexamples) or hallucinating results.

Evaluation and Results

The authors evaluated DUET by integrating it into an existing AI-assisted formal verification workflow and comparing it against a baseline flow that lacked experimentation tools.

• Experimental Setup:

  • Task: Formal verification of a simple round-robin arbiter design.
  • Scope: 10 distinct properties to verify.
  • Models: GPT-5.0 for LLM queries; Jasper for formal verification; Verilator for simulation.
  • Baseline: An agent that summarizes RTL, generates properties, creates formal testbenches, and iterates on them using Jasper feedback without external experimentation tools.
  • DUET Flow: The same workflow, but with explicit instructions to use experimentation (simulation, counterexample replication, formal tools) when debugging failures.

• Key Findings:

  • Performance Improvement: The DUET-enabled flow successfully proved 6 out of 10 properties, doubling the success rate of the baseline, which proved only 3.
  • Qualitative Improvements:
    • Property 1 & 6: DUET proved properties that the baseline could not prove at all.
    • Property 7: The baseline proved the property but it was "vacuous" (trivially true). DUET used experimentation to find a non-vacuous refinement and prove the meaningful version.
    • Property 2: The baseline failed to prove the property. DUET refined the property through experimentation, uncovering a path toward progress even if the final proof was not achieved in the trial.
  • Tool Usage Analysis:
    • Counterexample Replication: This was the most critical tool. Every improved property utilized this tool. In the case of Property 1, the agent replicated seven separate counterexamples to understand the timing behavior.
    • Simulation: Heavily relied upon, suggesting LLMs are better at crafting simulation testbenches than formal ones.
    • Formal Tools: Used sparingly, likely due to prompt engineering challenges or confusion between the "formal tool" and the standard verification loop.

Significance and Claims

The paper claims that DUET represents a shift from static code analysis to dynamic, iterative design understanding. Its primary significance lies in demonstrating that AI agents can learn deep, temporal behaviors of hardware designs not by reading syntax alone, but through "simple trial-and-error experimentation."

The authors emphasize that:

1. Deep Understanding is Achievable: Despite the complexity of RTL, agents can infer complex behaviors (like clock stretching in I2C or specific timing in arbiters) by generating hypotheses and testing them against EDA tools.
2. General Applicability: While evaluated on formal verification, DUET is a general tactic applicable to any AI workflow over code where experimentation tools exist (e.g., documentation generation, debugging).
3. Limitations: The paper acknowledges that agents still require careful prompting to avoid "cheating" or hallucinating. It also notes that the current implementation relies heavily on simulation, and formal verification tools could be utilized more effectively with better prompt engineering.

The work concludes that while LLMs are powerful, they fail in hardware domains without the ability to interact with the dynamic state of design execution. DUET provides the necessary framework to bridge this gap, enabling agents to replicate the iterative, tool-driven learning process of human experts.