Technical Summary: RTL-BenchMT

Problem Statement

The field of Electronic Design Automation (EDA) is undergoing a significant transformation with the integration of Large Language Models (LLMs) for automated Register-Transfer Level (RTL) generation. While current benchmarks are essential for measuring functional correctness and comparing model capabilities, this paper identifies two critical, systemic challenges that undermine their reliability:

1. Flawed Benchmark Cases: Existing benchmarks inevitably contain errors where the design description is inconsistent with the reference testbench, or critical implementation details are omitted. These flaws cause otherwise correct LLM-generated designs to be marked as failures, leading to unfair evaluations. Systematically identifying and fixing these issues manually is labor-intensive and requires deep RTL and verification expertise.
2. Overfitting: Public benchmarks are susceptible to overfitting, where LLMs achieve high performance by memorizing training examples or exploiting superficial patterns in descriptions rather than demonstrating genuine design capability. Currently, there is no practical framework to automatically detect and quantify this overfitting.

Methodology: RTL-BenchMT Framework

To address these challenges, the authors propose RTL-BenchMT, an agentic framework designed for the dynamic maintenance of RTL generation benchmarks. The system employs a multi-agent architecture that orchestrates specialized agents through three automated processes, supported by a human-in-the-loop for final approval.

1. Framework Architecture

The framework operates within two distinct environments:

• Agent Environment: A lightweight host where LLM agents and orchestration scripts run.
• EDA Environment: An isolated Docker container running commercial EDA tools (e.g., VCS, Icarus Verilog) on a license server.
  Agents interact with these environments via specific interfaces (Agent-Benchmark and Agent-Tool) to navigate files, edit descriptions, and execute compilation/simulation commands.

2. Core Processes

The framework utilizes an iterative reasoning paradigm (Thought-Action-Observation) across three main processes:

• Process 1: Failure Analysis

  • Goal: Identify flawed benchmark cases.
  • Mechanism: A Failure Analysis Agent inspects logs from standardized evaluations across diverse LLMs. It aggregates consistently failing cases and employs a three-step reasoning template:
    1. Code Mismatch Analysis: Compares generated code against reference code.
    2. Code Correctness Analysis: Verifies if the generated code correctly implements the design description (isolating LLM capability defects).
    3. Description Mismatch Detection: If the code is correct per the description but fails the testbench, the agent concludes the design description itself is flawed (e.g., inconsistent logic or missing module names).

• Process 2: Benchmark Revision

  • Goal: Automatically revise identified flawed cases.
  • Mechanism: A Description Revision Agent proposes candidate revisions to the design description. A Description Review Agent then validates these revisions against strict rules:
    1. The revision must not destroy basic semantics (though minor functional alignment with the testbench is permitted).
    2. The revision must not leak information from the testbench (e.g., copying reference code snippets) to prevent shortcutting.
  • Human engineers review and approve the final updates.

• Process 3: Overfitting Detection

  • Goal: Detect and update overfitting cases.
  • Mechanism: A Description Update Agent rewrites design descriptions while preserving their original semantics and functionality. This is achieved by shifting the description style (e.g., from "Technical/Formal" to "Educational/Tutorial" or "Problem/Task Solving").
  • Detection Logic: A model is flagged as overfitting if it passes the original description but fails the semantically equivalent, rewritten description. This provides an automatic signal for overfitting strength and increases benchmark diversity.

Key Contributions

1. RTL-BenchMT Framework: The first agentic framework capable of dynamically maintaining RTL benchmarks by automating the identification of flaws and the detection of overfitting.
2. Automated Flaw Identification: A systematic approach to pinpointing inconsistencies between design descriptions and testbenches, categorizing flaws into Syntax Ambiguity (e.g., undefined module names, port types), Functional Behavior Ambiguity (e.g., missing register initialization, trigger conditions), and Diagram Ambiguity (e.g., K-Map or FSM misinterpretations).
3. Overfitting Quantification: A method to automatically detect overfitting by evaluating model performance on semantically equivalent but stylistically varied descriptions.
4. Refined Benchmark Suite: The authors have applied RTL-BenchMT to analyze existing benchmarks (including VerilogEval, RTLLM, and CVDP), producing a refined, open-sourced benchmark suite that corrects identified flaws.

Results

The paper presents quantitative and qualitative analyses of the framework's impact:

• Flawed Case Statistics: Analysis reveals that approximately 5–10% of tasks across various benchmarks are ambiguous or flawed. For instance, in the CVDP benchmark, up to 36.4% of tasks showed consistent failure across LLMs, with a significant portion confirmed as flawed cases.
• Performance Re-evaluation: After applying RTL-BenchMT fixes:
  • GPT-4o showed significant improvements in functional accuracy (e.g., +3.8% on VerilogHuman v1, +3.5% on VerilogMachine v1), suggesting its prior performance was underestimated due to benchmark flaws.
  • Claude-3.7 showed minor variance, indicating its performance was less affected by the specific flaws identified.
• Overfitting Detection:
  • Models fine-tuned on RTL tasks, such as CodeV, exhibited the most severe performance degradation (9.6% drop) when evaluated on rewritten descriptions, indicating high overfitting.
  • DeepSeek also showed notable overfitting (FN rate), while open-source foundation models like LlaMA demonstrated greater robustness and even performance improvements on rewritten specs.
  • The framework successfully identified that models passing original specs but failing rewritten ones (False Negatives in the overfitting matrix) are strong indicators of overfitting.

Significance

The paper argues that the reliability of LLM-based RTL generation research is currently compromised by static, flawed benchmarks and undetected overfitting. RTL-BenchMT addresses these issues by shifting benchmark maintenance from a purely manual, static effort to an automated, dynamic process.

The significance of this work lies in:

1. Fairness: It enables a more accurate and fair evaluation of LLM capabilities by removing artificial barriers caused by benchmark errors.
2. Robustness: It provides a mechanism to distinguish between models that truly understand RTL design and those that merely memorize benchmark patterns.
3. Sustainability: It offers a scalable solution for the long-term maintenance of EDA benchmarks, reducing the human engineering cost required to keep evaluation suites relevant and error-free.

The authors conclude that a non-trivial fraction of LLM failures in current literature originates from benchmark flaws rather than model limitations, and they commit to open-sourcing both the refined benchmarks and the agentic maintenance framework to the community.