RuC: HDL-Agnostic Rule Completion Benchmark Generation

1. Problem Statement

While Large Language Models (LLMs) have shown promise in Electronic Design Automation (EDA), existing benchmarks for evaluating their code completion capabilities in Hardware Description Languages (HDLs) suffer from significant limitations:

• Lack of Granularity Control: Current benchmarks generally fall into two categories: Module Completion (MC), which masks an entire module (often too complex and context-heavy to reconstruct without specs), and Single-Line Completion (SLC), which masks a random line.
• Arbitrary Units: SLC relies on "lines" as a unit of masking, which is a formatting artifact rather than a semantically meaningful HDL construct. A random line might be trivial or syntactically incomplete, failing to test specific language features.
• Evaluation Rigor: Many benchmarks rely on exact-match metrics or simple simulation, which fail to capture partial correctness or rigorous functional equivalence in hardware designs.

The authors argue that there is a need for a framework that allows for arbitrary, semantically meaningful granularity (from a single assignment to a full logic block) and uses grammar-driven masking to systematically evaluate LLM understanding of specific HDL constructs.

2. Methodology: The RuC Framework

The authors propose RuC (Rule-based Completion), a grammar-driven, language-agnostic framework for generating rule-completion benchmarks.

A. Core Concept: Grammar-Driven Masking

Instead of masking random lines or whole modules, RuC parses HDL source code into a Parse Tree based on the language's formal grammar (e.g., SystemVerilog).

• Rule Selection: Users select specific grammar rules (non-terminals) to mask, such as port_declaration, continuous_assignment, case_statement, or always_construct.
• Granularity Control: By choosing different rules, the difficulty of the task can be tuned. For example, masking a simple_identifier is easier than masking an entire always_construct.
• HDL-Agnostic: The framework works for any language provided a grammar definition (e.g., via ANTLR) is available.

B. Task Construction Pipeline

The pipeline consists of three stages:

1. Preprocessing: Resolves compiler directives and merges multi-file projects into a unified source to provide complete context.
2. Parsing & Rule Sampling: Parses the source to generate a parse tree, identifies occurrences of selected grammar rules, and masks them. It filters out dead code or constructs removed during elaboration.
3. Context Budgeting: Addresses LLM context window limits. It extracts module dependencies (instantiations and imports) and prunes the context to include only relevant dependencies, allowing users to trade context size for structural information.

C. Prompting Strategies

RuC supports two prompting paradigms to align with different model training strategies:

• Chat-based: Replaces the masked region with a <MASK> token and provides surrounding context with an instruction.
• Fill-in-the-Middle (FIM): Divides the code into Prefix, Middle (masked), and Suffix. The model is prompted with Prefix + Suffix + <MIDDLE_TOKEN> to generate the missing middle. This aligns with the pre-training objectives of many coding LLMs.

D. Verification Pipeline

To ensure rigorous evaluation, RuC employs a two-step verification process:

1. Syntactic Validation (STX): Re-inserts the generated code and lints it (using Verilator) to ensure it compiles and adheres to language rules.
2. Functional Equivalence (EQV): Uses formal verification (SAT solvers via Yosys) to prove that the LLM-generated design is functionally equivalent to the original ground truth. It constructs a "miter circuit" and checks if any input sequence can produce different outputs.

3. Key Contributions

• RuC Framework: A novel, automated system for generating HDL code-completion benchmarks with controllable granularity based on formal grammar rules.
• HDL-Agnostic Design: The approach is applicable to any hardware description language with a defined grammar, not just SystemVerilog.
• Rigorous Evaluation: Moves beyond simple text similarity metrics to include formal functional equivalence checking, ensuring the generated code actually works as intended.
• Benchmark Generation: Created two new SystemVerilog benchmarks derived from the Tiny Tapeout TT07 shuttle and the CVE2 RISC-V core, covering a wide range of design complexities.

4. Experimental Results

The authors evaluated five open-source LLMs (including Qwen2.5/3 Coder, Seed Coder, and DeepSeek) on the generated benchmarks.

• Prompting Strategy: FIM prompting consistently outperformed Chat-based prompting. This suggests that the "infilling" objective is critical for HDL tasks where context is bidirectional.
• Model Variants: Surprisingly, Base models (pre-trained only) often outperformed their Instruction-tuned counterparts in rule completion. This indicates that instruction tuning, while helpful for general coding, may degrade the specific next-token prediction capabilities required for precise code infilling.
• Rule Difficulty: Performance varied significantly based on the grammar rule:
  • Easiest: PORT (port declarations) and PARAM (parameters), as these can often be inferred from interface usage.
  • Moderate: Assignment rules (CONT, BLK, NBLK).
  • Hardest: Control flow rules (COND, CASE, ALWAYS), which require reconstructing complex behavioral logic.
• Model Size vs. Performance: While larger models generally performed better, the correlation was not absolute. Smaller models (e.g., Seed Coder 8B) sometimes outperformed larger ones on specific rules, suggesting that rule-specific model selection is more effective than a one-size-fits-all approach.
• Codebase Influence: Performance was influenced by stylistic regularities in the codebase (e.g., non-blocking assignments in CVE2 followed a highly regular pattern, boosting scores).

5. Significance

• Granular Evaluation: RuC provides the first framework capable of isolating and testing an LLM's understanding of specific HDL constructs, moving beyond the "all-or-nothing" evaluation of whole modules or random lines.
• Scalability: By automating the generation of tasks from real-world open-source designs, it enables scalable benchmarking that was previously limited by manual dataset curation.
• EDA Workflow Integration: The findings highlight that FIM prompting and base models are currently superior for RTL copilot-style assistance, guiding the development of future EDA tools.
• Future Research: The framework opens the door for studying how LLMs handle varying context lengths and specific logic patterns, which is essential for building reliable AI assistants for hardware design.

In conclusion, RuC establishes a new standard for evaluating LLMs in hardware design by combining formal grammar analysis with rigorous functional verification, revealing that current models struggle with complex control logic but excel at structural reconstruction when prompted correctly.