🤖 AI

Structured Testbench Generation for LLM-Driven HDL Design and Verification-Oriented Data Curation

The paper introduces STG, a structured framework that leverages hardware design inherent structure to generate deterministic testbenches, significantly outperforming existing LLM-based approaches in speed, reliability, and coverage while serving as an efficient tool for data curation and test-time scaling in RTL workflows.

Original authors: En-Ming Huang, Yu-Hung Kao, Ren-Hao Deng, Wei-Po Hsin, Yao-Ting Hsieh, Cheng Liang, Hsiang-Yu Tsou, Mu-Chi Chen, Yu-Kai Hung, Shao-Chun Ho, Po-Hsuang Huang, Shih-Hao Hung, H. T. Kung

Published 2026-06-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: En-Ming Huang, Yu-Hung Kao, Ren-Hao Deng, Wei-Po Hsin, Yao-Ting Hsieh, Cheng Liang, Hsiang-Yu Tsou, Mu-Chi Chen, Yu-Kai Hung, Shao-Chun Ho, Po-Hsuang Huang, Shih-Hao Hung, H. T. Kung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master architect (an AI) who can instantly draw blueprints for new houses (computer chips) based on a simple description like "build a house with three bedrooms." This is great, but there's a huge problem: how do you know if the blueprint is actually safe and works?

Traditionally, to check a blueprint, you'd have to hire a team of human inspectors to build a full-scale model, run it through every possible storm, and see if it collapses. This takes forever and costs a fortune.

Recently, people tried to use another AI to write the inspection report. But this new AI is like a nervous, hallucinating intern. It guesses the test, sometimes makes up rules, and often says "Everything is fine!" when the house is actually on fire. It's slow, expensive, and unreliable.

Enter STG (Structured Testbench Generation).

The authors of this paper built a new tool called STG. Instead of asking an AI to "imagine" a test, STG acts like a smart, automated assembly line that builds a perfect test based on the actual structure of the blueprint.

Here is how it works, using simple analogies:

1. The "Template" Approach (No More Guessing)

Imagine you have a set of Lego instructions.

The Old Way (LLM): You ask a robot to "write a test." The robot tries to write a story about testing the Lego house. It might forget a step, invent a fake rule, or get confused. It's creative but messy.
The STG Way: STG looks at the Lego house, sees it has a door, a window, and a roof. It then grabs a pre-made, perfect "Inspection Kit" template. It automatically fills in the specific details (like "check the door," "check the window") and runs the test. It doesn't guess; it follows a strict, logical recipe.

2. Three Different "Inspection Kits"

The paper explains that different types of chips need different tests, just like different vehicles need different safety checks:

Combinational Designs (The Static House): These are chips that react instantly to inputs (like a light switch). STG treats these like a combinatorial lock. It tries every single combination of buttons to make sure the light turns on correctly. It's exhaustive and leaves no stone unturned.
General Sequential Designs (The Moving Car): These chips have a clock and memory (like a car engine). STG acts like a driving instructor. It doesn't just check the car at a stoplight; it checks how the car reacts while you are turning the wheel or pressing the gas mid-cycle. It catches "Mealy" bugs (instant reactions) that other tests miss.
FSM-Dominated Designs (The Maze): Some chips are like complex mazes with many states. Random driving won't get you to the exit. STG acts like a GPS with a map. It reads the internal map of the chip and guides the test specifically through every single path in the maze, ensuring no dead end is left unexplored.

3. Why This Matters (The Results)

The paper claims STG is a game-changer for three main reasons:

It's a Speed Demon: STG is 720 times faster than the old AI methods. If the old way took a whole day to check a batch of chips, STG does it in minutes. It's like swapping a snail for a race car.
It's a Truth-Teller: The old AI methods often gave "False Passes" (saying a broken chip was good). STG is much more reliable. It found 7.8% more broken chips that the old methods missed. It also helped fix a hidden bug in the standard testing tools used by the industry (a "race condition" where the test checked the wrong thing at the wrong time).
It Saves Massive Money and Energy: Because STG runs on a simple computer processor (CPU) instead of a massive, power-hungry AI supercomputer (GPU), it uses 127 times less energy. It's the difference between running a factory with a single efficient worker versus a factory with 100 screaming, overworked robots.

4. The "Teacher and Student" Analogy

The paper also uses STG to train smaller, smarter AI models (a process called "distillation").

Imagine a Teacher (a huge AI) trying to teach a Student (a small AI) how to design chips.
The Teacher generates thousands of practice problems.
STG acts as the strict grader. It instantly checks if the Teacher's answers are right or wrong.
Because STG is so fast and cheap, the Teacher can generate way more high-quality practice problems. The Student learns faster and becomes just as good as the Teacher, but without needing the Teacher's massive brainpower.

Summary

In short, STG stops trying to "dream up" tests for computer chips. Instead, it analyzes the blueprint and automatically builds a perfect, structured test based on the chip's actual design. It is faster, cheaper, more accurate, and more reliable than the current AI methods, making it a vital tool for the future of automated chip design.

Technical Summary: Structured Testbench Generation for LLM-Driven HDL Design and Verification-Oriented Data Curation

Problem Definition

The paper addresses a critical bottleneck in Large Language Model (LLM)-driven Register Transfer Level (RTL) workflows: the generation of reliable, high-coverage testbenches for verifying generated hardware designs. Existing approaches treat testbench generation as unconstrained code synthesis, relying on stochastic LLM prompting. This leads to three primary limitations:

High Cost and Low Reproducibility: Improving reliability requires iterative prompting or ensemble generation, significantly increasing token costs.
Insufficient Coverage: Stochastic generation often misses corner cases, particularly in control-flow and state-dependent behaviors.
Unreliable Verdicts: Inconsistent testbenches produce false-pass verdicts on incorrect Designs Under Test (DUTs) and fail to distinguish between partially correct and wholly incorrect designs.

These issues are exacerbated by emerging LLM-driven trends such as test-time scaling (where verification is part of an iterative search loop) and verification-oriented data curation (where large batches of candidate DUTs must be filtered for training data). In these contexts, verification is no longer just a downstream step but a core mechanism for model improvement, demanding low-cost, deterministic, and behaviorally meaningful testbenches.

Methodology: The STG Framework

The authors propose STG (Structured Testbench Generation), a framework that shifts testbench creation from unconstrained code synthesis to a deterministic, structure-aware process. STG exploits the inherent structure of hardware designs to generate testbenches via template-based rendering.

Core Workflow

Module Parsing and Analysis: STG parses the DUT and a trusted golden reference (or a "silver" reference model) using Icarus Verilog and lightweight heuristics.
- Design-Type Classification: Designs are categorized into three types: Combinational, General Sequential, or FSM-dominated.
- Signal Classification: Input ports are classified as Clock, Reset, Control, or Data using Longest Common Subsequence (LCS)-based fuzzy matching against name hints and width heuristics.
- FSM Detection: For sequential designs, STG uses deterministic pattern matching (scanning for always_ff blocks with state registers) or LLM-assisted analysis to extract Finite State Machine (FSM) structures.
Strategy-Specific Testbench Generation: Based on the classification, STG renders parameterized Jinja templates with specific stimulus strategies:
- Combinational: Exhaustive enumeration of all control signal combinations ( $2^{b_c}$ ), with randomized data signals ( $N_s$ samples) for each control vector.
- General Sequential: A two-pass, reset-aware strategy. It drives inputs within the clock period (not just at edges) to capture Mealy-style intra-cycle responses and checks outputs at both clock edges and after intra-cycle changes. It handles reset recovery by probabilistically inserting resets between cycles.
- FSM-Guided: For FSM-dominated designs, STG extracts a state-transition graph and performs a Depth-First Search (DFS) traversal. It separates drivable input conditions from internal wait conditions, driving inputs deterministically to satisfy transition guards, ensuring full transition coverage without random exploration.
Silver-Reference Extension: While primarily designed for known-reference settings (DUT vs. Golden RTL), STG can extend to scenarios without a trusted golden reference by replacing the golden module with an LLM-generated C++/SystemC "silver" reference model.

Key Contributions

STG Framework: A deterministic, structure-aware testbench generation framework that outperforms prompt-based LLM generation in efficiency, coverage, and reliability.
Verification-Oriented Data Curation: Demonstration that STG enables efficient filtering of large-scale training datasets, supporting the training of strong distilled RTL generation models using a simple Supervised Fine-Tuning (SFT) pipeline.
Test-Time Scaling Backend: Evidence that STG serves as an effective verification backend for LLM-guided refinement loops (e.g., MCTS), improving search quality and efficiency across multiple backbone models.

Experimental Results

The framework was evaluated on Verilog generation benchmarks (VerilogEval, RTLLM-v2, CVDP) across three scenarios:

1. Testbench Quality and Efficiency

Speed: STG generates testbenches 720× faster than an iterative LLM-based pipeline (0.13s vs. 92.4s per testbench).
Coverage: STG achieves higher line coverage (95.88% vs. 93.97%) and toggle coverage (95.77% vs. 85.40%).
Reliability: STG reduces false-pass verdicts on incorrect DUTs. In a study of 3,046 DUTs, STG correctly identified 7.8% more incorrect designs than the LLM-based approach, primarily by avoiding false passes on subtle bugs.
FSM Coverage: On deep-state FSM designs, random stimulus failed to reach deep states (S11–S14), whereas STG's FSM-guided traversal achieved 100% transition coverage.

2. Verification-Oriented Data Curation

Resource Efficiency: Filtering 115k problems using STG on a single CPU core took 5.6 hours and consumed 0.56 kWh of energy. In contrast, a pure-LLM approach on a high-end GPU took 59.1 hours and consumed 70.9 kWh (a 127× energy reduction and 10.6× speedup).
Model Performance: Distilled models trained on data curated by STG (using only a single SFT stage) achieved state-of-the-art or competitive performance on VerilogEval, RTLLM-v2, and the challenging CVDP benchmark. These models outperformed or matched complex pipelines involving multi-stage SFT and Reinforcement Learning (RL), suggesting that complex RL workflows may not yield consistent gains on newer benchmarks.

3. Test-Time Scaling

Search Efficiency: When integrated into an MCTS-based refinement loop, STG reduced the number of search nodes required to solve problems by 14–47% across four backbone models (ranging from 7B to 685B parameters).
Signal Quality: The deterministic and informative nature of STG's verification signals allowed the search loop to reject weak candidates earlier and converge on correct designs with fewer iterations.

Additional Findings

Benchmark Correction: STG's deterministic generation and human inspection revealed a systematic race condition in the VerilogEval testbenches (where stimulus and checkers triggered on the same clock edge without ordering guarantees). The authors corrected this, noting that previous evaluations may have been compromised by these false failures.
Reproducibility: The authors noted significant reproducibility issues with the VeriRL method, where their replicated checkpoint performed inconsistently across benchmarks compared to the original paper's claims.

Significance and Claims

The paper claims that STG establishes a practical, low-cost verification backbone for LLM-driven HDL workflows. By treating verification as a structured generation problem rather than unconstrained code synthesis, STG addresses the scalability and reliability issues that currently hinder the adoption of LLMs in hardware design.

The authors argue that the effectiveness of recent complex training and reinforcement learning workflows remains questionable, particularly on newer benchmarks, where their simpler, STG-powered pipeline provides competitive or superior performance with significantly lower computational overhead. They position STG as a critical enabler for continuous learning cycles in hardware design, where efficient and reliable data curation is essential for iterative model retraining.