🤖 AI

RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Models

This paper introduces RTL-BenchLS, a large-scale benchmark featuring over 10,000 formally verified Verilog designs and three novel self-supervised reasoning tasks that overcome the scalability limitations of existing benchmarks to rigorously evaluate and guide the development of LLMs for hardware design automation.

Original authors: Jing Wang, Shang Liu, Wenji Fang, Yuchao Wu, Yugao Zhu, Zhiyao Xie

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jing Wang, Shang Liu, Wenji Fang, Yuchao Wu, Yugao Zhu, Zhiyao Xie

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a brilliant but inexperienced apprentice how to build complex electronic circuits (called RTL designs) using only natural language instructions. To see if the apprentice is actually learning, you need a test.

For a long time, the tests used to evaluate these AI "apprentices" (Large Language Models or LLMs) were like kindergarten coloring books. They were small, simple, and the AI could easily get a perfect score just by memorizing the patterns. The paper argues that we need a real-world construction site to see if these AIs can actually do the job.

Here is the story of RTL-BenchLS, the new, much harder test introduced in the paper.

1. The Problem: The "Kindergarten" Tests

Existing tests were too easy. They had:

Too few designs: Like having only 50 tiny puzzles instead of thousands.
Too simple: Like asking the AI to build a single Lego brick instead of a whole castle.
One trick: They mostly just asked the AI to translate a sentence into code. Once the AI got good at that, the test stopped being useful because the AI was "saturating" (getting 100% on everything).

Furthermore, making a harder test was thought to be impossible. Why? Because to make a hard test, you usually need a human expert to write the "answer key" (a testbench) for every single puzzle. There aren't enough human experts to do that for 10,000 complex circuits.

2. The Solution: A Self-Checking "Magic Mirror"

The researchers built RTL-BenchLS, a massive library of over 10,000 verified circuit designs. These aren't tiny bricks; they are complex structures, some with nearly 500 lines of code.

To solve the "answer key" problem without hiring thousands of humans, they invented three new types of puzzles that act like a magic mirror. The AI has to prove it understands the circuit by doing things that must work if it truly understands the logic, rather than just guessing the right answer.

The Three New Puzzles:

Puzzle 1: The "Summarize and Rebuild" Challenge (Round-Trip Reasoning)

The Setup: You give the AI a complex circuit diagram.
The Task: The AI must first write a summary of how it works (like a recipe), and then, using only that summary, rebuild the exact same circuit from scratch.
The Catch: If the summary misses even one tiny detail, the rebuilt circuit will be different. The system checks if the new circuit is identical to the original.
Analogy: It's like asking a chef to taste a complex dish, write down the recipe, and then cook the dish again from that note. If the taste is different, they failed.

Puzzle 2: The "Missing Piece" Challenge (Masked-Content Reasoning)

The Setup: You show the AI a circuit, but you cover up a specific block of code with a "blank" sticker.
The Task: The AI must look at the surrounding code and the description of the blank spot to figure out exactly what was underneath the sticker and fill it in correctly.
Analogy: Imagine a jigsaw puzzle where one piece is hidden. You have to look at the picture around the hole and guess exactly what the missing piece looks like to complete the image.

Puzzle 3: The "Bug Detective" Challenge (Repository-Issue Reasoning)

The Setup: You give the AI a whole folder of code (a project) and a complaint from a user saying, "This part is broken!"
The Task: The AI must find the broken part in the massive code folder and fix it so it works exactly like the "golden" fix a human expert would make.
Analogy: You hand a mechanic a car with a weird noise and a note saying "it sounds like a squeak." The mechanic has to find the exact loose screw in the engine and fix it, without breaking anything else.

3. The Results: The AI is Still a "Novice"

The researchers tested 8 of the smartest AI models on this new, hard benchmark. The results were eye-opening:

On the old, easy tests: The best AIs got nearly 88% correct.
On the new, hard tests:
- Round-Trip (Summarize & Rebuild): The best AI only got ~23% right.
- Missing Piece: The best AI got ~28% right.
- Bug Detective: The best AI got only ~12% right.

What this means: Even the smartest AIs are currently terrible at truly understanding complex hardware logic. They are good at mimicking patterns on simple tasks, but when you ask them to reason through a complex problem, they struggle significantly.

4. Why This Matters

This paper doesn't say AI will never build chips. It says that we have been overestimating how good they are because our tests were too easy.

RTL-BenchLS is like moving the test from a kindergarten classroom to a professional engineering firm. It shows us exactly where the AI is failing (like getting confused by complex logic or missing small details in a bug report) and gives engineers a real tool to measure progress. It leaves a lot of room for improvement, proving that the journey to fully automated hardware design is just beginning.

Technical Summary: RTL-BenchLS

Problem Statement

The field of hardware design automation is increasingly leveraging Large Language Models (LLMs) for Register Transfer Level (RTL) generation and reasoning. However, progress in this domain is hindered by the limitations of existing benchmarks. Current benchmarks, such as VerilogEval and RTLLM, suffer from two primary deficiencies:

Limited Scale and Complexity: Existing datasets contain a small number of designs (typically <200) that are predominantly simple, single-module tasks with fewer than 100 lines of code (LoC). They fail to represent the complexity of real-world hardware subsystems.
Narrow Task Scope and Data Scarcity: Most benchmarks focus exclusively on specification-to-RTL generation. Scaling these benchmarks is fundamentally difficult because they require "aligned" data—specifically, the triad of natural language specifications, RTL code, and manually written testbenches. Such high-quality, aligned data is scarce in the public domain, causing frontier models to saturate performance on current tasks and making it impossible to distinguish between model capabilities or guide further development.

Methodology

To address these limitations, the authors introduce RTL-BenchLS, a large-scale benchmark comprising over 10,000 formally verified Verilog designs. The methodology focuses on two pillars: data construction and novel task design.

Data Construction

The dataset aggregates 10,028 designs from five sources:

Real-World Repositories (S1): 971 modules extracted from 25+ open-source projects (e.g., NVIDIA DLA, OpenC910, HummingBird E200), providing domain-specific complexity.
LLM-Generated Corpora (S2–S5): ~9,000 modules distilled from existing training corpora (OriGen, AutoVCoder, RTLPP, MG-Verilog) to provide scale and structural diversity.

The authors employ a rigorous filtering pipeline to ensure quality:

Complexity Filtering: Designs are filtered based on Lines of Code (LoC) and Number of Modules (NoM), resulting in an average of 63 LoC (max 479 LoC), significantly larger than prior benchmarks.
Similarity Pruning: Pairwise similarity (using TF-IDF and cosine similarity) is calculated to remove near-duplicates.
Formal Verification: Every retained design undergoes formal logic equivalence checking (LEC) or sequential equivalence checking (SEC) using industrial tools (Cadence Conformal and JasperGold). This ensures all designs are functionally correct without relying on manually curated testbenches.

Novel Benchmark Tasks

RTL-BenchLS introduces three tasks that jointly evaluate reasoning and generation, moving beyond simple specification-to-RTL generation:

Round-Trip Reasoning (Self-Supervised): The LLM must compress an original RTL design $D$ into an intermediate abstract $A$ (in formats like Natural Language, Diagram-style NL, Executable Code, or Free-form) and then reconstruct a functionally equivalent RTL $D'$ from $A$ . Success is defined by $D' \equiv D$ via formal equivalence checking. This eliminates the need for external labels.
Masked-Content Reasoning (Self-Supervised): A code block or sub-module is masked in the design. The LLM must generate a concise description of the masked logic and then reconstruct the missing code based on the surrounding context. Success is verified by checking if the reconstructed design is equivalent to the original.
Repository-Issue Reasoning: The LLM is given a real-world issue report (comment) and the repository context (including the buggy code) and must generate a fix that is functionally equivalent to the developer's golden patch. This task evaluates long-context understanding and cross-artifact debugging.

Key Contributions

Unprecedented Scale: The benchmark provides over 10,000 formally verified designs, two orders of magnitude larger than benchmarks supporting functional verification.
Diverse Complexity: Designs span a wider range of complexity (up to 495 LoC) compared to the predominantly simple modules in existing benchmarks.
Self-Supervised Paradigm: Two of the three tasks are self-supervised, resolving the data alignment bottleneck that has historically prevented benchmark scaling.
Formal Verification Infrastructure: The evaluation framework relies entirely on formal equivalence checking (LEC/SEC), removing the dependency on manual testbenches and ensuring rigorous, automated correctness verification.
Comprehensive Evaluation: The paper benchmarks eight state-of-the-art LLMs (including GPT-4o, Claude Sonnet/Haiku, DeepSeek, Qwen, and LLaMA) across all tasks.

Experimental Results

The evaluation reveals that current LLMs face significant challenges in RTL reasoning and generation, with performance far below the saturation levels seen in simpler benchmarks:

Round-Trip Reasoning: Even the best model (Claude Sonnet-4.5) achieves only ~23% pass rate on natural-language round-trip reasoning. Performance varies significantly by abstract format; executable code (Python/C) abstracts perform better (~39–52%) than pure natural language.
Masked-Content Reasoning: Performance is more uniform across models, with the best (LLaMA-3.3-70B) reaching ~28%.
Repository-Issue Fixing: This is the most challenging task. Even with a simplified setting where the buggy region is provided, the best model (DeepSeek-v3.2) achieves only ~12%.
Comparison: When the same models are tested on existing benchmarks (VerilogEval, RTLLM) using the new round-trip task, performance drops sharply (e.g., LLaMA-3.3 drops from 53.2% to 12.2%), confirming that RTL-BenchLS is substantially more challenging.

Significance and Claims

The authors claim that RTL-BenchLS represents a critical step forward for the hardware design automation community. By providing a large-scale, formally verified dataset with self-supervised tasks, the benchmark:

Breaks the Scaling Bottleneck: It demonstrates that effective task design can overcome the scarcity of aligned labels, enabling the creation of much larger and more diverse benchmarks.
Reveals True Capability Gaps: The low pass rates (12–28%) compared to near-saturation on older benchmarks indicate that current LLMs lack the necessary reasoning depth for complex, real-world hardware design tasks.
Guides Future Development: The results highlight specific areas for improvement, such as the need for better handling of long-context reasoning, precise bit-level logic, and the integration of formal verification into LLM agents.

The paper concludes that RTL-BenchLS leaves ample room for future improvement and serves as a foundational infrastructure for developing more capable LLM-based methods for hardware design. The dataset, evaluation scripts, and verification tools are made publicly available to support reproducible research.