OpenRTLSet: A Fully Open-Source Dataset for Large Language Model-based Verilog Module Design

Technical Summary: OPENRTLSET

Problem Statement
The application of Large Language Models (LLMs) to hardware design, specifically the generation of Verilog code from high-level specifications, is currently hindered by a critical scarcity of large-scale, high-quality, and fully open-source training datasets. Unlike software domains, where abundant open-source repositories exist, Verilog code is often restricted within proprietary industrial domains or lacks the transparency required for reproducible research. Existing datasets, such as MG-Verilog or PyraNet, either suffer from limited scale, restrictive licensing that prevents commercial use, or a lack of standardized, high-quality natural language pairings. This gap prevents the development of domain-specific models capable of generating syntactically and functionally correct hardware designs for both academic and commercial applications.

Methodology
The authors introduce OPENRTLSET, a comprehensive end-to-end workflow for constructing a fully open-source dataset of over 131,000 diverse Verilog modules. The methodology consists of three primary stages:

1. Data Collection and Curation:

   • Sources: The dataset aggregates code from three distinct sources: GitHub repositories containing native Verilog/SystemVerilog (approx. 102k modules), VHDL repositories converted to Verilog (approx. 5k modules), and synthesizable C/C++ repositories converted via High-Level Synthesis (HLS) tools (approx. 24k modules).
   • Licensing: A rigorous filtering process ensures all code is licensed for free use in both academic and commercial settings, removing repositories with restrictive or proprietary licenses.
   • Preprocessing: Multi-module files are decomposed into individual modules. Redundant comments and whitespace are removed, and metadata (module names, I/O pins) is extracted to create a structured JSONL format.

2. Dataset Labeling:

   • Model: The authors utilize DeepSeek-R1 70B, an open-source reasoning-focused LLM, to generate natural language descriptions for each Verilog module.
   • Context Augmentation: A novel aspect of the labeling flow involves using Verilator to generate C++ representations of the Verilog modules. The authors experiment with providing the LLM with both the Verilog code and the Verilator-generated C++ context to enhance semantic understanding during the description generation process.
   • Quantization: The labeling process evaluates the trade-offs between INT4 and BF16 quantization for the DeepSeek-R1 model to balance computational efficiency with labeling fidelity.

3. Fine-Tuning and Evaluation:

   • The labeled dataset is used to fine-tune various open-source LLM families, including the Granite series (8B) and Qwen series (7B and 32B).
   • Performance is evaluated using the VerilogEval benchmark (both Machine and Human variants), measuring Pass@1, Pass@5, and Pass@10 metrics.

Key Contributions

• OPENRTLSET Dataset: The creation of the largest fully open-source Verilog dataset (131k modules), ensuring complete reproducibility through traceable GitHub origins and permissive licensing.
• End-to-End Labeling Flow: A fully open-source pipeline that integrates reasoning-driven LLMs (DeepSeek-R1) and optional Verilator-generated C++ context to produce high-quality natural language descriptions for hardware code.
• Systematic Ablation Studies: The paper provides extensive analysis on the impact of dataset composition (Verilog vs. Verilog+C++), model size (7B–32B), and quantization strategies on code generation performance.
• Benchmarking: A comparative analysis demonstrating that models fine-tuned on OPENRTLSET outperform those trained on existing datasets like MG-Verilog, even when the latter are re-labeled using the same flow.

Results
The experimental results highlight several key findings:

• Performance Gains: Fine-tuning on OPENRTLSET yields significant improvements over baseline models. On the VerilogEval-Machine benchmark, fine-tuning Qwen2.5-32B on the full 131k dataset resulted in a Pass@10 of 89.3%, an absolute improvement of 5.4% over the best 11k downsampled configuration.
• Dataset Quality vs. Flow: While re-labeling the existing MG-Verilog dataset with the authors' flow improved performance, the OPENRTLSET dataset consistently outperformed it. This suggests that the diversity and quality of the source code in OPENRTLSET are as critical as the labeling methodology.
• Context Augmentation: The ablation study on labeling strategies revealed that for smaller models (7B–8B), augmenting Verilog with Verilator-generated C++ ("Verilog & C++") often yielded better results. However, for the larger Qwen2.5-32B model, using Verilog alone sometimes performed better, indicating that model capacity influences the utility of additional context.
• Quantization: INT4 quantization was selected for the labeling phase to save computational resources without significantly compromising the final fine-tuned model performance.
• Model Efficiency: The study noted that smaller models (e.g., Granite3.3-8B) could outperform larger closed-source models (e.g., Claude 3.7, GPT-3.5-turbo) on Verilog tasks after fine-tuning, though the 32B parameter models achieved the highest absolute scores.

Significance
The paper posits that OPENRTLSET establishes a new foundation for accessible research and commercial use in LLM-driven hardware design. By removing proprietary barriers and providing a standardized, high-quality dataset, the authors aim to accelerate the development of tools capable of generating correct hardware designs. The work demonstrates that open-source approaches, when combined with rigorous data curation and reasoning-based labeling, can achieve superior performance in hardware design tasks, addressing a fundamental bottleneck in the field. The dataset and the associated fine-tuned models are made available to the community to foster further innovation in automated hardware engineering.