💻 computer science

Synthesis-in-the-Loop Evaluation of LLMs for RTL Generation: Quality, Reliability, and Failure Modes

This paper introduces a synthesis-in-the-loop evaluation framework using the Hardware Quality Index (HQI) to assess 32 language models on RTL generation, revealing distinct performance tiers and systematic failure modes that highlight the gap between functional correctness and hardware implementation quality.

Original authors: Weimin Fu, Zeng Wang, Minghao Shao, Ramesh Karri, Muhammad Shafique, Johann Knechtel, Ozgur Sinanoglu, Xiaolong Guo

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Weimin Fu, Zeng Wang, Minghao Shao, Ramesh Karri, Muhammad Shafique, Johann Knechtel, Ozgur Sinanoglu, Xiaolong Guo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a team of architects to design a house. In the past, you might have just asked them to draw a picture of the house and checked if the drawing looked nice. If the drawing had a roof and doors, you'd say, "Great job!"

But in the real world of building chips (the brains of computers), a pretty drawing isn't enough. The design must be buildable, efficient, and safe. If the architect draws a wall made of glass that can't actually be built, or a staircase that takes up too much space, the whole project fails.

This paper is like a rigorous inspection report for 32 different "AI Architects" (Large Language Models) who are trying to write the blueprints for computer chips. Instead of just checking if the AI's drawing looks right, the researchers built a special testing machine that tries to actually build the design to see if it works.

Here is the breakdown of their findings using simple analogies:

1. The "Build-It" Test (Synthesis-in-the-Loop)

Most previous tests for AI coding were like a spelling bee: they checked if the AI wrote the words correctly (syntax) and if the story made sense (simulation).

This paper added a "construction phase." They took the AI's code and ran it through a factory process (called synthesis) that turns the code into a physical chip blueprint.

The Result: Many AIs passed the "spelling bee" but failed the "construction." They wrote code that looked correct but would crumble if you tried to build it.
The New Score: They created a "Hardware Quality Index" (HQI). Think of this as a score from 0 to 100 that measures not just if the design works, but how efficiently it uses space and time, and how many warnings the factory gave them.

2. The Three Tiers of Architects

When they ranked the 32 AI models, they didn't find a smooth line from "bad" to "good." Instead, the models fell into three distinct groups, like three different leagues of sports teams:

The Elite League (Tier 1): These 14 models are the "Master Architects." They consistently produce designs that are not only buildable but highly efficient. The top performer, Gemini-3-Pro, scored an impressive 85 out of 100.
The Amateur League (Tier 2): These 15 models are "Juniors." They can build simple houses, but when the design gets complex, they start making mistakes. They are about 10 points behind the Elite League.
The Novice League (Tier 3): These 3 models are "Interns." They struggle to even finish the basic structure. Their designs often fail the construction test entirely.

3. The "Best-of-Five" vs. "First Try" Problem

Imagine you ask an architect to design a house.

The "Best-of-5" Score: If you ask the architect to draw 5 different versions and pick the best one, they might do a great job.
The "First Try" Score: If you only let them draw one version and you have to use that immediately, the quality often drops significantly.

The paper found a huge gap here. Some models could produce a masterpiece if you gave them five tries, but their first attempt was often mediocre.

Why this matters: In the future, we want AI agents that work automatically (like a robot that designs a chip and moves to the next step without a human checking). If the robot has to wait for five tries to get a good result, it slows everything down. The paper says these AIs aren't quite ready for that "robot" job yet because their first try isn't reliable enough.

4. How They Fail: The "Late" vs. "Early" Crash

The researchers looked at why the designs failed and found a funny pattern based on where the AI came from:

Proprietary Models (The Big Tech Companies): These AIs are like students who studied very hard but got confused on the final exam. They usually write code that looks perfect at first, but when the factory tries to build it, they hit a wall late in the process (like realizing a door is in the wrong place after the walls are up).
Open-Weight Models (The Community Projects): These AIs are like students who skipped the basics. They fail very early. They often forget to put the "wrapper" around the design or try to use materials that don't exist in the real world (like trying to build a bridge out of water).

The paper suggests this happens because the "Open" AIs were trained mostly on code that was meant for testing (simulations), while the "Proprietary" AIs seem to have seen more code meant for building (synthesis).

5. Cost vs. Quality

Finally, they checked the price tag.

The Expensive Trap: Some of the most expensive AI models (like the ones that spend a lot of time "thinking" before answering) were not the best at building chips. They were slow and costly but didn't produce better blueprints.
The Value Pick: Some cheaper, faster models (like Gemini-3-Flash) produced top-tier designs for a tiny fraction of the cost. It turns out, for this specific job, "thinking harder" didn't mean "building better."

The Bottom Line

This paper tells us that while AI is getting very good at writing code that looks right, it still struggles to write code that is ready to build. To use AI for designing real computer chips, we need to stop just checking if the code passes a test, and start checking if it can actually be manufactured efficiently. Until the "First Try" reliability improves, we can't fully trust these AI architects to work alone.

Technical Summary: Synthesis-in-the-Loop Evaluation of LLMs for RTL Generation

Problem Statement

The generation of Register-Transfer Level (RTL) code using Large Language Models (LLMs) presents a unique challenge distinct from software synthesis. While software generation requires syntactic validity and functional correctness, RTL generation demands that designs be syntactically valid, synthesizable into gate-level netlists, functionally correct under simulation, and hardware-efficient in terms of area and timing. Existing benchmarks (e.g., VerilogEval, RTLLM) primarily evaluate models based on simulation pass rates ($pass@k$), which fail to distinguish between designs that merely pass a testbench and those that compile into efficient, synthesizable hardware. Furthermore, the rise of Agentic AI, which orchestrates end-to-end Electronic Design Automation (EDA) workflows, necessitates a shift from evaluating the "best-of- $k$ " capability to assessing the quality of a single attempt ($pass@1$), as autonomous agents cannot afford the combinatorial explosion of multi-sample strategies. Current evaluation methods do not capture the "quality-of-results" (QoR) metrics (area, delay, warnings) required for deployment reliability.

Methodology

The authors propose a Synthesis-in-the-Loop evaluation framework to address these gaps.

Benchmark and Models: The study evaluates 32 language models (including proprietary and open-weight variants from OpenAI, Google, Anthropic, Meta, DeepSeek, etc.) on 202 Verilog tasks drawn from VerilogEval (155 single-module tasks) and RTLLM (47 real-world multi-module tasks). Each model is prompted with five independent attempts per task.
Evaluation Pipeline: A generated design must pass three sequential gates to be assessed for quality:
1. Syntax Validity: Verified via Icarus Verilog.
2. Synthesizability: Verified via Yosys elaboration and mapping under the Nangate45 45nm standard-cell library.
3. Functional Correctness: Verified via testbench simulation with fail-first matching.
Hardware Quality Index (HQI): For designs passing all gates, a 0–100 score is calculated based on post-synthesis metrics relative to an expert "golden" reference. The score is derived from normalized area ( $\hat{a}$ ), delay ( $\hat{d}$ ), and excess warning count ( $\hat{w}$ ):
$\text{cost} = 0.5 \cdot \frac{\hat{a}}{a^*} + 0.5 \cdot \frac{\hat{d}}{d^*} + 0.1 \cdot \max(0, \hat{w} - w^*)$
$\text{HQI} = \min\left(\frac{100}{\text{cost}}, 100\right)$
Designs failing any gate receive an HQI of 0.
Aggregate Metrics: The study defines Complexity-weighted Coverage (task solve rate), Global HQI (best-of-5 capability ceiling), and Expected HQI (single-attempt quality). The gap between Global and Expected HQI quantifies the "deployment-readiness" gap.
Failure Taxonomy: Synthesis failures (passing Icarus but failing Yosys) are classified into a nine-class taxonomy based on Yosys diagnostic output.

Key Results

1. Three-Tier Capability Landscape

K-means clustering of the 32 models reveals three distinct capability tiers:

Tier 1 (Frontier): 14 models with Global HQI $\ge$ 66. Led by Gemini-3-Pro (87.5% coverage, 85.1 HQI), followed by GPT-5.4-Pro and Gemini-3-Flash. This tier includes top variants from OpenAI, Google, and Anthropic.
Tier 2 (Dense Cluster): 15 models with Global HQI between 43 and 66. Includes strong open-weight models like DeepSeek-V3.2 and Qwen3.5-397B-OSS, as well as mid-tier proprietary models.
Tier 3 (Low Performance): 3 models with Global HQI < 43, including Mistral-Nemo and GPT-5-Nano.

2. Simulation vs. Synthesis Discrepancy

Simulation-based pass rates systematically overstate hardware readiness. On average, the best-of-5 pass rate exceeds the Global HQI by 7.5 points, with gaps reaching up to 14.9 points for some models. Relying solely on simulation pass rates leads to ranking inversions; for instance, GPT-4.1 ranks 14th by pass rate but 17th by HQI, while GPT-5.4-Pro moves from 9th to 2nd when quality is considered.

3. The Deployment-Readiness Gap

A critical finding is the significant gap between Global HQI (best-of-5) and Expected HQI (single-attempt).

In Tier 1, the gap ranges from 3.7 to 17.8 points (median 8.7).
In lower tiers, gaps exceed 15 points, reaching a maximum of 22.1 points.
This indicates that a single model call may retain only half the quality of a multi-attempt strategy, posing a barrier to integration in autonomous agentic pipelines.

4. Failure Mode Taxonomy

Analysis of 195 synthesis failures reveals systematic differences between model types:

Proprietary Models: Fail late in the pipeline. Dominant failure modes are "late syntax errors" (code passes parser but fails elaboration due to structural issues) and "synthesis timeouts" (complex netlists). This suggests exposure to synthesis-grade RTL in training data.
Open-Weight Models: Fail early. Dominant modes are "undefined module references" (missing wrappers) and "non-synthesizable constructs" (e.g., while loops, initial blocks). This pattern aligns with training corpora skewed toward simulation-oriented code rather than synthesis-grade RTL.

5. Inference Characteristics

Cost and quality show poor correlation. High-cost models with heavy reasoning overhead (e.g., GPT-5-Pro) do not guarantee superior RTL quality. Gemini-3-Flash emerges as a cost-efficient frontier model, delivering Tier 1 quality (81.2 HQI) at a fraction of the cost of top-tier reasoning models.

Significance and Contributions

The paper claims three primary contributions:

A Synthesis-in-the-Loop Framework: It introduces a pipeline and the Hardware Quality Index (HQI) that moves beyond binary pass/fail metrics to assess implementation quality (area, delay, warnings) against expert references.
Empirical Characterization: It provides a comprehensive evaluation of 32 models, revealing a structured three-tier landscape and quantifying the "deployment-readiness gap" where single-attempt quality lags significantly behind best-of- $k$ potential.
Failure Taxonomy: It exposes a systematic divergence in failure modes between proprietary and open-weight models, linking these differences to training data composition (synthesis-grade vs. simulation-grade).

The authors conclude that while LLMs show promise for RTL generation, current models cannot yet be integrated into autonomous agentic pipelines without improving single-shot reliability. The gap between simulation pass rates and synthesis-aware quality metrics necessitates a shift in evaluation standards for hardware design automation. Future work should focus on iterative refinement with synthesis feedback and the curation of synthesis-grade training corpora to address the specific failure modes of open-weight models.