💻 computer science

AI-Generated RTL Code: A Functional Evaluation of Natural-Language and HLS Prompting

This study evaluates the ability of commercial LLMs to generate functional Verilog RTL from high-level specifications, finding that using HLS-C prompts significantly improves correctness over natural language while revealing persistent challenges with simulation errors and limited generalization across variable bit-widths.

Original authors: Giordano Santorum Lorenzetto, Vanderlei Bonato, Oliver Sinnen

Published 2026-08-05

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Giordano Santorum Lorenzetto, Vanderlei Bonato, Oliver Sinnen

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a tiny, super-fast factory inside a microchip. This isn't a factory that makes toys or cars; it's a digital factory that processes information at the speed of light. To build it, engineers write a special kind of instruction manual called RTL code (Register-Transfer Level). Think of this code as the blueprint for the factory's conveyor belts, assembly arms, and traffic lights. If the blueprint has even one tiny mistake, the whole factory might jam, catch fire, or just stop working entirely.

For a long time, writing these blueprints has been like speaking a secret, rigid language that only a few experts understand. But now, we have Artificial Intelligence (AI) that is incredibly good at writing code for software—like the apps on your phone or the websites you visit. People started wondering: "If AI can write software code so well, can it also write these tricky hardware blueprints?" This is the big question the researchers tackled. They wanted to see if a smart AI could look at a simple description of a hardware task and turn it into a working factory blueprint, or if it would just get confused by the strict rules of hardware.

The Experiment: Talking vs. Coding

The researchers set up a massive test to see how well different AI models could do this job. They gave the AI two very different ways to ask for a blueprint:

The "Chatty" Approach (Natural Language): They simply told the AI in plain English what the hardware should do. For example, "Make a machine that multiplies two numbers." It's like asking a friend, "Hey, build me a sandwich," without giving a recipe.
The "Programmer" Approach (HLS-C Prompting): Instead of plain English, they gave the AI a piece of C code (a common programming language) that described the logic. This is like handing the AI a detailed recipe written in a language it knows very well, and asking it to translate that recipe into the specific "factory language" (Verilog).

They tested this on seven different AI models (including big names like GPT-4 and Gemini) and asked them to build eight different types of digital circuits, ranging from simple traffic light controllers to complex math calculators. They ran each test 30 times to see if the AI could get it right by chance or if it truly understood the task.

The Results: The Recipe Wins

The findings were pretty clear, and they might surprise you. When the AI was just "chatted" with in plain English, it struggled. On average, only about 26.9% of the blueprints it generated actually worked. Most of the time, the AI made mistakes that were like trying to build a house with the wrong bricks—the code wouldn't even compile (start up), or it would run but produce the wrong results.

However, when the researchers switched to the C code recipe (HLS-C prompting), the AI's performance jumped up. The success rate climbed to 39.5%. That's a big improvement! It suggests that even though the AI is smart, it thinks more like a programmer than a hardware engineer. When you give it a structured code recipe, it knows exactly how to translate those steps into the hardware language. It's like the AI is saying, "Oh, you gave me a recipe in French? I can translate that to Spanish perfectly. But if you just say 'make a sandwich' in English, I might guess the wrong ingredients."

The "Weird Number" Test: Does it Memorize or Understand?

To see if the AI was just copying answers it had seen before (memorization) or if it actually understood how to build these machines, the researchers played a trick. They asked the AI to build a multiplier (a math machine) using weird, unusual numbers of bits, like 7 bits or 31 bits. These are numbers you almost never see in real life, so the AI couldn't have just memorized the answer from its training data.

The results here were fascinating. Some of the smarter AI models (like o4-mini and Gemini 2.5 Pro) handled these weird numbers almost perfectly, getting it right over 90% of the time. This suggests that these models aren't just copying and pasting; they actually understand the logic of how to build the machine, no matter the size. They can generalize, meaning they can apply the rules to new situations they've never seen before.

The Catch: It's Not Perfect Yet

Despite the improvements, the paper makes it very clear that we are not there yet. Even with the "recipe" approach, the AI still fails more often than it succeeds. The biggest problem isn't that the AI gets the syntax wrong (like spelling errors); it's that it gets the timing wrong.

Imagine the AI builds a factory where the conveyor belt moves one second too fast. The blueprint looks perfect, and the machine turns on, but the products fall off the line before they are packed. In the study, these "simulation errors" were the most common failure. The AI often creates code that looks right but behaves slightly wrong when the clock starts ticking. Fixing these errors is hard because it requires a deep understanding of how the hardware behaves over time, something the AI still struggles with.

The researchers also found that when the AI did get it right, the resulting hardware was often just as fast, or even faster, than blueprints written by human experts. This is a huge win, suggesting that if we can get the AI to stop making those tiny timing mistakes, it could be a super-powerful assistant for engineers.

The Bottom Line

So, what's the takeaway? AI is getting really good at helping us design computer chips, but it's not ready to take over the job completely just yet. It's like a brilliant intern who can draft a great design if you give them a clear, structured plan (like a C code recipe), but they still need a human supervisor to check the final details and make sure the factory doesn't crash.

The study suggests that using code-based prompts is a much better way to talk to these AI models than just chatting in plain English. While the AI still makes mistakes that require human debugging, it's already proving to be a powerful tool that can speed up the design process and help people who are new to hardware engineering get started. We aren't at the point where AI can build a chip from scratch without help, but we are definitely moving in the right direction.

Technical Summary: AI-Generated RTL Code: A Functional Evaluation of Natural-Language and HLS Prompting

Problem Statement
The increasing complexity of hardware design has spurred interest in using Large Language Models (LLMs) to generate Register-Transfer Level (RTL) code. While LLMs have demonstrated proficiency in software code generation, their applicability to hardware development remains underexplored. Hardware design imposes stricter requirements than software, necessitating functional correctness, resource optimization, and precise handling of concurrency and timing. Although High-Level Synthesis (HLS) tools allow designers to describe hardware in C/C++, they often require specific directives and offer limited control over the resulting RTL structure. This study investigates whether commercially available, general-purpose LLMs can generate functionally correct Verilog RTL directly from high-level specifications without fine-tuning, and evaluates how different prompting strategies influence generation quality.

Methodology
The research utilized the RTLLM benchmark, a dataset comprising natural language specifications, reference Verilog implementations, and functional testbenches for various hardware modules (ranging from combinational logic to sequential circuits). The authors evaluated seven distinct LLMs from OpenAI (GPT-4.1, GPT-4o, o4-mini) and Google (Gemini 1.5 Pro, 2.0 Flash, 2.5 Flash, 2.5 Pro).

Two prompting strategies were compared using a zero-shot approach:

Natural Language (NL) Prompting: Direct textual descriptions of functionality, I/O ports, and implementation guidelines.
HLS-style C Prompting: Structured C code snippets representing the desired hardware functionality, designed to be synthesizable by AMD Vitis HLS.

The evaluation workflow was fully automated:

Generation: Each model generated $n=30$ independent Verilog outputs per prompt.
Compilation: Code was checked for syntax errors using the Questa simulator.
Simulation: Successfully compiled code was integrated with testbenches and simulated.
Verification: Outputs were compared against expected results. Errors were categorized as Compilation, Integration, or Simulation errors.
Synthesis: Correctly functioning designs were synthesized on an Intel Cyclone V FPGA to measure maximum operating frequency ( $F_{max}$ ) and area utilization (ALMs) against human-written reference designs.

Additionally, an exploratory case study on bit-width scaling was conducted using a sequential multiplier design. The bit-width was varied across standard values (8, 32, 64) and unusual, sparse values (7, 12, 15, 23, 24, 31, 50, 63) to test the models' ability to generalize algorithmic logic versus merely recalling training patterns.

Key Results

Prompting Strategy Impact: HLS-C prompting significantly outperformed Natural Language prompting. The average functional correctness rate increased from 26.9% (NL) to 39.5% (HLS-C).
Error Distribution: HLS-C prompts substantially reduced compilation errors (from 33.5% to 24.2% on average) and integration errors. However, simulation errors remained the primary challenge, accounting for approximately 35% of attempts even with HLS-C, indicating that models still struggle with the cycle-accurate translation of complex hardware behavior.
Model Performance: The o4-mini and Gemini 2.5 Pro models consistently achieved the highest correctness rates across both strategies.
Generalization: In the bit-width scaling experiment, o4-mini and Gemini 2.5 Pro demonstrated robust generalization, maintaining high correctness (>90%) across non-standard widths (e.g., 7-bit, 23-bit), suggesting they are not simply memorizing training data but understanding underlying algorithms.
Hardware Quality: Synthesized designs often achieved $F_{max}$ values exceeding the human-written reference designs. However, HLS-C prompts tended to produce designs with lower performance variability but occasionally larger area footprints compared to NL prompts, likely due to the models translating procedural C constructs into rigid hardware structures.

Key Contributions

Comparative Analysis of Prompts: The study demonstrates that providing structured, code-centric inputs (HLS-style C) significantly improves the functional correctness of LLM-generated RTL compared to natural language descriptions.
Generalization Assessment: It provides an analysis of how LLMs handle uncommon design parameters, revealing that frontier models can abstract algorithmic logic effectively, while lighter models often fail to generalize beyond standard patterns.
Hardware Quality Evaluation: The work offers a quality analysis comparing LLM-generated designs against human references in terms of timing ( $F_{max}$ ) and area, highlighting that while LLMs can produce competitive architectures, they often require manual selection from multiple attempts to achieve optimal results.

Significance and Claims
The authors conclude that while LLMs are not yet capable of autonomously producing production-ready RTL code due to persistent simulation errors and the high cost of debugging, they represent a powerful assistive tool for hardware developers. The improved performance with HLS-C prompts suggests that leveraging the models' familiarity with software syntax can bridge the gap in hardware generation. The paper posits that LLMs are particularly valuable for generating draft implementations or "boilerplate" code for experienced engineers to review and optimize, and as educational tools to help software engineers transition into hardware design by mapping familiar C constructs to RTL. The study emphasizes that current limitations necessitate human oversight, particularly for complex sequential logic and timing-critical designs.

The Experiment: Talking vs. Coding

The Results: The Recipe Wins

The "Weird Number" Test: Does it Memorize or Understand?

The Catch: It's Not Perfect Yet

The Bottom Line

More like this