VeriInteresting: An Empirical Study of Model Prompt Interactions in Verilog Code Generation

This paper presents an empirical study mapping the interactions between model characteristics and prompt engineering strategies for Verilog code generation, revealing which trends generalize across diverse language models and benchmarks through controlled experiments.

The Gist

Imagine you are trying to teach a very smart, but slightly literal, robot how to build a complex machine. In the world of software, this robot is great at writing code for apps and websites. But in the world of hardware (like the chips inside your phone or computer), the rules are much stricter. A tiny mistake in a software app might just make a button not work; a tiny mistake in hardware code can cause the whole machine to catch fire, freeze, or fail years later when it's too late to fix.

This paper, titled "VeriInteresting," is like a massive field test where researchers tried to figure out the best way to talk to these AI robots to get them to build perfect hardware. They tested 18 different AI models (from small, cheap ones to giant, expensive ones) and tried many different ways of giving them instructions (called "prompts").

Here is the breakdown of their findings using some creative analogies:

1. The Challenge: The "Goldilocks" Problem

Hardware design is like building a house where the blueprint must be perfect before you pour a single drop of concrete. You can't just "try it out" and fix it later like you can with a website.

• The Problem: Most AI models are trained on general internet data. They are like general contractors who are great at painting walls but might not know the specific, rigid rules of electrical wiring.
• The Goal: The researchers wanted to see if we could get these general contractors to act like master electricians just by changing how we ask them to do the job, without having to retrain them (which is expensive and risky).

2. The Experiments: Trying Different "Instruction Manuals"

The researchers didn't just say "Build this." They tried five different ways of talking to the AI:

• The Basic Ask: "Here is the job, do it." (The standard approach).
• The Structured Blueprint: "Here is the exact format you must use, step-by-step." (Like giving the robot a strict checklist).
• The "Think Before You Speak" Method: Asking the AI to write down its reasoning first before writing the code. (Like asking an architect to sketch the plan before drawing the blueprints).
• The "Refine First" Method: Asking the AI to rewrite the instructions to make them clearer before building. (Like a translator making sure the client's vague idea is understood before construction starts).
• The "Show Me an Example" Method: Giving the AI a few examples of good work to copy. (Like showing a new apprentice a finished door so they know what to build).

3. The Big Discoveries

Size vs. Specialization (The "Generalist vs. Specialist" Debate)

• Finding: Bigger AI models (the "Generalists") are usually better, but Specialist models (AI specifically trained on hardware) are surprisingly good at their specific job.
• The Twist: However, the specialists are like a Formula 1 driver. They are amazing on a specific race track (the benchmark they were trained on), but if you put them on a dirt road or ask them to drive a truck, they crash. The generalist models are more like SUVs: they might not be the fastest on a race track, but they handle different terrains much better.
• Lesson: Don't just buy the most expensive specialist; sometimes a well-guided generalist is more reliable.

The "Over-Thinker" Trap

• Finding: Asking the AI to "think step-by-step" (Chain-of-Thought) worked great for some models but hurt others.
• The Analogy: Imagine asking a nervous student to explain their math homework while solving it. For some, it helps them focus. For others, it makes them panic and make silly mistakes.
• Lesson: There is no "one size fits all" instruction. Sometimes, just telling the AI to "do it" is better than making it write an essay first.

The "Refinement" Danger

• Finding: Asking the AI to "rewrite the instructions first" was the riskiest strategy.
• The Analogy: It's like asking a translator to rewrite a client's order before sending it to the chef. Sometimes the translator misunderstands the client and changes the order from "Spicy Tacos" to "Mild Tacos with extra cheese." The chef then makes the wrong dish.
• Lesson: In hardware, changing the instructions can introduce new errors. It's often safer to stick to the original request.

The "Example" Trap

• Finding: Showing examples (In-Context Learning) helped smaller models but sometimes confused the big ones.
• The Analogy: A junior employee loves seeing examples of past work to learn the ropes. A senior expert, however, might get annoyed by the examples and try to copy them too literally, ignoring the unique details of the new job.

4. The "Magic Wand" (Prompt Optimization)

The researchers also tried using an automated tool (called GEPA) to tweak the instructions automatically, like a robot tuning a radio to find the clearest signal.

• Result: It worked a little bit, but it wasn't a magic wand. It couldn't fix the fundamental gaps between what the AI knew and what the hardware needed.

5. The Final Verdict: "No Silver Bullet"

The most important takeaway from this paper is that hardware design is different from software design.

• In software, you can often fix bugs later. In hardware, you can't.
• Because of this, the tricks that work for writing Python code (like asking the AI to think out loud or rewriting the prompt) don't always work for hardware.
• The Best Strategy: There is no single "best" AI or "best" prompt. You have to match the right AI (Generalist vs. Specialist) with the right instruction style for your specific task. You also need to test your results on multiple different "tracks" (benchmarks) to make sure the AI isn't just memorizing the test answers.

In short: Building hardware with AI is like navigating a minefield. You can't just guess; you need to know exactly which AI you are using and exactly how to talk to it, because one wrong word could blow up the whole project.

Technical Summary

Here is a detailed technical summary of the paper "VeriInteresting: An Empirical Study of Model–Prompt Interactions in Verilog Code Generation."

1. Problem Statement

The rapid advancement of Large Language Models (LLMs) has revolutionized software code generation, but their application to Hardware Description Languages (HDLs), specifically Verilog, faces unique challenges:

• Strict Semantics: Unlike software, hardware design requires precise signal connectivity, timing constraints, and concurrency management. A plausible-looking Verilog module may simulate correctly in isolation but fail due to race conditions or timing violations during synthesis or fabrication.
• Data Scarcity: High-quality hardware designs are often proprietary, limiting the availability of open-source training data for fine-tuning.
• Evaluation Complexity: Verilog correctness cannot be fully validated by unit tests (simulation) alone; it often requires formal equivalence checking, which is computationally expensive.
• The Gap: It is unclear whether prompting strategies effective in software engineering (e.g., Chain-of-Thought, structured prompting) transfer effectively to hardware design, or if domain-specific fine-tuning is strictly necessary.

2. Methodology

The authors conducted a large-scale, controlled empirical study evaluating 18 Language Models across two distinct Verilog benchmarks using a factorial experimental design.

Models Evaluated

• Commercial LLMs: 7 models (e.g., GPT-4.1, GPT-5, Gemini 3, Claude Sonnet 4).
• Open-Source SLMs: 11 models, including general-purpose (Qwen2.5), code-specialized (Qwen-Coder, DeepSeek Coder), and Verilog-specialized variants (VeriReason, VeriThoughts) fine-tuned with Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL).
• Scale: Ranging from 3B to 16B parameters.

Benchmarks

1. Verilog Eval v2: Uses dynamic simulation (Icarus Verilog) with testbenches to check functional correctness against reference outputs.
2. VeriThoughts: Uses formal equivalence checking (Yosys) to prove functional identity against a golden reference for all input states.

Prompting Strategies (Inference-Time Adaptation)

The study isolated the impact of various inference-time techniques:

• Base: Standard single-pass prompts.
• Structured Prompting (Struct): Enforcing consistent task signatures and modular components (via DSPy).
• Chain-of-Thought (CoT): Explicit reasoning steps before code generation.
• Prompt Refinement (Refine): A two-stage pipeline where an LLM rewrites/clarifies the specification before generating code.
• In-Context Learning (ICL): Few-shot examples added to the prompt.
• Genetic-Pareto (GEPA): An automated evolutionary search method to optimize system prompts without retraining.

3. Key Contributions

• Empirical Map of Model-Prompt Interactions: The first systematic evaluation of how model scale, domain specialization, and prompting strategies interact specifically for Verilog generation.
• Benchmark Comparison: Demonstrated that results vary significantly between simulation-based (Verilog Eval) and formal verification-based (VeriThoughts) benchmarks, highlighting the risk of overfitting to a single evaluation protocol.
• Inference vs. Training Trade-offs: Provided data-driven insights into whether "strong prompting" can substitute for expensive domain-specific fine-tuning.
• Identification of Fragility: Showed that many techniques successful in software (like prompt refinement) can be detrimental in hardware due to semantic drift.

4. Key Results

RQ1: Scale vs. Specialization

• Scaling: Larger models generally perform better, but the trend is non-monotonic for some families (e.g., Qwen-3B outperformed Qwen-7B on VeriThoughts).
• Specialization: Verilog-specialized models (VeriReason, VeriThoughts) excel on their specific training distributions but show degraded robustness when faced with complex prompt transformations (e.g., structured outputs or CoT).
• Observation: Specialization often trades general instruction-following ability for peak performance on a narrow task distribution.

RQ2: Sensitivity to Prompting Strategies

• Structured Prompting: Generally beneficial for smaller, code-specialized open-source models but offers diminishing returns for top-tier commercial models.
• Chain-of-Thought (CoT): Highly variable. It acts as a stabilizer for structured prompts on Verilog Eval but can introduce "unverified assumptions" that degrade performance on formal verification benchmarks (VeriThoughts).
• Prompt Refinement: The least stable intervention. Rewriting specifications often introduces semantic drift, causing significant performance drops (e.g., GPT-5m and VeriThoughts models failed when asked to rewrite specs).
• In-Context Learning (ICL): Not a universal fix. It helps smaller models by anchoring output formats but can confuse specialized models or those with limited context windows.
• GEPA Optimization: Optimized prompts performed comparably to strong manual baselines (Struct + CoT) but did not yield breakthrough improvements, suggesting prompt evolution has limits in this domain.

RQ3: Fine-Tuning vs. Strong Prompting

• Inference-time methods (strong prompting) can narrow the gap to fine-tuned models for general-purpose LLMs.
• However, fine-tuning remains superior when the target distribution matches the training objective. For example, VeriReason models significantly outperformed prompted baselines on Verilog Eval v2, proving that training-time adaptation is irreplaceable for specific, high-stakes tasks.

RQ4: Benchmark Stability

• There is a strong positive correlation ($r=0.868$) between the two benchmarks, but Verilog Eval v2 is systematically harder than VeriThoughts for many models.
• Models that perform well on simulation (Verilog Eval) do not always translate to formal verification success, and vice versa. Relying on a single benchmark can lead to false conclusions about model reliability.

5. Significance and Implications

• Hardware is Distinct: Verilog generation is not merely a variant of software generation. The strict constraints of hardware (timing, concurrency, global correctness) make it more sensitive to semantic errors that standard prompting techniques cannot easily resolve.
• No "One-Size-Fits-All" Strategy: The optimal strategy depends heavily on the specific model class and the evaluation metric. A strategy that boosts a small open-source model might crash a fine-tuned specialized model.
• Robustness vs. Performance Trade-off: Domain-specialized models achieve high peak performance but lack robustness to prompt variations. General-purpose models are more adaptable but require stronger scaffolding.
• Deployment Guidance:
  • For low-cost deployment, strong prompting (Structured + CoT) on capable open-source models is a viable first step.
  • For high-assurance hardware, domain-specific fine-tuning is necessary, and evaluation must span both simulation and formal verification to ensure robustness.
  • Prompt Refinement should be used with extreme caution in hardware contexts.

This work serves as a foundational guide for researchers and engineers aiming to integrate LMs into hardware design flows, emphasizing that empirical validation across multiple benchmarks is critical before deployment.