🤖 AI

StepPRM-RTL: Stepwise Process-Reward Guided LLM Fine-Tuning for Enhanced RTL Synthesis

StepPRM-RTL is a novel framework that enhances LLM-based RTL code generation by integrating stepwise trajectory modeling, process-reward modeling, and retrieval-augmented fine-tuning to achieve superior functional correctness and long-horizon reasoning compared to prior methods.

Original authors: Prashanth Vijayaraghavan, Apoorva Nitsure, Luyao Shi, Ehsan Degan, Vandana Mukherjee

Published 2026-06-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Prashanth Vijayaraghavan, Apoorva Nitsure, Luyao Shi, Ehsan Degan, Vandana Mukherjee

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very talented but inexperienced apprentice how to build a complex digital clock using a specific, rigid language called "RTL" (which is like the blueprint language for computer chips).

The problem is that if the apprentice makes even one tiny mistake in the middle of the process—like forgetting to reset a counter or connecting a wire to the wrong place—the entire clock stops working. Traditional teaching methods usually only check the final clock at the end. If it doesn't work, the teacher says, "Try again," but doesn't explain which step went wrong. This is frustrating and inefficient.

StepPRM-RTL is a new, smarter way to train an AI (a Large Language Model) to build these digital designs. Here is how it works, broken down into simple concepts:

1. The "Step-by-Step" Recipe (Stepwise Trajectories)

Instead of asking the AI to write the whole code in one giant burst, StepPRM-RTL breaks the job down into small, logical steps.

The Analogy: Think of it like baking a cake. Instead of just handing the AI a list of ingredients and saying "Make a cake," the teacher shows a recipe card for every single step: "First, crack the eggs," then "Next, whisk them," then "Add the flour."
The Innovation: For every step, the AI must write a short note explaining why it is doing that step (the "rationale"). This forces the AI to think about the logic, not just copy-paste code.

2. The "Coach" Who Grades Every Move (Process Reward Model)

In old methods, the AI only got a grade at the very end (Pass/Fail). StepPRM-RTL introduces a "Coach" (called the StepPRM) that watches the AI work in real-time.

The Analogy: Imagine a chess coach who doesn't wait until the game is over to tell you if you played well. Instead, after every single move, the coach says, "That was a good move because it protects your king," or "That was a risky move because it leaves your queen open."
The Innovation: This coach gives immediate feedback on every small step. If the AI makes a logical error in the middle of the design, the coach catches it immediately, rather than waiting until the whole chip is broken.

3. The "What-If" Explorer (MCTS)

To get even better, the system uses a technique called Monte Carlo Tree Search (MCTS).

The Analogy: Imagine you are at a fork in a hiking trail. Instead of just picking one path and hoping it's right, the AI acts like a scout. It mentally simulates taking Path A, then Path B, and Path C. It asks, "If I take Path A, will I get stuck later?" It explores many different "what-if" scenarios to find the safest, most logical route before committing to it.
The Innovation: This helps the AI avoid dead ends and find the best way to solve complex problems that require many steps.

4. The "Library of Best Practices" (RAFT)

Finally, the system uses Retrieval-Augmented Fine-Tuning (RAFT).

The Analogy: Before the AI starts building, it quickly flips through a library of successful blueprints from similar projects. It doesn't just guess; it looks at how other experts solved similar problems and uses those patterns as a guide.
The Innovation: This ensures the AI isn't just making things up but is grounding its decisions in proven, real-world design patterns.

The Result

When the researchers tested this new method on standard benchmarks (using languages like Verilog and VHDL), the AI became significantly better at its job:

More Accurate: It produced working designs about 10% more often than previous best methods.
Better Reasoning: It didn't just get lucky; it actually understood the logic behind its steps, as shown by its ability to explain why it made certain design choices.

In short, StepPRM-RTL turns the AI from a "guess-and-check" machine into a "think-and-verify" apprentice, guiding it through every single step of the design process with a coach, a map, and a library of examples.

Technical Summary: StepPRM-RTL

Problem Statement

The automatic generation of Register-Transfer Level (RTL) code for digital hardware designs remains a significant challenge in Electronic Design Automation (EDA). Unlike general-purpose programming, RTL generation requires not only syntactic correctness but also precise temporal, concurrent, and structural behaviors. A single error in state updates or enable paths can propagate across modules, breaking the datapath despite the code remaining syntactically valid.

Current approaches primarily rely on supervised learning over code corpora, which capture surface-level patterns but fail to model the multi-step reasoning sequences required to assemble correct control and datapath logic. Furthermore, outcome-driven methods evaluate correctness only at the final design level, offering no supervision for intermediate decisions (e.g., structuring reset logic or aligning control-path transitions). This leads to struggles with long-horizon dependencies. Additionally, existing Process Reward Models (PRMs) often operate at the token level, which is fundamentally mismatched to hardware semantics where meaningful decisions span entire statements, modules, and signal groups.

Methodology: StepPRM-RTL Framework

The authors propose StepPRM-RTL, a reasoning-aware framework that treats RTL generation as a sequence of semantically meaningful design steps. The framework integrates four tightly coupled components in an iterative loop:

1. Stepwise Trajectory Construction

Canonical RTL implementations are decomposed into stepwise trajectories $\tau = \langle e_1, \dots, e_T \rangle$ . Each step $e_t$ consists of:

Rationale ( $r_t$ ): A natural-language description of a hardware design decision.
Code Edit ( $\delta_t$ ): The specific code modification applied to the partial implementation.
This decomposition, often assisted by Large Language Models (LLMs) and Abstract Syntax Tree (AST) analysis, creates high-quality, interpretable demonstrations used to bootstrap the system.

2. Step-Level Process Reward Model (StepPRM)

Unlike token-level reward models, StepPRM ( $V_\phi$ ) assigns scalar rewards to entire intermediate steps. It is trained using two objectives:

Supervised Preference Learning: Using a Bradley-Terry model to prefer canonical steps over perturbed or low-quality alternatives.
Reward Shaping via Partial Rollout Consistency: To address the lack of functional simulation for partial code, the model is trained to regress toward a shaped reward target. This target balances canonical supervision with a structural alignment score (computed via AST tree-edit similarity) between the partial implementation and the final canonical code.

3. PRM-Guided Monte Carlo Tree Search (MCTS)

To explore reasoning paths beyond canonical examples, the framework employs MCTS guided by StepPRM.

Search Structure: Nodes represent partial reasoning prefixes; edges represent RTL steps.
UCB Selection: The search maximizes an Upper Confidence Bound (UCB) score where the value estimate ( $Q_t$ ) is initialized directly by the StepPRM, providing dense semantic feedback even for partial code.
Feasibility Checks: The search discards branches violating structural invariants (e.g., unmatched blocks, combinational cycles) by assigning large negative penalties.
Output: The process generates a diverse set of high-value trajectories that extend beyond the initial canonical decompositions.

4. Retrieval-Augmented Fine-Tuning (RAFT)

The final component refines the generation policy ( $\pi_\theta$ ) using the expanded trajectory set.

Context Retrieval: For a given specification, the system retrieves relevant RTL files, design patterns, and verified trajectories from a repository.
Reward-Weighted Optimization: The policy is fine-tuned to maximize the likelihood of high-value trajectories, weighted by the cumulative StepPRM rewards. This ensures the model prioritizes semantically consistent and structurally valid reasoning sequences while grounding them in reusable design patterns.

The framework operates as an iterative loop: canonical trajectories bootstrap the StepPRM $\rightarrow$ StepPRM guides MCTS to expand the trajectory space $\rightarrow$ the expanded set refines the StepPRM $\rightarrow$ RAFT updates the policy $\rightarrow$ the improved policy guides the next MCTS iteration.

Key Contributions

Step-Level Process Rewarding for RTL: The introduction of StepPRM-RTL, the first framework to define and score semantically meaningful intermediate reasoning steps for Hardware Description Languages (HDL), resolving the mismatch between token-level scoring and hardware-level behaviors.
Unified Reasoning Pipeline: An integrated pipeline combining StepPRM-guided MCTS exploration with RAFT-based policy refinement, enabling stable long-horizon credit assignment and retrieval-grounded reasoning.
Comprehensive Evaluation: Extensive experiments demonstrating significant improvements in step-level reasoning quality, Pass@k, functional correctness, and generalization over supervised and reward-based baselines.

Experimental Results

The framework was evaluated on Verilog-Eval (156 tasks) and VHDL-Eval (202 tasks) benchmarks.

Performance: StepPRM-RTL achieved a Pass@1 of 0.857 for Verilog and 0.786 for VHDL, outperforming the best prior methods (including prompt-based GPT-4o and fine-tuned baselines like VeriThoughts) by over 10% in functional correctness.
Reasoning Fidelity: The model achieved reasoning fidelity scores of 82.5% (Verilog) and 80.2% (VHDL), significantly higher than baselines.
Ablation Studies:
- MCTS: Removing MCTS (replacing with sampling-only) reduced Pass@1 by ~4.7–5.0 percentage points, confirming MCTS's role in selecting high-quality intermediate steps.
- StepPRM: Removing step-level rewards (relying solely on outcome verification) caused a ~7.6 percentage point drop in Pass@1, highlighting the insufficiency of sparse outcome-only feedback for intermediate reasoning.
- Reward-Weighted RAFT: Treating all trajectories equally without reward weighting reduced Pass@1 by ~6 percentage points, demonstrating the necessity of prioritizing high-quality steps.
Hyperparameters: Performance peaked with 15 MCTS simulations per specification and a reward shaping weight ( $\lambda_{sh}$ ) of 0.3.

Significance

The paper claims that StepPRM-RTL establishes a new standard for LLM-assisted hardware design automation by bridging interpretable stepwise reasoning with improved RTL synthesis. By integrating step-level supervision, structured trajectory exploration, and retrieval-based context, the framework enables models to learn both how and why to construct correct RTL, effectively addressing long-horizon reasoning challenges that standard supervised or outcome-based training cannot resolve. The approach is presented as a scalable framework for high-fidelity, interpretable code generation across RTL languages.