🤖 machine learning

Reward-Weighted On-Policy Distillation with an Open Property-Equivalence Verifier for NL-to-SVA Generation

This paper introduces Reward-Weighted On-Policy Distillation (RWOPD), a novel training method that leverages a formal property-equivalence verifier to guide a 7B student model in generating SystemVerilog Assertions, achieving new state-of-the-art performance by prioritizing semantic correctness over token-level mimicry.

Original authors: Qingyun Zou, Yingze Li, Tianen Liu, Bingsheng He, Weng-Fai Wong

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Qingyun Zou, Yingze Li, Tianen Liu, Bingsheng He, Weng-Fai Wong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a talented but inexperienced apprentice (a 7-billion-parameter AI) how to write complex legal contracts for computer chips. These contracts are called SystemVerilog Assertions (SVA). They are the rules that tell a chip, "If this happens, then that must happen within 3 seconds," or "This signal must never be high."

For a long time, experts thought the best AI models had already mastered this task, reaching about 76% accuracy. But the authors of this paper discovered a hidden flaw: the AI was good at sounding like it knew the rules, but it was actually just memorizing a few simple sentence structures. When faced with complex timing rules, it would "collapse" into a default, generic template that looked right but was legally wrong.

Here is how the paper solves this problem, using simple analogies:

1. The Problem: Mimicry vs. Understanding

The old way of training these AIs was like a student copying a teacher's homework word-for-word. The student gets a grade based on how closely their spelling and grammar match the teacher's answer.

The Flaw: In this field, two sentences can look completely different but mean the exact same thing (equivalent). Conversely, two sentences can look almost identical but have opposite meanings. The old method rewarded the student for copying the words, not for understanding the logic.

2. The Solution: The "Reward-Weighted On-Policy Distillation" (RWOPD)

The authors created a new training method called RWOPD. Think of it as a three-step coaching process involving a Student, a Master Teacher, and a strict Judge.

Step A: The Student Practices (On-Policy)

Instead of just copying the teacher, the Student AI is asked to write its own contracts (called "rollouts") based on a prompt. It tries to solve the problem on its own first.

Step B: The Strict Judge (The Open Property-Equivalence Checker)

This is the paper's secret sauce. The authors built an open-source "Judge" (using tools called SymbiYosys and Z3) that doesn't just check if the grammar is correct. It checks the logic.

The Analogy: Imagine the Judge is a lawyer who takes the Student's contract and the Reference contract and runs them through a simulation.
The Verdict: The Judge asks: "Are these two contracts legally equivalent?"
- If the Student's contract is logically equivalent to the reference, the Judge says "Pass."
- If it's slightly stricter or more lenient, the Judge gives a "Partial Pass."
- If it's wrong, the Judge says "Fail."
Crucial Point: The Judge ignores the Student's answer if it's logically wrong. It acts as a filter, only letting "good" attempts move forward.

Step C: The Master Teacher (Distillation)

Here is where the magic happens. The Student doesn't just learn from the Judge's "Pass/Fail" score.

The Master Teacher (a much larger, 14-billion-parameter AI that is already very good) looks at the Student's successful attempts.
The Teacher says, "Okay, you got the logic right. Now, look at exactly how I would have written those specific words. I will show you the probability of every single word I would have chosen."
The Student then updates its brain to match the Teacher's style, but only on the attempts that the Judge approved.

Why is this better?

Old Way: The Student learns from every attempt, even the bad ones, trying to guess the right words.
New Way (RWOPD): The Student only learns from the "winning" attempts, and it learns the deep logic of how a Master would phrase those winning answers. It's like a student only studying the essays that won the prize, but learning from a Nobel Prize winner's commentary on why those essays were good.

3. The Results: Beating the Giants

The authors tested this method on a 7-billion-parameter model (the Student).

The Competition: They compared it against the previous best specialized model (a 14-billion-parameter AI) and even massive general-purpose models (671-billion parameters).
The Outcome: The small 7B Student, using this new coaching method, beat everyone. It achieved the highest accuracy on the benchmarks, surpassing models that are 100 times larger.
The "Hidden Gap" Fixed: The paper shows that the Student specifically got much better at the hardest types of rules (those involving time and delays), which is where the previous models were failing.

Summary

The paper argues that to teach an AI complex logic, you shouldn't just make it memorize answers. Instead, you should:

Let it try to solve the problem.
Use a strict logic-checker to filter out the wrong answers.
Have a Master Teacher show the Student exactly how to phrase the correct answers.

By combining a logic-checker with a Master Teacher, a small AI can learn to think like a giant, solving a problem that was previously thought to be nearly "saturated" (solved).

Technical Summary: Reward-Weighted On-Policy Distillation for NL-to-SVA Generation

Problem Statement
The generation of SystemVerilog Assertions (SVA) from natural language (NL) specifications using Large Language Models (LLMs) has reached a performance plateau, with the strongest specialized models (e.g., CodeV-SVA-14B) achieving approximately 76% accuracy on the NL2SVA-Human benchmark. However, this aggregate score masks a critical "temporal gap." While models perform well on combinational (C1) logic, they frequently fail on bounded-delay (C2) and liveness (C3) specifications. The dominant failure mode is "template collapse," where models default to a small set of implication templates (e.g., |->) regardless of the specific temporal requirements of the specification.

The root cause identified is the training objective. Existing systems rely on Supervised Fine-Tuning (SFT) on NL/SVA pairs, which optimizes for token-level mimicry of a single reference assertion. This approach fails to capture the semantic nature of SVA correctness, where two assertions can be syntactically distant yet semantically equivalent, or syntactically similar yet semantically opposed. Furthermore, standard Reinforcement Learning (RL) with verifier feedback has proven ineffective in this setting; sparse rollout-level rewards do not provide sufficient signal for small-parameter models (7B) with limited adaptation budgets (LoRA), often failing to surpass the SFT baseline.

Methodology: Reward-Weighted On-Policy Distillation (RWOPD)
The paper proposes Reward-Weighted On-Policy Distillation (RWOPD), a method that integrates a formal verifier into the distillation process to guide a student model toward property-equivalent behavior. The pipeline consists of four key components:

Stratified Curriculum SFT: Before distillation, a 7B student model (Qwen2.5-Coder-7B-Instruct) undergoes a curriculum-based SFT. This stage addresses the bias toward easier temporal classes by training in three stages (C1, C2, C3) with a replay buffer to prevent catastrophic forgetting. Crucially, it employs an operator-token-weighted cross-entropy loss, which up-weights temporal operator tokens (e.g., |->, s_eventually) to ensure the model learns the structural importance of these tokens.
Open Property-Equivalence Checker (PEC): To avoid the cost and closed-source nature of industrial tools like Cadence JasperGold during training, the authors introduce an open-source verifier built on SymbiYosys and Z3. This checker performs bounded model checking (BMC) on the C1 and C2 fragments. It outputs verdicts such as EQUIVALENT, IMPLIES_REF_TO_LM, IMPLIES_LM_TO_REF, NOT_EQUIVALENT, or UNSUPPORTED (for liveness cases outside BMC). The PEC is sound: any EQUIVALENT verdict it produces is guaranteed to be correct, though it may abstain on complex liveness properties.
RWOPD Training Loop:
- The student samples $K$ rollouts for a given prompt.
- The open PEC filters these rollouts, discarding those that are syntactically broken or semantically incorrect (NOT_EQUIVALENT).
- The remaining rollouts are assigned a reward weight based on the PEC verdict (e.g., 1.0 for EQUIVALENT, 0.6/0.4 for one-sided implications).
- A frozen 14B teacher model (CodeV-SVA-14B) provides a dense forward-KL (Kullback-Leibler) divergence target on the kept rollouts.
- The student is updated using a reward-weighted average of these KL losses. This ensures supervision is dense at the token level but grounded in property-equivalent behavior.
Evaluation Protocol: While training relies on the open PEC for filtering and reward shaping, final evaluation uses the industrial-grade Cadence JasperGold prop_eq_checker to ensure completeness and consistency with prior benchmarks.

Key Contributions

RWOPD Algorithm: A novel distillation method that restricts and reward-weights the teacher's dense supervision using a property-equivalence checker. This combines the dense token-level signal of distillation with the semantic grounding of formal verification.
Open SymbiYosys+Z3 PEC: A license-free, sound verifier for the bounded-model-checking fragment of SVA. It provides a reproducible, second-scale verification signal for training-time filtering, distinguishing it from the closed-source oracles used in prior work.
Empirical State-of-the-Art: The method establishes new records on the NL2SVA-Human and NL2SVA-Machine benchmarks.

Results
The authors evaluated a 7B student model with LoRA (0.53% trainable parameters) against specialized 14B models and 671B general-purpose baselines.

Performance: RWOPD achieved the highest pass@1, pass@5, and pass@10 scores on both benchmarks. On NL2SVA-Machine, it surpassed the specialized prior SOTA (CodeV-SVA-14B) by 2–4 percentage points (pp) and 671B models (DeepSeek-R1, GPT-5) by 4–7 pp on pass@1.
Temporal Breakdown: The gains were concentrated on the temporally non-trivial slices. RWOPD improved C2 (bounded delay) by 7.5 pp and C3 (liveness) by 11.5 pp over the 14B teacher, while C1 performance remained saturated.
Ablation Studies:
- Distillation vs. RL: Sparse reward optimization (GRPO, IPO) using the same PEC signal failed to exceed the SFT baseline, confirming that the dense teacher KL is the primary driver of performance.
- Filter vs. Teacher: A rejection-sampling SFT (filtering rollouts without the teacher) provided only marginal gains (~2.5 pp), isolating the dense teacher supervision as the dominant value-add.
- Filter Precision: RWOPD (using both EQUIVALENT and one-sided implications) outperformed strict EQUIVALENT-only filtering, balancing gradient availability with semantic correctness.

Significance and Claims
The paper claims that for the specific regime of 7B models with LoRA adaptation, the verifier is most effective not as a standalone reward signal for RL, but as a reward-shaped filter for dense teacher supervision. The authors argue that the "hidden temporal gap" in NL-to-SVA generation is not due to a lack of data or model size, but rather the misalignment of training objectives (token mimicry vs. property equivalence). By grounding the dense distillation signal in formal property equivalence, RWOPD enables smaller, efficient models to outperform both larger specialized models and massive general-purpose LLMs in generating semantically correct hardware assertions.

The work is presented as a "regime statement": at the 7B+LoRA scale, the combination of a sound open verifier and a strong teacher via on-policy distillation is a more effective use of verification signals than sparse reinforcement learning. The authors acknowledge limitations, including the indirect mechanism for C3 learning (relying on the curriculum seed rather than direct verifier feedback) and the dependence on JasperGold for final evaluation due to the open PEC's incompleteness on liveness properties.