🤖 AI

SpecAlign: A Semantic Alignment Framework for SystemVerilog Assertion Generation

This paper presents SpecAlign, a framework that enhances the semantic alignment of LLM-generated SystemVerilog Assertions with natural language specifications through iterative entailment-based evaluation, chain-of-thought reasoning, and self-consistency voting, thereby improving assertion accuracy without relying on golden RTL.

Original authors: Jaime Rafael Imperial, Hao Zheng

Published 2026-05-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jaime Rafael Imperial, Hao Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a master architect (the Design Specification) who has drawn up detailed blueprints for a new, complex house. You want to hire a very fast, very creative, but occasionally dreamy construction foreman (the Large Language Model, or LLM) to write a list of safety rules for the house. These rules are written in a strict, technical code called SystemVerilog Assertions (SVA).

The problem? The foreman is great at writing sentences that look like safety rules and pass the grammar check, but sometimes the rules he writes are actually nonsense or contradict your original blueprints.

For example, your blueprint says, "The front door must lock when the alarm is on." The foreman might write a rule that says, "The front door must lock when the front door is on." It sounds like a rule, and a computer might even say, "Yes, that's a valid sentence," but it makes no sense in the real world.

The Problem: "Valid" but Wrong

Traditionally, to check if these rules are good, engineers would build a physical model of the house (called Golden RTL) and run the rules against it. If the rule didn't break the model, they assumed it was good.

But the paper argues this is like checking if a rule works only by seeing if it fits in a specific model. If the model is perfect, great. But what if you don't have the model yet? Or what if the rule is technically "true" for that specific model but completely misses the point of what you asked for? The foreman might be hallucinating rules that sound smart but are useless.

The Solution: SpecAlign

The authors introduce SpecAlign, a new "Quality Control" framework. Instead of building a physical model to test the rules, SpecAlign acts like a super-smart translator and fact-checker that compares the foreman's rules directly against your original blueprints (the natural language specifications).

Here is how SpecAlign works, using a simple analogy:

1. The Translation Step (Normalizing)

The foreman writes rules in a strict code (SVA). SpecAlign first translates these back into plain English.

Analogy: Imagine the foreman writes in "Construction Code." SpecAlign translates it back to "English" so it can be compared directly to your English blueprints.

2. The Two-Step Fact Check (Alignment Loops)

SpecAlign runs two rounds of checking:

Round 1 (Property Check): It checks the ideas the foreman extracted from your blueprints. Do these ideas match what you wrote?
Round 2 (Rule Check): It checks the actual translated rules. Do they match the ideas and your blueprints?

3. The "Three-Box" Verdict

Instead of just saying "Pass" or "Fail," SpecAlign puts every rule into one of three boxes:

🟢 Entails (Green): The rule perfectly matches your blueprints. "Yes, this is exactly what you asked for."
🔴 Contradicts (Red): The rule directly fights your blueprints. "No, you said the door locks on the alarm, but this rule says it unlocks."
⚪ Unknown (Gray): The rule mentions things your blueprints never talked about. "You mentioned a 'smart lock,' but your blueprints only said 'door.' I don't know if this extra feature is okay or not."

4. The "Self-Correction" Loop

If a rule lands in the Red (Contradicts) box, SpecAlign doesn't just throw it away. It acts like a strict editor:

It tells the foreman exactly where the rule went wrong.
It asks the foreman to rewrite the rule based on the blueprints.
It checks the new rule again.
It repeats this until the rule is Green or, at the very least, clearly Gray.

To make sure the "editor" isn't making mistakes, SpecAlign asks the AI to think through the problem three different ways (like asking three different experts) and then takes a vote on the final answer. This is called Self-Consistency.

The Results: Cleaning Up the Mess

The authors tested this on two real-world designs (like communication protocols for chips, similar to how a USB or a network card talks to a computer).

Before SpecAlign: A competing method generated hundreds of rules. Most were either Red (contradicting the blueprints) or Gray (making up details). Only a tiny fraction were Green (actually matching the blueprints).
After SpecAlign:
- The number of Red (contradicting) rules dropped dramatically. For one design, they went from 148 bad rules down to just 6.
- The number of Green (aligned) rules went up significantly.
- The number of Gray (Unknown) rules went up. Why? Because SpecAlign stopped pretending that rules with made-up details were "good." It correctly identified them as "we don't have enough info to say this is good."

The Big Takeaway

The paper concludes that just because a rule is grammatically correct or passes a computer test doesn't mean it means what you think it means.

SpecAlign provides a way to check if the AI is actually listening to your instructions, without needing to build a physical model first. It turns a pile of "technically valid but useless" rules into a smaller, cleaner set of rules that you can actually trust, while clearly flagging the ones that are too vague to be sure about.

Technical Summary: SpecAlign – A Semantic Alignment Framework for SystemVerilog Assertion Generation

1. Problem Statement

The increasing complexity of modern hardware designs has driven the adoption of Large Language Models (LLMs) for generating SystemVerilog Assertions (SVA) to support Assertion-Based Verification (ABV). However, existing LLM approaches primarily focus on syntactic validity and formal verification (FV) outcomes (e.g., whether an assertion is proven or yields a counterexample on a given RTL). These metrics fail to address semantic alignment: whether the generated assertion accurately reflects the behavioral intent described in the natural language specification.

LLMs can produce assertions that are syntactically correct and formally provable on a specific RTL implementation but are semantically ineffective or hallucinated. For instance, an assertion might sample a clock on the same edge that triggers the property, rendering it vacuous despite passing FV. In the absence of a "golden" RTL model for comparison, manual inspection of such assertions is time-consuming and reduces confidence. Furthermore, FV engines face scalability issues (state-space explosion) when evaluating large sets of generated assertions for complex designs. There is a critical need for a framework that evaluates semantic consistency against the specification itself, independent of RTL implementation details.

2. Methodology: The SpecAlign Framework

SpecAlign is a specification-centric framework designed to evaluate and iteratively refine LLM-generated SVAs without relying on golden RTL. It operates through two primary iterative loops: a Property Alignment Loop and an SVA Alignment Loop. Both loops utilize three core stages: Alignment Checking, Inconsistency Analysis, and Evidence Extraction.

A. Architectural Overview

The framework normalizes the comparison between different representations of design intent. While the input consists of natural language properties ( $P$ ) and generated SVAs ( $A$ ), the framework converts every SVA into a Natural Language Summary ( $N$ ) via a specific extraction process. This normalization allows both properties and assertions to be evaluated on the same semantic level against the design specification ( $S$ ).

B. Core Components

Alignment Checking:
- Classification: The framework classifies natural language descriptions ( $X_i$ $X_{i}$ ) into three categories based on their relationship with the specification:
  - Entails: All elements are supported by the specification.
  - Contradicts: At least one element contradicts the specification.
  - Unknown: Elements are neither supported nor contradicted (often due to implementation-specific assumptions not present in the spec).
- Self-Consistency Voting: To improve reliability, the system employs a Chain-of-Thought (CoT) prompting strategy. It generates $k$ reasoning paths for a given input and aggregates the sub-verdicts using majority voting. In the event of a tie, a conservative priority is applied: Contradicts > Unknown > Entails.
- Reference Selection: The Property Loop uses the design specification ( $S$ ) as the reference. The SVA Loop uses an "Aligned Property Bank" (if available) to reduce noise from scattered specification documents; otherwise, it falls back to $S$ .
Inconsistency Analysis & Refinement:
- When a verdict is Contradicts, the system performs an Inconsistency Analysis. It identifies the specific conflicting elements in the description, references the correct behavior in the specification, and generates structured, actionable feedback ( $F_i$ ).
- This feedback guides the LLM to regenerate the property or assertion ( $P'_i$ or $A'_i$ ), explicitly defining trigger conditions and avoiding the introduction of new signals or behaviors not present in the specification.
- The refined output is re-evaluated in the alignment loop until a fixed number of iterations is reached or the output is aligned.
Evidence Extraction:
- For Entails cases, the system extracts supporting evidence (quotes and locations) from the specification to facilitate human review and strengthen confidence.
- For Unknown cases, the system categorizes which elements are ambiguous or undefined, distinguishing true contradictions from underspecified behavior.
Quantitative Metric:
- The framework defines an Alignment Score ( $a_t$ ) calculated as the ratio of Entailed assertions ( $E$ ) to the total number of assertions ( $n$ ) at a given iteration ( $a_t = E/n$ ). The goal of the refinement process is to maximize this score.

C. Natural Language Summary Extraction

A critical preprocessing step involves converting formal SVAs into structured natural language summaries. The LLM is prompted to identify specific SVA components (Trigger, Antecedent, Implication Operator, Consequent, and Disable If) and map them to equivalent natural language phrases. This ensures that the semantic intent of the formal code is preserved for comparison.

3. Key Contributions

The paper outlines the following contributions:

Semantic Evaluation Framework: A novel approach to assess LLM-generated SVAs against natural language specifications without relying on formal verification applied to golden RTL.
Iterative Refinement Pipeline: A structured framework comprising Alignment Checking, Inconsistency Analysis, and SVA Summary Extraction, designed for integration into existing LLM-based generation workflows.
Quantitative Alignment Metric: An alignment scoring metric based on semantic entailment classification to measure specification consistency, offering a scalable alternative to traditional FV correctness evaluation.
Inconsistency Analysis Mechanism: A method to generate specific, actionable feedback for contradicting properties and assertions, supporting both manual review and automated refinement.
Empirical Validation: Experimental results demonstrating the reduction of contradictory assertions in APB and UART designs through semantic refinement.

4. Experimental Results

The authors evaluated SpecAlign on two hardware designs: APB (APB to I2C Interface) and UART (UART to Bus Interface). The framework was compared against AssertionForge, a baseline method that uses a knowledge graph and incorporates RTL into generation.

APB Results:
- Baseline (AssertionForge): Generated 442 SVAs with an alignment score of 0.02. Only 8 were entailed, while 148 were contradictory.
- SpecAlign: Maintained 442 SVAs but increased the alignment score to 0.15. Crucially, the number of contradictory assertions dropped to 0, with 66 entailed assertions. The remaining 376 were classified as Unknown.
UART Results:
- Baseline: Generated 325 SVAs with an alignment score of 0.06 (21 entailed, 142 contradictory).
- SpecAlign: Generated 319 SVAs with an alignment score of 0.28. Contradictory assertions were reduced to 6, while entailed assertions increased to 92.

Analysis of Results:
The experiments revealed that baseline methods often generate assertions that are either contradictory to the specification or rely heavily on implementation-specific details (RTL) not found in the natural language specification. SpecAlign effectively filters out these contradictions. The increase in "Unknown" classifications in SpecAlign (e.g., 376 for APB) is interpreted not as a failure, but as a more conservative and accurate classification strategy that separates implementation-specific assumptions from specification-supported behavior. This distinction prevents the false positive labeling of unsupported behavior as "entailed."

5. Significance and Claims

The paper claims that SpecAlign provides a scalable complement to traditional formal verification. By separating semantic correctness from RTL proof outcomes, it addresses the limitation that formal verification alone cannot determine if an assertion captures the intended behavior described in the specification.

The authors emphasize that while formal verification is necessary to prove assertions hold on RTL, it is insufficient for evaluating the quality of the assertion's intent. SpecAlign improves confidence in LLM-generated assertions by:

Detecting semantic inconsistencies that formal tools might miss (e.g., vacuous truths).
Reducing debugging efforts by identifying contradictions before RTL simulation or formal proof.
Providing a mechanism to refine assertions iteratively without requiring a golden model.

The paper concludes modestly, noting that while the framework improves alignment, future work is needed to refine classification methodologies, better capture syntactical errors, and extend the approach to other hardware design automation tasks like RTL generation and debugging.