🤖 AI

NoTB: Oracle-Free Triage of LLM-Generated RTL via Cross-Model Formal Consensus

The paper introduces NoTB, an oracle-free framework that leverages sequential equivalence checking across multiple LLM-generated RTL designs to establish a formal consensus signal, enabling high-precision triage of functional correctness without relying on testbenches or golden references.

Original authors: Elisavet Lydia Alvanaki, Je Yang, Biruk Seyoum, Luca P. Carloni

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Elisavet Lydia Alvanaki, Je Yang, Biruk Seyoum, Luca P. Carloni

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of computer chip design, engineers traditionally write code that describes how electricity should flow through a circuit. This code, known as register-transfer level or RTL, acts as a blueprint for the physical hardware. For decades, creating this code has been a meticulous, human-driven process where every line is checked against a trusted reference to ensure the chip will function correctly. Recently, artificial intelligence has begun to write this code for us. Large language models, the same type of technology that can write stories or answer questions, are now being asked to generate these complex hardware blueprints from simple text descriptions. While these models can produce many versions of a design quickly, a critical problem remains: how do engineers know which of these AI-generated designs actually work? In the past, the answer was to compare the AI's output against a perfect, human-made reference or to run it through a simulated test. However, creating these perfect references is expensive and time-consuming, and the tests themselves can sometimes miss hidden errors.

A team of researchers at Columbia University has introduced a new way to sort through these AI-generated designs without needing a perfect reference or a pre-written test. They call their system NoTB. Instead of relying on a single judge to decide if a design is good, or running a simulation that might miss a critical flaw, NoTB asks multiple different AI models to write the same design independently. It then uses a rigorous mathematical tool called sequential equivalence checking to see if the different versions behave exactly the same way under every possible condition. The core idea is that if several different AI models, which have been trained in different ways, all arrive at the exact same behavior, it is highly likely that they have found the correct solution. This approach allows engineers to identify the most trustworthy designs early in the process, saving time and resources before they commit to building the actual chip.

The researchers tested this method on seventy-eight different hardware design tasks. They used four distinct families of large language models to generate multiple versions of each design. For every task, they compared the outputs to see which ones were mathematically identical. They found that when designs from all four different AI families agreed on the same behavior, the system was correct nearly ninety-five percent of the time. This high level of confidence came with a trade-off: the system only made a prediction for about twenty-seven percent of the tasks. However, by lowering the requirement to just three agreeing families, the system could cover thirty-three percent of the tasks while still maintaining an eighty-seven percent accuracy rate. This gives designers a flexible tool: they can choose to be extremely cautious and only accept the most certain designs, or they can accept a slightly higher risk to get more designs approved for further testing.

The study also revealed why older methods of checking AI designs were unreliable. One common approach involved asking a second AI model to act as a judge and predict whether a design was correct. The researchers found that this method was inconsistent; the same design could be marked as correct by one judge and incorrect by another, depending on which AI model was doing the judging. Another method involved running the designs through a test created by an AI. The researchers discovered that the quality of the results depended entirely on the quality of that test. If the test was weak, it might fail to catch errors, leading the system to believe that a flawed design was actually correct. In contrast, the new method does not rely on a test or a judge. It relies on the fact that the designs themselves are proven to be identical across the entire range of possible inputs, making the agreement a property of the design rather than the tool used to check it.

To make this process efficient, the system uses a smart filtering technique. Instead of comparing every single pair of designs, which would take a long time, it first removes exact duplicates and then groups designs that have already been proven to be the same. This reduces the number of comparisons needed significantly. The entire process, from generating the designs to checking them, takes on average between one and sixteen minutes per task, depending on how many versions are generated. The cost of running this system is also manageable, as it avoids the need for expensive, repeated calls to AI models just to act as judges. The researchers emphasize that this method does not replace the need for final verification. Instead, it acts as a triage system, a way to sort the pile of AI-generated designs so that the most promising ones can move forward with confidence, while the uncertain ones are set aside for more traditional, thorough checking.

The findings suggest that the diversity of the AI models used is key to the system's success. The researchers tested what would happen if they removed one of the four AI families from the mix. They found that the system remained robust and accurate even when one model was missing, proving that the confidence signal comes from the collective agreement of the group rather than relying on any single model. This makes the approach practical for real-world use, where access to specific AI models might vary. The study concludes that by shifting the focus from simulated tests to formal mathematical proof of agreement, engineers can build a more reliable pipeline for using artificial intelligence in hardware design. This allows the industry to harness the speed of AI generation while maintaining the rigorous safety standards required for the complex chips that power modern technology.

Technical Summary: NoTB – Oracle-Free Triage of LLM-Generated RTL via Cross-Model Formal Consensus

Problem Statement
Large Language Models (LLMs) are increasingly capable of generating synthesizable Register-Transfer-Level (RTL) designs from natural language specifications. However, a critical bottleneck remains: assessing the functional correctness of these generated designs early in the process without access to a trusted "oracle" (such as a reference testbench or golden RTL model). Constructing such oracles is expensive and error-prone, often unavailable at the start of a design task.

Existing oracle-free approaches suffer from fundamental limitations:

LLM-as-a-Judge: Replaces execution with model-based prediction but lacks formal guarantees and exhibits strong model-dependent bias, leading to inconsistent false-acceptance rates (varying by up to 45% across models).
Simulation-Based Self-Consistency: Relies on behavioral agreement under LLM-generated testbenches. This approach assumes the testbench covers the relevant input space and that incorrect implementations disagree with each other. In practice, weak testbenches can miss critical inputs, allowing incorrect designs to form a "false consensus," while shared error modes can cause multiple models to converge on the same wrong behavior.

Methodology: The NoTB Framework
The authors introduce NoTB, a framework that replaces heuristic agreement with formal cross-model consensus. The core insight is that if implementations produced by independently trained LLM families are proven equivalent over the entire input space via Sequential Equivalence Checking (SEC), their agreement is a property of the designs themselves, not of a finite set of test vectors.

The NoTB workflow consists of four stages:

Multi-Model RTL Generation: For a given specification, the system samples multiple RTL implementations from $M$ independently trained LLM families (e.g., Claude, GPT, Gemini, Qwen). Each model generates $K$ candidates, creating a diverse candidate set $D(s)$ .
Formal Equivalence Clustering: NoTB constructs a graph where nodes are candidate implementations and edges represent proven equivalence. It uses a formal tool (Cadence JasperGold) to perform Sequential Equivalence Checking (SEC) on pairs of designs.
- Two designs are connected only if the tool proves they produce identical outputs for all input sequences.
- Inconclusive results or interface mismatches do not create edges.
- Connected components in this graph form equivalence clusters.
Incremental Pruning: To manage computational cost ( $O(|D|^2)$ $O (∣ D ∣^{2})$ ), NoTB employs lightweight pruning:
- Syntactically identical files are merged via hashing before SEC.
- A union-find data structure tracks equivalence components; once a pair is proven equivalent, future comparisons within that component are skipped.
- Proven non-equivalences between component representatives prevent redundant cross-component checks.
Confidence Scoring via Model Diversity: The system identifies the largest equivalence cluster ( $C^*$ $C^{*}$ ) for a specification. The confidence score is defined as the model count: the number of distinct LLM families represented within that cluster.
- A higher model count indicates that independent models with different inductive biases converged on the same formally proven behavior.
- Designers can set a threshold $\tau$ (e.g., $\tau=4$ ) to accept only specifications where the dominant cluster contains implementations from at least $\tau$ distinct families. This creates a tunable tradeoff between precision (confidence) and coverage (number of accepted specs).

Key Contributions

Demonstration of Testbench Dependency: The paper shows that simulation-based oracle-free triage is highly sensitive to the testbench generator. On a fixed candidate pool, the precision of a four-way simulation agreement varied from 43% to 84% solely based on which LLM generated the testbench.
Introduction of NoTB: A novel framework that substitutes learned/simulation-based agreement with formal cross-model consensus. It clusters candidates by SEC and scores them by the diversity of LLM families in the cluster.
Calibrated Correctness Signal: Empirical evidence that model count is a calibrated signal for correctness. The system demonstrates a monotonic relationship: stronger cross-model formal agreement correlates with lower empirical error rates.

Experimental Results
Evaluated on 78 RTL-generation tasks from the NVIDIA CVDP benchmark using four LLM families (Claude 3.7, GPT-OSS-120B, Gemini 2.5, Qwen 3) with 5 samples each:

Four-Family Consensus ( $\tau=4$ ): Achieved 94.7% precision at 27% coverage.
Three-Family Consensus ( $\tau=3$ ): Achieved 87% precision at 33% coverage.
Comparison to Baselines:
- LLM-as-a-Judge: Showed high instability; false-positive rates varied significantly depending on the generator and the judge model.
- Simulation-Based Clustering: Precision fluctuated wildly (43%–84%) depending on the testbench generator, whereas formal SEC remained stable at ~95%.
Robustness: Leave-one-out ablation studies confirmed that the high-precision signal does not depend on any single specific model family; high precision was maintained even when one family was removed.
Cost: While SEC requires significant compute time (median 1–16 minutes per spec depending on samples), it avoids the massive token costs associated with running thousands of "judge" calls or generating multiple testbenches for every candidate.

Significance and Claims
The paper claims that NoTB provides a reliable, oracle-free basis for high-confidence triage in the early stages of hardware design. By leveraging formal guarantees rather than learned heuristics or finite test vectors, NoTB allows designers to selectively certify a subset of generated RTL designs that are likely correct, enabling them to bypass early-stage checks or prioritize them for final verification. The authors emphasize that NoTB does not replace the need for a trusted oracle or full verification flow; rather, it acts as a filter to identify high-confidence candidates before a golden reference is available. The method is presented as a practical solution to the "triage" problem, offering explicit control over the risk–coverage tradeoff without requiring model-dependent oracles.

Technical Summary: NoTB – Oracle-Free Triage of LLM-Generated RTL via Cross-Model Formal Consensus

More like this