💻 computer science

Autoformalizing Memory Specifications with Agents

This paper presents a method for automatically converting natural language DRAM specifications into a formal representation called DRAMPyML to enable end-to-end design verification tasks, accompanied by the release of the DRAMBench dataset to evaluate hardware autoformalization capabilities.

Original authors: Jan Ole Ernst, Dmitri Michelangelo Saberi, Derek Christ, Thomas Zimmermann, Rajath Salegame, Suhaas M. Bhat, Stanislav Levental, Thomas Dybdahl Ahle, Matthias Jung

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jan Ole Ernst, Dmitri Michelangelo Saberi, Derek Christ, Thomas Zimmermann, Rajath Salegame, Suhaas M. Bhat, Stanislav Levental, Thomas Dybdahl Ahle, Matthias Jung

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to build a incredibly complex, high-speed train system (a computer chip). The engineers have a massive, thick rulebook written in plain English that describes exactly how the train should behave, when it should stop, and how fast it can go. This is the specification.

The problem is that the people who actually build the train tracks and signals (the verification team) can't just read the English rulebook. They need a strict, mathematical "code" that a computer can check to ensure the train never crashes. Translating that thick English rulebook into this strict code is currently a slow, manual, and error-prone process. If they get it wrong, the train might derail later, costing millions to fix.

This paper introduces a new "AI translator" designed to solve this problem specifically for memory chips (the kind of RAM that powers your phone, computer, and AI servers).

Here is a breakdown of their approach using simple analogies:

1. The Problem: The "Lost in Translation" Gap

Currently, humans act as the translators. They read the English rulebook (like the JEDEC standards for DDR5 or HBM memory) and manually write the strict code.

The Issue: The rulebooks are huge (some are nearly 500 pages long) and full of subtle details. Humans get tired, and AI models often get confused by the specific jargon, leading to "hallucinations" where the AI invents rules that don't exist.
The Consequence: The verification process takes up more than half the time to build a chip, and mistakes can slip through.

2. The Solution: A Specialized "Middleman" Language

Instead of asking the AI to jump straight from "English Rulebook" to "Strict Code" (which is like asking someone to translate a novel directly into a programming language without a dictionary), the authors created a middleman language called DRAMPyML.

The Analogy: Think of DRAMPyML as a universal blueprint.
- The AI reads the English rulebook.
- It draws a blueprint in DRAMPyML (which looks like a flowchart of traffic lights and train stations).
- This blueprint is then automatically converted into the final strict code needed for testing.
Why this helps: It's easier for the AI to get the "flow" of traffic right in a blueprint than to get every single line of final code perfect immediately. If the blueprint is right, the final code will be right.

3. The "Agent": A Self-Correcting Intern

The paper doesn't just use a standard AI chatbot. They built an AI Agent that acts like a diligent, self-correcting intern.

How it works:
1. Reads: The agent reads the English rulebook.
2. Drafts: It writes a draft of the blueprint (DRAMPyML).
3. Tests: It runs its own tests on the blueprint to see if it makes sense (e.g., "Did I create a traffic jam where no trains can move?").
4. Fixes: If the test fails, the agent reads the rulebook again, finds its mistake, and edits the blueprint.
5. Repeats: It loops through this process until the blueprint passes all tests.

This is called an "Agentic" approach because the AI has tools to check its own work, rather than just guessing once and hoping for the best.

4. The "Benchmark": The Training Gym

To prove their method works, the team created a massive gym called DRAMBench.

They manually built 13 perfect "gold standard" blueprints for different types of memory chips (from old DDR2 to the newest HBM3).
They then let their AI Agent try to recreate these blueprints from the English rulebooks.
The Score: They measured how close the AI's blueprint was to the "gold standard" using a math score called the Jaccard Index (think of it as a "similarity percentage").

5. The Results: What Worked?

The "One-Shot" Fail: When they asked the AI to do it in one single try (without checking its work), it struggled, especially with complex memory types. It was like asking a student to write a thesis without being allowed to edit it.
The "Agent" Success: The self-correcting agent did much better. It could fix its own mistakes.
- With no examples to look at, it still managed to get the basics right for simpler chips.
- With examples (showing it a blueprint for an older chip), it became very accurate, sometimes creating perfect blueprints for complex chips.
The Cost: The more complex the chip, the more "brain power" (tokens) the AI needed. However, the agent was worth the extra cost because it produced much higher quality results than the one-shot attempts.

Summary

The paper claims that by using a specialized "blueprint" language (DRAMPyML) and an AI that can check and fix its own work (the Agent), they can automatically turn confusing English memory chip rulebooks into accurate, testable models. This reduces the need for humans to do the tedious translation work, potentially speeding up chip design and catching errors earlier.

They have released their "gym" (the dataset) so other researchers can test their own AI translators against these same memory chip standards.

1. Problem Statement

The Design Verification (DV) phase of semiconductor chip design is a major bottleneck, consuming over 50% of the design lifecycle. The core challenge lies in translating unstructured natural language specifications (e.g., JEDEC standards for memory chips) into formal, executable models required for verification.

Current AI approaches typically focus on generating isolated SystemVerilog Assertions (SVAs). However, this approach has significant limitations:

Incompleteness: SVAs check specific properties but do not generate complete behavioral models, stimulus generators, or functional coverage metrics.
Ambiguity: Natural language specifications often contain ambiguities that lead LLMs to generate syntactically invalid or protocol-misaligned assertions.
Complexity: Modern memory standards (DDR5, HBM3, GDDR7) involve intricate timing constraints, complex command interactions, and hierarchical structures that are difficult to capture with simple prompt-to-code generation.

2. Methodology

The authors propose an Agentic Autoformalization Framework that converts natural language memory specifications into a structured, executable intermediate representation called DRAMPyML, rather than directly generating SVAs.

2.1 The Intermediate Representation: DRAMPyML

Instead of jumping to assertions, the system targets DRAMPyML, a Python-based modeling language built on Timed Petri Nets.

Structure: It models memory devices as a tuple of places (banks, ranks), transitions (commands like ACT, RD, WR), and arcs (regular, inhibitor, reset, and timed).
Advantages: This representation is executable, allowing for the generation of command traces, stimulus, and timing checks. It serves as a unified source from which SVAs, testbenches, and coverage models can be derived downstream.
Formal Definitions: The paper defines DRAM Petri nets mathematically and introduces the concept of Trace Equivalence. It posits (and empirically validates) Conjecture 3.4: that verifying trace equivalence for small configurations (e.g., 1 or 2 banks) with short trace lengths ( $k=4$ ) is sufficient to prove equivalence for the full, complex device due to symmetry properties.

2.2 The Autoformalization Agent

The system employs an iterative Agentic Workflow (Figure 2 in the paper) rather than a one-shot generation approach:

Input: The agent receives JEDEC PDF specifications, vendor datasheets, and the DRAMPyML library.
Iterative Loop: The agent uses tools (MCP servers, LLM APIs) to:
- Read specific sections of the specification.
- Write or edit Python code to build the Petri net graph.
- Self-Validate: Run internal validation tests (checking for deadlocks, orphaned nodes, and generating command traces to cross-check against the spec).
Refinement: Based on validation feedback, the agent iteratively refines the model until structural consistency and timing correctness are satisfied.
Prompting Strategies: The study compares One-Shot prompting (single prompt with explicit conventions) vs. Agentic (iterative interaction with the environment and tools).

3. Key Contributions

DRAMBench: The release of the first open-source benchmark for hardware autoformalization, containing ground-truth DRAMPyML formalizations for 13 JEDEC standards (covering DDR2-5, LPDDR2-5, GDDR5-7, and HBM2-3).
Agentic Framework: Demonstration that an iterative agent using tool-augmented validation significantly outperforms one-shot generation, particularly for complex specifications.
Theoretical & Empirical Validation:
- Formal definition of DRAM Petri nets and trace equivalence.
- Empirical proof that checking short traces ( $k \le 4$ ) on minimal configurations (1-2 banks) is sufficient to detect semantic bugs in the generated models, bypassing the state-space explosion problem.
Benchmarking Results: A comprehensive evaluation of state-of-the-art models (Sonnet 4.5, GPT-5.2, DeepSeek-v3p2) showing the trade-offs between token cost and performance.

4. Experimental Results

The authors evaluated models on the DRAMBench dataset using two metrics:

Jaccard Index: Measures the overlap of legal command traces between the generated model and ground truth.
Timing Constraint Recall: Measures the fraction of ground-truth timing constraints correctly recovered.

Key Findings:

Agentic vs. One-Shot: The agentic approach consistently achieved higher peak performance, especially for complex standards (e.g., DDR5, HBM3) and when no examples were provided. One-shot generation struggled with the unfamiliar output format without strong priors.
Impact of Examples: Providing "cherry-picked" examples (e.g., using DDR3 to help generate DDR4) significantly boosted performance for both methods, sometimes allowing one-shot models to outperform agents on simpler tasks.
Model Performance:
- DeepSeek-v3p2 (open-weight) performed competitively with closed-source models (Sonnet 4.5, GPT-5.2), often surpassing them in the agentic setting.
- Sonnet 4.5 achieved the highest structural perfection for standards like HBM2 and DDR2-4 when provided with examples.
Failure Modes: Analysis of complex failures (e.g., DDR5) revealed that agents struggle with:
- Selecting correct primitives (e.g., wrong precharge commands).
- Modeling memory hierarchy (e.g., bank groups).
- Enforcing timing constraints at the correct abstraction level (coordinate mismatches causing timing arcs to become inactive).
Token Efficiency: While agents achieve higher accuracy, they consume more tokens. However, for complex specs, the "token vs. performance" curve favors the agent, as one-shot models fail to converge on correct structures regardless of token count.

5. Significance

This work represents a paradigm shift in hardware verification:

From Assertions to Models: It moves beyond generating isolated assertions to creating executable, structured behavioral models. This enables a "shift-left" in verification, allowing for early detection of design flaws and the automatic generation of diverse collateral (stimulus, coverage, SVAs).
Scalability: By leveraging the DRAMPyML intermediate representation and trace-equivalence checking, the method offers a scalable path to verifying modern, highly parallel memory devices that were previously too complex for automated formalization.
Open Science: The release of DRAMBench provides a standardized, professional-grade benchmark that was previously missing from the AI-for-Hardware literature, enabling the community to track progress in hardware autoformalization.

In conclusion, the paper demonstrates that agentic workflows combined with structured intermediate representations (Petri Nets) are the most viable path toward automating the verification of complex semiconductor standards, overcoming the limitations of current direct-to-assertion LLM approaches.