CktEvo: Repository-Level RTL Code Benchmark for Design Evolution

This paper introduces CktEvo, a repository-level benchmark and closed-loop framework that enables large language models to iteratively optimize Power, Performance, and Area (PPA) in complete RTL designs by preserving functional behavior across cross-file dependencies without human intervention.

The Gist

Imagine you are an architect who has just finished designing a massive, complex skyscraper. The building stands tall, the doors open, and the lights work perfectly. It's a "golden" design. But now, the city council (the chip manufacturer) says, "We need this building to use less electricity, be slightly smaller, and have faster elevators, but you cannot change how the rooms function or who lives where."

This is the daily struggle of RTL (Register-Transfer Level) coding in chip design. Engineers spend months tweaking code to make chips faster, smaller, and more power-efficient (known as PPA: Power, Performance, Area).

Enter CktEvo, a new tool and benchmark introduced in this paper. Think of CktEvo not as a magic wand that builds a new skyscraper from scratch, but as a super-intelligent renovation crew that helps you optimize an existing building.

Here is the breakdown of the paper using simple analogies:

1. The Problem: The "Isolated Room" Trap

Previously, AI tools for chip design were like interior decorators who only looked at one room at a time.

• The Old Way: You'd ask an AI, "Make this kitchen more efficient." The AI might suggest moving the fridge. But it didn't know that moving the fridge would block the hallway to the bedroom, causing a traffic jam in the whole house.
• The Reality: Real chip designs are huge "houses" with thousands of files (rooms) connected by complex wiring (hallways). If you change one file without understanding the others, the whole chip might break.
• The Gap: Existing AI could write code from scratch (which often fails because it hallucinates) or fix small bugs, but it couldn't handle the entire house renovation while keeping the structure safe.

2. The Solution: CktEvo (The Master Renovation Plan)

The authors created CktEvo, which is two things:

1. A Test Track: A collection of 11 real-world, complex chip designs (like a CPU, a USB controller, or a neural network engine) that are already built but could be optimized.
2. A Closed-Loop Framework: A system that acts like a foreman who constantly checks the work.

3. How It Works: The "Detective, Architect, and Inspector" Loop

The paper describes a "closed-loop" process. Imagine a team of three working together:

• Step 1: The Detective (Graph-Based Analyzer)
  The AI doesn't just read the code; it builds a map of the building. It looks at the "blueprints" and the "energy reports" from the construction tools. It finds the bottlenecks: "Hey, the elevator in the north wing is too slow because of a weird wiring loop in the basement."

• Step 2: The Architect (The LLM)
  The system asks a Large Language Model (the AI): "Here is the problem area. Can you suggest a way to rewire this specific section to make it faster, without changing what the elevator does?"
  The AI acts like a creative architect, suggesting small, targeted changes (like changing a "blocking" door to a "sliding" door).

• Step 3: The Inspector (Formal Verification)
  Before the changes are made permanent, a strict inspector checks the new plan.

  • Does the door still open? (Functional Equivalence)
  • Did we accidentally break the hallway? (Cross-file dependency check)
  • Is the building actually smaller/faster? (PPA metrics)
    If the answer is "No," the AI tries again. If "Yes," the change is saved.

4. The Results: Small Tweaks, Big Gains

The researchers tested this on their 11 "buildings" using different AI models (like DeepSeek, GPT-4o, and Qwen).

• The Magic: Without any human touching the code, the AI successfully renovated the designs.
• The Score: On average, they reduced the "Area-Delay Product" (a score combining size and speed) by 10.5% using open-source tools.
• The Catch: When they used super-expensive, industrial-grade tools (like the ones Intel or NVIDIA use), the gains were smaller (around 1.7%).
  • Why? Because those industrial tools are already so good at optimizing that there's very little room left to improve. However, even a 1.7% improvement on a chip that costs millions to manufacture is worth a fortune.

5. Why This Matters

Think of CktEvo as the bridge between AI that writes code and AI that engineers chips.

• Before: AI was a junior intern who could write a few lines of code but needed a senior engineer to check everything.
• Now: CktEvo shows that AI can act as a senior engineer that understands the whole system, finds the bottlenecks, and iteratively improves the design until it hits the target, all while ensuring the building doesn't collapse.

The Bottom Line

This paper proves that we can use AI to evolve existing chip designs rather than just generating new ones from scratch. It's like having a tireless, hyper-observant renovation crew that works 24/7 to make your chip faster and smaller, ensuring that every single change is safe and functional before it's ever built.

Technical Summary

Here is a detailed technical summary of the paper "CktEvo: Repository-Level RTL Code Benchmark for Design Evolution."

1. Problem Statement

Register-Transfer Level (RTL) design is an iterative, repository-scale process where Power, Performance, and Area (PPA) metrics emerge from complex interactions across multiple files and the downstream Electronic Design Automation (EDA) toolchain. While Large Language Models (LLMs) have been applied to hardware design, existing approaches suffer from two main limitations:

• Prompt-Based Generation/Debugging: Most efforts rely on natural language prompts to generate code from scratch or debug isolated snippets. This leads to ambiguity, hallucinations, and a lack of functional correctness, requiring extensive human review.
• Isolated Module Optimization: Other approaches start from formal inputs (e.g., HLS code) but optimize only single modules or high-level abstractions. They fail to account for cross-file dependencies, which are critical because PPA outcomes are emergent properties of the entire repository and toolchain, not just isolated files.

There is a significant gap between current LLM capabilities and the industrial need for repository-level, function-preserving, and PPA-driven design evolution.

2. Methodology

The authors propose CktEvo, a benchmark and a reference framework designed to bridge this gap.

A. Task Formulation

The task is defined as an iterative, multi-objective optimization problem:

• Input: An initial "golden" RTL repository ($D_0$) consisting of multiple source files and design constraints ($\Phi$).
• Goal: Generate a sequence of design variants ($D_1, \dots, D_T$) that improve non-functional metrics (specifically Area and Delay) while strictly preserving functional equivalence with the original design.
• Constraints: No new behaviors are allowed; the evolution must pass formal verification.

B. The CktEvo Benchmark

• Dataset: Comprises 11 high-quality, human-reviewed, multi-file Verilog repositories sourced from open-source projects (e.g., GitHub, OpenCore).
• Diversity: Includes diverse domains such as processors (RISC, simple_cpu), controllers (mem_ctrl, sdc_ctrl), interfaces (ethmac, usb, spi), and specialized blocks (hsm, nn_engine, vga_enh).
• Scale: Ranges from compact designs (600 lines) to large, complex cores (12,000 lines with 42 modules).
• Features: All designs are synthesizable by both open-source (Yosys) and commercial toolchains, ensuring baseline stability.

C. Closed-Loop Evolution Framework

The framework operates as an automated agent loop (see Figure 1 in the paper):

1. EDA Toolchain Execution: Runs Logic Synthesis (LS) and Static Timing Analysis (STA) to generate PPA reports.
2. Graph-Based Code Analyzer: Parses RTL source and tool reports to construct a Control Data Flow Graph (CDFG). It annotates the graph with critical timing paths and high-area modules to identify bottlenecks.
3. Prompt Generator: Acts as a "prompt engineer," translating the annotated CDFG and tool reports into a context-rich prompt for the LLM. It specifies the target file, the bottleneck, and the optimization goal.
4. LLM Mutation: The LLM proposes code revisions (diffs) to address the identified bottlenecks.
5. Revision & Rectification: The proposed diffs are applied. A rapid rectification phase uses the LLM to fix syntax or basic synthesis errors.
6. Formal Verification: The modified code undergoes formal equivalence checking (using Synopsys Formality) against the golden design. Only functionally equivalent changes are accepted.
7. Evolution Algorithm: Inspired by AlphaEvolve, the system uses an island-based population model. It maintains multiple independent populations (islands) to enhance search diversity. It employs a dual-cycle evaluation strategy (fast LS for filtering, full LS/STA/Formal for final selection) and uses a MAP-Elites archive to store the best designs found for specific structural feature bins.

3. Key Contributions

1. Formalization of Repo-Level Evolution: Defined the task of optimizing multi-file RTL repositories while preserving functional behavior and optimizing PPA, tied to end-to-end verification flows.
2. First Repo-Level Benchmark (CktEvo): Released a dataset of 11 complex, multi-file Verilog designs representing real-world engineering challenges, moving beyond isolated module benchmarks.
3. Reference Closed-Loop Framework: Provided a working agent framework that couples LLMs with EDA toolchain feedback, enabling cross-file modifications and iterative repair without human intervention.

4. Experimental Results

The framework was tested using three LLMs (DeepSeek-v3, GPT-4o, Qwen3-coder) with both open-source (Yosys) and commercial (Synopsys Design Compiler) toolchains.

• Open-Source Toolchain Results:
  • The framework achieved an average 10.50% reduction in Area-Delay Product (ADP) using DeepSeek-v3.
  • Delay reduction (7.92%) was significantly higher than Area reduction (2.80%), attributed to the graph-based analyzer's ability to pinpoint timing bottlenecks.
  • Design Dependency: Significant gains were seen in control-intensive designs (e.g., processors, controllers) where LLMs successfully refactored state machines. Gains were negligible for mature, standardized interfaces (e.g., USB, Ethernet) where logic redundancy is minimal.
• Commercial Toolchain Results:
  • Improvements were smaller (1.77% ADP reduction) because commercial tools (e.g., compile_ultra) are already highly aggressive, often collapsing the LLM's refactored RTL into the same gate-level netlist as the original.
  • However, the framework still achieved specific wins, such as a 10.61% delay reduction in the audio design and area reduction in hsm, proving that even with advanced tools, RTL-level evolution can unlock marginal gains.
• Evolution Strategies Observed:
  • Coding Style: Converting blocking to non-blocking assignments or optimizing loop constructs.
  • Logic Flattening: Refactoring deep, serialized logic chains into parallel expressions.
  • State Machine Optimization: Merging disparate state flags into unified binary-encoded registers.

5. Significance and Future Work

• Engineering Impact: CktEvo demonstrates that LLMs can autonomously optimize PPA in complex, multi-file repositories without human intervention, offering a viable path for "self-evolving" hardware design.
• Shift in Paradigm: It moves the role of LLMs from "code generators" (which often hallucinate) to "design experts" that iteratively refine existing, verified codebases.
• Future Directions:
  • Architectural Refactoring: Current LLMs are limited to local optimizations. Future agents must perform high-level architectural changes (e.g., altering data/control flows across the whole repository) to compete with commercial tools.
  • Domain Adaptation: Improving base models with specific PDK (Process Design Kit) and EDA toolchain knowledge via Retrieval-Augmented Generation (RAG) or fine-tuning on optimization traces.

In conclusion, CktEvo establishes a rigorous foundation for studying autonomous hardware engineering, proving that repository-level, function-preserving evolution is a feasible and valuable approach for next-generation chip design.