💻 computer science

MultModLM: A multi-modal benchmark for Large-Language Model based hardware schematic generation

This paper introduces MultModLM, a multi-modal benchmark and comprehensive evaluation framework for assessing Large Language Models' ability to generate hardware schematics from RTL descriptions, revealing that while models produce visually interpretable outputs, they struggle with functional correctness and that LLM-based evaluators are unreliable for judging structural precision in hardware design.

Original authors: Dhruv Kulkarni, Sai Manoj Pudukotai Dinkarrao

Published 2026-06-29✓ Author reviewed ⓘ

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Dhruv Kulkarni, Sai Manoj Pudukotai Dinkarrao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read librarian (the Large Language Model, or LLM) who knows how to write stories about machines. Now, imagine you ask this librarian to draw a blueprint of a machine based only on a written description of how that machine works.

This paper, titled MultModLM, is essentially a report card for these "librarians" when they try to draw these machine blueprints (called hardware schematics) from written code (called RTL).

Here is the breakdown of what the authors did and what they found, using simple analogies:

1. The Problem: The "Missing Link"

Usually, when engineers design computer chips, they write code (RTL) and then use expensive, specialized software to turn that code into a visual drawing (a schematic). It's like having a recipe and needing a professional chef to draw a picture of the dish.

Recently, AI models have gotten great at writing code, but nobody had tested if they could draw the machine diagrams from that code. There was no "test" to see if the AI could do this. The authors created this test, called MultModLM.

2. The Test: 99 Different Puzzles

The researchers gathered 99 different "puzzles." These were written descriptions of digital circuits, ranging from simple counters to complex number-checking machines.

The Challenge: They gave these written descriptions to two top AI models (GPT and Gemini) and asked them to draw the circuit diagrams.
The Twist: Unlike a math problem where there is only one right answer, there are many ways to draw the same machine. Just like you can draw a house with a red roof or a blue roof and it's still the same house, the AI could draw the circuit in different ways. This made grading the answers very tricky.

3. The Grading System: A Panel of Judges

Since there was no single "correct" drawing, the authors couldn't just use a computer to check if the drawing was right. Instead, they built a multi-stage judging panel:

The Artist Judges Themselves: The AI drew the picture and then graded its own work.
The Rival Judge: The other AI model looked at the drawing and graded it against the original code.
The Blind Judge: An AI looked at only the drawing (without seeing the code) to see if the drawing made sense on its own.
The Human Expert: Real engineers looked at the drawings to see if they were actually correct.

They used a checklist (rubric) to grade things like: "Did the wires connect correctly?" "Is the clock signal shown?" "Is the drawing easy to read?"

4. The Big Discovery: The AI Judges are Clueless

This is the most surprising part of the paper.

The Result: When the researchers compared the scores given by the AI judges against the scores given by the human experts, they found almost zero agreement.
The Analogy: Imagine you ask a robot to grade a painting. The robot gives it a 10/10 because it looks colorful. The human expert gives it a 2/10 because the perspective is wrong. The paper found that the AI judges were essentially "guessing" or hallucinating when trying to grade these technical drawings. They were completely out of sync with human experts.

5. The Conclusion: AI Can Draw, But Can't Verify

The paper concludes two main things:

AI can try to draw: The models could produce images that looked like circuit diagrams. They weren't perfect, but they were visually interpretable.
AI cannot be the referee: You cannot trust an AI to tell you if a hardware design is correct. The "AI-as-a-judge" method, which works well for writing essays or coding text, fails completely when it comes to checking the structural precision of hardware diagrams.

In short: The authors built a test to see if AI can turn code into pictures of machines. They found that while the AI can try to draw the pictures, it is terrible at grading them. If you want to know if a hardware design is correct, you still need a human expert, not another AI.

Based on the provided paper, here is a detailed technical summary of MultModLM.

Problem Statement

While Large Language Models (LLMs) have seen increasing adoption in hardware development for tasks like code generation, debugging, and text-based evaluation, a significant gap exists in multimodal hardware tasks. Specifically, there is no existing benchmark to evaluate the capability of LLMs to generate hardware schematics from RTL (Register Transfer Level) descriptions.

This gap is critical because:

Non-Uniqueness of Schematics: Unlike code, where a specific output is often required, multiple structurally different schematics can be functionally equivalent. This makes defining a single "ground truth" impossible.
Evaluation Challenges: Visual outputs cannot be directly simulated or formally verified in the same way code can. Traditional EDA tools (e.g., Synopsys, Cadence) required for schematic generation are expensive and require high domain expertise, limiting accessibility for researchers.
Reliability of Automated Judges: There is a lack of understanding regarding whether LLMs can reliably evaluate structural and functional correctness in hardware domains, a prerequisite for scalable automated evaluation.

Methodology

The authors introduce MultModLM, a novel benchmark and a multi-stage evaluation framework designed to address these challenges.

1. Dataset Construction

Scope: The dataset consists of 99 diverse RTL modules sourced from open repositories (e.g., OpenRISC) and human generation.
Composition: The modules cover arithmetic units, finite state machines (FSMs), control logic, and numeric property checkers (e.g., Armstrong number detection, palindrome detection).
Complexity: The designs range from small combinational circuits (1–20 lines) to multi-stage sequential designs (up to 430+ lines).
Ground Truth: No ground-truth schematics are provided due to the non-uniqueness of schematic representations. Correctness is assessed via functional equivalence to the RTL description through a structured rubric.

2. Evaluation Pipeline
To overcome the limitations of single-method evaluation, the authors propose a five-stage pipeline:

Prompting: Models are instructed to generate a clear, interpretable PNG image of the schematic based on the RTL code. Strict constraints are applied to prevent models from retrieving external images or providing only text explanations.
Self-Evaluation: The generating model evaluates its own output against a predefined rubric to establish a baseline for self-assessment bias.
Grounded Cross-Evaluation: A secondary model evaluates the schematic of the target model with access to the original RTL code. This approximates objective assessment by checking logical equivalence.
Blind Cross-Evaluation: The secondary model evaluates the schematic without access to the RTL code to measure visual coherence and self-consistency.
Human Evaluation: Human raters perform structured verification using the same rubric to provide an independent assessment.

3. Evaluation Metrics

Rubric-Based Scoring: A JSON-defined rubric with 8 criteria (e.g., Structural Fidelity, Data Path Representation, Clocking Accuracy, Signal Labeling) is used. Each criterion is scored as a Boolean True/False to avoid the subjectivity of graded scales.
Statistical Analysis: Cohen's Kappa ( $\kappa$ ) is calculated to measure the agreement between human raters and LLM-based evaluators.
Accuracy: Measured by the number of rubric criteria met (max 8 points per prompt).

Key Contributions

MultModLM Benchmark: The first multimodal benchmark specifically for evaluating RTL-to-schematic generation using LLMs.
Structured Evaluation Framework: A novel pipeline combining rubric-based scoring, cross-model assessment, and human validation to handle the non-uniqueness of schematic outputs.
Empirical Evidence on LLM-as-a-Judge: The work provides data demonstrating the limitations of using LLMs as evaluators in structurally precise domains.

Experimental Results

The study evaluated state-of-the-art models (GPT 5.2 Go and Gemini 3 Flash Pro) on the 99-module dataset.

LLM-as-a-Judge Reliability: The most significant finding is that LLM-based evaluators exhibit near-zero agreement with human raters.
- GPT vs. Human Kappa: 0.013
- Gemini vs. Human Kappa: 0.000
- This indicates that LLMs are currently unreliable as automated judges for hardware schematic correctness, even when provided with grounding RTL code.
Model Performance:
- Gemini showed marginally higher accuracy (0.314) than GPT (0.297) when rated by the secondary model.
- Gemini performed better on semantic and functional criteria (C1–C6), suggesting it preserves the logical core of the RTL better.
- GPT achieved slightly higher scores on visual-oriented criteria (C7–C8), though the authors note this may reflect layout cleanliness rather than strict adherence to schematic conventions.
Win/Loss Comparison: In pairwise comparisons, Gemini won 37 prompts, GPT won 33, and 29 resulted in ties.

Significance and Claims

The paper claims that MultModLM exposes a fundamental bottleneck in the field: the inability to reliably evaluate multimodal hardware outputs.

Limitation of LLM-as-a-Judge: The near-zero agreement between LLMs and humans reveals that current LLMs cannot be trusted to evaluate structural and functional correctness in hardware domains. This challenges the applicability of "LLM-as-a-judge" paradigms in fields requiring precise structural reasoning.
Need for New Methodologies: The findings motivate the development of more robust, domain-aware evaluation methodologies and tools for structural evaluation (e.g., formal equivalence checkers) that can serve as ground truth proxies at scale.
Future Direction: While the current dataset is intended for evaluation rather than training, the authors suggest that future work should focus on extending the dataset for fine-tuning and developing automated tools to bridge the gap between visual outputs and formal verification.

The paper concludes that while LLMs can generate visually interpretable schematics, their functional correctness remains constrained, and reliable evaluation of these outputs remains an open challenge.