KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware

KernelCraft introduces the first benchmark evaluating agentic LLM systems that use feedback-driven workflows to automatically generate and optimize low-level kernels for emerging hardware with novel ISAs, demonstrating their ability to produce valid, high-performance code that rivals or exceeds traditional compiler baselines.

The Gist

Imagine you are a master chef trying to cook a gourmet meal, but you've just been handed a brand-new, alien kitchen. This kitchen has never been used before. It doesn't have standard ovens, pots, or knives. Instead, it has weird, custom-made gadgets that only work if you press buttons in a very specific, microscopic order.

If you want to cook a simple dish (like a salad), you can't just follow a recipe from a standard cookbook. You have to invent a new way to use these alien gadgets from scratch. If you get the order wrong, the food burns, or the machine explodes. This is exactly the problem facing engineers who build new AI chips.

The Problem: The "Alien Kitchen" Dilemma

In the world of AI, companies are constantly building new, super-fast computer chips (accelerators) to make AI run faster. But every time they invent a new chip, they invent a new "language" (called an Instruction Set Architecture or ISA) that the chip understands.

• The Old Way: To make these chips work, human experts have to sit down and manually write thousands of lines of extremely low-level code (called kernels) to tell the chip how to do math. It's like writing a manual for every single ingredient in your alien kitchen. It's slow, boring, expensive, and prone to errors. If the human makes a typo, the whole chip is useless.
• The Bottleneck: Because this manual coding is so hard, new chips often sit on the shelf gathering dust because no one has the time or energy to write the software to make them run.

The Solution: KernelCraft (The "Super-Chef Apprentice")

The authors of this paper created a tool called KernelCraft. Think of it as hiring a Super-Chef Apprentice (an AI Agent) who is incredibly smart but needs a little guidance.

Instead of the human writing the code, they give the AI:

1. The Recipe: "Make a salad" (e.g., a specific math operation like Softmax or Matrix Multiplication).
2. The Manual: The instruction book for the alien kitchen (the new chip's specifications).
3. The Tools: A set of digital tools the AI can use to test its work.

How KernelCraft Works: The "Try, Fail, Fix" Loop

The genius of KernelCraft isn't just asking the AI to write the code once. It's about how the AI learns.

1. The Attempt: The AI writes a piece of code (a kernel) for the new chip.
2. The Taste Test: The AI uses its tools to run the code on a simulator (a digital twin of the chip).
   • Did it crash? The tool says "Syntax Error."
   • Did it give the wrong numbers? The tool says "Wrong Output."
   • Did it work but take too long? The tool says "Too Slow."
3. The Self-Correction: The AI reads the error message, thinks, "Oh, I forgot to align the memory," or "I used the wrong instruction," and rewrites the code.
4. Repeat: It does this over and over again until the code is perfect (100% correct) and fast.

This is like a chef tasting their soup, realizing it's too salty, adding water, tasting again, and repeating until it's perfect, all without the human chef touching a spoon.

What They Discovered

The researchers tested this "Super-Chef" on three different types of new, alien kitchens (hardware platforms) and asked it to cook over 20 different dishes (AI tasks).

• It Works (Mostly): The AI was surprisingly good. For simple tasks, it could figure out how to use the alien gadgets and write working code in just a few tries. In some cases, it even wrote code that was faster than the code written by human experts or standard compilers.
• It Needs Help: When the tasks got very complex (like cooking a whole 7-course meal instead of just a salad), the AI struggled. It needed more "thinking time" and sometimes needed a human to show it an example of a similar dish first.
• The "Co-Design" Surprise: In one cool experiment, the AI realized the alien kitchen was missing a specific tool needed to cook a new type of dish efficiently. It actually proposed a new tool (a new hardware instruction) to the human engineers. The engineers liked the idea, built the tool, and the AI then used it to cook the dish perfectly. This suggests AI can help design the hardware itself!

Why This Matters

KernelCraft is a big deal because it promises to democratize AI hardware.

• Before: Only huge companies with armies of engineers could build and use new AI chips.
• After: With tools like KernelCraft, we might be able to invent new, specialized chips for specific problems (like medical imaging or self-driving cars) and have an AI write the software for them in days instead of years.

The Bottom Line

Think of KernelCraft as the bridge between a brilliant new invention (a custom AI chip) and the real world. It uses a smart AI agent to act as a translator, turning high-level math problems into the microscopic, low-level instructions that new hardware needs to run. It turns the "impossible manual coding" problem into a "try, fail, and learn" game that AI is surprisingly good at playing.

In short: We are teaching AI to write the code that makes the next generation of AI chips possible.

Technical Summary

Here is a detailed technical summary of the paper "KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware."

1. Problem Statement

The rapid emergence of new AI accelerators with novel Instruction Set Architectures (ISAs) has created a critical bottleneck in software development. Unlike mature ecosystems (e.g., NVIDIA GPUs with CUDA), emerging hardware often lacks mature compiler toolchains capable of automatically mapping high-level tensor programs to optimized, low-level "close-to-metal" assembly kernels.

• The "Programmability Wall": Developers must manually craft assembly kernels, a process that is time-consuming, error-prone, and requires deep expertise in both the specific hardware constraints and the computational workload.
• Limitations of Current LLMs: While Large Language Models (LLMs) have shown promise in generating code for established languages (CUDA, Triton), it is unclear if agentic LLM systems (LLMs that can use tools and iterate) can autonomously generate correct and efficient assembly code for unseen ISAs without prior training data or human examples (zero-shot).
• Gap in Benchmarks: Existing benchmarks (e.g., KernelBench, TritonBench) focus on mature ecosystems and high-level languages. There is no standardized framework to evaluate LLM agents on generating assembly code for custom, emerging accelerators.

2. Methodology: KernelCraft Framework

The authors introduce KernelCraft, the first benchmark designed to evaluate LLM agents on generating low-level assembly kernels for customized accelerators.

A. System Architecture

KernelCraft operates as a feedback-driven agentic loop:

1. Input: The agent receives a task description (e.g., "Generate a GEMM kernel"), the ISA specification, and hardware configuration (memory layout, systolic array size).
2. Tool-Use Workflow: The agent interacts with a suite of native tools via function calling:
   • write_code: Saves assembly to a file.
   • check_syntax: Validates syntax and compilation.
   • run_evaluation: Executes the kernel on a simulator/hardware and compares output against ground truth.
   • view_output: Provides detailed numerical diffs for debugging.
   • grep_docs: Searches ISA documentation for instruction usage.
3. Iterative Refinement: If the kernel fails (syntax error, functional mismatch, or performance issue), the agent analyzes the feedback, consults documentation, and patches the code. This loop repeats until the kernel passes correctness criteria.

B. Benchmark Scope

• Hardware Targets: Three emerging platforms:
  • PLENA: A custom AI accelerator with a proprietary ISA.
  • AMD NPU: Based on the Versal AIE-ML architecture.
  • Coral NPU: A RISC-V based NPU with RVV (RISC-V Vector) extensions.
• Task Complexity: Tasks are categorized into three levels:
  • Level 1 (Primitive): Basic operations (SiLU, Softmax, GEMV, GEMM).
  • Level 2 (Composite): Complex algorithms (MLP, Attention, FlashAttention).
  • Level 3 (End-to-End): Full system blocks (Transformer Decoder, ConvBlock).
• Evaluation Metrics:
  • Success Rate: Functional correctness (output matches ground truth within numerical tolerance).
  • Performance: Speedup relative to the platform's standard compiler baseline (e.g., PLENA compiler, Peano, RVV intrinsics).

3. Key Contributions

1. First Assembly-Level Benchmark for Agentic Systems: KernelCraft evaluates LLMs on generating raw assembly code for novel ISAs, moving beyond high-level languages like CUDA or Python.
2. Tool-Use Evaluation: It isolates the capability of LLMs to autonomously use tools (compilers, simulators, documentation searchers) to debug and refine code, rather than relying on external orchestration loops.
3. Demonstration of Autonomous Optimization: The study shows that agents can not only generate correct code but also discover hardware-specific optimizations (e.g., loop unrolling, memory prefetching, instruction fusion) that outperform standard compiler baselines.
4. Co-Design Capability: The framework demonstrates that agents can identify architectural gaps in an ISA and propose new instructions to solve specific workload bottlenecks (e.g., for Diffusion LLMs).

4. Experimental Results

The authors evaluated four frontier models (GPT-5.2, Gemini-3-Flash, Sonnet 4, DeepSeek R1) across the three hardware platforms.

• Success Rates:
  • GPT-5.2 emerged as the top performer, achieving a 55% success rate on PLENA for primitive operations and significantly outperforming others on complex tasks.
  • Gemini-3-Flash was the second-best performer.
  • DeepSeek R1 struggled significantly, achieving only 1 successful kernel across all platforms, highlighting that reasoning capability alone does not guarantee tool-use proficiency.
  • Complexity Drop: Success rates dropped sharply as task complexity increased (Level 1 > Level 2 > Level 3). End-to-end blocks (Level 3) were largely unsolved, with only one successful generation (Gemini-3-Flash on Coral NPU).
• Performance Gains:
  • Agents generated kernels that matched or exceeded compiler baselines.
  • Coral NPU: Gemini-3-Flash achieved up to 7.93x speedup on ConvBlock by discovering "BatchNorm folding" (fusing normalization into integer arithmetic), a pass absent in the immature compiler toolchain.
  • PLENA: Normalization tasks saw consistent 1.06x–1.22x speedups.
• Ablation Studies:
  • Extended Reasoning: Enabling "thinking" tokens was critical for Level 2+ tasks; without it, success rates dropped to 0%.
  • In-Context Learning: Providing one-shot examples of related workloads significantly improved success rates for complex tasks, especially on platforms with sparse documentation (AMD NPU).
  • Tool-Use Necessity: Models that failed to use tools correctly (e.g., DeepSeek variants) failed completely, proving that tool-calling is a fundamental requirement for this domain.

5. Significance and Future Implications

• Accelerating Hardware Adoption: KernelCraft demonstrates that agentic LLMs can drastically reduce the "time-to-market" for new AI accelerators by automating the creation of low-level kernels, bypassing the need for massive manual engineering efforts.
• Beyond Code Generation: The framework shows potential for ISA Co-Design. In a case study, an agent identified that the existing ISA lacked instructions for efficient Diffusion LLM sampling, proposed new instructions (e.g., V_TOPK_MASK), and successfully implemented the kernel once the human provided the spec.
• Limitations: Current agents still struggle with highly complex, end-to-end system integration and require extensive documentation. Performance degrades significantly when documentation is sparse or when tasks require deep multi-step reasoning without examples.
• Future Directions: The authors suggest integrating formal verification for safety-critical guarantees, embedding ISA knowledge directly into model training, and exploring multi-agent systems where specialized agents handle testing, simulation, and coding separately.

In conclusion, KernelCraft establishes a new paradigm for evaluating AI in hardware development, proving that agentic systems can autonomously bridge the gap between novel hardware specifications and high-performance, low-level software implementation.