🤖 machine learning

KernelEvolve: Scaling Agentic Kernel Coding for Heterogeneous AI Accelerators at Meta

KernelEvolve is an agentic framework that automates the generation and optimization of high-performance kernels for diverse deep learning recommendation models across heterogeneous AI accelerators, significantly reducing development time while ensuring correctness and delivering substantial performance gains over existing baselines.

Original authors: Gang Liao, Hongsen Qin, Ying Wang, Alicia Golden, Michael Kuchnik, Yavuz Yetim, Jia Jiunn Ang, Chunli Fu, Yihan He, Samuel Hsia, Zewei Jiang, Dianshi Li, Uladzimir Pashkevich, Varna Puvvada, Feng Shi

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gang Liao, Hongsen Qin, Ying Wang, Alicia Golden, Michael Kuchnik, Yavuz Yetim, Jia Jiunn Ang, Chunli Fu, Yihan He, Samuel Hsia, Zewei Jiang, Dianshi Li, Uladzimir Pashkevich, Varna Puvvada, Feng Shi, Matt Steiner, Ruichao Xiao, Liyuan Li, Nathan Yan, Xiayu Yu, Zhou Fang, Roman Levenstein, Kunming Ho, Haishan Zhu, Alec Hammond, Richard Li, Ajit Mathews, Kaustubh Gondkar, Abdul Zainul-Abedin, Ketan Singh, Hongtao Yu, Wenyuan Chi, Barney Huang, Sean Zhang, Noah Weller, Zach Marine, Wyatt Cook, Carole-Jean Wu, Gaoxiang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Translation" Nightmare

Imagine Meta runs a massive global advertising system. To show you the perfect ad, this system has to do complex math billions of times a day. This math is done by specialized computer chips called AI accelerators (like NVIDIA GPUs, AMD GPUs, and Meta's own custom chips called MTIA).

However, there is a huge problem: These chips speak different languages.

An NVIDIA chip speaks "CUDA."
An AMD chip speaks "ROCm."
Meta's custom chip speaks a unique dialect of "Triton."

Furthermore, the "words" (math operations) needed for ads are incredibly diverse. Some are standard math (like multiplication), but many are weird, custom tasks specific to ads, like sorting user history or hashing data.

The Old Way:
To make the ads run fast, human engineers had to manually write a specific "translation" (a kernel) for every single math task on every single type of chip.

The Analogy: Imagine you are a chef trying to serve a meal to 1,000 guests, but each guest speaks a different language and has a different diet. You have to manually write a unique recipe and translate it into a different language for every single guest. If you miss one translation, that guest doesn't get fed. If you want to change the menu (update the AI model), you have to rewrite all 1,000 recipes from scratch. This takes weeks, is prone to errors, and is incredibly expensive.

The Solution: KernelEvolve (The "Super-Translator" Robot)

The paper introduces KernelEvolve, an AI agent system that automates this entire process. Instead of humans writing the code, KernelEvolve acts like a tireless, super-smart robot chef that can instantly learn any language and write the perfect recipe for any guest.

Here is how it works, step-by-step:

1. The "Tree Search" (Trying Many Paths)

When KernelEvolve needs to solve a math problem, it doesn't just guess once. It uses a Tree Search.

The Analogy: Imagine you are trying to find the fastest route through a giant maze. A human might try one path, get stuck, and give up. KernelEvolve is like a swarm of ants. It sends out thousands of "explorers" simultaneously. Some go left, some go right. If an explorer hits a wall (a code error), it stops. If an explorer finds a shortcut (a faster way to do the math), it marks that path as "good" and sends more ants down that route.
It keeps doing this, learning from every dead end and every success, until it finds the absolute best path.

2. The "Memory Library" (Learning from the Past)

KernelEvolve has a massive, organized library of notes called a Persistent Knowledge Base.

The Analogy: If you are trying to fix a leaky faucet, you don't just guess; you look up a manual. KernelEvolve does this, but its manual is huge. It contains specific instructions for every type of chip (NVIDIA, AMD, MTIA).
The Magic: Even for Meta's custom chip (MTIA), which is so new that no AI has ever seen it before, KernelEvolve has a special "instruction manual" injected into its library. It reads these notes, learns the chip's unique quirks, and writes code specifically for it. This allows it to work on hardware that doesn't exist in the public internet's training data.

3. The "Self-Correcting Loop" (Debugging Itself)

Once the robot writes a piece of code, it doesn't just trust it. It runs a Self-Correction Loop.

The Analogy: The robot writes a recipe, cooks the dish, and then tastes it.
- Taste Test (Correctness): Does it taste like the original? (Does the math match the expected result?)
- Speed Test (Performance): Did it cook faster than the old way?
- The Fix: If the dish is too salty (too slow) or tastes wrong (math error), the robot analyzes why. It looks at the "kitchen logs" (profiling data) to see if the stove was too hot or the ingredients were mixed wrong. It then rewrites the recipe and tries again.
It repeats this cycle hundreds of times for a single task, refining the code until it is perfect.

4. The "Remote Kitchen" (FaaS)

To test these recipes, the robot needs to use the actual expensive chips.

The Analogy: The robot (the brain) lives in a standard office computer. The chips (the ovens) are in a separate, high-tech kitchen. Instead of the robot walking to the kitchen, waiting for the oven, and walking back, it sends the recipe to a Remote Kitchen Service (FaaS). The remote kitchen cooks the dish and sends the results back instantly. This allows the robot to test thousands of recipes at the same time without getting stuck waiting for an oven.

The Results: Why It Matters

The paper claims that KernelEvolve is a game-changer for three reasons:

Speed: It reduced the time to create these specialized codes from weeks to hours.
Performance: The code it writes is incredibly fast. In tests, it made the AI run 1.2 to 17 times faster than the standard, human-written code.
- Example: For a specific data sorting task, it was 9.8 times faster. For a specific math operation on Meta's custom chip, it was 17 times faster.
Reliability: It passed 100% of the correctness tests. It didn't just make things fast; it made sure the math was right every single time.

Summary

KernelEvolve is an AI system that automates the boring, difficult, and expensive job of translating computer code for different types of AI chips. By using a "try-and-learn" strategy (Tree Search), a massive library of hardware rules (Knowledge Base), and a self-correcting loop, it can generate high-speed code for Meta's ad system in hours rather than weeks, making the whole system faster and cheaper to run.

Technical Summary: KernelEvolve

Problem Statement

The deployment of Deep Learning Recommendation Models (DLRMs) for large-scale advertising systems at Meta faces a "curse of dimensionality" driven by three intersecting factors: model architecture diversity (ranging from traditional MLPs to Transformer-based sequence models), kernel primitive diversity (extending beyond standard GEMM to hundreds of complex data preprocessing operators), and hardware heterogeneity (spanning NVIDIA GPUs, AMD GPUs, and proprietary Meta Training and Inference Accelerators like MTIA).

This complexity creates a critical bottleneck. Manual kernel development is economically infeasible, requiring 2–8 weeks of expert effort per operator-hardware combination. Furthermore, the lack of optimized kernels for specific preprocessing operators on proprietary accelerators forces "disaggregated serving" architectures. In these setups, preprocessing must run on under-provisioned host CPUs while neural network inference runs on accelerators, incurring 10–20ms of pure network latency overhead. This architectural tax consumes a significant portion of the sub-100ms latency budget required for ad-serving, directly impacting user experience and revenue. Existing AI-powered coding agents are limited by narrow scopes, synthetic benchmarks, single-platform focus, and a lack of autonomous, persistent knowledge management required for production-grade optimization.

Methodology

KernelEvolve is an agentic framework designed to automate the generation, optimization, and validation of high-performance kernels across heterogeneous hardware. It treats kernel optimization as a graph-based search problem rather than a simple code generation task.

Core Architecture

Tree Search and State Machine: The system models the optimization process as a search graph $G_t = (V_t, E_t)$ . It employs strategies including Greedy Search, Monte Carlo Tree Search (MCTS), and evolutionary algorithms. The process iterates through selection, transformation, and scoring:
- Fitness Function: Evaluates kernels based on speedup over PyTorch baselines, with correctness (numerical accuracy) as a hard constraint (failure = 0 fitness).
- Universal Operator: Unlike static multi-operator frameworks, KernelEvolve uses a single, context-aware operator. It dynamically synthesizes prompts based on runtime execution context (errors, profiling data) rather than fixed templates.
Agentic Retrieval & Self-Managed Context:
- Context Memory Sub-Agent: Analyzes runtime artifacts (logs, profiling metrics, correctness results) to diagnose bottlenecks and synthesize optimization directives. It maintains a persistent metadata store (relational database) tracking the search tree, fitness scores, and parent-child relationships, enabling fault tolerance and cross-session knowledge reuse.
- Deep Search Sub-Agent: Performs targeted retrieval from a Persistent Knowledge Base. This knowledge base encodes hardware-specific constraints (NVIDIA, AMD, MTIA), optimization patterns, and debugging methodologies in a hierarchical file system. For proprietary hardware like MTIA, which lacks representation in LLM training corpora, this injection of domain-specific documentation (e.g., MTIA's SFU operations, inter-PE communication primitives) is critical for generating correct code.
Evaluation and Tooling Framework:
- Dual Implementation Interface: Generated kernels are wrapped in a standardized interface alongside a PyTorch baseline for automated correctness validation (torch.allclose) and performance benchmarking.
- Hardware Interpreters: Dedicated environments (Bento-based) for NVIDIA, AMD, and MTIA ensure reproducible execution with up-to-date toolchains.
- Unified Profiling (Triton MPP): Meta's Multi-Pass Profiler unifies instrumentation across abstraction layers (MLIR, assembly, runtime), providing structured, instruction-level performance data without the fragmentation of traditional toolchains.
- FaaS Integration: Evaluation is offloaded to a Function-as-a-Service platform, decoupling CPU-bound generation from GPU/accelerator-bound evaluation to maximize resource utilization.

Key Contributions

Production-Grade AI Kernel System: KernelEvolve is the first AI-powered kernel optimization system deployed at industrial scale for recommendation model training and inference. It autonomously generates optimized Triton kernels for hundreds of models serving billions of users.
Automated Optimization for Proprietary Hardware: By integrating a persistent knowledge base with retrieval-augmented prompt synthesis, the system effectively generates kernels for proprietary architectures (MTIA) absent from LLM training data, bridging the gap between model capabilities and hardware constraints.
Performance and Efficiency Gains: The system reduces development time from weeks to hours. It achieves substantial speedups (1.2x to 17x) over PyTorch baselines across diverse workloads, including data preprocessing, convolutional transformers, and ranking models.
Operational Insights: The paper shares failure mode analyses, debugging strategies for incorrect kernel generation, and organizational patterns for integrating automated workflows into production environments.

Results

KernelEvolve was validated on the publicly available KernelBench suite and a custom suite of 160 PyTorch ATen operators across three heterogeneous platforms (NVIDIA H100, AMD MI350, MTIA v3).

Correctness: Achieved a 100% pass rate on all 250 KernelBench problems and 100% correctness across all 480 operator-platform configurations.
Performance: Demonstrated speedups ranging from 1.25x to 17x over PyTorch baselines. Notable improvements include:
- Data Preprocessing: MapId (4.1x), MergeBucketizedDense (9.3x), Batch Event Truncate (9.8x).
- Convolutional Transformers: Conv1d (6.5x), Conv2d (4.7x).
- Ranking Models: WuKong Optimized FM (4.0x), InterFormer PFFN (2.5x).
- MTIA Specifics: RMSNorm 2D backward (17x).
Search Efficiency: Optimization trajectories show that graph-based search with execution feedback consistently improves fitness scores over iterative steps, discovering fusion strategies and tiling configurations that manual development would take weeks to explore.

Significance

The paper argues that KernelEvolve fundamentally transforms kernel development from a manual, expertise-dependent process into an automated, scalable service. Its significance lies in:

Enabling Monolithic Accelerator Deployment: By automating the generation of kernels for all operators (including low-arithmetic-intensity preprocessing), it removes the binary constraint that forces disaggregated CPU-accelerator architectures. This eliminates the 10–20ms network latency tax, directly improving serving efficiency and reducing Total Cost of Ownership (TCO).
Scalability to Heterogeneity: It demonstrates that agentic AI can effectively navigate the combinatorial explosion of operator-hardware pairs, making the deployment of new models on new hardware generations feasible without waiting for manual kernel porting.
Bridging the Knowledge Gap: The approach of injecting proprietary hardware knowledge into a persistent knowledge base allows LLMs to generate high-quality code for custom accelerators without retraining, offering a blueprint for future AI system design and optimization at scale.