MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?

This paper introduces MobileKernelBench, a framework revealing that current LLMs struggle to generate efficient mobile kernels due to compilation failures and hallucinations, and proposes MoKA, a multi-agent system that significantly improves compilation success and kernel performance.

The Gist

📱 The Big Question: Can AI Write Code for Your Phone?

Imagine you have a super-smart robot (a Large Language Model, or LLM) that is amazing at writing code. It can write scripts for giant, powerful supercomputers (like server GPUs) with ease. But can this same robot write code for your smartphone?

The authors of this paper asked: "Can LLMs write efficient 'kernels' (the tiny, high-speed engines that make apps run) for mobile devices?"

They found that while the robot is smart, it's currently terrible at this specific job. But they built a new tool to fix it.

---

🏗️ The Problem: The "Mobile Jungle" vs. The "Server City"

To understand why this is hard, imagine two different worlds:

1. The Server City (GPUs): This is like a massive, well-organized city with wide highways, unlimited fuel, and strict rules. Everyone drives the same type of car (CUDA). It's easy for the robot to navigate here because the roads are clear and the rules are simple.
2. The Mobile Jungle (Phones): This is a chaotic, fragmented jungle.
   • Different Terrain: Some phones use one type of engine, others use another.
   • Tiny Fuel Tanks: Phones have very limited battery and memory.
   • No Maps: There are very few "instruction manuals" (high-quality examples) for how to drive in this jungle.

The Robot's Failure:
When the researchers asked top-tier AI models to write code for the "Mobile Jungle," they failed miserably.

• Hallucinations: The robot tried to use tools that didn't exist (like trying to drive a boat on a dirt road).
• Compilation Failures: Over 54% of the code the robot wrote wouldn't even start (it wouldn't compile).
• Slow Performance: Even when the code worked, it was often slower than the standard code humans wrote.

Why? The robot was trained on data from the "Server City." It didn't know the specific, messy rules of the "Mobile Jungle."

---

🛠️ The Solution: Building a Test Lab (MobileKernelBench)

Before they could fix the robot, they needed a way to test it properly. They couldn't just ask the robot to "write code" and hope it worked.

They built MobileKernelBench, which is like a high-tech driving test track specifically for mobile phones.

• The Track: It has 190 different "obstacle courses" (tasks) representing 95 different types of math operations needed by apps.
• The Car: They used a specific mobile framework called MNN (Mobile Neural Network).
• The Auto-Pilot: They built a robot arm that automatically:
  1. Takes the code the AI writes.
  2. Installs it on a real phone (a Xiaomi 13).
  3. Runs it to see if it crashes.
  4. Times how fast it is.

This allowed them to see exactly where the AI failed: it couldn't handle the specific, messy details of mobile development.

---

🤖 The Fix: Introducing "MoKA" (The Multi-Agent Team)

Since a single robot couldn't do the job, the researchers created a team of specialized robots called MoKA (Mobile Kernel Agent).

Instead of one robot trying to do everything, they broke the job down into three roles, like a construction crew:

1. The Builder (Coder): This robot writes the initial code.
2. The Inspector (Debugger): If the code crashes or has errors, this robot reads the error message, looks at the "blueprints" (the code repository), and tells the Builder exactly what to fix. It doesn't guess; it checks the facts.
3. The Tuner (Accelerator): Once the code works, this robot looks at the speed. It says, "Hey, this part is slow. Let's try a different gear or a better path."

How they work together:
They use a "Plan-and-Execute" loop.

• The Builder writes code.
• The Inspector checks it. If it fails, they fix it.
• The Tuner checks the speed. If it's slow, they optimize it.
• They repeat this until the code is perfect.

The Result:
MoKA was a massive success!

• Compilation Success: It got the code to run 93.7% of the time (up from ~47% for standard AI).
• Speed: 27.4% of the time, MoKA wrote code that was actually faster than the human-written standard code.

---

🎯 The Takeaway

The Analogy:
Think of standard AI models as generalist chefs. They can cook a great steak in a fancy restaurant (Server GPUs). But if you ask them to cook a meal in a tiny, broken-down food truck with a specific, weird stove (Mobile Phones), they burn the food or use the wrong ingredients.

MoKA is like giving that chef a team of assistants:

• One assistant checks the stove instructions.
• Another tastes the food and says, "Too salty, fix the recipe."
• A third checks the timer and says, "Cook it faster."

Conclusion:
AI can write efficient code for mobile devices, but only if you give it the right tools, the right team structure, and a way to learn from its mistakes in real-time. You can't just ask it to "do it"; you have to guide it through the messy reality of the mobile world.

Technical Summary

1. Problem Statement

While Large Language Models (LLMs) have shown promise in generating code and optimizing kernels for server-grade GPUs (e.g., CUDA), their ability to generate efficient kernels for mobile devices remains unexplored and faces unique challenges. Mobile kernel development differs significantly from server-side development due to:

• Compatibility Priority: Mobile frameworks (like MNN, NCNN) require support for a vast diversity of operators to ensure cross-framework interoperability, unlike the homogeneous CUDA ecosystem.
• Engineering Complexity: The mobile ecosystem is fragmented, involving heterogeneous backends (CPU, GPU, NPU) and strict resource constraints (power, memory, bandwidth).
• Data Scarcity: There is a lack of high-quality, open-source reference implementations for mobile-specific operators, creating a "data-poor" environment that hinders LLM training and generalization.

The central research question is: Can LLMs write efficient, functional kernels for mobile devices?

2. Methodology

A. MobileKernelBench (The Benchmark)

To systematically investigate this, the authors introduced MobileKernelBench, a comprehensive evaluation framework consisting of:

• Dataset: A curated set of 190 tasks derived from 95 primitive operators across 12 categories (e.g., Convolution, Matrix, Reduction). Unlike previous benchmarks focusing on algorithmic complexity, this prioritizes operator diversity and cross-framework interoperability.
• Format: Tasks are defined as (PyTorch, ONNX) pairs to ensure a universal functional definition, bridging the gap between training frameworks and mobile inference engines.
• Automated Evaluation Pipeline: A novel "host-to-device" pipeline that automates the entire lifecycle:
  1. Registration: Injecting generated C++ code into the MNN framework.
  2. Compilation: Cross-compiling via Android NDK.
  3. Verification: Differential testing against ONNX baselines for functional correctness.
  4. On-Device Benchmarking: Running on a real mobile device (Xiaomi 13 with Snapdragon 8 Gen 2) to measure latency and speedup.

B. MoKA (Mobile Kernel Agent)

Recognizing that standard LLMs struggle with this domain, the authors proposed MoKA, a multi-agent system designed to navigate the complexities of mobile kernel development. MoKA operates on a plan-and-execute paradigm with three specialized agents:

1. Coder: Generates initial C++ implementations and applies code modifications based on feedback.
2. Debugger: Activated upon compilation or functional failures. It uses repository-aware tools (to parse codebases and error logs) and model parsing tools (to compare ONNX/MNN graphs) to diagnose and fix syntax or semantic errors.
3. Accelerator: Activated after functional correctness is achieved. It analyzes performance profiling data to identify bottlenecks (e.g., memory access, thread contention) and proposes specific optimization strategies (e.g., SIMD, cache blocking).

• Reflective Memory: Agents share a history of strategies and outcomes to avoid repetitive mistakes and prune ineffective optimization paths.

3. Key Contributions

1. First Systematic Study: Extends automated kernel generation research to the mobile domain, highlighting the distinct gap between server-side and mobile-side development.
2. MobileKernelBench: Introduces the first benchmark dedicated to cross-framework mobile kernel implementation, featuring an automated, hardware-in-the-loop evaluation pipeline.
3. MoKA Framework: Proposes a multi-agent system that effectively handles data scarcity and engineering complexity through iterative reasoning, repository awareness, and performance-driven optimization.
4. Empirical Insights: Demonstrates that standard LLMs fail significantly in this domain due to hallucinations and lack of domain grounding, while agentic workflows can overcome these barriers.

4. Experimental Results

Baseline Performance (Standard LLMs)

The authors evaluated state-of-the-art models (including GPT-5, Claude-Sonnet-4.5, Llama-3.1, DeepSeek, and Qwen) on MobileKernelBench:

• High Failure Rates: Over 54.7% of generated kernels failed at the compilation stage due to hallucinated APIs or invalid framework usage.
• Low Correctness: Functional correctness rates were negligible (e.g., ~13.7% for top models).
• Poor Optimization: Only 16.3% of successful kernels matched the native baseline speed, with a mere 4.7% achieving significant speedups (>1.5x).
• Fine-tuning Limitations: Standard fine-tuning (LoRA) and Reinforcement Learning (GRPO) yielded only marginal improvements, confirming that data scarcity prevents static models from mastering the domain.

MoKA Performance

MoKA, initialized with a strong base model (Claude-Sonnet-4.5) and iterated 10 times, achieved State-of-the-Art (SOTA) results:

• Compilation Success Rate (CSR): Increased from ~46% (baseline) to 93.7%.
• Functional Correctness Rate (FCR): Jumped from ~34% to 75.3%.
• Performance Speedup: 27.4% of kernels achieved a speedup of >1.5x over native libraries (compared to <5% for baselines).
• Case Study: On the LayerNorm2D operator, MoKA achieved a 6.82x speedup over the baseline by iteratively applying SIMD vectorization, FMA instructions, and 64KB cache blocking.

5. Significance

This work fundamentally shifts the paradigm of automated kernel generation for edge computing. It proves that while raw LLMs lack the specific, low-level knowledge required for mobile engineering, agentic frameworks equipped with tool use, repository awareness, and iterative feedback loops can successfully bridge this gap.

The findings suggest that for specialized, data-scarce domains like mobile inference, dynamic context retrieval and multi-agent collaboration are superior to static fine-tuning. This paves the way for automated optimization of mobile AI applications, potentially reducing the engineering burden of deploying models on resource-constrained devices and enabling more efficient on-device AI.