🤖 AI

BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal

The paper introduces BaseRT, a native Metal inference runtime for Apple Silicon that leverages chip-specific optimizations to achieve record-breaking throughput for large language models, outperforming existing frameworks like llama.cpp and MLX by up to 1.56x.

Original authors: Prabod Rathnayaka, Fabian Waschkowski, Lukas Wesemann

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Prabod Rathnayaka, Fabian Waschkowski, Lukas Wesemann

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Building a Custom Race Car

Imagine you want to drive a race car (a Large Language Model, or LLM) on a very specific track (Apple Silicon chips like the M3 or M4).

Currently, most people drive these cars using a "universal adapter" or a "rental car kit" (existing software like llama.cpp or MLX). These kits are great because they work on many different tracks, but they aren't perfectly tuned for this specific Apple track. They have extra weight, unnecessary steps, and a driver who doesn't know the shortcuts.

BaseRT is a new, custom-built race car designed only for the Apple track. The team at Base Compute stripped away the heavy adapters and built a vehicle from the ground up to match the track's unique curves and rules. The result? It drives significantly faster.

The Problem: Why Current Software is "Slow"

The paper argues that existing software for running AI on Macs is like a delivery truck making too many stops.

The "Middleman" Problem: Most software uses a "framework" (like a middleman) to talk to the computer's graphics chip (the GPU). This middleman adds extra steps, like a manager checking a clipboard before every single task.
The "Memory" Problem: Apple computers have a special feature called "Unified Memory," where the CPU and GPU share the same big pool of RAM. Existing software often treats them as if they are in separate rooms, forcing data to walk back and forth unnecessarily.

The Solution: How BaseRT Works

BaseRT cuts out the middleman and builds a direct highway between the brain and the muscle. Here are the four main tricks they used:

The "Data-Driven" Blueprint:
Instead of writing a new set of instructions for every different AI model (like Qwen, Llama, or Gemma), BaseRT uses a single, flexible blueprint. Think of it like a universal remote control that automatically detects which TV you are pointing at and changes its buttons instantly, rather than needing a different remote for every brand. This keeps the engine running smoothly without stopping to reconfigure.
The "Zero-Stop" Loop:
When you ask an AI to write a sentence, it generates one word at a time. In old software, the computer has to find a parking spot (memory allocation) for every single new word it creates. BaseRT pre-parks all the spots before the race starts. Once the AI starts talking, it never stops to find a parking spot; it just keeps driving. This eliminates "traffic jams" caused by memory management.
The "Fusion" Kitchen:
Usually, an AI has to cook ingredients in separate pots (Matrix Multiplication, Attention, Normalization) and move the food between them. BaseRT fuses these steps into one giant pot. It cooks the ingredients together in a single motion, saving the time it takes to move the food around.
The "Custom Driver":
The software knows exactly which Apple chip it is running on (M1, M2, M3, etc.). It adjusts its driving style based on the specific engine size, ensuring it gets the most power out of every drop of fuel.

The Results: Who Won the Race?

The team tested BaseRT against the two biggest competitors: llama.cpp (the most popular open-source runner) and MLX (Apple's own official framework).

Speed: BaseRT was the fastest in almost every test.
- On small models, it was up to 1.56 times faster than llama.cpp.
- On large "Mixture-of-Experts" models (complex AI brains), it was up to 1.78 times faster at processing the initial prompt.
The "Small Model" Sweet Spot: The biggest wins happened on smaller models. This is because, on small models, the "overhead" (the time wasted on administrative tasks) makes up a huge chunk of the total time. By cutting out that waste, BaseRT made a massive difference.
The "Big Model" Reality: On very large models, the speed is mostly limited by how fast data can move through the memory (bandwidth). Even with a perfect engine, you can't go faster than the speed limit of the road. However, BaseRT still managed to squeeze out extra speed here, proving that previous software wasn't using the road efficiently.

Why Does This Matter? (The "Edge" Shift)

The paper suggests we are moving toward a future where AI runs on your own device (your laptop or phone) rather than in a distant cloud server.

Privacy: Your data never leaves your computer.
Speed: No waiting for the internet to send your request back and forth.
Cost: You don't pay a monthly fee per word; you just pay for the hardware once.

BaseRT proves that Apple Silicon is much more powerful for running AI locally than we thought, if you use software built specifically for it.

What BaseRT Does Not Do (The Limitations)

The paper is honest about what this tool isn't yet:

Single User Only: It's designed for one person using the AI at a time. It doesn't handle hundreds of people asking questions simultaneously (server load).
Apple Only: It only works on Macs and iPads. It doesn't work on Windows, Android, or NVIDIA graphics cards yet.
No "Vision" Yet: It currently handles text models, not models that can "see" images.

The Bottom Line

The authors built a custom engine (BaseRT) that removes all the unnecessary baggage from running AI on Apple computers. By doing so, they unlocked the full speed potential of the hardware, making local AI faster, more private, and more efficient than ever before. You can try it out yourself on GitHub.

Technical Summary: BaseRT – Best-in-Class LLM Inference on Apple Silicon via Native Metal

Problem Statement

The paper identifies a performance gap in Large Language Model (LLM) inference on Apple Silicon. While Apple's unified memory architecture (UMA) offers high bandwidth and capacity suitable for local LLM deployment, existing runtimes fail to fully exploit this hardware.

Framework Overhead: Dominant runtimes like llama.cpp (originally CPU-first with a Metal backend) and MLX (a general-purpose array framework with lazy evaluation) introduce architectural constraints. These include unnecessary abstraction layers, framework-level scheduling, and dispatch overheads that are not optimized for Metal's execution model.
Suboptimal Resource Utilization: These frameworks often leave performance "on the table," particularly regarding kernel launch overheads, memory layout inefficiencies, and the inability to perform chip-specific kernel fusion.
Edge Inference Context: As the industry shifts toward edge inference driven by privacy, latency, connectivity, and cost concerns, the inability of current runtimes to maximize throughput on consumer hardware limits the viability of on-device AI.

Methodology

BaseRT is a native C++ inference runtime built directly on Apple's Metal GPU API, eliminating dependencies on intermediate frameworks like MLX, PyTorch, or CoreML. The design is guided by four core principles to minimize overhead:

Data-Driven Architecture Descriptors: Instead of hard-coding architecture variations (e.g., attention mechanisms, normalization) into the inference loop via conditional branches, BaseRT captures these differences as data. A compact descriptor drives the engine, ensuring the "hot path" remains identical across different model families (LLaMA, Qwen3, Gemma, etc.), reducing branching and improving uniformity.
Zero-Allocation Decode Loop: To eliminate memory management contention, all intermediate buffers (residuals, attention scratch, logits, KV cache) are pre-allocated at load time. The decode loop allocates zero bytes during operation, relying entirely on GPU command dispatch and lightweight CPU-side patching.
Kernel Fusion and Specialization:
- Fusion: BaseRT fuses operator sequences (e.g., within attention paths and feed-forward blocks) that are typically dispatched as separate kernels in other runtimes. This reduces global memory round-trips and kernel launch overhead, which is critical during the decode phase where individual kernel runtimes are short.
- Specialization: The runtime includes a library of hand-written Metal shaders with dedicated GEMV (for decode) and GEMM (for prefill) variants for each quantization format (Q2–FP16). Dequantization is integrated directly into the inner loop to avoid materializing weights in global memory.
- Hardware Adaptivity: Kernel selection, threadgroup sizes, and fusion decisions are dynamically configured based on the specific chip (M1–M5) and GPU core count.
Prefill Optimization: Prefill operations utilize tiled GEMM kernels with simdgroup matrix intrinsics. A FlashAttention implementation computes attention in tiles using online softmax to achieve $O(n)$ memory complexity, avoiding the materialization of the full $O(n^2)$ attention matrix.

Key Contributions

Native Metal Runtime: The first LLM inference runtime for Apple Silicon built natively on Metal without reliance on MLX or other intermediate frameworks.
Chip-Specific Optimizations: Implementation of kernel fusion strategies and unified memory-aware data layouts specifically tailored to Apple Silicon's topology.
Custom Dispatch Logic: A low-overhead command dispatch mechanism that removes per-operator scheduling and graph-evaluation work found in framework-based runtimes.
Comprehensive Benchmarking: A rigorous evaluation across multiple model families (Qwen3, Llama 3.2, Gemma 4) and quantization levels (Q4, Q8) on M3 and M4 Pro devices, establishing new throughput baselines.

Results

The paper reports that BaseRT achieves the highest inference throughput on Apple Silicon to date, outperforming llama.cpp, MLX, and the native Metal runtime uzu (Mirai).

Decode Throughput (Token Generation):
- BaseRT achieves 1.04× to 1.56× higher throughput than llama.cpp across tested models. The improvement is most significant on smaller dense models (e.g., 56% faster on Qwen3-0.6B) where fixed per-token overhead is a larger fraction of latency.
- BaseRT achieves 1.01× to 1.35× higher throughput than MLX on most models.
- On M3 Base devices, BaseRT leads llama.cpp by 1.13× to 1.34× and MLX by 1.01× to 1.22×.
Prefill Throughput (Prompt Processing):
- Dense Models: Performance is closely matched between runtimes, as prefill is compute-bound and saturates GPU matmul units. BaseRT leads on shorter prompts but trails slightly on longer prompts for some dense models.
- Mixture-of-Experts (MoE) Models: BaseRT demonstrates a substantial lead, achieving up to 1.78× higher throughput than MLX and 1.81× higher than llama.cpp on MoE models (e.g., Qwen3-30B-A3B) at shorter prompt lengths.
Comparison with uzu: BaseRT outperforms uzu in decode throughput on most configurations (up to 1.19× on Qwen3-0.6B). However, uzu leads in prefill throughput on dense models, likely due to its utilization of the Apple Neural Engine (ANE) via MPSGraph, a capability BaseRT (GPU-only path) does not currently exploit.

Significance and Claims

The paper claims that BaseRT establishes a new performance baseline for on-device LLM inference on Apple Silicon, demonstrating that the hardware's potential has been systematically underutilized by framework-based approaches.

Redefining Hardware Capabilities: The results suggest that Apple Silicon is a more capable inference platform than previously reported, with the performance ceiling significantly higher when framework abstraction overhead is removed.
Enabling Edge Inference: By delivering consistent best-in-class throughput from sub-1B to 30B parameter models, BaseRT serves as a critical enabling layer for the transition toward on-device deployment. This supports the emerging edge inference paradigm driven by privacy requirements, latency constraints, and cloud cost pressures.
Open Availability: The runtime is publicly available, allowing the community to leverage these native optimizations.

The authors remain modest regarding limitations, noting that BaseRT currently supports single-device inference only, lacks continuous batching or tensor parallelism, and is exclusive to Apple Metal (no CUDA/Vulkan backend yet). Future work aims to extend architecture coverage and add backend support for broader platform compatibility.