🤖 AI

SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions

This paper introduces SMEPilot, an LLM inference engine that optimizes performance on CPUs with Scalable Matrix Extensions (SME) by dynamically selecting between CPU-only, SME-only, or cooperative execution strategies and employing tile-level partitioning and layout reuse to achieve up to 3.94× speedup across diverse models and platforms.

Original authors: Feiyang Chen, Haibo Chen

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Feiyang Chen, Haibo Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: A New Tool for Old Machines

Imagine you have a very busy kitchen (a modern computer CPU). For years, this kitchen has relied on a team of generalist chefs (standard CPU cores) to chop vegetables, stir pots, and plate food. They are good at everything, but they aren't super-fast at one specific task: chopping huge piles of vegetables into perfect squares (which is what Large Language Models, or LLMs, need to do to "think").

Recently, the kitchen got a new piece of equipment: a high-speed industrial vegetable slicer (called SME, or Scalable Matrix Extension). This machine can slice vegetables incredibly fast. However, the paper's authors discovered a problem: just because you have a slicer doesn't mean you should use it for every task.

The Problem: If you try to use the slicer for everything, you might clog the conveyor belt (memory bandwidth) or waste time setting it up for small jobs. Sometimes, the generalist chefs are actually faster at the small, tricky tasks.
The Solution: The team built SMEPilot. Think of SMEPilot as a super-smart Kitchen Manager. It doesn't just hand everything to the slicer; it decides exactly which chef does which job, when to use the machine, and how to keep the whole kitchen running without anyone standing around waiting.

How SMEPilot Works: The Three Magic Tricks

The paper identifies three main ways SMEPilot makes the kitchen run faster. Here are the analogies for each:

1. The "Team Split" (Tile-Level Work Partitioning)

The Problem: Imagine you have a giant pizza (a huge math problem) to cut.

If you give the whole pizza to the industrial slicer, the chefs sit idle.
If you give it to the chefs, the slicer sits idle.
If you just cut the pizza in half and give one half to the slicer and the other to the chefs, the slicer might finish in 1 second while the chefs take 10 seconds. The slicer just waits, wasting time.

The SMEPilot Fix: SMEPilot cuts the pizza into tiny slices (tiles). It gives a few slices to the slicer and a few to the chefs. Crucially, it calculates exactly how many slices each should get so that both finish at the exact same time. This keeps both the machine and the chefs busy 100% of the time.

2. The "Assembly Line" (Phase-Aware Pipeline Execution)

The Problem: Making a sandwich involves two steps:

Step A: Slicing the bread (needs the industrial slicer).
Step B: Spreading the mustard (needs a human hand).

In a bad kitchen, you wait for the entire loaf of bread to be sliced before you start spreading mustard. The human sits idle while the machine works. Then, the machine sits idle while the human spreads mustard. This is called a "temporal bubble" (a gap in time where no work is done).

The SMEPilot Fix: SMEPilot creates an assembly line. As soon as the slicer finishes one slice of bread, it passes it to the human to spread mustard, while the slicer immediately starts on the next slice. The machine and the human work at the same time on different parts of the sandwich. This eliminates the waiting time.

3. The "Pre-Packed Lunch" (Layout-Aware Runtime)

The Problem: The industrial slicer only works if the vegetables are arranged in a very specific, tight grid (a "packed" layout). But the rest of the kitchen stores vegetables in loose, standard bags.

The Bad Way: Every time you want to use the slicer, you stop, take the vegetables out of the bag, arrange them perfectly in a grid, slice them, and then put them back in a bag. This "re-arranging" takes almost as much time as the slicing itself, ruining the speed benefit.

The SMEPilot Fix: SMEPilot is smart about when to rearrange.

For static ingredients (like model weights): It arranges them into the perfect grid once when the kitchen opens, so they are ready instantly for every order.
For fresh ingredients (like new data): It arranges them into the grid immediately when they are created, so they are ready for the slicer without any extra steps later. This avoids the "re-arranging" penalty.

The Results: How Much Faster?

The researchers tested SMEPilot on phones, laptops, and servers using popular AI models (like Llama and Qwen).

The Speed: SMEPilot made the AI up to 3.94 times faster than the standard way of running these models on CPUs.
The Energy: Because it finishes the work so much faster, it also uses less than half the energy compared to the standard method.
The Comparison: In some tests, the CPU running SMEPilot was almost as fast as a dedicated graphics card (GPU), which is usually the "heavy lifter" for AI.

The Takeaway

The paper argues that SME (the new machine) is not a magic wand that replaces the old chefs. Instead, the best performance comes from a smart manager (SMEPilot) that knows:

When to use the machine and when to use the chefs.
How to split the work so everyone stays busy.
How to organize the workspace so no time is wasted on setup.

By doing this, ordinary computers can run powerful AI much faster and more efficiently than before.

Technical Summary: SMEPilot

Problem Statement

Modern CPUs are increasingly integrating matrix extensions, such as the Arm Scalable Matrix Extension (SME), to provide high-throughput matrix execution within the CPU package. While these extensions offer a middle ground between GPU accelerators and conventional vector cores, they are not a universal replacement for CPU cores in Large Language Model (LLM) inference.

The core challenge identified is a mismatch between the heterogeneous requirements of LLM operators and the static capabilities of SME units.

Diverse Operator Characteristics: LLM inference involves distinct phases (prefill vs. decode) and operators (attention, FFN, MoE) with varying arithmetic intensities, vector behaviors, and layout requirements.
Resource Contention: SME units and CPU cores share memory bandwidth and cache hierarchies. Treating SME as a simple "drop-in" replacement for CPU kernels leads to suboptimal performance because:
1. Spatial Underutilization: Large matrix-heavy operators (e.g., prefill GEMMs) can utilize both SME and CPU cores simultaneously, but assigning an entire operator to one backend leaves the other idle.
2. Temporal Bubbles: Operators like attention interleave matrix-heavy phases (benefiting from SME) and vector-heavy phases (better suited for CPU cores). Executing these sequentially creates idle time ("bubbles") for one hardware unit while the other works.
3. Layout Overhead: SME kernels require packed, tile-friendly layouts, whereas standard LLM pipelines use conventional tensor layouts. Naive conversion before every SME invocation adds memory traffic and disrupts data locality, negating throughput gains.

Methodology

The paper introduces SMEPilot, an LLM inference engine designed to systematically schedule SME and CPU cores based on a unified performance model.

1. Unified Roofline Characterization

The authors develop a multi-ceiling roofline model that captures the coupled nature of SME and CPU resources:

Additive Compute, Shared Bandwidth: SME and CPU cores provide near-additive matrix throughput but share a single memory bandwidth roof.
Regime Classification: The model classifies operators into three regimes based on arithmetic intensity ( $I$ $I$ ):
- Memory-bound: Operators below the CPU ridge point. SME offers no benefit as bandwidth is the bottleneck.
- Backend-asymmetric Ridge-crossing: Operators between the CPU and SME ridge points. These are compute-bound on CPU but memory-bound on SME. SME lifts performance to the shared memory limit.
- Compute-bound: Operators above the SME ridge point. Both SME and CPU cores can contribute, and mixed execution is optimal.
Dynamic Scheduling: The model dictates that backend selection (CPU-only, SME-only, or Mixed) must be dynamic, changing with request shapes (e.g., sequence length, batch size) and operator types.

2. SMEPilot Runtime Design

To bridge the gap between the roofline-guided plan and actual execution, SMEPilot employs three key techniques:

Tile-Level Work Partitioning (Addressing Spatial Utilization):
- Instead of assigning an entire operator to one backend, SMEPilot partitions matrix work at the tile granularity.
- For GEMM operations, it splits the output dimensions ( $M$ or $N$ ) between SME workers and CPU cores based on the operator shape and measured throughput.
- This allows both backends to contribute to the same operator without cross-worker synchronization overhead.
- In MoE layers, experts with few tokens (thin matrices) remain on CPU, while experts with many tokens utilize mixed execution.
Phase-Aware Pipeline Execution (Addressing Temporal Bubbles):
- Applied primarily to Attention mechanisms, which contain matrix phases ( $QK^\top$ , $PV$) and vector phases (Softmax, masking).
- SMEPilot pipelines these phases: while SME workers compute the matrix stage for a specific tile, CPU cores simultaneously process the vector stage (e.g., Softmax) for a preceding tile.
- This overlaps hardware-specific work, eliminating idle time caused by coarse-grained operator boundaries.
Layout-Aware Runtime (Addressing Layout Compatibility):
- The runtime tracks tensor layouts as state to minimize conversion overhead.
- Offline Packing: Static model weights are packed into SME-friendly layouts once at load time, removing this cost from the request critical path.
- Producer-Side Packing: Dynamic activations (e.g., KV-cache entries) are packed at the producer side when the downstream consumer is an SME kernel. This avoids repeated unpacking and repacking on critical paths.

3. Execution Plan Generation

SMEPilot uses an Offline Profiler to measure hardware metrics (throughput, bandwidth) and an Online Planner to generate execution plans. The planner evaluates the roofline model for each operator given the current request shape to determine:

Backend selection (CPU, SME, or Mixed).
Thread allocation and partition ratios.
Tile sizes to balance kernel efficiency with pipeline overlap.

Key Results

SMEPilot was evaluated on three platforms (Apple M4 Pro, MediaTek Dimensity 9500, KunPeng 920 Server) using Llama-3.2-3B, Qwen3-4B, and Qwen3-30B-A3B.

End-to-End Performance: SMEPilot achieves up to 3.94× speedup over the llama.cpp baseline across various device-model-workload configurations.
Prefill Phase:
- FFN GEMM: 1.42–1.67× faster than highly optimized Apple Accelerate (SME-only) and 6.06–7.43× faster than GGML (CPU-only).
- Attention: 1.56–3.50× faster than FlashAttention CPU.
- Gains are driven by the ability to utilize both SME and CPU cores for large matrix operators.
Decode Phase:
- Achieves up to 3.48× speedup over CPU-only inference.
- Improvements are significant for larger batch sizes where operators enter the "ridge-crossing" regime, allowing SME to alleviate memory-bound constraints.
Energy Efficiency: On the Apple M4 Pro, SMEPilot reduces end-to-end energy consumption to 0.485× of the llama.cpp baseline, primarily due to faster completion times.
GPU Comparison: On Apple M4 Pro, SMEPilot achieves 0.72–0.96× the performance of GPU inference, positioning it as a viable alternative when GPUs are unavailable or reserved.

Significance and Claims

The paper claims that SME is not a universal substitute for CPU vector cores but rather a specialized resource that requires careful orchestration. The significance of SMEPilot lies in its shift from static kernel replacement to dynamic, shape-aware scheduling:

Systematic Characterization: It provides the first comprehensive roofline-based characterization of SME-enabled CPUs for LLM inference, revealing that operator placement must be dynamic and shape-dependent.
Cooperative Execution: It demonstrates that the highest performance is achieved not by choosing either SME or CPU, but by cooperatively executing both within a single operator (via tile partitioning) and across operator phases (via pipelining).
Runtime Overhead Management: It highlights that raw throughput gains from matrix extensions can be erased by layout conversion costs, necessitating layout-aware runtime strategies that keep packing off critical paths.

The authors conclude that CPU matrix extensions are most effective when their throughput is applied selectively and amortized across suitable inference regimes, rather than being applied uniformly to all operations.