🤖 machine learning

What actually runs: a measurement study of language model placement and decode speed on the Apple Neural Engine

Through a rigorous measurement study of language model placement and decode speed on the Apple Neural Engine, the author demonstrates that computational expression and weight encoding, rather than model architecture alone, dictate accelerator residency and performance, leading to a design procedure that prioritizes encoding efficiency to achieve significantly smaller and faster ternary models.

Original authors: Shahir M A

Published 2026-08-25✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shahir M A

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a smartphone trying to understand a conversation. To do this, it must run a massive digital brain, a language model, right on the device. For this to work smoothly, the phone needs to use a special, ultra-fast processor built just for this kind of thinking, rather than its general-purpose main brain. The challenge is that this special processor is picky; it will only run certain types of calculations, and it refuses to run others, even if those others are mathematically identical. For years, developers have tried to guess which calculations will work and how to make them fast, often relying on rules of thumb that turn out to be wrong. They have assumed that if a model is small enough, it will run on the fast processor, or that making the numbers smaller will always help. But no one had actually watched the processor to see what it was really doing, or to measure exactly how the size of the model and the way its numbers are stored changed its speed.

A researcher named Shahir M A decided to stop guessing and start watching. Using an Apple computer with an M1 chip, they built a series of experiments to see exactly what happens when a language model tries to run on the phone's special processor, known as the Neural Engine. They did not just look at what the software said it would do; they measured the actual electricity and data moving through the chip to see what was really happening. They tested dozens of different ways to build the same mathematical operation, trained real models of different sizes, and changed how the numbers inside those models were stored, ranging from standard precision to very compressed, low-precision formats. Their goal was simple: to find out what actually gets a model onto the fast processor and how fast it can speak once it gets there.

The first thing they discovered was that the processor does not care about what a calculation means, only how it is written. They found that if you write a specific type of normalization—a step that helps the model keep its numbers stable—in one way, the processor accepts it immediately and runs it at full speed. But if you write the exact same mathematical step using a different, slightly more complicated set of instructions, the processor refuses to touch it and forces the phone to use its slower, general-purpose brain instead. It is as if the processor speaks a specific dialect of math; if you use the right dialect, it listens, but if you use a different one, even if the meaning is the same, it walks away. This means that the way a developer writes the code is just as important as the math itself.

The second, and perhaps most surprising, finding was that the size of the model is not the only thing that decides if it can run on the fast processor. The researchers found that a model with about twenty-six million parameters, written in standard precision, was too small to run on the special processor at all. It was forced to run on the slow brain, taking over a second to generate each word. However, when they took that exact same model and compressed the numbers inside it to use fewer bits, the processor suddenly accepted it. The compressed version ran on the fast processor and generated words in less than a second. In fact, for smaller models, compressing the numbers was the only way to get them onto the fast processor at all. The processor had a hidden rule: it would not run small models unless they were compressed. This flipped the common assumption that bigger models are always the ones that need the fast processor; here, the small ones needed the compression to get in the door.

Once the model was inside the fast processor, the speed was determined almost entirely by how much data had to be moved, not by how complex the math was. The researchers measured the flow of data and found that for every word the model generated, it had to stream the entire set of its weights—the numbers that make up its knowledge—through the processor. This happened for every single word, no matter how long the conversation went on. Because of this, the speed was directly tied to how many bits were in those weights. A model with compressed, low-precision numbers moved much less data and was therefore much faster. They found that a model using two-bit numbers was nearly three times faster than a model using standard numbers, simply because it had to move less data. The type of math the model used, such as whether it focused more on attention or on convolution, mattered very little for speed once the model was already running on the processor. The only thing that mattered was the size of the data being moved.

The researchers concluded that the best way to build a language model for a phone is to start with the compression, not the size. Instead of building a large model and then trying to shrink it, one should choose the most compressed format possible first, and then spend the available memory budget on adding more parameters. They found that a model with twenty-five million parameters, using a specific type of compressed math, could fit into just ten megabytes of space and generate words in about six-tenths of a millisecond. This was nearly ten times smaller and three times faster than the standard, uncompressed models that developers usually start with. The study showed that the path to a fast, on-device language model is not about making the model bigger or more complex, but about choosing the right way to write the math and the right way to store the numbers. By measuring the actual flow of data, they proved that the key to speed is not just having a fast processor, but knowing exactly how to feed it.

Technical Summary: What Actually Runs: A Measurement Study of Language Model Placement and Decode Speed on the Apple Neural Engine

Problem Statement

Practitioners deploying Large Language Models (LLMs) on Apple devices face two unresolved questions: what architectural and encoding choices ensure a model runs on the Apple Neural Engine (ANE) at all, and how fast will it decode? Existing literature characterizes the ANE's datapath and roofline but fails to measure operator placement and decode speed with weight encoding as a design variable. Furthermore, standard diagnostic tools are unreliable: the MLComputePlan is a compile-time estimate of compiler intent rather than a record of execution, and power telemetry can be misleading due to channel mismatches across silicon generations. Consequently, practitioners lack a reliable method to determine if a specific model configuration will actually utilize the accelerator or fall back to the CPU.

Methodology

The study employs three independent measurement paths to ensure robustness, supporting every headline claim with at least two of them:

64-Shape Primitive Matrix: The author constructed a matrix of 64 LLM primitives, varying the mathematical expression (e.g., fused vs. decomposed operations) while holding the computed function constant. These were exported to CoreML to record per-operation device placement and supported-device sets. This isolates "expression" effects from semantic ones.
Matched Model Training: Instead of using synthetic weights, the author trained a family of models (LFM2 architecture) across varying sizes (5M–50M parameters) and precisions (fp16, int8, ternary/2-bit). Quantized checkpoints were engineered to be byte-identical in structure to their fp16 counterparts, ensuring that deployment measurements reflect real trained artifacts without confounding structural differences.
Hardware Byte Counters: To bypass the unreliability of MLComputePlan, the author read the ANE's memory-controller byte counters (AMC Stats) directly via IOReport during inference. This establishes whether the engine actually streamed weights. The study also documents a critical cross-chip calibration hazard where channel name mismatches (e.g., between M1 and M3) can silently report zero activity, necessitating explicit channel enumeration per chip.

Measurements were conducted on an Apple M1 (macOS 26.1) for primary data and replicated on an Apple M3 (macOS 26.5.2) using ANE energy counters to verify placement behavior.

Key Findings

1. Placement is a Property of Expression, Not Semantics

Placement on the ANE depends on how a computation is expressed, not what it computes.

Fused vs. Decomposed: A fused RMSNorm is 100% ANE-eligible. An arithmetically identical decomposition (using pow, reduce_mean, rsqrt, etc.) is 0% eligible (CPU-only).
Convolution Limits: Short convolutions with kernels $\le 12$ are eligible; kernels $\ge 15$ fall off the engine.
Embedding Gather: This operation is CPU-only across all index dtypes.

2. Encoding Gates Accelerator Placement

Weight encoding is the primary determinant of whether a model runs on the ANE, creating a "size cliff" for specific architectures.

The Conv-Heavy Cliff: A 25.85M-parameter conv-heavy model in fp16 runs entirely on the CPU (0% ANE residency). The same architecture in int8 or 2-bit (ternary) achieves ~83% residency.
Architecture Matters: An all-attention model with only 22.29M parameters (below the conv-heavy cliff) achieves 98.9% residency in fp16.
Mechanism: The CoreML segmenter uses a cost heuristic involving max(flops/gflops, bytes/bw). Quantization reduces the bytes/bw term, allowing smaller models to amortize the dispatch overhead and clear the threshold for ANE execution.

3. Decode Cost is Driven by Bytes Streamed

Decode latency is a data-movement problem. The ANE streams the entire weight set for every token generated.

Constant Reuse Factor: Measured bytes per parameter per token track nominal encoding width at a constant ~0.77 factor across fp16, int8, and 2-bit.
- fp16: ~1.54 B/param (79% of theoretical bandwidth)
- int8: ~0.75 B/param (63%)
- 2-bit: ~0.20 B/param (31%)
Implication: Lower bit-widths reduce total bytes but incur a "dequantization tax" (reconstitution overhead), preventing them from reaching the theoretical roofline. However, the reduction in total bytes still yields significant speedups.

4. Operator Mix and the "Ternary Frontier"

Once a model is resident on the ANE (typically achieved via quantization), the specific operator mix (attention vs. convolution) has minimal impact on decode speed at matched sizes.

GQA Advantage: Grouped-Query Attention (GQA) with 2 KV heads is parameter-efficient (~~2.5 $d^2$ ) compared to gated-conv blocks (~~4 $d^2$ ). This makes GQA the "smaller" operator, allowing it to clear the fp16 placement bar where conv-heavy models fail.
Performance: At matched sizes (e.g., 25M parameters), ternary models with different operator mixes (conv-heavy, half-attention, dense attention) land within a narrow band of 10.0–10.8 MB and 0.62–0.64 ms/token.
Headline Results: A 25M ternary half-attention model achieves 10.5 MB size and 0.63 ms/token decode latency. Compared to the initial 25.85M conv-heavy fp16 baseline (which failed to run on the ANE), this represents a 9.8× reduction in size and a 3.0× speedup.

Significance and Design Procedure

The paper argues that the standard practice of "train then quantize if too slow" is suboptimal for the ANE. Instead, the author proposes a design procedure:

Choose Encoding First: Quantization is the reliable mechanism to ensure small models land on the ANE.
Spend Byte Budget on Parameters: Since decode cost is bytes/token, lower precision allows for more parameters within a fixed latency budget.
Treat Operator Mix as Secondary (at matched size): Once resident, operator choice has little effect on speed; the choice should be driven by accuracy (measured in a companion study) or placement requirements at fp16.
Express Operations Natively: Use fused operations (e.g., fused RMSNorm) and avoid unsupported decompositions or dynamic shapes.

The study establishes that on the Apple Neural Engine, the "wrong" architectural choice does not merely run slower; it may not run on the accelerator at all. The findings invert CPU-centric intuitions (where conv-heavy models are often preferred) and provide the first hardware-confirmed evidence that weight encoding is a primary control for accelerator residency in LLMs.