Non-Monotonic Latency in Apple MPS Decoding: KV Cache Interactions and Execution Regimes
This paper reveals that Apple's MPS backend exhibits unexpected non-monotonic latency spikes during autoregressive inference, where decoding performance can abruptly degrade by up to 21x for specific configurations due to backend execution dynamics, contrasting with the smooth scaling observed on CPU and NVIDIA CUDA systems.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are driving a car on a highway that is supposed to get slower and slower as you add more passengers (tokens) to the vehicle. In the world of AI, this is how we usually expect things to work: the longer the conversation, the more time it takes to generate the next word, but it should be a smooth, predictable increase.
However, this paper discovered that on Apple's computers (specifically those with M-series chips), the "highway" behaves strangely. Instead of a smooth slope, the drive hits sudden, massive speed bumps that appear out of nowhere and disappear just as quickly.
Here is a breakdown of the paper's findings using everyday analogies:
1. The "Ghost Speed Bump"
The researchers found that when using Apple's graphics system (called MPS) to generate text, the time it takes to produce words doesn't always go up steadily.
- The Normal Expectation: If you ask the AI to write 100 words, it takes 1 second. If you ask for 200, it takes 2 seconds. If you ask for 300, it takes 3 seconds. This is monotonic scaling (a smooth, predictable line).
- The Apple Reality: The AI might write 496 words in 9 seconds. But if you ask it to write just 16 more words (512 total), it suddenly takes 89 seconds. Then, if you ask for just 16 more (528 total), it snaps back to being fast again.
- The Analogy: Imagine driving down a road where, for no apparent reason, you hit a patch of mud that slows you down to a crawl for exactly 100 feet, but the moment you pass that patch, you are back to full speed. The paper found these "mud patches" happen at very specific word counts (like 512 or 384), and they can make the AI 21 times slower than usual.
2. The "Magic Memory Box" (KV Cache)
To make AI faster, engineers use a trick called KV Caching. Think of this as a "Magic Memory Box" where the AI stores the context of the conversation so it doesn't have to re-read the whole story every time it writes a new word.
- Usually: This box makes the car drive much faster.
- The Problem: The paper found that when the AI hits one of those "Ghost Speed Bumps" (the 512-word mark), the Magic Memory Box stops working as well as it should.
- The Result: Normally, using the box makes the AI 20 times faster than not using it. But at the "bad" word counts, that advantage shrinks to almost nothing (only 1.9 times faster). The box is still there, but it's stuck in traffic.
3. It's Not About Running Out of Gas (Memory)
When the AI slows down, you might think, "Oh, the computer is running out of memory."
- The Test: The researchers checked the computer's memory usage. They saw that memory usage went up smoothly and steadily, like a balloon inflating. It didn't suddenly spike or crash at the moment the AI got slow.
- The Conclusion: The slowdown isn't because the computer ran out of space. It's because the "engine" (the software backend) suddenly decided to switch gears in a very inefficient way at specific moments. It's like a car engine suddenly deciding to run on a different, terrible fuel mixture for a few seconds, even though the gas tank is full.
4. It Happens Everywhere (Not Just One Car)
The researchers tested this with different types of AI models (GPT-2, BLOOM, OPT) and different Apple computers (M3 Max and M3 Pro).
- The Finding: The "Ghost Speed Bumps" appeared in all of them. The exact location of the bump changed slightly depending on the car model and the engine size, but the phenomenon was the same.
- The Comparison: When they tested the same AI on a standard computer (CPU) or an NVIDIA graphics card (CUDA), the drive was smooth. The "bumpy road" is a specific quirk of Apple's current graphics software.
5. Why This Matters for You
The paper warns that if you only test an AI at one or two specific lengths (e.g., "Let's see how fast it is at 100 words"), you might miss these massive slowdowns entirely.
- The Trap: If you benchmark an AI and it looks fast at 500 words, but you don't check 512 words, you might think the system is perfect. But in real life, if a user's conversation hits that specific length, the system could freeze for a moment.
- The Lesson: You can't just look at the "average" speed. You have to check every step of the way, because on Apple chips, the speed can change abruptly and unpredictably based on exactly how many words are being generated.
In summary: Apple's AI chips are powerful, but their software has a hidden quirk where, at very specific moments, the system suddenly becomes incredibly slow for a brief period before snapping back to normal. This happens even when there is plenty of memory available, and it makes the usual "speed-up tricks" (like KV caching) much less effective during those moments.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.