From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
This survey classifies over thirty KV cache management systems for LLM serving into five architectural archetypes based on four key axes, identifies ownership as a primary driver of design variance, and highlights seven critical measurement gaps that impede progress in fault tolerance, isolation, and advanced serving techniques.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are running a massive, high-speed library where a single librarian (the AI model) is trying to write a story one word at a time. To write the next word, the librarian needs to remember everything written so far. In the world of Large Language Models (LLMs), this "memory" is called the KV Cache (Key-Value Cache).
For a long time, this memory was treated like a temporary sticky note: the librarian grabbed it, wrote a few words, and then threw it away when the story was done. But now, stories are getting incredibly long (context windows), and the library is getting crowded with hundreds of people asking for stories at once (high concurrency). The sticky notes have become too big to fit on the librarian's desk, and throwing them away every time is wasting a huge amount of time.
This paper is a survey (a big review) of how different computer systems are solving this "memory crisis." The authors argue that we are shifting from treating the KV cache as a simple, local note to treating it as a complex, distributed memory system that needs careful management.
Here is the breakdown of their findings using simple analogies:
1. The Four Questions Every System Must Answer
The authors say that every system trying to manage this memory is answering four specific questions. They call these the "Four Axes":
- Locality (Where does the memory live?): Is the memory sitting right on the librarian's desk (local GPU), or does the librarian have to walk to a different room, or even call a friend in another city to get it?
- Lifetime (How long does it stay?): Does the memory disappear the second the story is finished? Does it stay for the whole conversation with one person? Or does it stay forever so that anyone can reuse it later?
- Ownership (Who is in charge?): Is the librarian the only one who can decide what to keep or throw away? Is there a central manager (like a head librarian) making the rules? Or is everyone in the library making their own rules?
- Substrate (What carries the memory?): Is the memory moving via a super-fast cable inside the building (GPU memory), a high-speed fiber optic line between buildings (RDMA), or a slower truck on the highway (hard drive/SSD)?
2. The Five "Archetypes" (The Five Library Styles)
When the authors looked at over 30 different systems, they found they all fell into five main "styles" or archetypes based on how they answered the four questions above:
- Local-Paged (The Efficient Desk): The memory stays on the librarian's desk, but they use a clever filing system (paging) to swap notes in and out quickly without throwing them away. This is the most common style right now (e.g., vLLM).
- Disaggregated-Pipeline (The Assembly Line): The library splits the work. One team of librarians writes the beginning of the story (Prefill), and a different team finishes the rest (Decode). They pass the notes back and forth. This prevents the desk from getting cluttered.
- Shared-Store (The Global Archive): The library has a giant, shared archive room. If two people ask for the same story start, they don't rewrite it; they just grab the existing notes from the archive. This saves a ton of time.
- Memory-Pool (The Shared Warehouse): Instead of moving notes between rooms, the library builds a giant, shared warehouse (using new tech like CXL) that everyone can reach into directly. It's like having one giant desk that everyone shares.
- Hybrid-Tier (The Super-System): This is the "Swiss Army Knife." It combines the assembly line, the shared archive, and the warehouse all at once. It's complex but very powerful (e.g., Mooncake).
3. The Big Discovery: "Ownership" is the Key
The authors found that once you fix the hardware and the type of work, the biggest difference between systems is Ownership.
- Some systems have a Central Manager (a Head Librarian) who decides exactly where every note goes.
- Others use a Distributed Team where every librarian decides for themselves.
- The paper argues that this choice determines how well the system scales and what happens if a computer crashes.
4. The Missing Pieces (The Blind Spots)
The paper points out a major problem: We don't have good rulers to measure these systems.
Currently, researchers just say, "Our system is faster!" but they don't explain why. The authors found seven missing measurements that we need to see to truly understand these systems:
- We don't know how much time is wasted looking up where the notes are (Metadata cost).
- We don't know exactly how long notes sit around before being thrown away (Lifetime).
- We don't have good public records of how real people use these libraries (Public traces).
5. What's Next?
The authors propose a research agenda. They say we need to stop just guessing and start measuring these specific things. If we do, we can figure out:
- How to handle it if a computer crashes in the middle of a story (Fault Tolerance).
- How to keep secrets safe so one user doesn't accidentally see another user's notes (Isolation).
- How to manage the memory when the library gets huge.
In short: The KV cache has grown from a small sticky note into a massive, distributed memory problem. The paper organizes all the current solutions into five clear categories, identifies that "who is in charge" is the most important design choice, and calls for better tools to measure exactly how well these solutions are working.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.