SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving
SplitZip is a GPU-friendly, lossless compressor that accelerates KV cache transfer in disaggregated LLM serving by exploiting floating-point exponent redundancy through a fixed-length codebook and sparse escape stream, achieving significantly higher throughput and reduced latency compared to existing methods.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are running a massive, high-speed library where two different teams work together to answer complex questions.
- Team A (The "Prefill" Team): They are the researchers. They read a huge, long document (the user's prompt) and take detailed notes. This part is very fast and requires a lot of brainpower (computing power).
- Team B (The "Decode" Team): They are the writers. They use those notes to write the answer, one word at a time. This part is slower and requires a lot of memory to hold the notes.
In modern AI systems, these two teams often work in different buildings (different servers) to save money and balance the workload. The problem? Team A has to mail their notes to Team B before Team B can start writing.
If the notes are huge (like when the user asks about a whole book), mailing them takes too long. The writer (Team B) sits idle, waiting for the mail to arrive. This "mailing delay" is the bottleneck that slows down the whole system.
The Problem with Current "Mailing" Methods
Previously, people tried to shrink these notes (compress them) to make them faster to mail.
- The "Lossy" Method: Some tried to throw away parts of the notes to save space. But this is like summarizing a novel by deleting random sentences. It might save space, but the story (the AI's answer) could become wrong or nonsensical.
- The "Old" Lossless Method: Others tried to zip the notes perfectly without losing a single letter. But the software they used was like a slow, old-fashioned typist. It was too slow to keep up with how fast Team A was generating the notes. By the time the notes were zipped, Team A had already generated a new batch.
The Solution: SplitZip
The authors created a new system called SplitZip. Think of it as a super-smart, high-speed courier service designed specifically for these AI notes.
Here is how it works, using a simple analogy:
1. The "Note" Structure
The notes the AI generates are made of numbers. In the format they use (called BF16), every number has three parts:
- The Sign: Is it positive or negative? (Like a plus or minus sign).
- The Mantissa: The specific details of the number.
- The Exponent: The "power" or scale of the number (like whether it's 10, 100, or 1,000).
2. The Secret Discovery
The researchers noticed something funny: While the "details" (Mantissa) are all over the place, the "power" (Exponent) is very repetitive. It's like if you were writing a book, and 99% of the time, you only used the words "very," "quite," and "extremely." You rarely used "somewhat" or "barely."
3. The SplitZip Strategy
Instead of writing out every single word, SplitZip does this:
- The Shortcut: It creates a "Top 16" list of the most common "powers" (exponents). It assigns each of these 16 common powers a tiny, 4-letter code (like a secret handshake).
- The Packing: It packs two of these tiny codes into a single byte of space. This is like fitting two secret handshakes into the space of one letter.
- The "Rare" List: For the 1% of times a "rare" power appears, it doesn't try to force it into the shortcut. Instead, it writes down a small "escape note" that says: "At position #50, the power was actually 'Rare-Value-99'."
4. Why It's Fast
- For Team A (Encoding): Because the system uses fixed, short codes (4 bits) instead of complex, variable-length codes, it can pack the notes incredibly fast. It's like a robot arm that knows exactly where to put every item, rather than a human trying to figure out the best way to fold a shirt every time.
- For Team B (Decoding): When the notes arrive, Team B can unpack them instantly. They look up the 4-letter code, get the power, and combine it with the details. If there's an "escape note," they just overwrite that one spot. It's a straight line of work with no confusing detours.
The Results
The paper claims that SplitZip is a game-changer:
- Speed: It compresses and decompresses data at speeds of 613 GB/s and 2181 GB/s respectively. To put that in perspective, it's hundreds of times faster than previous methods that tried to do this on the CPU.
- Perfect Accuracy: It is "lossless." The notes Team B receives are bit-for-bit identical to what Team A wrote. No information is lost.
- Real-World Impact: When they tested this in a real AI system (using the SGLang framework), they saw:
- 1.32x faster transfer of notes between servers.
- 1.30x faster time to get the first answer word (TTFT).
- 1.23x more total requests handled per hour.
Summary
SplitZip is like a specialized, high-speed courier that knows the AI's notes are mostly repetitive. Instead of mailing the whole heavy box, it sends a tiny, coded list of the common items and a small note for the rare ones. It does this so fast that the AI server never has to wait, allowing it to answer long, complex questions much quicker without ever losing a single detail.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.