🤖 machine learning

Haiku to Opus in Just 10 bits: LLMs Unlock Massive Compression Gains

This paper demonstrates that LLM-generated text can be compressed to extreme ratios (as low as 0.0006) by leveraging domain-adapted arithmetic coding for lossless compression and an interactive "Twenty Questions" protocol for lossy compression, where a small model recovers significant capability from a large model by transmitting only a few bits of binary answers.

Original authors: Roy Rinberg, Annabelle Michael Carrell, Simon Henniger, Nicholas Carlini, Keri Warr

Published 2026-04-06

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Roy Rinberg, Annabelle Michael Carrell, Simon Henniger, Nicholas Carlini, Keri Warr

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a massive, incredibly smart library (a Large Language Model, or LLM) that can write perfect answers to any question. But there's a problem: sending these answers over the internet is like mailing a 100-pound encyclopedia when you only need to send a single postcard. It's slow, expensive, and wasteful.

This paper, titled "Haiku to Opus in Just 10 bits," is about a team of researchers who figured out how to shrink those massive encyclopedia answers down to the size of a few postcards—sometimes even smaller than a single pixel! They did this by changing how we send information, not just by squishing the text tighter.

Here is the story of their three main tricks, explained with simple analogies.

1. The "Specialized Librarian" (Lossless Compression)

The Problem: Imagine a general librarian who knows a little bit about everything. If you ask them to summarize a book about quantum physics, they might stumble a bit because they aren't a physics expert. If they stumble, they use more words (or "bits") to explain things, making the message longer.

The Solution: The researchers created LoRA adapters. Think of these as "specialty vests" the librarian can put on.

If the message is about coding, the librarian puts on the "Coder Vest."
If it's about cooking, they put on the "Chef Vest."

Because the librarian is now an expert in that specific topic, they can predict exactly what words will come next. This allows them to send the message using a highly efficient code (arithmetic coding).

The Result: They cut the file size in half just by giving the librarian the right "vest" for the job.

2. The "Bullshit Detector" (Lossy Compression via Rewriting)

The Problem: Sometimes, the answer doesn't need to be the exact same words the computer generated. It just needs to be the right idea. Imagine a student writing a 5-page essay to solve a math problem. They use a lot of filler words like "Step 1," "Therefore," and "In conclusion."

The Solution: The researchers asked the AI to rewrite its own answer, but with a twist: "Make it as short as possible, but keep the answer correct."

It's like asking a verbose friend to summarize their story for a text message.
They also tried a game called "Shortest-of-N": Ask the AI to write 10 different versions of the answer, then pick the one that is the most "squishable" (compressible).

The Result: By stripping away the fluff and redundancy, they shrunk the answers to about 3% of their original size. The meaning stayed the same, but the "weight" of the message dropped dramatically.

3. The "20 Questions" Game (Interactive Compression)

The Problem: This is the paper's most magical trick. Imagine you are trying to describe a complex painting to a friend over a phone line with terrible static. You can't send the picture. You can't even send a long description. You only have 10 seconds to talk.

The Solution: Instead of sending the answer, the small AI (the "Student") plays a game of 20 Questions with the big AI (the "Teacher").

The Student tries to solve a hard math problem but gets stuck.
Instead of sending the whole wrong answer, the Student asks the Teacher a Yes/No question: "Did I forget to carry the one in step 3?"
The Teacher says "Yes." (That's 1 bit of information).
The Student asks: "Is the final answer an even number?"
The Teacher says "No." (Another bit).

After just 10 questions (10 bits of data), the Student has enough clues to fix their answer and get it right.

The Analogy:

Old Way: Sending a 5,000-word essay (The Encyclopedia).
New Way: Sending a list of 10 "Yes/No" clues (The Postcard).

The Result: This is the most shocking part. They achieved compression ratios of 0.0006.

To put that in perspective: If the original answer was a 100-page book, this new method sends the answer using only 0.06 pages of data.
They proved that for math and science problems, you don't need to send the solution; you just need to send the corrections to the solution.

Why Does This Matter?

The authors suggest this could be a superpower for security.
Imagine a company wants to protect its super-smart AI from being stolen. They can put a "bit limit" on the internet connection leaving their server.

Before: A hacker could steal the AI's "brain" (weights) or huge amounts of data.
Now: If the company says, "You can only send 10 bits out per second," the hacker gets nothing. But a legitimate user asking a question? The AI can answer them perfectly using the "20 Questions" trick, sending just enough bits to guide the user to the answer without ever revealing the AI's internal secrets.

The Big Takeaway

We used to think that to send a smart answer, you had to send a lot of data. This paper shows that if you let the sender and receiver talk to each other (interact), you can transmit knowledge with almost zero data. It's the difference between mailing a dictionary to teach someone a word versus just whispering the definition to them.

In short: They turned the internet from a "dump truck" of data into a "whisper network" that is 100 times more efficient.

1. Problem Statement

Large Language Models (LLMs) generate text that is highly redundant, yet standard compression algorithms (like GZIP) and even state-of-the-art LLM-based arithmetic coding struggle to compress this data efficiently.

Lossless Limitations: Traditional lossless compression requires exact reconstruction of the token sequence. While LLMs can predict tokens well, domain mismatch (general models vs. specific text) and non-determinism in inference pipelines limit compression ratios to roughly 0.1 (10% of original size).
The Opportunity: In many LLM applications (chatbots, coding assistants), users do not need the exact token sequence generated by a model; they only need a response that is semantically equivalent and useful. This opens the door to lossy compression, where the goal is to transmit the information or capability of a large model using significantly fewer bits than the full text output.

The paper investigates the compression-compute frontier: how much can we compress LLM outputs if we are willing to trade off computational resources (e.g., running a small model iteratively) and accept lossy reconstruction?

2. Methodology

The authors propose a spectrum of compression techniques ranging from lossless to extreme lossy, utilizing a hierarchy of models (Small Language Model - SLM, and Large Language Model - LLM). They use Anthropic's Claude 4.5 family (Haiku as SLM, Opus as LLM) as a testbed.

A. Lossless Compression: Domain-Adapted LoRA

To improve lossless arithmetic coding, the authors address distribution mismatch. A general LLM is suboptimal for specific domains.

Technique: They train lightweight LoRA (Low-Rank Adaptation) adapters on specific topic clusters (e.g., Code, Science, Creative Writing).
Routing: A RAG-based router selects the appropriate LoRA adapter based on the input prompt.
Mechanism: The selected adapter provides a probability distribution closer to the true data distribution, reducing the KL divergence and thus the bits required for arithmetic coding.

B. Lossy Compression: Response Rewriting

This approach sacrifices exact token reconstruction for semantic preservation.

Shortest-of-N: Generate $N$ candidate responses and select the one with the lowest compression ratio.
Summarization ("Just Ask"): Explicitly prompt the model to rewrite the verbose solution "as succinctly as possible while preserving enough information to infer the correct answer."
Result: This produces denser text (higher bits-per-character) but significantly shorter length, yielding better total compression.

C. Extreme Lossy Compression: Question-Asking (QA) Protocol

Inspired by the game "Twenty Questions," this is the paper's core contribution. It treats knowledge transfer as an interactive protocol.

Setup: An SLM (Haiku) attempts to solve a problem. It fails or is uncertain.
Interaction: The SLM iteratively formulates Yes/No questions about its reasoning strategy or the problem constraints.
Oracle: A stronger LLM (Opus) answers these questions with a single bit (0 or 1).
Reconstruction: The SLM uses these bits to refine its reasoning and generate a final correct answer.
Transmission Cost: Only the $N$ binary answers (bits) are transmitted. The receiver (who also has the SLM) can reconstruct the entire interaction transcript by running the same SLM locally, meaning the communication cost is exactly $N$ bits, independent of the response length.

3. Key Contributions

Lossless Improvement via LoRA: Demonstrated that domain-specific LoRA adapters improve lossless arithmetic coding by 2× over baseline models (compression ratio ~0.09 vs 0.18) on conversational datasets.
Lossy Rewriting: Showed that "Just Ask" (summarization) achieves compression ratios of 0.03, a 2× improvement over selecting the most compressible candidate from multiple generations. Crucially, this does not degrade answer accuracy on math/code benchmarks.
The QA Protocol (10 Bits): Introduced an interactive protocol where a small model recovers the capability of a large model using only 10 binary questions.
- Achieved compression ratios between 0.0006 and 0.004.
- This represents a >100× improvement over prior SOTA LLM-based compression (which hovered around 0.08).
Compression-Compute Frontier: Mapped the trade-off where increased computation (iterative questioning, larger models) yields exponentially better compression.

4. Key Results

The results are evaluated on 8 benchmarks (Math, Code, Science) using Claude 4.5 models.

Method	Compression Ratio	Recovery Rate (Medium/Hard Problems)	Notes
GZIP	~0.44	N/A	Baseline for text.
LLM Arithmetic Coding (Base)	~0.18	N/A	Lossless baseline.
LLM + LoRA (Lossless)	~0.09	N/A	2× improvement over base.
Summarization (Lossy)	~0.034	High	Preserves answer quality.
QA Protocol (10 bits)	0.0006 – 0.004	23% – 72% (Easier) 7% – 38% (Harder)	Transfers 10 bits of knowledge.

Recovery: On easier benchmarks (GSM8K, MATH), the QA protocol allows Haiku to recover 23–72% of the performance gap between Haiku and Opus using only 10 bits.
Hard Problems: On very hard problems (AIME, HLE), recovery is lower (7–38%), suggesting that if the SLM lacks the fundamental reasoning structure, 10 bits of guidance are insufficient.
Comparison: The QA protocol is over 100× more efficient than transmitting the full text of the large model's response.

5. Significance and Implications

Model Security & Egress Limiting: The paper highlights a critical application for model security. If an adversary can only extract a limited number of bits from a datacenter (egress limiting), they cannot steal model weights. However, if text is highly compressible (e.g., to 10 bits), strict bit limits might inadvertently allow attackers to steal "knowledge" or high-quality outputs without stealing weights. Conversely, defenders can use these compression techniques to enforce stricter egress limits on legitimate users without degrading utility.
Interactive Proofs & Knowledge Transfer: The work formalizes the connection between interactive proof systems (IP = PSPACE) and LLM compression. It suggests that interaction is a more efficient way to transfer complex knowledge than transmitting static data.
Scientific Process Analogy: The authors draw a philosophical parallel to the scientific method: a scientist (SLM) proposes hypotheses and asks binary questions to nature (LLM) to refine their understanding, rather than trying to memorize the entire universe of facts.
Future Directions: The paper suggests that future compression gains will depend on improving the SLM's ability to ask targeted questions. If the SLM can better identify its own reasoning gaps, the efficiency of the QA protocol will increase.

Conclusion

This paper demonstrates that by moving from lossless to interactive lossy compression, we can achieve massive gains in efficiency. The "Haiku to Opus in 10 bits" result proves that the information content of a large model's reasoning can be distilled into a tiny number of bits, provided the receiving model has the capacity to interpret those bits through an iterative, question-answering process. This redefines the limits of data transmission in AI systems.