🤖 AI

Lossless Tensor Compression as Program Synthesis

The paper introduces Brevis, a novel lossless tensor compression system that formulates the problem as program synthesis using a typed domain-specific language and a learned production prior to automatically generate compact, self-contained programs that achieve significant storage reductions and competitive throughput across diverse model checkpoints.

Original authors: Jieke Shi (James), Junda He (James), Wenjia Jiang (James), Weifeng Sun (James), Shidong Pan (James), Zhensu Sun (James), Chengran Yang (James), Peixin Zhang (James), Yifan Jia (James), Zhou Yang (Jame

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Jieke Shi (James), Junda He (James), Wenjia Jiang (James), Weifeng Sun (James), Shidong Pan (James), Zhensu Sun (James), Chengran Yang (James), Peixin Zhang (James), Yifan Jia (James), Zhou Yang (James), Thong Hoang (James), Xiwei Xu (Sherry), Zhenchang Xing, David Lo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to ship a massive library of digital blueprints across the ocean. These aren't just any blueprints; they are the intricate, mathematical instructions that teach computers how to speak, draw, or sing. In the world of artificial intelligence, these blueprints are called "model checkpoints," and they are growing so huge and numerous that shipping them is becoming a logistical nightmare, costing a fortune in storage space and transfer time.

To solve this, engineers usually try two things. The first is "lossy" compression, which is like taking a photo of a blueprint, squishing the details, and hoping the architect can guess the missing parts. This saves space but ruins the exactness; you can't rebuild the original blueprint perfectly. The second is "lossless" compression, like zipping a file. This keeps every single bit of information safe, but standard zipping tools treat the blueprint like a random pile of paper scraps, ignoring the fact that the blueprint actually has a clever, repeating structure. They miss the patterns.

The big question is: Can we do better? Instead of just squishing the file or blindly zipping it, can we look at the blueprint, understand its hidden patterns, and write a tiny, custom instruction manual that tells a computer exactly how to rebuild the original file from scratch? If we could write a unique, perfect recipe for every single blueprint, we might save a massive amount of space without losing a single detail.

This is exactly what a team of researchers has done with a new tool called Brevis. They treat the problem of compressing these giant AI blueprints not as a file-packing job, but as a "program synthesis" challenge. Think of it like this: instead of just stuffing a toy into a box, Brevis looks at the toy, realizes it's made of three red blocks and two blue blocks, and writes a tiny note that says, "Take three red blocks, stack them, then add two blue ones."

Brevis uses a special, custom-made language (a "Domain-Specific Language" or DSL) to write these notes. This language has special commands that can spot when a pattern repeats, when a number is just a copy of the one before it, or when a section of data is perfectly uniform. When Brevis compresses a file, it doesn't just pick a pre-made zipper; it acts like a detective, searching through millions of possible "recipes" to find the shortest, most efficient one that can rebuild the file bit-for-bit. To make this search fast, it learns from a small sample of the file to guess which recipes are most likely to work, kind of like a chef knowing that a specific soup usually needs salt before it even tastes it.

The results are impressive. The researchers tested Brevis on 10 different public AI models, including those for language, audio, and images, covering a total of 2.13 TB of data. They found that Brevis could shrink this massive amount of data down to 1.41 TB. That is a 33.93% reduction in storage space. To put that in perspective, Brevis created archives that were up to 30.87% smaller than those made by popular, general-purpose tools like gzip and zstd. It even beat specialized tools designed specifically for AI data, creating archives up to 2.90% smaller than the best competitor, ZipNN.

Perhaps most exciting is that Brevis doesn't just save space; it does it fast. It can compress data at a speed of 3.60 GB per second and decompress it at 6.61 GB per second. And because the "recipe" it writes is a self-contained program, the computer can run it directly to rebuild the file perfectly, without needing to search for patterns again. The researchers suggest that this approach of "writing a program to represent data" could be a powerful new way to handle the exploding size of AI models, offering a way to store and move them efficiently without losing a single byte of the original information.

Technical Summary: Lossless Tensor Compression as Program Synthesis (Brevis)

Problem Statement

The rapid proliferation of large language models (LLMs) and other deep learning architectures has led to an exponential growth in the number and size of model checkpoints. Storing, transferring, and deploying these assets has become increasingly costly. While general-purpose compressors (e.g., gzip, zstd) are widely used, they treat tensor weights as generic byte sequences, ignoring the specific structural properties of tensors (e.g., data types, shapes, floating-point layouts, and element relationships). Conversely, existing tensor-specific compressors often rely on fixed, format-specific pipelines or predefined transformations that may fail to capture diverse, tensor-specific patterns such as repeated values, recurring subsequences, and simple relationships among elements.

The core challenge is to achieve lossless (bit-exact) compression that adapts to the unique structure of each tensor without relying on rigid, pre-defined schemes.

Methodology: Brevis

The authors present Brevis, a system that formulates lossless tensor compression as a program synthesis problem. Instead of selecting a fixed codec, Brevis synthesizes a compact, self-contained program for each tensor that, when executed, reconstructs the original tensor bit-for-bit.

1. Typed Domain-Specific Language (DSL)

Brevis employs a typed DSL with reversible operators designed to capture recurring tensor structures:

Lit: A universal fallback that stores arbitrary word streams using various physical encodings (raw words, bit packing, Huffman, rANS) to select the smallest representation.
Const: Stores a single word and its repetition count.
Concat: Joins programs for adjacent regions.
Repeat: Expands a non-empty child program multiple times to capture repeated subsequences.
Map: Applies width-preserving bijections (e.g., XOR, modular addition, ZigZag, Gray coding) to expose relationships not visible as simple strings.
Scan: Stores an initial word followed by updates (XOR or modular addition), exposing adjacent correlations.
Merge: Combines equal-length children whose widths sum to the parent width, useful for splitting floating-point fields (sign, exponent, fraction) or bit planes.

2. Target-Directed Synthesis with Bounded A* Search

The synthesis process searches for a program $P$ such that $Exec(P) = Bits(X)$, minimizing the serialized size $L(P)$ .

Backward Search: Rather than enumerating programs, Brevis searches backward from the target tensor. Every "hole" in the partial program is paired with the exact stream it must generate. An operator is only applied if its decomposition of the target stream is valid (i.e., the composition of the children's outputs reconstructs the target exactly).
Checkpoint-Specific Production Prior: To guide the search efficiently, Brevis learns a probability distribution over DSL productions from a small, deterministic sample of tensors within a checkpoint. This prior captures the specific structural tendencies of the model (e.g., frequent use of Repeat or Merge for specific layers).
Bounded A Algorithm: The search uses a bounded A algorithm where states are prioritized by $g(s) + h(s)$ $g (s) + h (s)$ .
- $g(s)$ is the accumulated cost based on the learned prior.
- $h(s)$ is an admissible heuristic for the remaining completion cost.
- A byte lower bound ( $LB_L$ ) prunes states that cannot improve the current best solution.
- The search is constrained by an expansion budget (e.g., 1 expansion per tensor in the main evaluation) and node/depth limits.

3. Archive Format and Reconstruction

The resulting archive preserves the original safetensors header and layout. Each tensor record contains its name, dtype, shape, and the synthesized program. Decompression involves parsing the program, validating type rules, and executing it directly without search or the learned prior, ensuring bit-exact reconstruction.

Key Contributions

Formulation: The paper formulates bit-exact checkpoint compression as program synthesis, representing each tensor as a compact, self-contained DSL program.
DSL Design: A typed domain-specific language with reversible operators that capture diverse tensor structures (values, sequences, fields, and planes) while guaranteeing bit-exact reconstruction.
Synthesis Algorithm: A bounded A* synthesis algorithm guided by a checkpoint-specific production prior learned from a small sample, enabling efficient discovery of compact programs within a practical budget.
Evaluation: Comprehensive evaluation on 10 public checkpoints (language, audio, image) totaling 2.13 TB of data.

Results

The authors evaluated Brevis on 10 public checkpoints spanning language, audio, and image generation models (2.13 TB of source data).

Storage Reduction: Brevis reduced the total data size from 2.13 TB to 1.41 TB, achieving a 33.93% storage reduction.
Comparison with General-Purpose Compressors: Brevis produced smaller archives than zstd, gzip, LZ4, and Snappy on all 10 checkpoints. The average reduction over these baselines ranged from 12.94% (vs. zstd) to 30.87% (vs. Snappy).
Comparison with Tensor-Specific Compressors:
- Compared to ZipNN, Brevis produced smaller archives on all 10 checkpoints, saving 10.21 GB in aggregate. The pooled saving is 0.72% (with a mean per-checkpoint saving of 0.71%).
- Compared to DFloat11 (on the Llama-3.1-8B checkpoint where a validated native result was available), Brevis was 2.90% smaller (316.27 MB saved).
Throughput: Under a practical concurrency configuration, Brevis achieved 3.60 GB/s compression and 6.61 GB/s decompression. It lies on the Pareto frontier, offering better storage reduction than general compressors while maintaining competitive throughput.
Statistical Significance: Improvements over all paired baselines remained statistically significant after Holm correction ( $p_H = 0.0098$ ).

Significance and Claims

The paper claims that by treating compression as program synthesis, Brevis overcomes the limitations of fixed pipelines. It demonstrates that a self-contained program can serve as a highly efficient, bit-exact representation of a tensor, adapting to specific structural redundancies that generic codecs miss.

The authors emphasize that their approach:

Preserves every source byte (bit-exact), making it suitable for archival and exact checkpoint transfer, unlike lossy methods (quantization/pruning).
Does not require hardware-specific decoders or predefined codec families.
Achieves significant storage savings across diverse model types and data formats (BF16, FP32, FP8).

The paper modestly acknowledges limitations, noting that the evaluation covers public checkpoints rather than all tensor workloads, and the current implementation synthesizes tensors independently (without cross-tensor synthesis). The authors position Brevis as a step toward more adaptive, structure-aware lossless compression for the growing ecosystem of large models.