OmniZip: Learning a Unified and Lightweight Lossless Compressor for Multi-Modal Data

OmniZip is a unified, lightweight, and efficient lossless compressor that leverages a lightweight backbone with specialized tokenization and routing mechanisms to effectively compress diverse multi-modal data (including images, text, speech, and more), outperforming state-of-the-art methods while enabling near real-time inference on resource-constrained edge devices.

The Gist

Imagine you have a massive digital library. Inside, you have books (text), paintings (images), voice recordings (speech), medical scans, DNA sequences, and even recordings of how things feel to the touch (tactile data).

Currently, if you want to shrink these files to save space, you need a different "compressor" for every single type of file. You need a specific tool for your photos, another for your voice notes, and a third for your DNA data. It's like having a different pair of scissors for every type of paper you own. This is messy, expensive, and slow.

OmniZip is the solution proposed in this paper. Think of it as a "Universal Swiss Army Knife" for digital data compression. Instead of needing a different tool for every job, OmniZip is one lightweight, smart tool that can shrink any kind of file efficiently.

Here is how it works, broken down into simple concepts:

1. The Problem: The "One-Size-Fits-None" Dilemma

• The Old Way: Most AI compressors are like specialists. A "Text Expert" is great at shrinking books but terrible at shrinking photos. A "Photo Expert" is the opposite. If you want to compress a whole library, you need to hire a whole team of specialists, which takes up a lot of space and time.
• The New Way (OmniZip): OmniZip is a Generalist. It doesn't just know one type of data; it knows them all. But usually, generalists are slow or require massive computers. OmniZip is special because it is lightweight (small and fast) yet powerful enough to beat the specialists.

2. How OmniZip Thinks: The Three Magic Tricks

To make this "Universal Compressor" work, the researchers gave it three superpowers:

A. The Universal Translator (Modality-Unified Tokenization)

Imagine you have a book written in English, a painting, and a song. They all look different.

• OmniZip's Trick: It has a translator that converts everything into a single, simple language of "tokens" (like Lego bricks).
  • A word becomes a brick.
  • A pixel of color becomes a brick.
  • A sound wave becomes a brick.
• Why it matters: Once everything is just a pile of Lego bricks, the computer doesn't have to worry about whether it's looking at a face or a sentence. It just sees a sequence of bricks. Crucially, this translation is reversible, meaning you can turn the bricks back into the original file perfectly without losing a single detail.

B. The Smart Traffic Controller (Modality-Routing Context Learning)

Now that everything is Lego bricks, the computer needs to predict what brick comes next to shrink the file.

• The Problem: Predicting the next word in a sentence is different from predicting the next pixel in a photo.
• OmniZip's Trick: It uses a "Mixture of Experts" system. Imagine a busy office with different departments.
  • When the computer sees a "DNA brick," it routes the task to the "Genetics Department."
  • When it sees a "Voice brick," it routes it to the "Audio Department."
  • When it sees a "SQL Database brick," it routes it to the "Data Department."
• The Magic: It doesn't wake up the whole office for every task. It only wakes up the specific experts needed for that specific type of data. This keeps the computer running fast and cool, even on a phone.

C. The Flexible Brain (Modality-Routing Feedforward)

After the experts do their job, the computer needs to combine their thoughts.

• OmniZip's Trick: Just like the traffic controller, the "thinking" part of the brain also routes tasks to different mini-experts. This allows the model to be flexible. It can learn the complex patterns of a gene sequence and the smooth patterns of a voice recording without getting confused.

3. The "Training Gym" Secret (Reparameterization)

To make OmniZip smart enough to handle all these different tasks, the researchers used a clever training trick.

• The Analogy: Imagine a student studying for a math test. During the study session (training), they use a giant, heavy backpack full of extra notes and practice problems to learn deeply. But when they take the actual test (inference), they put on a light, empty backpack.
• How it works: During training, OmniZip uses extra "branches" to learn more complex patterns. Once it's done learning, it folds those extra branches back into the main body. The result? It has the knowledge of a giant model but the size and speed of a tiny model.

4. Why This is a Big Deal

• Speed: It's fast enough to run on your iPhone or a laptop in real-time. You could compress a whole movie or a database while you wait for your coffee.
• Efficiency: It beats the old standard tools (like gzip) by huge margins. On some datasets, it shrinks files 40% to 60% more than the best tools we have today.
• Simplicity: Instead of managing a dozen different compression programs, you just install OmniZip. It handles your photos, your voice memos, your medical records, and your code all in one go.

Summary

OmniZip is like a master chef who can cook a steak, bake a cake, and brew coffee with the same small, portable kitchen kit. It doesn't need a massive factory (supercomputer) to do it; it just needs a smart recipe (the routing mechanism) and a universal translator (tokenization) to turn any ingredient into a delicious, compact meal.

This technology means that in the future, we might not need to worry about file sizes or different formats. One small, smart tool could handle all our digital storage needs, saving us space, money, and time.

Technical Summary

1. Problem Statement

Lossless compression is critical for efficient data storage and transmission. While learning-based compressors have shown promise, they face two significant limitations:

• Modality Fragmentation: Existing methods are typically designed for a single modality (e.g., text-only or image-only). In multi-modal systems, this necessitates deploying multiple separate compressors, increasing software complexity and hardware costs.
• Computational Complexity: State-of-the-art (SOTA) learning-based compressors often rely on massive Large Language Models (LLMs) with billions of parameters. These models suffer from excessive inference latency (e.g., taking minutes to compress a single image) and are too large for practical deployment on edge devices, often exceeding the size of the data they compress.

The core challenge is designing a unified compressor that can handle diverse data types (images, text, speech, tactile, gene sequences, databases) while remaining lightweight enough for real-time inference on resource-constrained devices.

2. Methodology: OmniZip

OmniZip is a unified, lightweight lossless compressor built on a RWKV-7 backbone (a linear-time recurrent architecture). It addresses modality heterogeneity through three key architectural innovations and a training strategy:

A. Modality-Unified Tokenization

To handle diverse data formats within a single model, OmniZip maps all inputs into a unified, fully reversible token space:

• Text-like Data (Language, Gene, Database): Uses a SentencePiece BPE tokenizer (16K vocab) with domain-specific symbols added (e.g., nucleotide bases for genes, SQL keywords for databases).
• Image-like Data (Natural, Medical, Tactile): Images are divided into $16 \times 16 \times 3$ patches. Pixels are flattened in raster order, and RGB values are treated as individual sub-pixel tokens (vocab size 256). Tactile force data is mapped to pseudo-RGB images.
• Speech: Treated as a raw byte stream, with each byte as a token (vocab size 256).
• Modality Prefixes & Masking: Each sequence is prefixed with a modality identifier (e.g., <image>, <speech>). Before arithmetic coding, modality masking zeros out probabilities for non-target tokens, reducing estimation errors.

B. Modality-Routing Context Learning

Different modalities exhibit distinct contextual dependencies. OmniZip integrates a Mixture-of-Experts (MoE) mechanism into the Time Mixing module of the RWKV backbone:

• Routing Strategy: A learnable router assigns tokens to specific experts.
• Targeted Application: Experiments show that the Value (V) projection layer benefits most from routing diversity, as it encodes specific memory content, whereas the Key (K) and Receptance (R) layers serve as semantic indices and gating mechanisms.
• Efficiency: Only the V layer uses MoE (4 experts, top-k=2), keeping the computational overhead negligible while allowing the model to adapt to specific modality dynamics.

C. Modality-Routing Feedforward

Standard MLPs in RWKV blocks treat all tokens uniformly. OmniZip replaces the standard feedforward layer with an MoE-based modality-routing module:

• Structure: Similar to the context learning module, but the experts are small MLPs (with a hidden factor of $2\times$) rather than projection layers.
• Benefit: This allows the model to learn distinct non-linear representations for different modalities without increasing the number of activated parameters per token.

D. Reparameterization Training Strategy

To enhance model capacity without increasing inference cost:

• Training: High-rank matrix decomposition branches are added to the R, K, and V layers to increase representational power.
• Inference: These branches are merged back into the main path via structural reparameterization ($W = W_0 + AB$), ensuring the inference complexity remains identical to the lightweight baseline.

3. Key Contributions

1. Unified Architecture: The first lightweight lossless compressor capable of handling seven distinct modalities (natural/medical images, tactile signals, text, gene sequences, databases, and speech) within a single model.
2. Efficient Design: A novel combination of RWKV-7, modality-unified tokenization, and sparse MoE routing that achieves SOTA compression with significantly fewer parameters than LLM-based approaches.
3. Edge Deployment: Demonstrates near real-time inference capabilities on resource-constrained devices (MacBook CPUs and iPhone NPUs), bridging the gap between high-performance compression and practical usability.
4. Comprehensive Evaluation: Validated on 16 datasets across 7 modalities, outperforming both classical compressors (gzip, bzip2) and specialized learning-based methods.

4. Experimental Results

OmniZip was evaluated on 16 datasets (e.g., Kodak, enwik9, LibriSpeech, WikiSQL) and compared against classical compressors (gzip, zstd, FLAC) and learning-based SOTA (Llama3, P2LLM, DLPR).

• Compression Efficiency:
  • Image-like: OmniZip-L achieves 42% higher efficiency than gzip on the CLIC-Mobile dataset and outperforms specialized image compressors on medical and tactile data.
  • Text-like: On the enwik9 dataset, OmniZip-L achieves 62% higher efficiency than gzip and competes with specialized text compressors (NNCP, CMIX) despite having far fewer parameters.
  • Speech: On LibriSpeech, OmniZip improves efficiency by 42% over gzip and 23% over the specialized FLAC codec.
• Inference Speed & Hardware:
  • Server (A100 GPU): Achieves MB/s-level speeds (up to ~4 MB/s for OmniZip-S).
  • Edge Devices: Achieves near real-time speeds (~1 MB/s) on MacBook CPUs and iPhone NPUs.
• Parameter Efficiency: OmniZip models (S, M, L) range from 4.8M to 152M parameters, drastically smaller than the 8B+ parameter LLMs used in competing multi-modal approaches, yet they match or exceed their compression performance.

5. Significance

OmniZip represents a paradigm shift in lossless compression by proving that unification does not require massive complexity.

• Practicality: It enables the deployment of a single, lightweight compressor across diverse data types in real-world applications (e.g., mobile apps, IoT devices, medical imaging systems), eliminating the need for multiple specialized tools.
• Scalability: By leveraging efficient recurrent architectures (RWKV) and sparse MoE routing, it offers a scalable path for multi-modal AI that is both performant and deployable on the edge.
• Future Impact: The work provides a blueprint for handling heterogeneous data in the era of multi-modal AI, demonstrating that specialized domain knowledge can be effectively integrated into a unified, lightweight framework.