💻 computer science

MixCompress: Mixture of Experts for Variable Rate Learned Image Compression

MixCompress is a unified variable-rate image compression framework that overcomes feature entanglement and computational bottlenecks by integrating sparse Mixture-of-Experts routing, a dynamic Mixture-of-Depths extension for scalable capacity, and Conditional Auxiliary Transforms, achieving performance that rivals or surpasses individually optimized single-rate models.

Original authors: Calvin-Khang Ta, Praneet Singh, Tong Shao, Peng Yin

Published 2026-07-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Calvin-Khang Ta, Praneet Singh, Tong Shao, Peng Yin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to send a photo to a friend, but you have a strict limit on how much data you can send. If you want to send a tiny, blurry thumbnail, you can compress the file heavily. If you want to send a crystal-clear, high-definition masterpiece, you need to send much more data. In the world of "Learned Image Compression," computers use smart AI models to figure out the best way to shrink these photos without losing too much quality. Think of these AI models as super-smart chefs who know exactly how to chop, blend, and package ingredients (pixels) to fit them into a lunchbox (the file).

For a long time, these AI chefs had a major problem: they needed a different, completely separate chef for every single size of lunchbox. If you wanted a tiny thumbnail, you needed Chef Tiny. If you wanted a giant poster, you needed Chef Giant. This meant storing dozens of different "recipe books" (models) on your phone or server, which took up a lot of space and was a pain to manage. Scientists tried to fix this by teaching one "Super Chef" to cook for all sizes at once. They did this by giving the chef a dial to turn up or down the flavor. But here's the catch: trying to make a perfect soup for a tiny bowl and a massive banquet at the same time often confused the chef. The instructions for "make it smooth and simple" (for small sizes) clashed with "keep every tiny detail sharp" (for big sizes), leading to a meal that wasn't great at either.

This is where the new paper, MixCompress, comes in to save the day. The researchers, working at Dolby Laboratories, realized that instead of forcing one chef to do everything poorly, they should build a kitchen with a team of specialized experts who can step in only when needed. They created a system where the AI doesn't just turn a dial; it actually swaps out parts of its brain depending on how big the photo needs to be.

Here is how their magic kitchen works:

The Team of Experts (Mixture of Experts)
Imagine a school project where you have a group of friends. For a simple poster, you only need two people to draw the background. But for a complex, detailed map, you need the whole team, including the person who is great at tiny details. MixCompress uses a "Mixture of Experts" (MoE). It has a main brain that handles the basics for everyone, but it also has a team of specialized "expert" sub-networks. When the computer needs to compress a photo for a small file size, it only activates the experts good at smoothing things out. When it needs a huge, detailed file, it activates the experts who are obsessed with preserving every tiny hair and texture. Crucially, these experts don't all work at the same time; the system picks the best ones for the job. This stops the "confusion" where the instructions for a blurry image mess up the instructions for a sharp one.

The Deepening Team (Mixture of Depths)
Sometimes, even picking the right experts isn't enough; you might need more brainpower for the hardest jobs. The authors added a feature called "Mixture of Depths" (MoD). Think of this as a ladder. For easy tasks, the data takes a short, quick path through the kitchen. For the hardest, most detailed tasks, the data is routed through a longer, deeper path with more steps and more processing power. This allows the system to grow its brainpower exactly when it's needed, without wasting energy on simple photos.

The Dynamic Seasoning (Conditional Auxiliary Transforms)
Finally, the team added a special tool called "Conditional Auxiliary Transforms" (CAT). Imagine that while the main chef is cooking, a sous-chef is constantly tasting the soup and adding just the right amount of salt or pepper based on the size of the bowl. In the computer's case, this tool adjusts the energy of different parts of the image (like the smooth sky vs. the noisy grass) based on the target file size. It helps the system decide what to keep and what to toss out more efficiently, acting like a smart filter that changes its settings automatically.

The Results
The researchers tested this new system on thousands of images. They found that MixCompress didn't just solve the problem of having too many models; it actually made the pictures better. By using this team of specialized experts, the system could handle all the different file sizes in a single model without the "confusion" that plagued previous methods. In fact, the single MixCompress model performed so well that it often beat the old method of having separate, individually trained chefs for each size.

The paper shows that by letting the AI switch between different specialized "brains" and adjust its own depth, we can get high-quality images at any size without needing to store a massive library of different models. It's a smarter, more flexible way to pack our digital photos, proving that sometimes, having a team of specialists is far better than trying to be a jack-of-all-trades.

Technical Summary: MixCompress

Problem Statement

Learned Image Compression (LIC) has achieved high performance by jointly optimizing nonlinear transforms and entropy models. However, state-of-the-art LIC models typically require separate, independently trained networks for each rate-distortion operating point. This leads to significant storage overhead, increased training costs, and deployment complexity.

Existing Variable Bit-Rate (VBR) approaches attempt to unify these operating points within a single model by conditioning a shared dense backbone on a rate parameter (e.g., via latent gain scaling or affine transformations). The paper identifies a fundamental flaw in this "dense parameter modulation" strategy: gradient conflict.

The Conflict: Low-bit-rate optimization requires aggressive smoothing to reduce bitrate, while high-bit-rate optimization requires the preservation of high-frequency textural details.
The Consequence: Forcing a shared backbone to approximate these divergent mappings causes "feature entanglement." The gradients from low-rate samples destructively interfere with the parameters needed for high-fidelity reconstruction, leading to sub-optimal performance that falls short of individually optimized single-rate models.

Methodology

The authors propose MixCompress, a unified VBR framework that replaces dense parameter modulation with sparse structural specialization. The architecture integrates three core components:

1. Sparse Structural Routing (MoE and MoD)

Instead of modulating all parameters, MixCompress replaces bottleneck layers in the analysis transform, synthesis transform, and hyper-prior networks with Mixture-of-Experts (MoE) modules.

Mechanism: A gating network, conditioned solely on the rate embedding ( $\lambda_e$ ), routes input features to a subset of $K$ experts out of $N$ total experts.
Shared Expert: A global shared expert ( $E_{global}$ ) is always active to ensure stable information flow for universally useful features.
Gradient Mitigation: By isolating the optimization trajectories of different rates into separate expert subspaces, the framework mitigates gradient interference. The inner product of gradients for conflicting rates is strictly bounded by the intersection of active experts, shifting the optimization dynamics toward orthogonality.
Mixture-of-Depths (MoD): To address the increased representational demands of higher bit-rates, the authors introduce an MoD extension. In this variant, experts are not just parameter-specialized but also depth-specialized. High-rate samples are routed to deeper, more complex expert networks (stacked Bi-RWKV or convolutional blocks), dynamically scaling model capacity based on the target rate.

2. Conditional Auxiliary Transforms (CAT)

To complement structural routing, the authors introduce Conditional Auxiliary Transforms (CAT), an extension of wavelet-based shortcuts (AuxT).

Function: CAT performs a 2D Haar discrete wavelet transform (DWT) on intermediate features, decomposing them into frequency sub-bands (LL, LH, HL, HH).
Rate-Adaptivity: Unlike static shortcuts, CAT uses a FiLM (Feature-wise Linear Modulation) head to generate scaling parameters ( $\gamma_\lambda, \beta_\lambda$ ) conditioned on the rate embedding $\lambda_e$ .
Effect: This allows for dynamic modulation of sub-band energy. Low-rate regimes benefit from stronger energy compaction, while high-rate regimes preserve finer details, all without altering the main entropy model or arithmetic coding pipeline.

3. Inference Efficiency

A critical design constraint is that the gating networks and CAT modulations depend strictly on the rate embedding $\lambda_e$ , which is independent of the input image $x$ .

Zero Overhead: Consequently, the expert selection and modulation parameters can be precomputed and cached in a lookup table before inference. This introduces zero routing overhead during the actual encoding/decoding process, as the network parameters do not need to be recomputed per image.

Key Contributions

Unified VBR Framework: MixCompress achieves a new Pareto frontier for computationally efficient image coding, matching or surpassing individually optimized single-rate baselines while using a single unified model.
Gradient Conflict Resolution: The paper demonstrates that sparse structural routing (MoE/MoD) effectively resolves the gradient interference inherent in dense modulation methods, as evidenced by layer-wise gradient cosine similarity analysis shifting toward orthogonality.
Dynamic Capacity Scaling: The MoD extension allows the model to dynamically expand representational capacity (via depth) for high-bit-rate tasks, addressing a limitation of fixed-capacity MoE models.
Deployment Efficiency: The framework reduces storage overhead significantly. For example, a single MixCompress-MoD model covers the same rate-distortion curve as six independently trained single-rate models but requires only ~156M total parameters (with ~120M active per pass) compared to the ~397M parameters required for the ensemble of single-rate models.

Experimental Results

The authors evaluated MixCompress on the Kodak, CLIC (Professional validation/test), and Tecnick datasets against state-of-the-art single-rate models (LALIC, LIC-TCM, FAT) and dense VBR baselines (QRAF, CondConv).

Performance vs. Single-Rate Baselines: MixCompress variants consistently match or surpass individually optimized single-rate checkpoints. Notably, MixCompress-LALIC-MoD achieved BD-rate reductions of -25.73% (CLIC Valid), -28.71% (CLIC Test), and -28.88% (Tecnick) relative to VTM-23.1, outperforming the single-rate LALIC anchor by significant margins.
Performance vs. Dense VBR: Dense modulation baselines (QRAF, CondConv) suffered substantial performance degradation (3.7–12.0 percentage points worse than single-rate LALIC). MixCompress effectively closed this gap, with the MoD variant even exceeding the single-rate counterpart.
Visual Quality: Visual comparisons showed that dense methods failed to reconstruct high-frequency details (e.g., thin cables) at high bit-rates due to over-smoothing from low-rate gradient interference. MixCompress successfully preserved these details.
Efficiency: Despite supporting multiple rates, MixCompress was trained for the same total duration as a single single-rate model. Inference latency saw only a slight increase, while memory usage for the full model remained manageable compared to storing multiple checkpoints.

Significance and Claims

The paper claims that conditional structural allocation is fundamentally more effective than dense weight modulation for variable-rate learned image compression. By decoupling the optimization trajectories of competing rate points through sparse gating and dynamic depth adaptation, MixCompress resolves the "feature entanglement" that has historically limited unified VBR models.

The authors assert that this approach establishes a new standard for practical neural image compression, achieving superior representational density without a proportional increase in inference cost. The work suggests that future VBR systems should prioritize structural specialization over simple parameter conditioning to fully realize the potential of learned compression across diverse operating points.