⚡ electrical engineering

MoECodec: Image Compression for joint human and machine perception via Mixture-of-Experts

MoECodec is a token-aware image compression framework that integrates a Mixture-of-Experts architecture with a stable routing strategy and lightweight Group Shuffle MLP to dynamically adapt computation based on input content and task objectives, thereby achieving superior performance in both human and machine perception tasks within a single unified model.

Original authors: Jiancheng Zhao, Xiang Ji, Yifan Zhan, Zunian Wan, Yinqiang Zheng

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiancheng Zhao, Xiang Ji, Yifan Zhan, Zunian Wan, Yinqiang Zheng

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant library of photos. Traditionally, when we want to shrink these photos to save space (compression), we use a "one-size-fits-all" approach. We treat every single pixel in the image exactly the same way, smoothing out the grass, the sky, and the text on a sign with the same level of care.

But here's the problem: Humans and Machines see the world differently.

Humans care about smooth colors and natural textures.
Machines (like the AI driving a car or identifying a disease) care about specific shapes, edges, and semantic details. They don't care if the grass is perfectly smooth; they care that the "stop sign" is clearly defined.

The paper introduces MoECodec, a new way to compress images that acts like a smart, dynamic team of specialists rather than a single, rigid worker.

The Core Idea: The "Specialist Team" (Mixture of Experts)

Think of a standard image compression model as a single chef trying to cook a massive banquet. They chop every vegetable the exact same way, whether it's a delicate herb or a tough potato. It's inefficient and often misses the mark.

MoECodec replaces that single chef with a team of four specialized chefs (Experts) working in the same kitchen.

Expert 1 might be great at preserving sharp edges (perfect for machine vision).
Expert 2 might be great at smoothing out colors (perfect for human eyes).
Expert 3 and 4 have their own unique skills.

The magic isn't just having the team; it's the Manager (The Router).
In old systems, the manager forced every single pixel to be chopped by the same chef. In MoECodec, the Manager looks at each tiny piece of the image (called a "token") and asks: "What does this specific piece need right now?"

If the token is part of a car's license plate, the Manager sends it to the "Sharp Edge Expert."
If the token is part of a blurry sky, the Manager sends it to the "Smooth Color Expert."

This means the computer only uses the specific "chef" needed for that specific part of the image, saving energy and time.

Solving the "Chaos" Problem

The authors realized that if you just let these experts pick their own work randomly, the results would look like static on an old TV (noisy and fragmented). One expert might grab a pixel here, and another grabs the pixel right next to it, creating a messy patchwork.

To fix this, they introduced two clever rules:

The "Expert-Choice" Rule: Instead of the pixels picking the experts, the experts pick the pixels. Imagine the "Sharp Edge Expert" scanning the whole image and saying, "I claim all the edges!" This ensures that all the edges are handled by the same expert, creating a smooth, coherent plan rather than a chaotic one.
The "Neighborhood" Rule: They added a rule that says, "If you are handling a pixel, you should probably handle your neighbors too." This prevents the "salt-and-pepper" noise and ensures that a whole region (like a tree) is processed by the same specialist.

The "Lightweight" Trick

Usually, having a team of experts makes the system huge and slow because you have to store the brainpower for all of them. The authors solved this with a Group Shuffle MLP.

Think of this as a rotating shift system. Instead of giving every expert a full, separate brain (which is expensive), they give them a shared, modular toolkit. They break the work into small groups, shuffle the tools between the groups, and let the experts work efficiently without needing a massive amount of memory. This keeps the system small enough to run on real devices while still being powerful.

What Did They Find?

The paper tested this system on two fronts:

Human Vision: When they tried to make the images look good for humans, MoECodec actually did a better job than previous methods, saving more space while keeping the image clear.
Machine Vision: When they tested it on tasks like recognizing objects or finding cars, the machine performed significantly better than before. Because the system knew exactly which parts of the image were important for the machine, it didn't waste space on irrelevant details.

The Bottom Line

MoECodec is like upgrading from a factory assembly line where every car gets the same paint job, to a custom workshop.

It looks at every tiny part of the image.
It decides which specialist is best for that specific part.
It does this in a way that keeps the whole team organized and the tools lightweight.

The result is a single, smart system that can compress images perfectly for both human eyes and machine brains, without needing to build a separate, expensive system for each task.

Technical Summary: MoECodec

Problem Statement

Machine-oriented image compression has emerged as a critical research area to support downstream computer vision tasks such as classification, detection, and segmentation. However, existing approaches face significant limitations:

Task-Specific Designs: End-to-end optimization for specific tasks often requires dedicated encoder-decoder pairs or multi-branch architectures, leading to substantial parameter overhead and deployment complexity.
Transfer-Based Adaptation: Methods that adapt human-oriented base codecs using small trainable parameters (e.g., adapters) remain externally attached and rely on task-specific heuristics rather than being intrinsically learned within a unified model.
Static Computation Patterns: Both lines of work largely apply uniform transformations across all image tokens. This fails to account for the fact that different image regions exhibit markedly different semantic importance and complexity for machine perception, leading to inefficient resource allocation.

Methodology

The authors propose MoECodec, a unified image compression framework that integrates token-wise Mixture-of-Experts (MoE) into transformer-based Learned Image Compression (LIC) models. The core innovation is replacing standard Feed-Forward Network (FFN) layers in transformer blocks with MoE layers, enabling dynamic, token-level computation conditioned on input content and task objectives.

Key Architectural Components

Token-Wise MoE Integration:
- MoECodec replaces FFN layers in both the encoder and decoder of transformer-based LIC backbones (e.g., TIC, TinyLIC, TCM).
- Input tokens are adaptively routed to specialized expert networks, allowing the model to support heterogeneous perceptual objectives within a single unified model.
Routing Strategy (Expert-Choice + Spatial Regularization):
- Challenge: Standard Transformer-based LIC models often use fine-grained tokenization (e.g., 1×1 patches). Naive token-choice (TC) routing leads to noisy, fragmented expert assignments (salt-and-pepper artifacts) due to the lack of spatial context.
- Solution: The authors adopt Expert-Choice (EC) routing, where each expert actively selects its top- $k$ preferred tokens from the entire sequence. This reverses the decision perspective, allowing experts to consider global context and ensuring balanced utilization without auxiliary load-balancing losses.
- Spatial Total Variance (TV) Regularization: To further address local spatial inconsistencies in EC routing, a spatial TV regularization term is added to the training loss. This penalizes high-frequency variations in the spatial expert affinity maps, encouraging piecewise-smooth and coherent expert assignments.
Lightweight Expert Architecture (Group Shuffle MLP):
- Challenge: Replacing FFNs with $E$ independent experts typically introduces an $E\times$ parameter overhead, which is prohibitive for compression models where FFNs dominate the parameter count. Simply reducing expert size degrades channel aggregation capabilities.
- Solution: The authors propose Group Shuffle MLP (GShMLP). This architecture partitions input channels into $G$ groups and applies independent linear projections within each group. A parameter-free channel shuffle is inserted between the two projection layers to restore cross-channel interaction. This design significantly reduces parameter complexity while maintaining effective channel aggregation.

Key Contributions

Unified Framework: MoECodec is the first unified LIC framework to integrate token-wise MoE into transformer-based codecs, enabling a single model to jointly optimize for reconstruction quality and multiple machine downstream tasks.
Addressing LIC-Specific Challenges: The paper identifies and solves two specific challenges of introducing MoE into LIC:
- Spatial Discontinuity: Solved via Expert-Choice routing combined with Spatial Total Variance Regularization.
- Parameter Overhead: Solved via the lightweight Group Shuffle MLP (GShMLP) architecture.
Dynamic Compute Allocation: The framework establishes a dynamic computation paradigm where heterogeneous task objectives are accommodated intrinsically, allowing the model to focus computation on task-relevant regions.

Experimental Results

Extensive experiments demonstrate consistent improvements over strong baselines in both reconstruction and machine tasks.

Image Reconstruction: Applied to three transformer-based backbones (TIC, TinyLIC, TCM), MoECodec achieved BD-Rate improvements of 11.54%, 8.12%, and 6.42% respectively on the Kodak dataset (PSNR), while simultaneously reducing total parameters.
Machine Downstream Tasks: Using TIC as a base, MoECodec outperformed Full Finetune, TransTIC, and AdaptICMH baselines:
- Classification (ImageNet-1K): 7.74% BD-Rate improvement.
- Object Detection (COCO): 8.98% BD-Rate improvement.
- Instance Segmentation (COCO): 9.72% BD-Rate improvement.
Efficiency: MoECodec reduced activated parameters from 7.51M to 6.25M and achieved a 142ms inference latency on an RTX 4090, outperforming the original TIC (155ms) by 8.4%.
Ablation Studies:
- Switching from Token-Choice to Expert-Choice routing improved BD-Rate from 19.57% to 15.90%.
- Adding Spatial TV regularization further reduced BD-Rate to 15.44%.
- GShMLP provided a better efficiency-performance trade-off than reduced FFN expansion ratios.
- Placing MoE layers in both encoder and decoder yielded the best performance.

Significance and Claims

The paper claims that MoECodec offers a novel approach to end-to-end multi-task codec training, moving away from static, task-specific custom training paradigms. By enabling sparsely activated, dynamic compute allocation, the framework allows a single model to adaptively focus computational resources on semantically important regions for specific tasks. The authors assert that this approach successfully balances the trade-off between model cost (parameters and latency) and performance across both human perception (reconstruction) and machine vision tasks, demonstrating that token-level granularity is effective for heterogeneous computation in multi-task image compression.