InfoTok: Information-Theoretic Regularization for Capacity-Constrained Shared Visual Tokenization in Unified MLLMs

1. Problem Statement

Unified Multimodal Large Language Models (MLLMs) aim to integrate image understanding (perception) and image generation (synthesis) within a single framework using a shared visual tokenizer. However, a fundamental tension exists:

• Understanding requires semantic abstraction (high-level concepts).
• Generation requires fine-grained perceptual details (textures, colors, exact layouts).

Existing shared-token approaches are largely architecture-driven, relying on end-to-end task losses without an explicit criterion for what information to preserve. Consequently, the shared tokenizer often operates as a "compute-bounded learner" with a finite representational budget, inadvertently wasting capacity on high-entropy, hard-to-exploit visual variations (redundancy) rather than prioritizing reusable structures (semantic entities and compositional cues). This leads to suboptimal performance in both tasks.

2. Methodology: InfoTok

The authors propose InfoTok, a framework grounded in the Information Bottleneck (IB) principle to explicitly regulate information flow from images to shared tokens and finally to multimodal outputs.

Core Concept: Capacity-Constrained Perspective

InfoTok views the shared tokenizer as a learner that must compress high-dimensional images into a limited token budget. It enforces a trade-off between compression (removing redundancy) and task relevance (preserving information necessary for downstream tasks).

The Information-Regularized Objective

The method extends the classical IB principle to handle dual objectives (Understanding $u$ and Generation $g$). The objective function ($\mathcal{L}_{InfoTok}$) minimizes a Lagrangian composed of three terms for each task branch:

1. Compactness ($I(\tilde{Z}; I)$): Penalizes redundant input information. It encourages the tokenizer to discard high-entropy variations and retain only essential visual content.
2. Sufficiency ($I(\tilde{Z}; Y^{GT})$): Maximizes the mutual information between the tokens and the ground-truth targets (text for understanding, image for generation). This ensures the tokens retain task-critical semantics and perceptual cues.
3. Alignment ($I(\tilde{Z}; T)$): Maximizes mutual information between visual tokens and text embeddings to enforce cross-modal consistency within the unified next-token prediction pipeline.

The total loss is:
$$\mathcal{L}_{InfoTok} = \mathcal{L}^{(u)}_{IB} + \mathcal{L}^{(g)}_{IB} + \mathcal{L}_{distill}$$

Practical Instantiation

Since Mutual Information (MI) is intractable for high-dimensional continuous visual data, InfoTok uses two differentiable estimators:

• Variational Information Bottleneck (VIB): Uses a variational posterior (Gaussian) and KL-divergence to bound MI.
• Hilbert Schmidt Independence Criterion (HSIC): A kernel-based dependence measure that avoids explicit density modeling.

Additionally, a Distillation Constraint is introduced to ensure the information regularization applied to task-specific projections ($\tilde{Z}$) propagates back to the shared encoder tokens ($Z$), stabilizing the shared visual encoder ($V_{enc}$).

3. Key Contributions

1. Capacity-Constrained Perspective: The paper introduces a new theoretical viewpoint that shared tokenizers in unified MLLMs are compute-bounded learners. It argues that explicit criteria are needed to allocate representational budgets toward reusable structures rather than raw visual noise.
2. Information-Theoretic Formulation: It formalizes shared token learning using the IB principle, creating an explicit regularization objective that balances compression, sufficiency, and cross-modal alignment.
3. Tractable Implementation: The authors derive practical, differentiable instantiations using VIB and HSIC, making the theory applicable to deep neural networks without requiring closed-form distribution knowledge.
4. Plug-and-Play Framework: InfoTok is designed as a regularization module that can be integrated into existing unified MLLMs without architectural changes or additional training data.

4. Experimental Results

The authors evaluated InfoTok on three representative unified MLLMs: Harmon, OpenUni, and Show-o2.

• Performance Gains:
  • Understanding: Consistent improvements across benchmarks like MME, POPE, SEED, GQA, and MMMU. For example, Harmon's MME score increased from 1411 to 1565 (+154).
  • Generation: Significant improvements on Geneval, Geneval++, and WISE. Show-o2's Geneval score improved from 0.60 to 0.75.
  • Qualitative: Generated images showed better instruction following (e.g., correct attribute binding, spatial relationships) and higher semantic coherence.
• Ablation Studies:
  • Components: The core IB terms (Compactness + Sufficiency) drove the majority of gains. Alignment and Distillation provided stability and further boosts.
  • Estimators: Both VIB and HSIC variants yielded consistent improvements, proving robustness to the specific MI estimator used.
  • Data Efficiency: InfoTok achieved peak performance with 1.5M samples; scaling to 2M yielded diminishing returns, indicating it does not rely on massive fine-tuning datasets.
  • Comparison: InfoTok outperformed standard fine-tuning, entropy regularization, and token dropout, which often degraded performance or failed to provide consistent gains.
• Frequency Analysis: CKA analysis revealed that InfoTok shifts token representations toward low-frequency components (structure/layout) and away from high-frequency components (noise/texture), confirming the "compression of redundancy" hypothesis.

5. Significance

• Paradigm Shift: The paper moves the field from "architecture-driven" tokenization (e.g., larger codebooks, separate encoders) to "principle-driven" tokenization. It suggests that controlling what information is preserved is more critical than simply scaling model size.
• Unified Representation: It provides theoretical and empirical evidence that a single, information-regularized token space can effectively support both semantic abstraction and perceptual synthesis, challenging the notion that these tasks require fundamentally disjoint representations.
• Practical Utility: Since InfoTok requires no new data and works as a fine-tuning regularizer, it offers a low-cost, high-impact method for improving existing unified MLLMs, making it highly applicable for future model development.

In conclusion, InfoTok demonstrates that explicitly regulating information flow via information-theoretic constraints creates a more optimal shared token space, leading to state-of-the-art performance in unified multimodal learning without architectural bloat.