Adaptive Transform Coding for Semantic Compression

1. Problem Statement

The paper addresses the challenge of compressing semantic feature embeddings (learned representations from vision models) rather than raw pixel data.

• Context: With the rise of edge AI and foundation models (e.g., CLIP), visual data is increasingly processed locally to extract semantic meaning (embeddings) before transmission. These embeddings must be compressed for downstream tasks like classification, retrieval, or fusion.
• Limitations of Existing Methods:
  • Task-Agnostic vs. Task-Specific: Many existing compression methods are tied to specific tasks or require retraining when operating conditions change.
  • Neural Codec Complexity: State-of-the-art neural compression methods (learned codecs) are often complex, black-box, and lack interpretability.
  • Inefficiency of Standard Methods: Traditional linear methods like Principal Component Analysis (PCA) fail to capture the heterogeneous, non-Gaussian nature of semantic feature distributions effectively.
• Goal: Develop a task-agnostic, interpretable, and efficient compression scheme for semantic embeddings that outperforms or competes with learned neural codecs without requiring complex neural network training.

2. Methodology

The authors propose an Adaptive Transform Coding (ATC) scheme grounded in the conditional rate–distortion theory of a Gaussian Mixture Model (GMM).

A. Theoretical Foundation

• GMM Modeling: The paper models the semantic feature space $X$ as a mixture of $K$ Gaussian components: $X \sim \sum \pi_c \mathcal{N}(\mu_c, \Sigma_c)$.
• Conditional Rate-Distortion: Instead of treating the source as a single Gaussian distribution, the authors derive a coding scheme where the transform and quantization are selected based on the inferred mixture component (mode) $C$.
• Genie-Aided Bound: They derive an upper bound on the rate-distortion function assuming the component index $C$ is known (or transmitted losslessly). This leads to a "reverse water-filling" solution where distortion is allocated across components and dimensions based on a global quality parameter $\theta$.

B. The Coding Scheme

The method operates in two stages:

1. Offline Design Stage:

   • Feature Extraction: A frozen semantic encoder (e.g., CLIP, ResNet) extracts embeddings from a representative dataset.
   • GMM Fitting: A GMM with $K$ components is fitted to these embeddings.
   • Transform & Quantizer Selection: For each component $c$:
     • Compute the covariance matrix $\Sigma_c$ and perform eigendecomposition to obtain the Karhunen–Loève Transform (KLT) matrix $U_c$.
     • Apply whitening to the data.
     • Determine the target distortion for each dimension using the reverse water-filling rule.
     • Select the optimal scalar Lloyd–Max quantizer (from a pre-defined set) for each whitened coefficient to meet the distortion target.

2. Online Coding Stage:

   • Inference: For a new image, extract the embedding $x$.
   • Mode Selection: Estimate the most likely mixture component $\hat{c}$ using Maximum A Posteriori (MAP) estimation (minimizing the distance to the component mean adjusted by covariance and prior probability).
   • Transformation & Quantization:
     • Transform the embedding using the KLT specific to $\hat{c}$.
     • Quantize the coefficients using the pre-selected Lloyd–Max quantizers for that mode.
   • Entropy Coding: Encode the quantizer indices and the mode index $\hat{c}$ using arithmetic coding based on their respective probabilities.
   • Reconstruction: The decoder recovers the mode, de-quantizes, and applies the inverse transform to reconstruct the feature vector.

C. Complexity Optimization

To address the $O(N^2K)$ computational cost of fitting GMMs to high-dimensional vectors, the authors propose a complexity-aware variant:

• First, apply a global PCA to reduce dimensionality from $N$ to $M$.
• Fit the GMM in the reduced $M$-dimensional space.
• This significantly reduces storage and computation while maintaining performance.

3. Key Contributions

1. Theoretical Derivation: Derived a genie-aided upper bound for the rate-distortion function of a GMM, showing that mode-dependent transforms and quantizers governed by a single reverse-water-filling parameter are optimal.
2. Practical Algorithm: Translated the theory into a practical, neural-network-free adaptive transform coder using mode-dependent KLTs and scalar Lloyd–Max quantizers.
3. Superior Performance: Demonstrated that this simple, interpretable method outperforms or is competitive with state-of-the-art learned neural semantic codecs (like PQVAE and MBT) across various vision backbones (ResNet, MobileNet, ViT) and foundation models (CLIP).
4. Flexibility: The method is task-agnostic, meaning it does not need retraining for different downstream tasks (classification, retrieval, etc.).

4. Experimental Results

The authors evaluated the method on embeddings from ImageNet, Oxford-IIIT Pet, and Food-101 datasets using models like ViT-B/32, ViT-L/14, ResNet50, and MobileNetV3.

• Rate-Distortion (MSE): The adaptive scheme (ATC) consistently outperformed non-adaptive transform coding (TC, $K=1$) and PCA-based baselines. It approached the theoretical rate-distortion bound more closely than non-adaptive methods.
• Semantic Preservation (Cosine Similarity):
  • ATC showed significant gains in cosine similarity as the number of mixture components ($K$) increased (up to 5% gain for $K=20$ vs $K=1$).
  • It achieved higher cosine similarity than the neural baseline PQVAE across most operating points.
• Downstream Task Performance (Zero-Shot Classification):
  • On ImageNet, ATC maintained accuracy close to the uncompressed upper bound at high bitrates and outperformed PQVAE in the medium-bitrate regime.
  • On Oxford-IIIT Pet and Food-101, ATC demonstrated better generalization than both PQVAE and the neural MBT codec.
• Comparison with Neural Codecs:
  • While MBT (a neural codec with entropy coding) showed comparable or slightly better performance in some metrics, ATC achieved similar results when entropy coding was applied to ATC.
  • Crucially, ATC achieves this without the heavy training overhead and black-box nature of neural codecs.

5. Significance and Conclusion

• Paradigm Shift: The paper challenges the assumption that semantic compression must rely on complex learned neural networks. It proves that classical information-theoretic principles (transform coding) are highly effective when adapted to the statistical structure (GMM) of semantic embeddings.
• Interpretability & Efficiency: The proposed method is transparent (interpretable transforms and quantizers) and computationally efficient (especially with the PCA variant), making it suitable for deployment on edge devices.
• Standardization Relevance: The work aligns with emerging standards like JPEG AI and MPEG Feature Coding for Machines (FCM), offering a robust, task-agnostic solution for compressing machine-readable visual data.

In summary, Adaptive Transform Coding provides a simple, effective, and theoretically grounded alternative to deep learning-based semantic compression, achieving state-of-the-art performance while retaining the benefits of classical coding theory.