Technical Summary: Benchmarking and Enhancing VLM for Compressed Image Understanding

Problem Statement

With the rapid expansion of multimedia services, efficient image compression is critical for managing transmission bandwidth and storage. While traditional and learned compression methods have advanced significantly for human perception, their impact on machine vision tasks remains under-explored. Specifically, existing Vision-Language Models (VLMs) are predominantly trained and evaluated on high-bitrate, uncompressed images. Their ability to interpret low-bitrate compressed images—where visual artifacts and information loss are prevalent—has not been systematically investigated. Furthermore, current research on "Image Coding for Machines" (ICM) often focuses on specific tasks (e.g., object detection) with fixed backbones or specific codecs, lacking generalization across diverse compression types and VLM architectures. There is a critical gap in understanding how different compression codecs (traditional, learning-based, and generative) degrade VLM performance and how to bridge this gap without retraining entire models or optimizing codecs for every specific model.

Methodology

1. Comprehensive Benchmarking

The authors introduce a large-scale benchmark comprising over one million compressed images to evaluate VLM performance.

• Codecs: The study covers 11 state-of-the-art codecs across three categories:
  • Traditional: JPEG, HM (HEVC), VTM (VVC).
  • Learning-based: ELIC, TCM, MLICpp.
  • Generative: HiFiC, MS-ILLM, DiffEIC, RDEIC, StableCodec.
• Models: Nine VLMs of varying scales (1B to 32B parameters) are evaluated, including Qwen2.5-VL, InternVL3, and Janus-pro.
• Tasks: Seven evaluation metrics cover both coarse-grained (semantic understanding, spatial reasoning) and fine-grained (OCR, object hallucination) tasks, utilizing benchmarks such as POPE, GQA, OCRBench, MME, MMBench, and SEEDBench.

2. Decomposition of Performance Gap

The paper proposes a theoretical framework to decompose the performance degradation observed in compressed images into two distinct components:

• Information Gap: The irreducible loss of task-relevant information caused by the compression algorithm itself. This gap exists regardless of the model's architecture and cannot be fixed by model adaptation alone; it requires improvements in the compression algorithm.
• Generalization Gap: The performance loss resulting from the VLM's failure to generalize to distorted inputs. This gap is attributed to the model's lack of exposure to compression artifacts during training and is theoretically reducible through model adaptation.

3. Proposed Solution: Universal VLM Adapter

To address the generalization gap, the authors propose a lightweight, universal VLM adapter.

• Architecture: The method modifies the Vision Transformer (ViT) encoder of the VLM. It introduces a conditional embedding mechanism that encodes both the codec type (one-hot encoded) and the compression bitrate level.
• Mechanism: These conditions are mapped to a latent space and added to the positional embeddings (using Rotary Positional Embedding, RoPE) of the visual tokens. This allows the encoder to condition its feature extraction on the specific distortion characteristics of the input.
• Training Strategy: The adapter is trained using a distillation loss. The goal is to minimize the Mean Squared Error (MSE) between the features extracted from compressed images (by the adapted encoder) and the features extracted from the original uncompressed images (by the original encoder). This approach avoids full fine-tuning of the VLM, focusing instead on adapting the vision encoder to handle diverse compression distortions.

Key Findings and Results

Benchmark Observations

1. Performance Degradation: VLMs exhibit significant performance drops on heavily compressed images (bitrates < 0.1 bpp), struggling with semantic understanding and task accuracy.
2. Model Robustness: Stronger VLMs (larger parameter counts) generally perform better on compressed images, maintaining relative performance rankings similar to uncompressed baselines. However, model scaling does not consistently eliminate compression-induced degradation.
3. Codec Differences: Generative codecs (particularly diffusion-based ones like RDEIC and StableCodec) offer superior semantic reconstruction at low bitrates compared to traditional and learning-based codecs, making them more suitable for VLM-oriented coding. However, they often underperform on fine-grained tasks like OCR.
4. Scaling Laws: The standard scaling laws (where larger models generalize better to unseen inputs) do not strictly apply to compressed images; increasing model size does not guarantee reduced degradation from compression artifacts.
5. Metric Decoupling: There is a decoupling between human-centric pixel-level metrics (e.g., PSNR) and VLM task performance. High pixel fidelity does not always correlate with strong semantic understanding.

Adapter Performance

• Quantitative Gains: The proposed adapter consistently improves VLM performance across all tested codecs and bitrates.
  • Improvements range from 10% to 30% depending on the task and compression level.
  • For example, on the POPE metric with JPEG compression at low bitrates, the adapter achieved approximately a 30% gain.
• Generalization: The adapter demonstrates strong generalization capabilities. When trained on a subset of codecs (e.g., JPEG, ELIC, ILLM), it successfully improves performance on unseen codecs (e.g., HM, MLICpp, DiffEIC) without retraining.
• Ablation Studies: The inclusion of both codec identity and distortion-level conditions yields the best results. However, the adapter remains effective even without explicit metadata (blind adaptation), though performance stability across bitrates improves with conditional inputs.

Significance and Claims

The paper claims to provide the first comprehensive benchmark specifically designed to evaluate VLMs on compressed images, filling a gap in the literature where existing benchmarks focus on high-quality inputs. By decomposing performance loss into "information" and "generalization" gaps, the authors offer a new analytical framework for understanding machine vision under compression.

The primary contribution is the demonstration that the generalization gap can be effectively mitigated through a lightweight, universal adapter. This approach offers a practical and scalable solution for real-world VLM applications where bandwidth constraints necessitate the use of compressed visual inputs. Unlike previous ICM methods that require training separate models for specific codecs or tasks, this method provides a unified enhancement framework that adapts to diverse compression distortions, bridging the gap between current VLM capabilities and the realities of compressed data transmission.

The authors acknowledge limitations, such as the exclusion of proprietary models due to API costs, but assert that their findings and methods provide valuable insights for advancing image coding for machines and semantic compression.