LoC-Path: Learning to Compress for Pathology Multimodal Large Language Models

1. Problem Statement

Whole Slide Images (WSIs) in digital pathology are gigapixel-scale images that, when processed by Multimodal Large Language Models (MLLMs), present unique challenges distinct from standard vision-language tasks:

• Extreme Sequence Length: A single WSI is partitioned into thousands of tiles (patches), resulting in visual token sequences that are orders of magnitude longer than standard image inputs.
• High Redundancy: Visual tokens exhibit significant redundancy at both global (entire slide) and local (neighboring patches) scales.
• Sparse Task Relevance: Only a small fraction of these tiles contains diagnostically relevant information for a specific query (e.g., a specific diagnosis or report section). Most tiles are irrelevant to the text supervision.
• Computational Bottleneck: Existing slide-level MLLMs (e.g., WSI-LLaVA, SlideChat) typically use a "concatenate-and-attend" strategy (LLaVA-style), feeding long visual prefixes directly into the LLM. This leads to:
  • Quadratic attention complexity $O((T+L)^2)$, where $T$ is text length and $L$ is the massive number of visual tokens.
  • High memory and FLOP costs for both training (multimodal tuning) and inference.
  • Inefficiency under limited computational resources, making end-to-end development difficult.

2. Methodology: LoC-Path

The authors propose LoC-Path, a resource-efficient framework that adopts a "compress-then-fuse" paradigm. Instead of concatenating all visual tokens, LoC-Path reduces redundancy before fusion and routes only task-relevant information into the LLM. The architecture consists of four key components:

A. Multi-Stage Token Redundancy Reduction

To address the sheer volume of tokens, LoC-Path employs two stages of compression:

1. Sparse Token Merger (STM):
   • Goal: Reduce local redundancy.
   • Mechanism: Before resampling, STM merges highly correlated neighboring tiles within non-overlapping $s \times s$ windows on the 2D grid.
   • Effect: Reduces the sequence length significantly (e.g., $N \to N/s^2$) while preserving coarse spatial layout and tissue diversity.
2. MAE-Pretrained Resampler:
   • Goal: Reduce global redundancy and compress the sequence into a compact latent set.
   • Mechanism: Inspired by how pathologists scan slides, this module uses a Masked Autoencoder (MAE) pre-training strategy. It learns to reconstruct masked tiles from partial observations, forcing the model to "glance" broadly and capture distinct features rather than averaging them out.
   • Architecture: Uses a small set of learnable latent queries ($L \ll N$) with sparse Top-K cross-attention to aggregate information from the STM output.
   • Regularization: Includes Coverage Loss (ensuring latents attend to diverse regions) and Diversity Loss (preventing latents from collapsing to a global average).

B. Efficient Task-Relevant Multimodal Fusion

To address the sparsity of relevant evidence, LoC-Path replaces standard concatenation with a Select-Then-Fuse strategy:

1. Token Importance Scorer (TIS):
   • Goal: Dynamically select the most relevant visual latents based on the specific text query.
   • Mechanism: A lightweight router scores the projected visual latents ($V$) using the mean-pooled text query embedding. It selects the top-$M$ most relevant latents ($M \le L$).
   • Training: Uses knowledge distillation from the final fusion layers (CARA) to train the scorer without extra labels, plus a pairwise ranking loss to ensure correct ordering.
2. Cross-Attention Routing Adapter (CARA):
   • Goal: Fuse the selected latents into the LLM efficiently.
   • Mechanism: Instead of concatenating visual tokens with text, CARA injects the selected $M$ latents into specific decoder layers of the LLM via gated cross-attention.
   • Complexity: Reduces the fusion cost from $O((T+L)^2)$ to $O(T^2 + TM)$. Since $M$ is small and constant, the cost becomes linear with respect to the visual input size, avoiding the quadratic scaling of long prefixes.

3. Key Contributions

1. Regime Characterization: The paper identifies and quantifies the specific "WSI regime" for MLLMs: extreme multi-scale redundancy combined with sparse, query-dependent task relevance. This challenges the assumption that all visual tokens must be preserved and jointly attended to.
2. LoC-Path Framework: A novel architecture combining:
   • STM for local merging.
   • MAE-Resampler for global compression (trained with MAE pre-training to handle weak supervision).
   • TIS for dynamic, query-dependent routing.
   • CARA for efficient cross-modal fusion.
3. End-to-End Efficiency: The design significantly lowers both training costs (multimodal tuning) and inference latency/memory, making slide-level MLLM development feasible on limited hardware (e.g., 4x A6000 GPUs) without sacrificing performance.

4. Experimental Results

The authors evaluated LoC-Path on WSI-Bench, Slide-Bench, and Path-Bench datasets, comparing against SOTA models like WSI-LLaVA, SlideChat, and GPT-4o.

• Performance:

  • LoC-Path achieves competitive or superior performance compared to SOTA slide-level MLLMs.
  • On WSI-Bench, it matches WSI-LLaVA (0.739 vs 0.754 average score) while using significantly fewer resources.
  • It achieves the highest WSI-Precision (0.384) and WSI-Relevance (0.473) in report generation tasks, outperforming models with much larger token budgets.
  • In zero-shot generalization (Slide-Bench), it performs comparably to specialized baselines.

• Efficiency:

  • Inference: Reduces inference TFLOPs by ~82% and peak memory usage by ~39% compared to the mean of general baselines.
  • Training: Reduces peak reserved memory during stage-2 fusion training (19.36 GB vs. 21.23 GB for LLaVA-style) and avoids the steep memory growth seen in models like SlideChat (which uses 20k tokens).
  • Latency: Maintains nearly constant Time-to-First-Token (TTFT) and memory scaling as visual input size increases, whereas baselines scale linearly or quadratically.

• Ablation Studies:

  • MAE Pre-training: Essential for preventing latent collapse; models without it or with dense attention perform significantly worse.
  • Routing (TIS/CARA): Removing TIS or the ranking loss degrades performance, confirming that selecting and ordering the right latents is crucial.
  • STM: Aggressively reduces FLOPs (up to 14x reduction) with negligible accuracy loss.

5. Significance

LoC-Path represents a paradigm shift in how pathology MLLMs are built. By acknowledging that gigapixel slides are inherently redundant and that clinical relevance is sparse, the authors move away from "brute-force" token concatenation toward intelligent compression and routing.

• Practicality: It enables the training and deployment of high-performance slide-level MLLMs on standard hardware, democratizing access to advanced computational pathology tools.
• Scalability: The architecture is theoretically scalable to even larger WSIs without the quadratic cost explosion of current methods.
• Clinical Alignment: The "glance-and-select" mechanism mimics the diagnostic workflow of human pathologists, who focus on specific regions of interest rather than processing every pixel equally, leading to more interpretable and clinically relevant outputs.