Technical Summary: Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation

Problem Statement

Vision-language models (VLMs) offer a promising avenue for automating radiology report generation. However, applying these models to full 3D Computed Tomography (CT) volumes presents significant computational challenges. Modern foundation vision encoders (VEs) can generate tens of thousands of vision tokens per scan, creating a primary bottleneck for the downstream Large Language Model (LLM). While vision-to-language projectors can compress this sequence to reduce computation, aggressive compression risks discarding clinically relevant details. Conversely, preserving high-resolution inputs requires effective compression strategies to maintain a fixed downstream token budget.

The central design question addressed by this work is how the "vision-token budget" should be optimally allocated across three competing dimensions:

1. Input Field of View (FOV): Whole-volume encoding vs. anatomy-guided Region of Interest (ROI) cropping.
2. Spatial Resolution: Standard resolution vs. higher-resolution inputs.
3. Vision-to-Language Projection: The choice of projector architecture and its compression ratio.

Methodology

The authors propose a systematic evaluation framework based on a standard three-stage LLaVA-style VLM architecture: a frozen pretrained Vision Encoder, a trainable vision-to-language projector, and an instruction-tuned LLM.

Experimental Setup

• Datasets: Two large-scale CT report datasets were used: CT-RATE (50k+ chest CTs) and Merlin (25k+ abdominal CTs).
• Vision Encoders (VEs): Four heterogeneous medical foundation models were benchmarked:
  • Primus: A 3D Vision Transformer (ViT).
  • Curia: A 2D ViT encoding axial slices independently.
  • Merlin: A 3D CNN (ResNet-based).
  • CT-FM: A 3D CNN (SegResNet-based).
• Projectors: Five token-reducing projectors were evaluated alongside a token-preserving MLP baseline:
  • Grid-based: AveragePool, ConcatAdapter, TokenPacker.
  • Query-based: AttentionPool, PerceiverResampler.
  • Compression Ratios: Evaluated up to 64× compression.
• LLMs: Five instruction-tuned models ranging from 1.7B to 4B parameters (Qwen3, Qwen3-VL, Phi-4-mini, Gemma3).
• Evaluation Metric: Clinical Macro F1 was the primary metric, calculated using fine-tuned classifiers (RadBERT for CT-RATE, ModernBERT for Merlin) trained on LLM-extracted labels. Lexical metrics (BLEU, ROUGE) were reported only for final comparisons.

Input Regimes

The study compared three input strategies under a fixed LLM token budget:

1. Whole: Encoding the entire resized volume.
2. ROI: Cropping to an anatomical ROI (lungs for CT-RATE, whole-body foreground for Merlin) using TotalSegmentator.
3. High Resolution: Increasing effective resolution via multi-block encoding (Primus-4B) or denser axial feature aggregation (Curia-64S).

Key Contributions

1. Systematic Evaluation of Design Choices: This is the first controlled study to decouple input FOV and resolution from the LLM token budget, systematically evaluating the interplay between VEs, projectors, and LLMs across two large datasets.
2. ROI Cropping as a Consistent Strategy: The study identifies anatomy-guided ROI cropping as the most robust intervention. It improves clinical macro F1 in 19 of 20 configurations, with average gains of +3.7 points for Primus and +1.1 points for Curia.
3. Projector-Dependent Resolution Benefits: The paper demonstrates that increasing input resolution is not universally beneficial; its success is heavily dependent on the projector's aggregation mechanism. Only projectors capable of effective strong compression (TokenPacker and PerceiverResampler) benefit from higher-resolution inputs.
4. Encoder Dominance: The choice of Vision Encoder is shown to have a larger impact on performance than the choice of LLM. ViT-based encoders (Primus, Curia) significantly outperformed CNN-based encoders (Merlin, CT-FM).
5. State-of-the-Art Performance: The proposed configurations achieve new state-of-the-art results on both benchmarks, reaching a clinical macro F1 of 49.5 on CT-RATE and 49.0 on Merlin.

Results

• ROI vs. Whole Volume: ROI cropping consistently outperformed whole-volume encoding. For the 3D ViT (Primus), the gain was substantial (+3.7 F1), likely due to the increased effective resolution within the fixed token budget. For the 2D ViT (Curia), which already operates near native in-plane resolution, the gain was smaller (+1.1 F1).
• High Resolution Impact:
  • On CT-RATE, higher resolution did not improve performance for Primus (hitting a resolution ceiling) but benefited Curia when paired with the PerceiverResampler.
  • On Merlin, higher resolution consistently benefited both encoders when paired with TokenPacker or PerceiverResampler, particularly for detecting small pathologies like renal cysts and gallstones.
  • Grid-based projectors like ConcatAdapter and AveragePool often degraded with higher resolution due to information loss or parameter explosion.
• Token Compression:
  • Token-preserving MLPs generally outperformed token-reducing projectors for the Primus encoder.
  • For Curia, the PerceiverResampler at 64× compression surpassed the uncompressed MLP baseline (49.2 vs. 48.5 on CT-RATE).
  • TokenPacker was the strongest compressed projector for Primus on the Merlin dataset.
  • ConcatAdapter showed severe performance degradation under high compression due to the scaling of its input dimension and parameter count.
• LLM Influence: While LLM choice affected performance (variations of 2.2–5.3 F1), it was less influential than the VE choice (variations >10 F1). Notably, models initialized from multimodal pretraining (Qwen3-VL) outperformed text-only counterparts (Qwen3-1.7B) even when the native vision tower was replaced.

Significance and Claims

The paper claims that the allocation of the vision-token budget is a critical design variable for scalable 3D radiology report generation. The authors emphasize that:

• ROI cropping is the most consistent and effective strategy to improve clinical performance without increasing computational cost.
• Projector architecture dictates whether higher-resolution inputs are beneficial; simple pooling or concatenation strategies fail to leverage high-resolution data under strong compression, whereas query-based or structured aggregation (PerceiverResampler, TokenPacker) succeeds.
• Foundation Model Adaptation: The choice of the Vision Encoder is the dominant factor in system performance, suggesting that future efforts should focus on adapting robust medical foundation models rather than solely optimizing LLMs.
• Clinical Relevance: The study highlights that lexical metrics are unreliable for medical report quality, and clinical macro F1 derived from automated classifiers provides a more meaningful evaluation of diagnostic correctness.

The authors conclude that while hardware capabilities are increasing, the demand for efficient vision-token utilization will grow with higher-resolution imaging and larger models. Therefore, careful design of the input regime and projector remains central to achieving state-of-the-art 3D radiology report generation.