Align Your Query: Representation Alignment for Multimodality Medical Object Detection

1. Problem Statement

Medical object detection faces a significant challenge when a single detector is trained on mixed multimodality datasets (e.g., combining Chest X-rays, CT scans, MRI, and Pathology).

• Heterogeneity: Different imaging modalities possess distinct statistical properties and visual characteristics, leading to disjoint representation spaces.
• Performance Degradation: Training a unified model on these mixed data often results in degraded performance compared to modality-specific models because the model struggles to generalize across the "domain shift" inherent in different imaging types.
• Limitations of Existing Methods: Standard multimodal integration techniques (like cross-attention with text) often require heavy architectural changes or pre-training the image encoder, which adds complexity and latency.

2. Methodology

The authors propose a detector-agnostic framework that aligns object query representations with modality context using two primary components: Modality Tokens, Multimodality Context Attention (MoCA), and QueryREPA.

A. Modality Tokens

Instead of using complex language streams, the authors define Modality Tokens: compact, text-derived embeddings that encode the imaging modality (e.g., "CXR", "CT") and the target class (e.g., "Aortic enlargement").

• Construction: Text prompts in the format "{Class} in {Modality}" (e.g., "Brain tumor in MRI") are passed through a pretrained text encoder (e.g., CLIP, BiomedCLIP).
• Projection: The [CLS] token embedding is projected into the model's latent space to create a fixed vector $m_{d,c}$.
• Advantage: These tokens are lightweight, require no extra annotations, and act as stable semantic anchors shared across the dataset.

B. Multimodality Context Attention (MoCA)

MoCA is a novel mechanism to inject modality context directly into the detection process without altering the backbone or head architecture.

• Mechanism: For a given image, the relevant modality token is appended to the set of object queries ($Q$) to form an augmented set $\tilde{Q} = [Q, f_\theta(m_{d,c})]$.
• Self-Attention Mixing: This augmented set is fed into the decoder's Self-Attention layers. Unlike cross-attention (where queries attend to a separate text stream), MoCA allows object queries to attend to the modality token via self-attention.
• Effect: This enables every object query to dynamically incorporate modality-specific context during the iterative refinement process, effectively guiding the query to be "modality-aware."

C. QueryREPA (Query Representation Alignment)

To further strengthen the alignment, the authors introduce a pretraining stage before the main detection training.

• Objective: A contrastive loss (InfoNCE) aligns the mean-pooled object query representations of an image with its corresponding modality token.
• Process:
  1. Extract query embeddings from a specific decoder layer.
  2. Mean-pool them to get a single image-level query representation.
  3. Project this representation and compute a contrastive loss against the modality token.
• Modality-Balanced Batching: To ensure effective contrast, mini-batches are constructed using a round-robin sampler to ensure each batch contains images from distinct modalities. This forces the model to learn to separate queries based on modality.
• Goal: This shapes the query manifold to be modality-aware and class-faithful, maximizing the mutual information between queries and their modality tokens.

3. Key Contributions

1. Detector-Agnostic Modality Tokens: A simple formulation using text-derived tokens that explicitly encode modality and class context with minimal overhead and no architectural changes.
2. MoCA (Multimodality Context Attention): A drop-in self-attention mechanism that fuses modality tokens with object queries, allowing for intra-query mixing of representations. This is the first method to explore this specific intra-query mixing for object detection.
3. QueryREPA: A pretraining strategy that aligns query representations to modality tokens using a contrastive objective. This enables the utilization of large-scale, weakly labeled (image-level only) medical datasets to improve downstream detection performance.
4. Synergistic Framework: The combination of MoCA and QueryREPA creates a robust system that outperforms existing state-of-the-art methods on mixed multimodality datasets.

4. Experimental Results

The method was evaluated on a comprehensive mixed dataset comprising CXR, CT, MRI, Colonoscopy, and Pathology images.

• State-of-the-Art Performance:
  • Using the DINO detector as a baseline, the proposed method (MoCA + QueryREPA) achieved a total AP of 41.1 (vs. 37.7 baseline) and AP50 of 65.5 (vs. 58.6 baseline).
  • It outperformed other SOTA detectors including Deformable DETR, Sparse R-CNN, DiffusionDet, GLIP, and Grounding DINO across all metrics (AP, AP50, AP75, and scale-specific APs).
• Robustness:
  • The method showed consistent improvements regardless of the text encoder used (CLIP, BiomedCLIP, or PubMedCLIP), proving the architectural contribution is the primary driver of performance.
  • It improved performance across all object scales (small, medium, large).
• Efficiency:
  • Latency: MoCA adds negligible latency (e.g., DINO runtime increased from 9.37ms to 9.59ms per image) because it only appends a single token to the self-attention mechanism.
• Unlabeled Data Utilization:
  • Experiments demonstrated that QueryREPA can leverage large datasets with only image-level labels (no bounding boxes) during pretraining, significantly boosting final detection accuracy. This addresses the scarcity of annotated medical data.
• Qualitative Improvements:
  • Visualizations showed that the method significantly reduced false positives and improved bounding box localization for subtle lesions compared to baselines.
  • UMAP visualizations confirmed that QueryREPA successfully disentangled the query space, separating queries by modality, whereas the baseline queries were highly entangled.

5. Significance

• Practical Clinical Application: The approach offers a practical path toward building generalist medical AI models that can handle diverse imaging modalities within a single framework, a critical step for real-world clinical deployment where multiple scan types are common.
• Efficiency: By avoiding heavy architectural modifications or expensive cross-modal pretraining of image encoders, the method is computationally efficient and easy to integrate into existing pipelines.
• Data Efficiency: The ability to utilize weakly labeled data (image-level only) via QueryREPA addresses a major bottleneck in medical AI, where obtaining pixel-level bounding box annotations is expensive and time-consuming.
• Paradigm Shift: The paper shifts the focus of representation alignment from the image encoder (backbone) or region proposals to the object queries themselves, demonstrating that aligning the decision-making embeddings is a highly effective strategy for multimodal detection.