PRS-Med: Position Reasoning Segmentation in Medical Imaging

1. Problem Statement

Current medical image segmentation and multimodal large language models (MLLMs) face two critical limitations in clinical settings:

• Lack of Spatial Reasoning: Existing prompt-based segmentation methods (e.g., MedSAM) rely on explicit prompts like bounding boxes or points. They struggle to interpret complex, free-form natural language queries regarding spatial relationships (e.g., "Find the tumor in the upper-left quadrant").
• Brittleness of General MLLMs: While general-domain MLLMs (like LLaVA) show promise, they lack the specialized training to understand medical anatomy and often fail at precise localization or "pixel-perfect" coordinate regression, which is prone to hallucination and lacks the structured reliability required for diagnosis.
• Data Scarcity: There is a significant lack of large-scale, expert-validated datasets that pair medical images with spatially grounded question-answer pairs necessary to train models for position reasoning.

2. Methodology

The authors propose PRS-Med, a unified framework that integrates a medical Vision-Language Model (VLM) with a segmentation decoder to perform "Position Reasoning Segmentation."

A. The PosMed Dataset

To address the data gap, the authors introduced PosMed, a large-scale dataset comprising 116,000 expert-validated question-answer pairs across six imaging modalities (Ultrasound, MRI, CT, X-ray, Endoscopy, and Skin).

• Construction Pipeline: A semi-automated pipeline uses GPT-4 to generate diverse question templates based on tumor/anatomy names and positional information.
• Anatomical Action Zones (AAZ): Instead of complex coordinate regression, the system divides images into discrete zones (Top-Left, Top-Right, Bottom-Left, Bottom-Right, and Center). This "categorical spatial grounding" mimics radiologists' search patterns and reduces spatial hallucination.
• Expert Validation: Three board-certified radiologists validated the dataset, ensuring medical accuracy, clinical plausibility, and correct positional reasoning.

B. PRS-Med Architecture

The framework consists of three primary modules:

1. Vision Backbone: Utilizes TinySAM (based on TinyViT) to extract efficient, pixel-level features from medical images. This lightweight encoder is chosen for its balance of performance and computational efficiency.
2. Multimodal-LLM (MLLM): Employs LLaVA-Med as the language backbone. To adapt it for position reasoning without catastrophic forgetting, the authors use LoRA (Low-Rank Adaptation) to fine-tune the query, key, value, and output projection matrices. The MLLM processes the image and text prompt to generate reasoning embeddings ($z_{emb}$).
3. Prompt Mask Decoder: A novel fusion module that combines image features ($z_{image}$) and MLLM embeddings ($z_{emb}$) via a Cross-Attention mechanism. This allows the model to dynamically align spatial features with medical vocabulary. The fused representation is then passed through transposed convolutions to generate the final segmentation mask.

C. Objective Function

The model is trained with a multi-task loss function:
$$L = \lambda_{seg}L_{seg} + \lambda_{txt}L_{text}$$

• $L_{seg}$: Combines Binary Cross-Entropy and Dice Loss for segmentation accuracy.
• $L_{text}$: Uses Categorical Cross-Entropy for the textual reasoning output.
  This dual-loss approach creates a feedback loop where segmentation informs reasoning and vice versa.

3. Key Contributions

1. PosMed Dataset: The release of the first large-scale, multi-modal dataset specifically designed for position reasoning segmentation, featuring 116k expert-validated pairs across six modalities.
2. PRS-Med Framework: A unified model that performs spatially-aware tumor segmentation using implicit natural language prompts, mimicking the "search patterns" of radiologists.
3. Categorical Spatial Grounding: A shift from brittle pixel-coordinate regression to a robust, zone-based reasoning approach (AAZ) that ensures clinical actionability and interpretability.
4. Open Source: Full release of the dataset pipeline, model weights, and codebase to facilitate future research in spatially-aware medical AI.

4. Experimental Results

Extensive experiments were conducted on six medical imaging datasets, comparing PRS-Med against state-of-the-art (SOTA) methods like MedSAM, BiomedParse, LISA, and various medical MLLMs.

• Segmentation Performance: PRS-Med achieved significant improvements in mean Dice (mDice) and mean IoU (mIoU). Notably, it showed a +31.2% mDice improvement on Lung CT scans and +13.6% on Brain MRI compared to the second-best method.
• Position Reasoning: In terms of text generation quality (ROUGE) and reasoning accuracy (ACC), PRS-Med outperformed all baselines. For example, on the Polyp dataset, it improved accuracy by +16.79% over the previous best.
• Qualitative Analysis: Visualizations demonstrate that PRS-Med successfully captures small lesions and anatomical structures that generalist models miss, providing high-confidence interpretability layers.
• Ablation Studies:
  • Vision Backbone: TinySAM (pretrained) outperformed SAM-Med variants in efficiency and accuracy.
  • MLLM Backbone: LLaVA-Med significantly outperformed LLaVA-1.5/1.6, highlighting the importance of domain-specific pretraining.
  • LoRA Configuration: Targeting all projection matrices (q, k, v, o) yielded the best results, confirming the need for comprehensive cross-modal alignment.

5. Significance

PRS-Med represents a paradigm shift in medical AI by prioritizing functional reliability over technical complexity.

• Clinical Relevance: By mimicking the structured language of radiologists (using zones rather than coordinates), the model produces outputs that are directly actionable and interpretable in a clinical workflow.
• Scalability: The proposed dataset pipeline allows for the generation of massive, diverse datasets across multiple modalities, overcoming the bottleneck of manual spatial annotation.
• Future Impact: This work lays the foundation for the next generation of intelligent medical assistants capable of not just identifying pathologies, but reasoning about their location and context, ultimately aiding in early diagnosis and automated screening systems.