RAPTOR+: A Visually Grounded Vision-Language Framework to Improve Clinical Trust and Auditability in Automated Cancer Referral Processing

Technical Summary: RAPTOR+

Problem Statement
Urgent suspected colorectal cancer (CRC) referrals often rely on semi-structured clinical documents (scanned forms, faxes, handwritten notes) that create operational bottlenecks due to the need for manual review and transcription. While previous systems like the original RAPTOR utilized Large Language Models (LLMs) for structured extraction, they relied on a decoupled Optical Character Recognition (OCR) stage. This architecture is vulnerable to handwriting, layout variations, and poor scan quality. More critically, it lacks visual grounding, meaning the system cannot link extracted data back to specific regions in the source document. This absence of evidence localisation prevents clinician validation, introduces risks of "ungrounded extractions" (hallucinated values), and undermines the auditability required for safety-critical clinical workflows.

Methodology
The authors introduce RAPTOR+, a multimodal, end-to-end vision-language framework designed to perform structured extraction with visual evidence localisation.

• Dataset: The study utilizes a clinically curated dataset of 223 urgent CRC referral forms, reflecting real-world variability including heterogeneous layouts, free-text, handwritten annotations, and editorial marks. The data is split into 176 training forms and 47 held-out test forms. Ground truth includes structured field values and precise bounding-box annotations for visual evidence.
• System Architecture: RAPTOR+ replaces the OCR-first pipeline with a multimodal transformer-based encoder-decoder. It processes visual features via a Vision Transformer and aligns them with linguistic tokens using cross-attention. The output is a structured JSON object where every field value is accompanied by a bounding box indicating the source evidence.
• Models Evaluated: The study benchmarks a diverse set of Vision-Language Models (VLMs):
  • Commercial Zero-Shot: Gemini 2.5 Flash and Claude Opus 4.6.
  • Open-Weight Zero-Shot: Qwen3-VL variants (2B, 4B, 8B, 32B parameters).
  • Open-Weight Fine-Tuned: Qwen3-VL variants (2B, 4B, 8B) trained on paired document images and structured JSON targets containing values and bounding-box coordinates.
• Evaluation Protocol: Performance is measured using a multi-criteria protocol separating semantic extraction from evidence localisation:
  • Reading Accuracy (NM): Semantic correctness of the extracted value.
  • Evidence Precision (EP): The proportion of predicted bounding boxes that overlap with any ground-truth region.
  • Strict Safety (Joint IoU): The proportion of fields where both the value is correct and the bounding box has an Intersection-over-Union (IoU) > 0.5 with the ground truth.
  • Audit Precision (Joint IoP): A more permissive metric (IoP > 0.5) assessing utility for human-in-the-loop review.

Key Contributions

1. Grounding-Aware Evaluation Framework: The authors propose a novel evaluation method that jointly quantifies structured extraction accuracy and visual evidence localisation, addressing the "grounding gap" where models read well but point poorly.
2. Characterization of Failure Modes: The study identifies specific clinically relevant failure modes, particularly "confident mislocalisation" in zero-shot models (e.g., Claude Opus 4.6), where models generate coordinates for most fields but with near-zero accuracy, creating a false sense of reliability.
3. Analysis of Adaptation: The work analyzes how model scale and task-specific fine-tuning affect grounding reliability, demonstrating that fine-tuning acts as a structural switch for enabling evidence-grounded extraction.

Results

• Zero-Shot Limitations: While zero-shot models demonstrated strong semantic reading capabilities (e.g., Claude Opus 4.6 at 95.2%, Qwen3-VL-32B at 95.3%), they exhibited severe grounding failures. Gemini 2.5 Flash achieved 92.6% reading accuracy but only 1.2% Strict Safety. Claude Opus 4.6 showed even worse grounding (0.2% Strict Safety), frequently pointing to incorrect regions despite high reading accuracy.
• Impact of Fine-Tuning: Fine-tuning significantly improved both reading accuracy and grounding. The fine-tuned Qwen3-VL-8B achieved 96.1% Reading Accuracy and 60.6% Strict Safety, a substantial improvement over its zero-shot counterpart (88.4% accuracy, 6.2% safety) and all zero-shot baselines.
• Evidence Precision: Fine-tuning increased Evidence Precision from 70.1% (zero-shot Qwen3-VL-8B) to 88.5%. Smaller models (2B/4B) that produced no bounding boxes in zero-shot mode became functional after fine-tuning, achieving high coverage (82.8% and 87.6%).
• Trade-offs: The study found no trade-off between extraction quality and evidence reliability; fine-tuning improved both simultaneously. Fine-tuned models also learned to abstain (output null) when uncertain, prioritizing reliable pointers over forced extraction.

Significance and Claims
The paper claims that task-specific fine-tuning is essential for reliable, auditable clinical document understanding. The results demonstrate that semantic extraction capability alone is insufficient for safety-critical deployment; without verifiable evidence grounding, automated systems cannot support clinician trust or accountability.

RAPTOR+ enables extracted referral decisions to be linked directly to visual evidence, supporting a human-in-the-loop workflow where clinicians can rapidly verify decisions against source regions. The authors conclude that while zero-shot models may serve as audit assistants to direct attention, they are currently unsuitable for fully automated verification. The proposed framework offers a path toward safer, more efficient cancer referral triage by ensuring that automation is paired with verifiable evidence.