GAZE: Grounded Agentic Zero-shot Evaluation with Viewer-Level Tools and Literature Retrieval on Rare Brain MRI

1. Problem Statement

Current Vision-Language Models (VLMs) typically operate in a single-forward-pass manner (one image in, one text response out). This contrasts sharply with the clinical workflow of radiologists, who perform iterative examinations (zooming, adjusting contrast, comparing sides) and cross-reference medical literature before committing to a diagnosis.

This gap is particularly critical for rare neurological conditions (e.g., on the NOVA benchmark), where diagnostic uncertainty is high, and errors have severe clinical consequences. Existing agentic systems for radiology often rely on modality-specific AI tools (like segmentation models) rather than the simple viewer controls radiologists use daily, and they rarely integrate literature retrieval in a way that allows for joint evaluation of diagnosis, lesion localization, and captioning.

2. Methodology: The GAZE Framework

The authors introduce GAZE, an agentic framework designed to bridge the gap between single-pass VLMs and iterative clinical reasoning.

Core Architecture

• Agentic Loop: The VLM decides, turn-by-turn, whether to call a tool, which tool to call, or to stop and generate a final report.
• Decoupled Design: Task definitions (prompts, schemas, validators) are separated from the agent machinery (tool registry, logging, error recovery), making the framework reusable across different medical VLM tasks.
• Structured Outputs: Every model response is validated against a strict JSON schema. If the output is malformed, the framework injects a "nudge" message to correct it, ensuring auditability and robustness.

Tool Set

GAZE exposes two categories of tools via a unified function-calling interface:

1. Viewer-Level Visual Tools (7 tools): These mimic standard radiology workstation actions without requiring new trained models:
   • Zoom/Crop: Magnify or extract regions of interest.
   • Window/Level & Contrast: Adjust intensity ranges to enhance tissue contrast.
   • Edge Detection & Symmetry Diff: Highlight boundaries and asymmetries.
   • Rotate/Flip/Reset: Reorient or restore the image view.
   • Note: The framework tracks coordinate system changes to ensure bounding box scoring remains accurate.
2. Retrieval Tools (2 tools): Backed by the U.S. National Library of Medicine (NLM):
   • search_web: Queries PubMed for guidelines, reviews, and case reports (prioritized by evidence type).
   • search_images: Queries Open-i to retrieve radiological images and captions for visual comparison.

Evaluation Protocol

The framework is evaluated on the NOVA benchmark (906 brain MRI cases, 281 rare conditions).

• Joint Protocol: Unlike previous baselines that evaluate tasks independently (often using ground-truth captions as "oracle" inputs), GAZE evaluates captioning, diagnosis, and localization simultaneously from the image alone.
• Metrics:
  • Localization: Mean Average Precision (mAP) at IoU thresholds (0.3, 0.5, and 0.5–0.95).
  • Captioning: METEOR and Clinical F1.
  • Diagnosis: Top-1 and Top-5 accuracy (using GPT-4o as a semantic judge).

3. Key Contributions

1. Reusable Agentic Framework: GAZE unifies viewer-level visual tools, NLM-backed retrieval, schema-validated multi-turn outputs, and auditable tool-call traces into a single registry.
2. Joint Multi-Task Evaluation: Introduces a rigorous protocol scoring captioning, diagnosis, and localization jointly, revealing that framework design (prompting, schema constraints) is itself a significant experimental variable.
3. Discovery of Model-Dependent Trade-offs: Identifies a critical failure mode where retrieval improves diagnosis but degrades localization in weaker models, arguing against single-task evaluation for retrieval-augmented medical AI.

4. Key Results

Experiments were conducted on Gemini 2.0 Flash, Gemini 3 Flash, and Qwen3.5 variants (MoE models).

• Framework Impact (Zero-Shot): Even without tools, the GAZE framework (structured prompting + schema validation) improved the published Gemini 2.0 Flash baseline on NOVA from 20.2 to 29.4 mAP@0.3 for localization. This proves that better scaffolding alone yields significant gains.
• Tool Effectiveness:
  • Gemini 3 Flash achieved the best overall performance: 58.2 mAP@0.3 and 34.9% Top-1 diagnostic accuracy.
  • Qwen3.5-397B-A17B nearly matched Gemini 3 Flash (57.6 mAP@0.3).
  • Rare Disease Gains: Tool usage disproportionately helped rare pathologies. For conditions with ≤3 examples, the fraction of cases with IoU > 0.3 rose from 17% to 58% (a 241% relative gain), compared to a 172% gain for common conditions.
• The Retrieval Trade-off (Critical Finding):
  • Strong Models (Gemini 3 Flash): Successfully integrated textual retrieval with visual evidence, showing modest gains in both diagnosis and localization.
  • Weaker Models (Qwen3.5-122B-A10B): Exhibited a negative trade-off. Adding search improved diagnosis (Top-1: 20.4% → 23.3%) but severely degraded localization (mAP@0.3: 39.1 → 20.9, a ~47% drop).
  • Mechanism: Weaker models appeared to shift bounding boxes toward spatial patterns described in retrieved text (e.g., abstracts describing subcortical patterns) rather than the actual lesion visible in the image, effectively "hallucinating" the location based on text.

5. Significance and Implications

• Clinical Realism: GAZE demonstrates that agentic workflows (iterative tool use) are essential for handling rare diseases where single-pass inference fails.
• Evaluation Necessity: The paper argues that joint evaluation is mandatory. Evaluating retrieval systems solely on diagnostic accuracy can mask catastrophic failures in spatial grounding (localization), which is vital for clinical utility.
• Auditability: The framework's trace logging and schema validation align with regulatory requirements (e.g., EU AI Act) for medical AI, providing a clear audit trail of how a diagnosis was reached.
• Future Direction: The results suggest that future medical VLMs must be capable of resolving conflicts between textual evidence and visual data, and that benchmarking must account for the "engagement" quality of the model with tools, not just the frequency of tool calls.

In conclusion, GAZE establishes that combining viewer-level tools with literature retrieval in a structured, agentic loop significantly outperforms single-pass inference for rare brain MRI analysis, provided the model is capable of integrating these modalities without sacrificing spatial accuracy.