Can Multimodal Large Language Models Understand OCT?

Technical Summary: OCT-Bench

Problem Statement
While Multimodal Large Language Models (MLLMs) have demonstrated significant potential in general vision tasks and medical image analysis, their ability to interpret Optical Coherence Tomography (OCT) images remains insufficiently evaluated. Existing benchmarks largely reduce OCT understanding to coarse-grained disease classification or isolated visual question answering (VQA). This approach fails to capture the complete cognitive workflow required in clinical practice, which progresses from low-level visual perception to high-level clinical reasoning. Consequently, current evaluations cannot distinguish whether a model's failure stems from inadequate visual perception, deficient medical knowledge, or unreliable clinical inference. Furthermore, there is a lack of systematic benchmarks dedicated specifically to the fine-grained, hierarchical nature of OCT interpretation.

Methodology
To address these limitations, the authors introduce OCT-Bench, a comprehensive benchmark designed to evaluate MLLMs on OCT image understanding through a clinically grounded, hierarchical framework.

• Dataset Construction: The benchmark is built from 4,137 high-quality OCT images sourced from seven public datasets (OCT5k, OIMHS, OCT-C8, AMD-SD, OCTDL, MMC-AMD, and GOALS). These images cover diverse diseases, anatomical structures, and lesion types.
• Question Generation: The authors generated 10,076 expert-verified multiple-choice questions (MCQs). The generation pipeline involved collecting medical knowledge from authoritative guidelines (e.g., AAO, CMA) and references, followed by task-driven generation using GPT-4o. A two-stage quality control process (automated checks by GPT-4o and manual review by domain experts) ensured medical correctness, visual answerability, and task alignment.
• Hierarchical Taxonomy: Following the clinical interpretation workflow, OCT-Bench establishes a three-dimensional capability taxonomy:
  1. Perception: Evaluates the extraction of visual evidence, including imaging attributes, retinal anatomy, lesion morphology, reflectivity, spatial relationships, and quantitative estimation.
  2. Cognition: Assesses the transformation of visual information into medical knowledge, focusing on anatomical understanding, pathological recognition, and linking findings to disease states.
  3. Reasoning: Tests the integration of imaging evidence with medical knowledge for clinical decision-making, including disease assessment, therapeutic planning, and prognostic management.
     This structure comprises 9 capability groups and 20 fine-grained tasks (L3-T01 to L3-T20).

Key Contributions

1. OCT-Bench Benchmark: The introduction of a large-scale, high-quality benchmark containing over 10,000 questions across 4,000+ images, moving beyond simple disease classification to enable comprehensive assessment of OCT understanding.
2. Clinical Taxonomy: The development of a hierarchical framework that decomposes OCT understanding into Perception, Cognition, and Reasoning. This allows for the precise identification of capability bottlenecks at different stages of the cognitive process.
3. Systematic Evaluation: A rigorous evaluation of 20 representative MLLMs, including proprietary models (e.g., GPT-5.4-mini, Gemini-2.5-flash), open-source general-purpose models (e.g., LLaVA, InternVL, Qwen2.5-VL), and medical-domain models (e.g., HealthGPT, MedGemma).

Experimental Results
The evaluation of 20 models on OCT-Bench reveals several critical findings:

• Overall Performance Gap: Current MLLMs remain far from reliable OCT understanding. The best-performing model, GPT-5.4-mini, achieved an overall accuracy of only 62.0%, meaning it failed nearly 38% of the questions.
• Performance Degradation: There is a marked decline in performance as tasks progress from Perception to Reasoning. The highest Perception score was 75.8%, while the highest Reasoning score was only 42.9%. Strong visual perception does not guarantee clinical reasoning capabilities.
• Limited Impact of Specialization and Scaling:
  • Domain Adaptation: Medical-domain models did not consistently outperform general-purpose models across all dimensions. While some medical models excelled in specific reasoning tasks, others underperformed compared to general models.
  • Model Scaling: Increasing model scale (e.g., from 7B to 32B parameters) yielded uneven gains. While scaling often improved Cognition scores, it frequently failed to resolve bottlenecks in Reasoning or even slightly decreased Perception performance in some cases.
• Fine-Grained Analysis:
  • Models performed well on basic modality and annotation recognition but struggled with fine-grained morphological descriptions and spatial orientation.
  • Anatomical cognition showed strong biases; models often recognized the retina well but failed significantly on the choroid or specific retinal layers.
  • Disease diagnosis accuracy varied widely by category, with models performing poorly on conditions requiring subtle differential diagnosis (e.g., AMD, Glaucoma) compared to more distinct pathologies.
  • Clinical reasoning tasks (treatment planning, follow-up adjustment) showed consistently low ceilings, indicating an inability to reliably synthesize evidence into coherent clinical strategies.

Significance
The paper posits that OCT-Bench provides a necessary foundation for identifying specific capability bottlenecks in MLLMs, moving beyond aggregate accuracy to reveal how errors propagate from visual perception to clinical reasoning. By demonstrating that current models, regardless of scale or domain specialization, struggle with reliable, image-grounded clinical reasoning, the authors highlight the need for future research focused on better visual grounding, image-knowledge alignment, and multi-step clinical inference. The benchmark serves as a tool to guide the development of MLLMs capable of supporting genuine ophthalmic decision-making rather than merely pattern matching.