LMOD+: A Comprehensive Multimodal Dataset and Benchmark for Developing and Evaluating Multimodal Large Language Models in Ophthalmology

This paper introduces LMOD+, a large-scale multimodal ophthalmology benchmark dataset and evaluation framework featuring 32,633 annotated instances across 12 conditions and 5 imaging modalities, designed to advance and systematically assess the capabilities of multimodal large language models in vision-threatening disease diagnosis, staging, and bias detection.

The Gist

Imagine the world of eye care (ophthalmology) as a vast, complex library. For years, the librarians (doctors) have been overwhelmed. There are too many books (patients), not enough staff, and many people are waiting too long to get their eyes checked, leading to preventable blindness.

Enter Artificial Intelligence (AI). Specifically, a new generation of "super-readers" called Multimodal Large Language Models (MLLMs). Think of these not just as text-chatting robots, but as brilliant interns who can read a patient's history and look at a picture of their eye simultaneously to make a diagnosis.

However, there's a problem: How do we test if these interns are actually good?

The Problem: The Wrong Test

Previously, researchers tried to test these AI interns using old, simple quizzes designed for much older, dumber computers.

• The Old Way: Show a picture of an eye, and the computer just had to say "Yes" or "No" (e.g., "Is there glaucoma?"). It was like a multiple-choice test where you just circle a letter.
• The New Reality: Modern AI is like a human doctor. It needs to look at the picture, explain why it thinks there's a disease, describe the anatomy, and even guess the patient's age or gender based on the eye. The old "Yes/No" tests were too simple and didn't measure the AI's true intelligence or its ability to explain its reasoning.

The Solution: LMOD+ (The Ultimate Eye-Exam Simulator)

The authors of this paper created LMOD+, which is essentially a massive, high-tech "simulator" or "training ground" for these AI doctors.

Here is what makes LMOD+ special, using some everyday analogies:

1. A Massive, Diverse Library (The Dataset)
Instead of just a few photos, LMOD+ contains over 32,000 eye images.

• The Variety: It's not just one type of photo. It includes:
  • Color Fundus Photos (CFP): Like a standard photo of the back of the eye (the most common type).
  • OCT Scans: Like a high-tech "slice" of the eye, showing layers like a loaf of bread.
  • Surgical Scenes: Videos and photos of actual eye surgeries.
  • Lens Photos: Close-ups of the eye's front lens (for cataracts).
• The Annotations: Every image is tagged with detailed notes: "Here is the optic nerve," "Here is a tumor," "This patient is 60 years old." It's like having a textbook where every diagram is labeled by a team of expert doctors.

2. The Four-Part Exam (The Tasks)
LMOD+ doesn't just ask the AI to guess a disease. It puts the AI through a rigorous four-part board exam:

• Anatomy 101: "Point to the optic nerve and the retina." (Can the AI see the parts?)
• Diagnosis: "Does this patient have diabetic retinopathy? Explain why." (Can it spot the disease and talk about it?)
• Staging: "If they have the disease, how bad is it? Is it Stage 1 or Stage 4?" (Can it judge severity?)
• Demographics: "Based on this eye, guess the patient's age and gender." (This tests if the AI is biased or if it can pick up subtle clues).

3. The "Zero-Shot" Challenge
The most difficult part of this test is that the AI is asked to take the exam without any prior studying on these specific questions. This is called "zero-shot."

• Analogy: Imagine handing a medical student a brand new, complex eye scan and asking them to diagnose it before they've ever seen a similar case in class. Most AI models failed this test, performing barely better than random guessing.

The Results: The AI is Smart, But Not a Doctor Yet

The researchers tested 24 different AI models (including famous ones like GPT-4o, Qwen, and InternVL) on this new exam.

• The Good News: Some models, like Qwen and InternVL, showed promise. They could screen for diseases with about 58% accuracy without any special training. That's better than a coin flip, but not good enough to trust with a human life yet.
• The Bad News:
  • The "Hallucination" Problem: When the AI didn't know the answer, it sometimes made things up. It might confidently say, "I see a tumor," when there was none, or it might get stuck in a loop repeating the same word forever.
  • The "Medical Knowledge" Gap: Some models that were specifically trained on medical texts (like "LLaVA-Med") actually performed worse than general models. It turns out, just reading medical books isn't enough; you need to learn how to look at medical pictures.
  • The "Staging" Struggle: While the AI could sometimes guess "Yes/No" for a disease, it was terrible at figuring out how bad the disease was. This is crucial because treatment depends on severity.

Why This Matters

Think of LMOD+ as a standardized driving test for AI cars. Before, we were testing self-driving cars on empty parking lots (simple datasets). Now, LMOD+ throws them into a busy city with rain, pedestrians, and complex traffic signs (real-world ophthalmology).

The Takeaway:
Current AI is like a very smart student who has read all the textbooks but has never actually practiced on real patients. It knows the theory but struggles with the messy reality of a real eye exam.

The authors are releasing this "exam" and the "study materials" to the public. Their goal is to help developers build better AI doctors. If we can train these models to pass the LMOD+ exam, we could eventually have AI assistants that help doctors in remote villages diagnose eye diseases instantly, saving millions of people from going blind.

In short: We built a better test to see if AI is ready to be a doctor. The answer is: "Not quite yet, but we now have the tools to teach them how to get there."

Technical Summary

1. Problem Statement

The rising global burden of vision-threatening eye diseases is exacerbated by workforce shortages and diagnostic delays. While Artificial Intelligence (AI), specifically Multimodal Large Language Models (MLLMs), offers promise for automated diagnosis and clinical documentation, their advancement in ophthalmology is hindered by a lack of unified, comprehensive benchmarks.

• Limitations of Existing Benchmarks: Most existing datasets were designed for traditional Convolutional Neural Networks (CNNs) or specific tasks (e.g., single disease classification). They typically output fixed disease labels rather than free-text responses, making them unsuitable for evaluating the generative and reasoning capabilities of modern MLLMs.
• Gap in Multimodality: Text-only benchmarks fail to capture the critical visual patterns required for ophthalmic diagnosis (e.g., retinal imaging), while existing multimodal benchmarks often lack diversity in imaging modalities or disease coverage.

2. Methodology

The authors present LMOD+, a significantly expanded and unified benchmark derived from a systematic data curation pipeline.

A. Dataset Construction (LMOD+)

• Scale: The dataset contains 32,633 instances, a ~50% increase from the preliminary LMOD benchmark (21,933 instances).
• Modalities: It integrates 5 distinct imaging modalities:
  1. Color Fundus Photography (CFP) - 43.2%
  2. Scanning Laser Ophthalmoscopy (SLO) - 30.6%
  3. Optical Coherence Tomography (OCT) - 11.8%
  4. Lens Photography (LP) - 7.5%
  5. Surgical Scenes (SS) - 6.9%
• Annotations: Includes multi-granular annotations covering 12 common ophthalmic conditions, anatomical structures, and patient demographics (age, sex).
• Curation Pipeline: The authors developed a unified pipeline to repurpose existing heterogeneous datasets (e.g., REFUGE, IDRiD, OIMHS) into MLLM-compatible formats. This involves:
  • Extracting raw annotations (bounding boxes, segmentation masks).
  • Converting them into structured prompt-response pairs.
  • Generating free-text explanations and justifications alongside diagnostic labels.

B. Evaluation Tasks

The benchmark defines four primary tasks to test MLLM capabilities:

1. Anatomical Structure Recognition: Identifying and localizing specific ocular components (e.g., optic disc, macular hole) across different modalities.
2. Ophthalmic Disease Diagnosis:
   • Binary: Detecting the presence/absence of specific conditions (12 tasks, e.g., Diabetic Retinopathy, AMD).
   • Multi-class: Distinguishing between multiple diseases (e.g., Cataract vs. Glaucoma vs. DR vs. Normal).
3. Disease Staging Assessment: Classifying severity levels (e.g., ICDR and SDRG grading for Diabetic Retinopathy; Macular Hole staging).
4. Demographic Prediction: Inferring patient age and sex from images to assess potential model bias.

C. Experimental Setup

• Models Evaluated: 24 state-of-the-art MLLMs were tested, including general-domain models (GPT-4o, InternVL, Qwen, DeepSeek, LLaVA, Yi) and medical-specific models (LLaVA-Med, Med-Flamingo).
• Setting: Primarily zero-shot evaluation to test generalization without task-specific fine-tuning.
• Metrics: Accuracy, Precision, Recall, F1 Score, and a custom Hallucination Resistance (HR) metric to measure the frequency of invalid or hallucinated outputs.

3. Key Contributions

1. LMOD+ Dataset: The largest and most comprehensive multimodal ophthalmology benchmark to date, covering 5 imaging modalities, 12 conditions, and free-text generation tasks.
2. Unified Curation Pipeline: A reproducible framework for transforming legacy, label-based datasets into generative MLLM benchmarks.
3. Systematic Benchmarking: The first large-scale evaluation of 24 diverse MLLMs across anatomical recognition, diagnosis, staging, and bias assessment in ophthalmology.
4. Error Taxonomy: A detailed analysis categorizing MLLM failures into five types: Text Generation Failures, Medical Knowledge Errors, Inconsistent Reasoning, Misinterpreted Visual Features, and Absent Visual Processing.

4. Key Results

• Overall Performance: Current MLLMs perform suboptimally in ophthalmology.
  • Anatomical Recognition: Average F1 score was 0.2113. GPT-4o led with 0.5761, followed closely by InternVL-1.5-4B (0.5716). Medical-specific models (LLaVA-Med, Med-Flamingo) performed poorly, often below random baselines.
  • Disease Diagnosis: Binary diagnosis accuracy hovered near random chance (50%). The best models (Qwen-7B, InternVL-2.5-8B) achieved 58% accuracy in zero-shot settings. Multi-class diagnosis was even lower (28%).
  • Disease Staging: Performance was critically low (best accuracy ~26.67%), indicating models struggle to distinguish subtle severity gradations.
  • Demographics: Models failed to infer age or sex better than random chance, suggesting they do not rely on demographic bias for decisions (unlike some CNNs).
• Modality Variance: Models performed best on CFP and LP but struggled significantly with Surgical Scenes (SS) and OCT, where domain-specific anatomical knowledge is crucial.
• Medical vs. General Models: Contrary to expectations, general-domain models (e.g., InternVL, Qwen) generally outperformed medical-specific models (e.g., LLaVA-Med). This suggests that pre-training on general data provides better visual grounding, while medical fine-tuning on limited text-heavy data (PubMed) is insufficient for visual tasks.
• Error Analysis: Misinterpreted Visual Features accounted for 50% of all errors, followed by Inconsistent Reasoning (21%). This highlights that models often "see" the image but fail to interpret the clinical significance correctly.
• Prompting Strategies: Techniques like Chain-of-Thought (CoT) and Few-Shot prompting showed mixed results. While CoT improved recall, it severely degraded format compliance (Hallucination Resistance dropped), making it impractical for structured clinical outputs without post-processing.

5. Significance and Future Directions

• Clinical Reality Check: The study demonstrates a substantial gap between general-domain MLLMs and the specialized requirements of ophthalmology. Current models are not yet ready for reliable clinical deployment in zero-shot settings.
• Resource for Community: By releasing the dataset, pipeline, and a dynamic leaderboard, the authors enable the community to develop domain-specific models and evaluate them rigorously.
• Future Work: The authors emphasize the need for:
  • Domain-specific fine-tuning of MLLMs on high-quality ophthalmic data.
  • Development of datasets for end-to-end patient management (e.g., treatment planning), which currently lacks due to privacy constraints.
  • Improving model architectures to better align visual features with clinical reasoning.

In conclusion, LMOD+ serves as a critical stepping stone, providing the necessary infrastructure to transition MLLMs from general-purpose tools to specialized, reliable assistants in ophthalmic care.