💻 computer science

Comparative Performance of General-Purpose Large Language Models and a Specialized Clinical AI Tool on OKAP-Style Ophthalmology Questions

This study evaluates five large language models on OKAP-style ophthalmology questions, revealing that general-purpose models, particularly Gemini-Pro-3, outperformed the specialized clinical AI tool OpenEvidence in accuracy and calibration, while highlighting significant performance heterogeneity and shared reasoning failures across all systems.

Original authors: Alon Moore Galindo, Amit Ginsberg, Razan Saadi, Tomer Kerman, David Schwartzman, Anat Loewenstein, Igal Leibovitch

Published 2026-07-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Alon Moore Galindo, Amit Ginsberg, Razan Saadi, Tomer Kerman, David Schwartzman, Anat Loewenstein, Igal Leibovitch

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a high-stakes medical exam for eye doctors, known as the OKAP. Now, imagine five different "super-smart" computer programs trying to take this test. The goal of this study was to see which computer program is the best at answering these tricky eye-doctor questions and, just as importantly, which one knows when it might be wrong.

Here is the breakdown of what happened, using some everyday analogies:

The Contestants

The researchers lined up five different AI "students":

OpenEvidence: A specialized student who only studies medical textbooks and has a direct line to famous medical journals. Think of this as a medical librarian who has memorized the library but doesn't have a general brain for everything else.
Gemini-Pro-3, Claude-Opus-4.6, ChatGPT-5.4-Pro, and DeepSeek-v3.2: These are "general-purpose" students. They read everything from cooking recipes to physics papers. Think of them as polymaths—people who know a little bit about a lot of things.

The Test

The test consisted of 659 multiple-choice questions about eye health. These weren't easy trivia; they ranged from simple facts (like "What is the name of this eye part?") to complex scenarios requiring deep reasoning (like "If a patient has symptom X and history Y, what is the best treatment?").

The Results: Who Won?

The results were surprising to some experts:

The Champion: Gemini-Pro-3 (the general-purpose student) got the highest score, answering 95.8% of the questions correctly. It was so good that it beat everyone else by a significant margin.
The Runner-Up: Claude-Opus-4.6 (another general-purpose student) came in second.
The Specialist: OpenEvidence (the medical librarian) came in third. It did very well (88%), but it did not beat the general-purpose students. In fact, the general students outperformed the specialist.
The Rest: ChatGPT and DeepSeek followed, with DeepSeek scoring the lowest (73.7%).

The Big Takeaway: Having a tool built specifically for medicine didn't guarantee it would be the smartest at answering medical questions. The general-purpose AI was actually better at this specific test.

The "Confidence" Check

The researchers also asked the AI: "How sure are you that your answer is right?" (On a scale of 0 to 100). This is like asking a student to raise their hand if they are 100% confident.

The Best "Self-Aware" Student: Gemini-Pro-3 was the most honest. When it said it was confident, it was usually right. Its confidence matched its actual performance perfectly.
The "Over-Confident" vs. "Under-Confident": Some other models were good at knowing when they were right (discrimination), but not as good at matching their confidence level to their actual accuracy (calibration).
The Warning Sign: One model, DeepSeek, was terrible at this. It often said it was confident when it was actually wrong, or unsure when it was right. This is dangerous because if a doctor trusts a confident but wrong answer, it could lead to mistakes.

Where Did They All Fail?

The researchers looked at the 9 questions that every single AI got wrong. They tried to figure out why.

What they got right: They understood the words and the basic topic.
Where they broke down: They failed at complex reasoning and checking their sources.
- Analogy: Imagine a student who can read a question perfectly but gets lost when they have to connect three different facts to solve a puzzle. Or, they might guess an answer without actually checking if the rulebook supports it.
- The AI struggled most with questions that required "thinking steps" rather than just remembering a fact.

The Limitations (The Fine Print)

The authors were careful to say what this study didn't prove:

No Pictures: The test was only text. In real eye care, doctors look at photos of eyes. None of these AIs were tested on looking at images during this study.
Not Real Patients: These were exam questions, not real-life patients with messy, complicated histories.
One-Time Test: The AI took the test once. If you asked it again, it might give a different answer.

Summary

In simple terms: General-purpose AI is currently beating specialized medical AI at answering eye-doctor exam questions. However, the AI still struggles with the hardest, most complex reasoning tasks. Also, just because an AI gives a high score doesn't mean it's safe to use blindly; you need to check if it knows when it's unsure. The study suggests that for now, we should be careful about using these tools for real medical decisions until we know more about how they handle real-world complexity.

Technical Summary: Comparative Performance of General-Purpose LLMs and Specialized Clinical AI on OKAP-Style Ophthalmology Questions

Problem Statement
While Large Language Models (LLMs) are increasingly integrated into healthcare, there is a critical gap in understanding how general-purpose LLMs compare to specialized, clinically oriented AI platforms within specific medical subspecialties. In ophthalmology, OpenEvidence has emerged as a specialized tool designed for clinical use with partnerships from major medical organizations (e.g., NEJM Group, AAO). However, recent benchmarking suggests general-purpose models may outperform specialized ones in general medical knowledge. The authors sought to address the lack of direct comparative evidence between OpenEvidence and state-of-the-art general-purpose LLMs specifically on ophthalmic knowledge, evaluating not just raw accuracy but also reliability metrics such as calibration and confidence-based discrimination.

Methodology
The study employed a comparative design evaluating five LLM-based systems on a dataset of 659 unique, text-based multiple-choice questions (MCQs) derived from the Ophthalmic Knowledge Assessment Program (OKAP) and structured to reflect the OphthoQuestions bank. The models evaluated were:

OpenEvidence (March 2026 version): A specialized clinical AI platform.
ChatGPT-5.4-Pro: General-purpose LLM.
Gemini-Pro-3: General-purpose LLM.
Claude-Opus-4.6: General-purpose LLM.
DeepSeek-v3.2: General-purpose LLM.

Data and Evaluation Protocol:

Dataset: 659 questions manually categorized by subspecialty (13 domains), clinical domain, question type, Bloom's taxonomy level, and difficulty (Easy to Very Hard).
Access: Models were accessed via official APIs (Single Run) except OpenEvidence, which was accessed manually via its web interface due to the lack of a public API at the time.
Prompts: Standardized prompts required definitive answers (A–D). Most models were instructed to provide a confidence score (0–100); OpenEvidence did not provide confidence scores.
Statistical Analysis: Primary outcomes included binary accuracy. Multivariable Generalized Estimating Equations (GEE) logistic regression were used to assess independent effects of model type, difficulty, and taxonomy on correct responses, with Holm-Bonferroni correction for multiple comparisons. Secondary analyses included Cochran's Q and McNemar's tests for pairwise comparisons, ROC analysis for confidence-based discrimination, and Expected Calibration Error (ECE) and Brier scores for calibration.
Error Analysis: A post hoc descriptive analysis was conducted on 9 questions answered incorrectly by all models, involving structured re-evaluation to identify failure patterns.

Key Results

Overall Accuracy: Performance varied significantly across models. Gemini-Pro-3 achieved the highest accuracy at 95.8% (95% CI, 93.9–97.0). The ranking was as follows:
1. Gemini-Pro-3 (95.8%)
2. Claude-Opus-4.6 (89.2%)
3. OpenEvidence (88.0%)
4. ChatGPT-5.4-Pro (84.8%)
5. DeepSeek-v3.2 (73.7%)
  In multivariable analysis, all other models had significantly lower adjusted odds of a correct response compared to Gemini-Pro-3 ( $P < 0.001$ ). Notably, general-purpose models (Gemini-Pro-3 and Claude-Opus-4.6) outperformed the specialized clinical platform (OpenEvidence).
Difficulty and Complexity: Accuracy declined significantly with increasing question difficulty. Higher-order Bloom's taxonomy questions and logical questions showed lower accuracy compared to lower-order and informative questions, though the latter did not remain statistically significant after Holm correction.
Reliability Metrics:
- Calibration: Gemini-Pro-3 demonstrated the best calibration (lowest Brier score: 0.038; ECE: 0.008). DeepSeek-v3.2 showed the poorest calibration.
- Confidence-Based Discrimination: Claude-Opus-4.6 and ChatGPT-5.4-Pro showed the strongest ability to distinguish correct from incorrect answers based on self-reported confidence (highest AUCs: 0.779 and 0.775, respectively). DeepSeek-v3.2 showed weak discrimination (AUC: 0.607).
- Thresholding: At a $\ge 90\%$ confidence threshold, Claude-Opus-4.6 reached 99.6% accuracy (with 34.3% coverage), while Gemini-Pro-3 maintained 97.0% accuracy with 96.4% coverage.
Subspecialty Performance: No subspecialty differences remained statistically significant after Holm correction, though descriptive trends suggested Retina/Vitreous was more challenging and General Medicine was better performing relative to Fundamentals.
Failure Patterns: The post hoc analysis of the 9 universally missed questions revealed that shared failures were concentrated in logical reasoning (64.4% success in re-evaluation) and reference validity (53.3% success), rather than basic question semantics or topic comprehension.

Key Contributions and Significance
The paper claims several significant findings:

Specialization Does Not Guarantee Superiority: A specialized clinical AI platform (OpenEvidence) did not outperform leading general-purpose LLMs in ophthalmic knowledge; in fact, Gemini-Pro-3 and Claude-Opus-4.6 surpassed it in overall accuracy.
Heterogeneity of Performance: There is substantial variation in accuracy and reliability metrics across contemporary LLMs, necessitating model-specific validation rather than assuming uniform capability.
Reliability Beyond Accuracy: The study highlights that accuracy alone is insufficient. The highest-accuracy model (Gemini-Pro-3) had the best calibration, while other models (Claude-Opus-4.6, ChatGPT-5.4-Pro) excelled at confidence-based discrimination. This suggests that self-reported confidence can serve as a reliability filter, particularly in well-calibrated models.
Nature of Errors: Current LLMs perform well on factual recall but struggle with higher-order synthesis and reference validity, which are critical for real-world clinical decision-making.

Limitations and Scope
The authors explicitly note that the study is limited to text-based OKAP-style questions and does not evaluate multimodal image interpretation, which is central to ophthalmology. The evaluation was a single run per question, not capturing intra-model variance. Furthermore, OpenEvidence was accessed manually, and its lack of confidence scores limited direct comparison on calibration metrics. The authors conclude that while these models show promise, further studies using real-world clinical scenarios are required before deployment in clinical decision support.