A Fine-Grained Benchmark for Ophthalmic Foundation Models with Point-Based Classification and Retrieval
This paper introduces a fine-grained benchmark for ophthalmic foundation models that utilizes point-supervised classification and retrieval tasks to reveal that while these models outperform general-purpose counterparts, they still struggle with identifying specific clinical signs despite strong disease classification performance.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of medical imaging, computers have become remarkably skilled at looking at pictures of the human body and telling a doctor what they are seeing. For years, the standard way to test these artificial intelligence systems has been to ask them a simple question: "What disease does this patient have?" If the computer can correctly label an image as "diabetic retinopathy" or "age-related macular degeneration," it is often considered a success. However, this approach misses a crucial part of how human doctors actually think. When a specialist examines an eye scan, they do not jump straight to a diagnosis. Instead, they first hunt for specific, tiny clues—small spots of bleeding, patches of swelling, or subtle changes in the tissue layers. These individual signs are the evidence that leads to the final conclusion. If a computer model can name the disease but cannot point to the specific evidence that caused that name, it is like a student who can recite the answer on a test but cannot explain the steps to get there.
This gap between naming a disease and understanding the evidence behind it is the focus of a new study led by researchers from institutions including University College London and Moorfields Eye Hospital. The team set out to build a more rigorous test for artificial intelligence models designed to read eye images. Rather than just asking the computer to guess the final diagnosis, they created a benchmark that forces the models to prove they understand the specific clinical signs hidden within the picture. They tested eleven different AI models, ranging from those built specifically for eyes to general-purpose vision systems trained on millions of everyday photos. The researchers evaluated these models on two main tasks: first, classifying the severity of diseases like diabetic retinopathy and macular degeneration using tiny dots to mark where the disease evidence is; and second, a retrieval task where the computer had to find other images in a massive database that contained the exact same specific sign, such as a particular type of fluid leakage or a specific kind of scar.
The results revealed a surprising split in how these models think. The general-purpose AI models, which are trained on vast collections of ordinary photographs of cats, cars, and landscapes, performed quite well when asked to simply label the disease. With a little bit of training on eye images, they could often match the accuracy of models built specifically for ophthalmology. However, when the test changed to require the models to find specific signs or point to the exact location of a disease clue, the general-purpose models struggled significantly. They could recognize the overall "vibe" of a sick eye, but they failed to isolate the tiny, critical details that define the condition. In contrast, the models that had been pre-trained on huge datasets of actual eye scans showed a much stronger ability to locate and retrieve these specific clinical signs. They were better at understanding that a small red dot in one corner of an image meant something different than a large patch of white in the center.
The study also explored the value of the "point" annotations used in the testing. By giving the AI a simple dot to mark where a sign was located, the researchers found that the models learned to focus on the right areas much more effectively. This method helped the models avoid taking shortcuts, such as guessing a disease based on the type of camera used to take the picture or the general brightness of the image. When the models were trained with these simple dots, they became better at generalizing to new patients and different types of equipment. The researchers found that this approach was particularly helpful for the general-purpose models, which showed the biggest improvement when guided by these specific points. It suggests that teaching an AI to look at the specific evidence, rather than just the final outcome, is a powerful way to make it more reliable.
Ultimately, the paper argues that the current way of evaluating medical AI is too narrow. Just because a model can correctly label a disease does not mean it truly understands the image. The ability to retrieve an image based on a specific sign, like finding all cases of a particular type of fluid buildup, is a more honest test of whether the computer has learned the visual language of the eye. The findings suggest that while general-purpose AI is adaptable, it lacks the deep, fine-grained understanding that comes from training on domain-specific data. For these tools to be truly useful in a hospital, they need to be able to do more than just guess; they need to be able to point to the evidence, just as a human doctor does. By releasing this new benchmark to the public, the researchers hope to push the field toward developing models that are not just accurate classifiers, but true interpreters of the complex visual signs that define eye health.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.