FundusSigns-47: Prompt-Tuned Residual Visual Verification for Multi-Label Ophthalmic Sign Recognition
This paper introduces FundusSigns-47, a large-scale multi-label ophthalmic dataset with 47 signs, and proposes SCVPT, a novel framework combining prompt-tuned SigLIP2 with a learned InceptionV3 residual verifier that achieves superior macro-F1 and recall performance for multi-label sign recognition.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The human eye is a window to the body's health, and the retina, the light-sensitive layer at the back of the eye, often reveals the first signs of disease. To see this layer, doctors use a special camera to take pictures of the fundus, the interior surface of the eye. These images are rich with detail, showing blood vessels, the optic nerve, and the macula, the area responsible for sharp central vision. For decades, researchers have tried to teach computers to read these pictures automatically, hoping to spot conditions like diabetic retinopathy or glaucoma before they cause permanent damage. Most of these computer systems have been trained to look for just one thing at a time, like a security guard checking a single door. They are excellent at saying "yes" or "no" to a specific disease, but they struggle when a single image contains a complex mix of different problems, such as bleeding, scarring, and swelling all at once. This limitation leaves a gap between how a computer sees an image and how a doctor actually reads it, where multiple clues often appear together to tell a complete story.
A team of researchers from Xiamen University and the Xiamen Eye Center has taken a significant step toward closing that gap. They created a new collection of 7,111 fundus images, each carefully labeled with up to 47 different types of eye signs, ranging from common changes like a tessellated appearance to rare and serious issues like retinal detachment. This dataset, called FundusSigns-47, reflects the messy reality of clinical practice, where signs are not evenly distributed; some appear in thousands of images while others are so rare they show up in only a handful. To make sense of this complexity, the researchers developed a new method called Selective Candidate-Verified Prompt Tuning. Instead of forcing a computer to guess all 47 signs at once, they built a two-step system that mimics a more careful human thought process. First, a large, pre-trained model acts as a broad scanner, using language-based prompts to suggest which signs might be present. Then, a second, specialized visual verifier steps in to double-check those suggestions, looking for specific visual evidence to confirm or correct the initial guess.
The researchers found that simply asking a powerful computer model to look at these images without any special training did not work well. The model, which had learned from millions of general pictures, failed to recognize the subtle, specific details of eye disease when left to its own devices. However, when the team taught the model to pay attention to the right visual features using a technique called deep visual prompt tuning, it became much better at generating a list of possible signs. This first step produced a strong set of candidates, but it still missed some rare signs and occasionally guessed wrong on signs that looked similar. The breakthrough came with the second step: adding a visual verifier. This second computer model, trained specifically on the fundus images, acted as a fact-checker. It examined the image again and provided extra visual evidence to support or adjust the first model's list. Crucially, the researchers designed the system so that this verifier only checked the signs that the first model thought were likely, rather than trying to re-evaluate every single possibility. This selective approach allowed the system to focus its attention where it mattered most.
The results of this approach were measured against a fixed set of test images that the computer had never seen before. The new system achieved a score of 56.15 percent on a metric that balances how well it found the right signs against how often it made mistakes, known as the macro-F1 score. While a simple combination of two different models working independently performed slightly better at ranking the most likely signs, the new two-step system was superior at actually finding the signs that were there, particularly the harder-to-spot ones. The researchers showed that the improvement came primarily from the visual verifier adding new evidence, not just from the first model getting better at guessing. They also demonstrated that the system did not simply become more cautious or more reckless; instead, it corrected specific errors, recovering missed signs and removing false alarms. The study suggests that for complex medical imaging tasks, a strategy that combines a broad, language-aware scanner with a focused, visual fact-checker offers a more balanced and reliable way to understand the full picture of a patient's eye health.
Despite these promising results, the authors are careful to note that this is a starting point rather than a finished solution. The dataset comes from a single institution, and the researchers emphasize that the system needs to be tested on images from other hospitals and different patient populations before it can be used in real clinical settings. They also point out that the current system identifies observable signs, such as a specific type of bleeding or a scar, but it does not yet make the final medical diagnosis, which requires a doctor to combine these signs with the patient's history and other tests. The work highlights that while artificial intelligence can be a powerful tool for spotting patterns, the most effective approach for complex, multi-faceted problems may be one that uses different types of models to check each other's work, ensuring that the final answer is both comprehensive and accurate.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.