💻 computer science

Predicting Visual Acuity from Fundus Photographs Using a Domain-Pretrained Vision Transformer: Anatomical Alignment of Attention Patterns

This study demonstrates that a domain-pretrained Vision Transformer (RETFound) fine-tuned on over 21,000 fundus photographs achieves substantial agreement in predicting four-class visual acuity while developing anatomically organized attention patterns—shifting from retinal vessels in low-vision eyes to the macula in normal-vision eyes—that correlate with classification correctness and offer interpretability for uncertain predictions.

Original authors: Jin Hyun Kim, Yong Seop Han, Kuk Jin Jang, Seoung Jin Lee, Hyonyoung choi, Insup Lee

Published 2026-07-17

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Jin Hyun Kim, Yong Seop Han, Kuk Jin Jang, Seoung Jin Lee, Hyonyoung choi, Insup Lee

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess how well someone can see just by looking at a single, colorful map of the back of their eye. This is the world of ophthalmic AI, where computers are taught to read retinal photographs to predict visual acuity—basically, how sharp a person's vision is. Usually, checking your vision requires you to sit in a doctor's office, squint at a chart, and read letters out loud. But what if a computer could look at a photo of your eye and say, "Ah, your vision is perfect," or "You might need glasses," without you ever saying a word?

To do this, scientists use Vision Transformers. Think of these as super-smart digital detectives. Unlike older computer models that just scan an image like a camera taking a snapshot, a Vision Transformer breaks the picture into tiny puzzle pieces and asks, "How does this piece relate to that piece?" It learns to pay attention to specific details, like a detective focusing on a fingerprint or a muddy shoe print. The big question this paper tackles is: Can a computer learn to predict vision quality just by looking at these eye maps, and more importantly, can we trust where the computer is looking? If the computer says your vision is bad, is it actually looking at the damaged parts of your eye, or is it just guessing based on random noise?

The Detective's New Glasses

In this study, a team of researchers decided to give their digital detective a pair of special "glasses" called RETFound. Imagine you have a detective who has spent years studying millions of crime scene photos. They know exactly what a fingerprint looks like, how blood spatters, and where the clues usually hide. That's what RETFound is: a massive AI model that has already "read" 1.6 million unlabeled eye photos. It's already an expert at spotting the tiny vessels and textures of the retina before it even tries to solve the specific puzzle of visual acuity.

The researchers took this expert detective and fine-tuned it to solve a specific mystery: sorting eye photos into four vision categories.

C0: Total or functional blindness.
C1: Severe impairment (very blurry).
C2: Moderate to mild impairment.
C3: Normal, sharp vision.

They tested this on 21,602 photos from 3,654 patients. To make sure the detective wasn't just cheating by memorizing the answers, they were very strict: they split the data by patient, not by photo. This means if a patient had five photos, all five went into either the training group or the test group, never both. This is like making sure a student taking a test never sees the same question twice in different forms.

The Results: A "Substantial" Success

The results were promising. The RETFound detective got the right answer 55.6% of the time. While that might not sound like a perfect score, in the world of medical AI, it's a big deal. More importantly, when it did make a mistake, it was usually a small one. 89.6% of its errors were just one step away from the correct answer (like guessing "severe impairment" when the answer was "moderate"). It rarely made wild guesses, like calling a normal eye "blind."

The paper also compared RETFound to two other detectives: DINOv2 (a general-purpose AI that hasn't studied eye photos specifically) and EfficientNet (an older style of computer vision). RETFound beat them both by a significant margin. The researchers calculated a score called Quadratic Weighted Kappa, which measures how well the predictions match the real order of things. RETFound scored 0.612, which experts call "substantial agreement."

The "Where Are You Looking?" Mystery

The most exciting part of the paper isn't just that the computer got the right answer, but how it looked at the eye to get there. The researchers used a technique called Attention Rollout. Imagine the computer is a spotlight shining on the eye photo. This technique lets the researchers see exactly where the spotlight is focused.

They found a beautiful pattern that matches what human doctors have known for decades:

For eyes with terrible vision (C0): The computer's spotlight focused heavily on the blood vessels around the edge of the eye. This makes sense because severe vision loss is often caused by damaged or leaking blood vessels. The computer learned to look at the "messy" parts of the map.
For eyes with perfect vision (C3): The spotlight moved to the very center of the eye (the macula). This is the part of the eye responsible for sharp, central vision. The computer learned to ignore the edges and focus on the center.

It's like a detective who knows that if a house is on fire, they should look at the smoke and flames (the vessels), but if the house is pristine, they should look at the front door and windows (the macula) to confirm everything is fine.

The "Trust Me" Test

Here is the clever twist: The researchers wanted to know if this "spotlight" behavior was just a lucky accident or if it actually meant the computer was thinking correctly. They compared the photos the computer got right versus the ones it got wrong.

When the computer was right: The spotlight behaved perfectly. If it guessed "blind," the light was on the vessels. If it guessed "normal," the light was on the center. The connection between the spotlight and the answer was very strong.
When the computer was wrong: The spotlight was confused. It was all over the place, looking at random spots. The connection between where it looked and what it guessed was weak.

This suggests that the computer's "attention" is a reliable clue. If the computer is looking at the right place, it's probably right. If it's looking everywhere at once, you should be skeptical.

The "Blurry Middle" Problem

The study also found a tricky spot. The computer struggled the most with the C1 category (severe impairment, but not total blindness). It often confused these eyes with the "moderate" category. The researchers suggest this isn't because the computer is dumb, but because the eye photos for this specific level of vision are just naturally ambiguous. It's like trying to tell the difference between a slightly foggy day and a very cloudy day; sometimes, the picture just doesn't have enough clear clues to be 100% sure.

What This Means

This paper doesn't claim to have solved vision loss or replaced doctors. Instead, it shows that we can build AI that doesn't just guess, but actually "sees" the eye the way a human doctor does. It learns to focus on blood vessels for bad eyes and the center for good eyes.

The authors suggest that in the future, we could use this "spotlight" behavior as a warning system. If an AI makes a prediction but its spotlight is all over the place, doctors could flag that case for a human to double-check. It turns the AI from a black box into a partner that tells you not just what it thinks, but why it thinks it, and how much you should trust that guess.

Technical Summary: Predicting Visual Acuity from Fundus Photographs Using a Domain-Pretrained Vision Transformer

Problem Statement
Visual acuity (VA) is the primary metric for functional vision but typically requires patient cooperation and standardized charting, limiting its utility in large-scale screening or resource-limited settings. While deep learning has been applied to predict VA from fundus photographs, prior approaches often treated the task as binary classification or continuous regression, failing to capture the ordinal, multi-class nature of clinical VA grading. Furthermore, previous studies by the authors and others suffered from methodological flaws, specifically image-level train-test splits that introduced patient-level data leakage, inflating performance metrics. Additionally, existing models often operate as "black boxes," lacking insight into whether they attend to clinically relevant retinal structures (e.g., macula vs. peripheral vessels) or spurious image statistics. This study addresses the need for a robust, patient-level validated model that predicts four-class ordinal VA and characterizes the anatomical alignment of its internal attention mechanisms.

Methodology

Dataset: The study utilized a retrospective dataset of 21,602 fundus photographs from 3,654 patients. Images were classified into four ordinal VA classes: C0 (0.0, blindness), C1 (0.1–0.2, severe impairment), C2 (0.3–0.7, moderate-mild), and C3 (0.8–1.0, normal). Strict patient-level splitting was enforced, with a fixed held-out test set of 564 patients (3,236 images) to prevent data leakage.
Model Architecture: The primary backbone was RETFound, a Vision Transformer (ViT-L/16, 307M parameters) pretrained on 1.6 million unlabeled fundus images via masked autoencoding (MAE). This was compared against a general-purpose ViT (DINOv2, ViT-B/14-reg4) and a convolutional baseline (EfficientNet-B4).
Training Strategy: RETFound was fine-tuned using a progressive unfreezing schedule (head only $\to$ last 25% blocks $\to$ last 50% blocks) to preserve pretrained retinal representations. The primary loss function was OrdinalFocalLoss, combining focal cross-entropy with an ordinal regression penalty (MSE on expected class values). Ablation studies compared this against class-weighted cross-entropy, focal cross-entropy, and Earth Mover's Distance (EMD).
Attention Analysis: To interpret the model, Attention Rollout was computed across all 24 transformer blocks to aggregate spatial attention flow. Quantitative metrics included:
- Anatomical Enrichment: Ratios of attention weight within masks for retinal vessels (Frangi filter), macula (fixed circular mask), and optic disc.
- Spatial Metrics: Center Proximity Ratio (CPR) and Spatial Entropy.
- Individual-Level Analysis: A log Attention Dominance Ratio (log-ADR) was defined to quantify the balance between macular and vessel attention per sample.
Validation: Robustness was tested via 180 mask parameter combinations, negative controls (spatial shuffling and random-weight ViT), cross-method comparison (GradCAM++), and region occlusion experiments.

Key Results

Classification Performance: RETFound achieved the highest performance with 55.6% accuracy and a Quadratic Weighted Kappa (QWK) of 0.612 ("substantial agreement"), significantly outperforming DINOv2 and EfficientNet-B4 (McNemar $p < 0.0001$ ). Notably, 89.6% of errors were confined to adjacent VA classes, preserving clinical ordering.
Loss Function Trade-offs: While OrdinalFocalLoss yielded the best top-1 accuracy and lowest Mean Absolute Error (MAE), class-weighted cross-entropy achieved the highest QWK and weighted F1, indicating a trade-off between ordinal consistency and agreement-oriented metrics.
Anatomical Alignment: A clear, monotonic shift in attention patterns was observed across VA classes:
- Low VA (C0): Attention was enriched in peripheral retinal vessels (Enrichment $\approx$ 1.75) and depleted in the macula.
- Normal VA (C3): Attention shifted to the macula (Enrichment $\approx$ 1.17) and depleted in vessels.
- Intermediate VA (C1/C2): Attention was more diffuse, with C1 showing a lack of a dominant anatomical anchor.
Individual-Level Reliability: Correctly classified samples exhibited a significantly stronger correlation between log-ADR and VA class ( $\rho = 0.644$ ) compared to misclassified samples ( $\rho = 0.198$ ). Misclassified samples across all classes were predominantly "diffuse-dominant," lacking a clear anatomical focus.
Causal Verification: Region occlusion experiments confirmed that masking the macula caused a significant drop in C3 recall (from 0.65 to 0.48), providing functional evidence that macular attention is causally necessary for normal vision classification. Conversely, vessel occlusion had minimal impact on C3, while macular occlusion had no systematic effect on C0.
Robustness: The observed attention patterns were stable across 180 mask parameter combinations and were absent in spatially shuffled or randomly initialized models, confirming they arise from learned features rather than image statistics.

Significance and Claims
The paper claims that fine-tuning a domain-pretrained Vision Transformer (RETFound) for ordinal VA prediction not only achieves state-of-the-art performance but also develops anatomically organized attention patterns that mirror established ophthalmic knowledge. Specifically, the model learns to prioritize vascular pathology in severely impaired eyes and macular integrity in normal eyes.

Crucially, the authors argue that the alignment between anatomical attention and prediction correctness serves as a potential proxy for classification reliability. The finding that misclassified samples consistently lack a class-congruent anatomical focus suggests that attention-based interpretability could be used to flag uncertain predictions for human review in clinical settings. The study also highlights that the under-performance on the C1 class (VA 0.1–0.2) likely reflects a fundamental imaging ambiguity in this range rather than a model limitation, as even correctly classified C1 samples lacked a distinct anatomical anchor.

Finally, the paper distinguishes between spatial attention (where the model looks, captured by Rollout) and discriminative gradients (what drives the decision, captured by GradCAM++), noting that these can diverge (e.g., vessel attention is high in C0 for Rollout but high in C3 for GradCAM++), providing complementary insights into model behavior.