🤖 AI

Deep Learning for Retinal Degeneration Assessment: A Comprehensive Analysis of the MARIO Challenge

This paper presents a comprehensive analysis of the MARIO challenge at MICCAI 2024, detailing how 35 teams utilized multi-modal OCT and clinical data to benchmark AI performance in retinal degeneration assessment, revealing that while AI matches physician accuracy in detecting AMD progression, it currently falls short in predicting future disease evolution.

Original authors: Rachid Zeghlache, Ikram Brahim, Pierre-Henri Conze, Mathieu Lamard, Mohammed El Amine Lazouni, Zineb Aziza Elaouaber, Leila Ryma Lazouni, Christopher Nielsen, Ahmad O. Ahsan, Matthias Wilms, Nils D. F

Published 2026-08-04

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Rachid Zeghlache, Ikram Brahim, Pierre-Henri Conze, Mathieu Lamard, Mohammed El Amine Lazouni, Zineb Aziza Elaouaber, Leila Ryma Lazouni, Christopher Nielsen, Ahmad O. Ahsan, Matthias Wilms, Nils D. Forkert, Lovre Antonio Budimir, Ivana Matovinović, Donik Vršnak, Sven Lončarić, Philippe Zhang, Weili Jiang, Yihao Li, Yiding Hao, Markus Frohmann, Patrick Binder, Marcel Huber, Taha Emre, Teresa Finisterra Araújo, Marzieh Oghbaie, Hrvoje Bogunović, Amerens A. Bekkers, Nina M. van Liebergen, Hugo J. Kuijf, Abdul Qayyum, Moona Mazher, Steven A. Niederer, Alberto J. Beltrán-Carrero, Juan J. Gómez-Valverde, Javier Torresano-Rodríquez, Álvaro Caballero-Sastre, María J. Ledesma Carbayo, Yosuke Yamagishi, Yi Ding, Robin Peretzke, Alexandra Ertl, Maximilian Fischer, Jessica Kächele, Sofiane Zehar, Karim Boukli Hacene, Thomas Monfort, Béatrice Cochener, Mostafa El Habib Daho, Anas-Alexis Benyoussef, Gwenolé Quellec

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery that happens inside a tiny, intricate city: the human eye. Specifically, you are looking at the retina, the light-sensitive screen at the back of the eye. Sometimes, this city gets invaded by a sneaky, leaky growth called neovascular age-related macular degeneration (AMD). It's like a plumbing disaster where tiny, unwanted pipes sprout and leak fluid, blurring the center of your vision. To fix this, doctors use special "anti-leak" injections (anti-VEGF) that act like a super-glue to stop the pipes. But here's the tricky part: the leakiness changes every day. Sometimes the pipes are quiet, sometimes they are gushing, and sometimes they are just a little damp. Doctors have to take high-resolution 3D pictures of the eye (called OCT scans) every few weeks to see if the glue is working. If they guess wrong, they might inject a patient who doesn't need it, or worse, wait too long on a patient who is getting worse.

For a long time, computers have been getting really good at looking at a single picture and saying, "Hey, this eye has a leak!" But the real challenge is like watching a movie instead of a still photo. The computer needs to look at two pictures taken weeks apart and decide: "Did the leak get better, stay the same, or get worse?" Even harder, can the computer look at just one picture today and predict what will happen three months from now? This is the question a group of scientists asked the world's best AI detectives to solve in a global contest called the MARIO challenge. They wanted to know if artificial intelligence could become a reliable sidekick for doctors, helping them decide exactly when to give those injections and when to let the patient rest.

The Great Eye Detective Contest

In 2024, a team of researchers organized a massive game called the MARIO challenge (Monitoring Age-Related macular degeneration with Intelligent Ophthalmology). They gathered 35 teams of AI experts from around the world, including students, university researchers, and tech companies. The goal was to build a computer program that could analyze optical coherence tomography (OCT) scans—those super-detailed 3D cross-sections of the retina—to track the disease.

The challenge had two levels, like a video game with an easy mode and a hard mode.

Level 1: The "Spot the Change" Game
In this level, the AI was given two pictures of the same eye taken at different times (let's say, Visit A and Visit B). The computer had to look at both and decide: Did the disease get Reduced (better), stay Stable (the same), or Worsened (got worse)?
The results here were surprisingly exciting. The best team, called MIPLAB, built a model that got it right about 86% of the time. To put that in perspective, this is almost as good as two human eye doctors agreeing with each other! The AI successfully learned to spot the subtle differences in fluid levels between two scans. It was like teaching a computer to notice that a puddle on the sidewalk had shrunk or grown just by looking at two photos. This suggests that for checking if a treatment is working right now, AI is ready to be a helpful assistant.

Level 2: The "Crystal Ball" Game
This level was much, much harder. The AI was given only one picture (Visit A) and had to predict what would happen in the next three months. Would the eye get better, stay the same, or get worse?
The results here were a reality check. The best AI model only managed a score of about 0.30, which is barely better than just guessing "Stable" every time because most eyes are stable. The teams' predictions were so different from each other that they agreed with one another no better than chance. It was as if the AI was trying to predict the weather three months in advance using only a single photo of the sky today; there just wasn't enough information in that one picture to know what was coming. The paper concludes that predicting the future from a single visit is still a problem that AI hasn't solved yet.

The "Foreign Language" Problem

There was one more twist in the story that taught the researchers a very important lesson about fairness and reliability. The main group of patients came from Brest, France. But to test if the AI was truly smart or just memorized the French data, the organizers threw in a tiny, secret test set from Tlemcen, Algeria. These patients had different backgrounds, and their eye scans were taken with slightly different equipment settings.

When the winning AI from the French data tried to solve the Algerian puzzles, it stumbled badly. The team that came in first place in France dropped to fifth place in Algeria. Some teams' scores crashed so hard they were barely better than random guessing. It was like a student who aced a math test in one classroom but failed completely when the teacher switched to a different textbook with slightly different symbols. This showed that the AI models were too sensitive to where the data came from. They hadn't learned the universal rules of the eye; they had just learned the specific "accent" of the French scans.

What Does This Mean?

The MARIO challenge gave us a clear map of where we stand.

Short-term monitoring is possible: If a doctor wants to know if an eye is better or worse compared to last month's scan, AI can do a great job. It's close to being ready to help doctors save time and reduce unnecessary injections.
Long-term prediction is not ready: If a doctor wants to know what will happen three months from now based on a single scan, AI is currently useless. The signal is too weak, and the models are just guessing.
One size does not fit all: An AI that works perfectly for one group of people might fail miserably for another. Before we can trust these tools in hospitals everywhere, we need to make sure they work for everyone, regardless of where they live or what machine scanned their eyes.

The paper doesn't claim to have solved the mystery of AMD, but it did solve the mystery of what AI can and cannot do right now. It tells us that while we have built a very good "spotter" for current changes, we still need to invent a much smarter "prophet" for the future. And until we do, we must be careful not to trust our digital detectives too much when they are in a new neighborhood.

Technical Summary: Deep Learning for Retinal Degeneration Assessment: A Comprehensive Analysis of the MARIO Challenge

Problem Definition
The paper addresses the critical need for automated, longitudinal monitoring of neovascular age-related macular degeneration (nAMD) in patients undergoing anti-VEGF therapy. While Optical Coherence Tomography (OCT) is the standard imaging modality, current AI applications largely focus on cross-sectional disease classification or segmentation. There is a significant gap in algorithms capable of detecting subtle temporal changes in neovascular activity (fluid dynamics) to support "treat-and-extend" protocols. The MARIO (Monitoring Age-Related macular degeneration with Intelligent Ophthalmology) challenge, organized at MICCAI 2024, was established to benchmark AI methods on two specific clinical tasks:

Task 1 (Activity Change Classification): Classifying the evolution of neovascular activity between two consecutive 2D OCT B-scans (baseline $t_0$ vs. follow-up $t_1$ ) into three classes: Reduced, Stable, or Worsened (plus an Uninterpretable class).
Task 2 (Prognostic Prediction): Predicting disease progression over the subsequent three months based on a single visit (one OCT B-scan), representing a more difficult, single-visit prognostic task.

The challenge uniquely incorporates a cross-domain evaluation, utilizing a primary cohort from Brest, France (136 patients, Heidelberg Spectralis device) and a held-out external test set from Tlemcen, Algeria (5 patients), to assess generalization across populations and imaging conditions.

Methodology and Data

Dataset: The dataset comprises high-resolution 2D B-scans and 3D volume sequences paired with infrared (IR) localizer images and clinical variables. Annotations were performed by ophthalmologists, distinguishing between fluid types (subretinal, intraretinal) and activity states. The data was simplified into the three primary classes for the challenge.
Participating Teams: 35 teams registered, with 12 reaching the final phase.
Architectural Trends:
- Task 1: Most finalists adopted Siamese or late-fusion architectures to compare paired B-scans. A dominant strategy involved using foundation models pretrained on retinal imaging (e.g., RETFound, BiomedCLIP) as backbones, often fine-tuned with pretext tasks like change detection or segmentation. Multi-modal fusion (combining OCT, IR, and clinical data) was attempted by only one team (MIPLAB).
- Task 2: Given the lack of a second time-point image, teams employed three main strategies: (1) treating the single image as a variant of Task 1; (2) attempting to "hallucinate" or reconstruct the future state using generative approaches (e.g., Patch Progression Masked Autoencoders); or (3) framing the problem as multiple-instance learning over the OCT volume.
Evaluation: Performance was measured using macro-F1, specificity, Matthews Correlation Coefficient (MCC), and Rank Correlation Coefficient (Rk). Task 2 additionally utilized Quadratic Weighted Kappa (QWK) to account for the ordinal nature of the classes. A composite score weighted Task 2 more heavily (65%) than Task 1 (35%). Statistical significance was assessed via stratified bootstrap resampling and McNemar tests.

Key Results

Task 1 (Short-term Monitoring): The challenge demonstrated that short-term activity monitoring is tractable. The best team (MIPLAB) achieved a composite score of 0.833 (macro-F1 = 0.859) on the Brest test set, approaching inter-expert agreement (Cohen's $\kappa \approx 0.49$ among top teams). This suggests current deep learning models can reliably track short-term changes in neovascular activity.
Task 2 (Single-Visit Prognosis): This task proved significantly harder. The best composite score was only 0.306, with QWK values near zero. Inter-team agreement collapsed (mean $\kappa = 0.10$ ), indicating that no method successfully recovered a reliable prognostic signal for 3-month progression from a single time point. Models largely defaulted to predicting the majority "Stable" class.
Domain Generalization: A critical finding was the fragility of models under domain shift. When transferred to the Algerian cohort, performance degraded substantially for all teams. The Task 1 winner (MIPLAB) dropped to fifth place on the external set, and the ranking order reshuffled entirely. This highlights unsolved challenges in cross-population and cross-device generalization.
Error Analysis: Errors were often shared across teams rather than method-specific. "Uninterpretable" scans and subtle fluid shifts were frequently misclassified as "Stable." In Task 2, the "Worsened" class was almost universally missed (mean F1 $\approx$ 0.018), a clinically critical failure mode.

Contributions and Significance
The paper establishes MARIO as the first international benchmark for automated longitudinal monitoring of nAMD. Its primary contributions are:

Benchmarking Longitudinal AI: It provides a standardized framework for evaluating AI on temporal OCT data, moving beyond static classification.
Defining the State-of-the-Art: It demonstrates that while short-term activity classification is within reach of current methods, single-visit progression prediction remains an open problem, likely due to insufficient information in a single snapshot rather than architectural limitations.
Highlighting Generalization Gaps: The challenge explicitly reveals that models trained on homogeneous European data fail to generalize to diverse populations and different acquisition protocols, underscoring the need for domain-robust AI and diverse datasets.
Clinical Relevance: The results suggest that AI can currently support clinical decision-making for interval extension in stable eyes (Task 1) but is not yet ready for predicting future progression without richer inputs (e.g., full longitudinal history or volumetric data).

The authors conclude that while the "ceiling" for Task 1 is approaching clinical utility, Task 2 requires richer data modalities and more diverse training sets to overcome the current information deficit. The challenge emphasizes that robustness across populations cannot be inferred from single-site performance, advocating for external validation as a default component of medical AI benchmarks.

The Great Eye Detective Contest

The "Foreign Language" Problem

What Does This Mean?

More like this