💻 computer science

GazeLT: Visual attention-guided long-tailed disease classification in chest radiographs

The paper introduces GazeLT, a novel framework that leverages radiologists' temporal visual attention patterns through an integration-disintegration mechanism to significantly improve long-tailed disease classification in chest radiographs, outperforming existing methods on large-scale public datasets.

Original authors: Moinak Bhattacharya, Gagandeep Singh, Shubham Jain, Prateek Prasanna

Published 2026-07-28

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Moinak Bhattacharya, Gagandeep Singh, Shubham Jain, Prateek Prasanna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Detective's Gaze: Teaching Computers to See What Humans See

Imagine you are trying to teach a robot how to spot hidden objects in a giant, messy room. If you just show the robot a thousand photos of the room, it might get really good at finding the big, obvious things like a red armchair or a blue rug. But if you ask it to find a tiny, rare silver coin hidden under a pile of laundry, it will likely miss it every time. This is a classic problem in the world of artificial intelligence called "long-tailed classification." It happens when some things are super common (the "head" of the tail) and others are incredibly rare (the "tail"), making it hard for computers to learn the rare stuff.

To solve this, scientists often look at how humans solve problems. In medicine, doctors called radiologists look at X-rays to find diseases. But they don't just stare at a picture and guess; their eyes move in a specific dance. They scan the whole image, zoom in on suspicious spots, glance at the edges, and sometimes look at things that turn out to be harmless. This movement of the eyes, known as "gaze," holds a secret map of how an expert thinks. The big question is: Can we teach a computer to copy this eye-dance so it doesn't just look at the obvious stuff, but also hunts down those rare, tricky diseases that it usually misses?

The Paper's Big Idea: GazeLT

This paper introduces a new method called GazeLT (Gaze-guided Long-Tailed classification). The researchers, working with chest X-rays, wanted to see if they could use a radiologist's eye movements to help an AI become a better detective. Instead of just showing the AI the X-ray, they showed it the X-ray plus a video of where a real doctor's eyes looked while diagnosing it.

The core of their idea is that looking at a medical image isn't a static, frozen moment; it's a process that happens over time. The authors realized that previous AI models treated a doctor's attention as a single, blurry snapshot. But in reality, a doctor's gaze changes. At first, they might do a quick, broad sweep of the whole chest (looking for big, obvious problems). Later, they might zoom in on tiny, specific details (looking for rare, subtle issues).

To capture this, the team split the doctor's viewing time into four equal windows. They created two special "attention modes":

Integration: This is like the doctor zooming in to connect the dots on a specific, detailed clue.
Disintegration: This is like the doctor stepping back to get a broad, global view of the whole picture.

They built a "Teacher-Student" system to teach the AI. The Teacher is a smart AI that learns directly from the radiologist's eye-gaze data. It practices looking at the X-ray through those four time windows, learning to integrate fine details and disintegrate into a broad view. Once the Teacher is trained, it doesn't need the eye-gaze data anymore. It then teaches the Student (a simpler, faster AI) how to look at the X-ray. The Student learns to mimic the Teacher's focus, but it only needs the X-ray image to work. This means the final AI can be used in a hospital without needing to track a doctor's eyes in real-time.

What They Found

The team tested GazeLT on two massive collections of chest X-rays: the NIH-CXR-LT dataset (with 89,237 images) and the MIMIC-CXR-LT dataset (with 111,898 images). These datasets are "long-tailed," meaning they have many common diseases (like pneumonia) and very few rare ones (like pneumomediastinum).

The results were quite promising. GazeLT didn't just do okay; it significantly outperformed the best existing methods.

On average, GazeLT beat the top "long-tailed loss" methods by 4.1%.
It crushed the previous "visual attention" baselines by a huge 21.7%.

Most importantly, the AI got much better at finding the rare diseases. On the NIH dataset, it improved the detection of rare "tail" classes by 10.9%, and on the MIMIC dataset, by 12.2%. While it was slightly less accurate at spotting the most common diseases (a drop of about 5%), the gain in finding the rare, dangerous conditions was considered a major win, as missing those is often the most critical error in medicine.

How They Proved It Worked

The researchers didn't just guess that this would work; they ran a series of experiments to prove it. They compared GazeLT against standard AI models that use basic math to handle rare classes, as well as other models that tried to use eye-gaze data but didn't split it into time windows.

They also ran "ablation studies," which is a fancy way of saying they took parts of their machine apart to see what was doing the heavy lifting. They found that:

Using just the "Integration" part or just the "Disintegration" part didn't work as well as using both together.
Splitting the time into 4 windows was the "sweet spot." Splitting it into 2 windows wasn't detailed enough, and splitting it into 8 windows created too much confusion and redundancy. The 4-window setup perfectly balanced the need to see the big picture and the small details.

The Catch and the Future

The paper is very clear about one limitation: to train the "Teacher" AI, you actually need a radiologist to wear eye-tracking glasses and look at X-rays. This is hard to do in the real world because it requires special hardware and willing doctors. However, the authors emphasize that once the Teacher is trained and has taught the Student, the Student can work perfectly fine without any eye-tracking data at all. You just feed it an X-ray, and it gives you a diagnosis.

The researchers suggest that this approach could eventually be used for other types of medical images, like CT scans or MRIs, not just chest X-rays. They also note that they didn't factor in how confident the doctors were in their diagnoses, which is something they might explore in the future.

In short, GazeLT suggests that if we teach AI to "look" the way human experts look—taking time to scan broadly and then zoom in deeply—we can build smarter medical tools that don't miss the rare, life-saving clues hidden in the noise.

Technical Summary: GazeLT – Visual Attention–Guided Long-Tailed Disease Classification in Chest Radiographs

Problem Statement

Long-tailed classification in medical imaging remains a significant challenge due to the inherent class imbalance between common (head) and rare (tail) disease pathologies. In chest radiographs, common conditions like Pneumothorax or Atelectasis appear frequently, while rare conditions such as Pneumomediastinum or Subcutaneous Emphysema occur in less than 1% of cases. Standard deep learning models tend to perform well on head classes but struggle to identify tail classes, often learning shortcuts or focusing on disease-irrelevant features. While existing methods address this through loss re-weighting, re-sampling, or decoupling strategies, they generally overlook the potential of clinical experts' visual search patterns. Furthermore, previous attempts to integrate eye gaze data have treated visual attention as a static, aggregated map, failing to capture the dynamic, temporal nature of how radiologists interpret images over time.

Methodology: GazeLT

The authors propose GazeLT, a novel teacher-student deep learning framework that integrates the temporal dynamics of radiologists' visual attention to improve long-tailed classification. The core hypothesis is that radiologists' gaze patterns evolve during diagnosis, shifting from coarse global searches to fine-grained inspections, and that decomposing these patterns into time windows can capture both integration (fine-grained) and disintegration (coarse/global) attention mechanisms.

1. Temporal Decomposition of Attention

The method divides the total duration of a radiologist's image viewing into $n$ equal time windows (set to $n=4$ in experiments). Within each window, two distinct attention mechanisms are modeled:

Time Windowed Integration (TW-I): Captures fine-grained attention. Fixation points within a specific distance threshold are clustered to form a "main" attention region, while others form "substitute" regions. These are combined and smoothed to create an integration heatmap. This mimics the radiologist's focused search for specific abnormalities.
Time Windowed Disintegration (TW-D): Captures coarse, global attention. Gaussian filtering is applied directly to raw fixation points without clustering, representing a broader, global search pattern akin to "disintegration" in human psychology.

2. Teacher-Student Knowledge Distillation

The framework operates in two stages:

Teacher Training (t-HVA): A teacher network, composed of TW-I and TW-D blocks, is trained using radiologists' eye gaze data.
- The TW-I block utilizes a FocalNet transformer architecture to learn integration attention.
- The TW-D block utilizes a GC-ViT (Global Context Vision Transformer) architecture to learn disintegration attention.
- Both blocks are supervised by Temporal Visual Attention Loss (tVAL) functions ( $L_{I-tVAL}$ and $L_{D-tVAL}$ ), which minimize the distance between the model's intermediate feature maps and the ground-truth gaze-derived attention heatmaps.
Student Training: A student model (ResNet-50) is trained for the downstream long-tailed classification task. The teacher's weights are frozen. The student learns via Knowledge Distillation (KD), where the fused features from the teacher's TW-I and TW-D blocks ( $J(f_I, f_D)$ $J (f_{I}, f_{D})$ ) are transferred to the student.
- The distillation loss ( $L_{BD}$ ) minimizes the Bhattacharyya distance between the student's feature distribution and the teacher's joint representation.
- The student is also optimized using a Label-Distribution-Aware Margin (LDAM) loss ( $L_{LDAM}$ ) to handle class imbalance directly.
- The final student loss is: $L_s = L_{LDAM} + \lambda L_{BD}$ .

3. Inference

During inference, the student model operates solely on the input chest X-ray image. The eye gaze data and the teacher network are not required, ensuring the system is deployable in clinical settings without additional hardware.

Key Contributions

Temporal Attention Decomposition: The paper introduces a novel approach to decompose static visual attention into Time Windowed Integration and Disintegration mechanisms, capturing the dynamic shift from global to local search patterns in radiologists.
Novel Loss Functions: The authors propose Visual Attention Loss (VAL) functions specifically designed to train the teacher network to mimic the temporal evolution of human gaze.
Teacher-Student Framework for Long-Tailed Learning: This is the first work to leverage radiologists' gaze behavior specifically to mitigate long-tail classification problems, using a distillation strategy to transfer temporal diagnostic priors to a student model.
Empirical Validation: The method is validated on two large-scale, publicly available long-tailed datasets: NIH-CXR-LT ( $n=89,237$ ) and MIMIC-CXR-LT ( $n=111,898$ ).

Results

The authors report significant performance improvements over state-of-the-art baselines, including standard long-tailed losses (Softmax, Focal, LDAM, RW LDAM-DRW) and visual attention-based baselines (RadioTransformer, GazeRadar).

NIH-CXR-LT: GazeLT achieved an average accuracy of 0.410 on the balanced test set, outperforming the best long-tailed loss baseline (LDAM) by 4.1% and the visual attention baseline (RadioTransformer) by 21.7%. Notably, it improved tail-class accuracy by 10.9%.
MIMIC-CXR-LT: GazeLT achieved an average accuracy of 0.418, an 8.01% improvement over the best baseline. Tail-class accuracy improved by 12.2%.
Ablation Studies: Experiments confirmed that the combination of TW-I and TW-D (Time Windowed) outperforms standalone Integration or Disintegration blocks, validating the necessity of modeling both temporal attention types. A sensitivity analysis showed that 4 time windows offered the optimal trade-off between temporal detail and redundancy compared to 2 or 8 windows.
Multiclass Performance: On the NIH-CXR-LT imbalanced test set, GazeLT achieved a balanced accuracy of 0.421, an MCC of 0.391, and an AUC of 0.829, significantly surpassing GazeRadar and RadioTransformer.

Significance and Claims

The paper claims that GazeLT represents a significant advancement by integrating the temporal visual attention patterns of radiologists into a deep learning framework. The authors argue that:

Dynamic vs. Static: Decomposing attention into time windows captures the "context-rich" nature of diagnostic reasoning better than static aggregated maps.
Clinician-in-the-Loop: The approach enhances model interpretability and clinical relevance by aligning model focus with expert visual search strategies, particularly for rare diseases where data is scarce.
Practical Deployment: While gaze data is required for training the teacher, the final student model requires no gaze input during inference, making it feasible for clinical environments lacking eye-tracking hardware.
Modality Agnostic: Although demonstrated on 2D chest X-rays, the authors posit that the framework is extensible to other modalities (CT, MRI, ultrasound) and 3D data, provided gaze data can be acquired.

The authors acknowledge limitations, primarily the dependency on high-quality eye-tracking data for the training phase and the current focus on 2D chest radiographs. They conclude that incorporating expert temporal dynamics is crucial for developing robust, accurate, and interpretable diagnostic tools for long-tailed medical imaging tasks.