GazeXPErT: An Expert Eye-tracking Dataset for Interpretable and Explainable AI in Oncologic FDG-PET/CT Scans

This paper introduces GazeXPErT, a comprehensive 4D eye-tracking dataset capturing expert search patterns on 346 oncologic FDG-PET/CT scans, which demonstrates that integrating human gaze data significantly enhances the performance and interpretability of AI models for tumor segmentation and localization.

The Gist

🕵️‍♂️ The Big Idea: "Teaching AI to See Like a Detective"

Imagine you are trying to teach a robot how to find a needle in a haystack.

• The Old Way: You show the robot a picture of the haystack and say, "Find the needle." The robot looks at the whole picture, guesses where the needle might be, and draws a box around it. Sometimes it gets it right; sometimes it guesses wrong. The problem is, the robot doesn't know how it found the needle, so doctors don't trust it.
• The GazeXPErT Way: Instead of just showing the robot the haystack, you strap a camera to a master detective's head. You watch exactly where the detective looks, how long they stare at a spot, and how they move their eyes to find the needle. Then, you teach the robot to copy the detective's eye movements, not just the final answer.

This paper introduces GazeXPErT, a massive new dataset that does exactly this for cancer scans.

---

🏥 The Problem: The "Black Box" of Cancer Scans

FDG-PET/CT scans are like super-powered flashlights that show where cancer cells are burning sugar (metabolism) inside the body. They are crucial for finding tumors.

However, reading these scans is hard work. Doctors have to look through hundreds of 3D slices of the body, looking for tiny spots of "heat" that might be cancer.

• The Bottleneck: There aren't enough expert doctors to read all these scans.
• The AI Struggle: We have AI models that can try to find tumors automatically, but they are like "black boxes." They give an answer, but they can't explain why. Doctors are scared to use them because if the AI makes a mistake, no one knows why, and it might miss a tumor or flag a harmless spot as cancer.

👁️ The Solution: Recording the "Eye of the Expert"

The researchers built a special system to record where expert doctors look while they read these scans.

Think of it like recording a GPS trail of a master chef's eyes while they are cooking a complex meal.

1. The Setup: They took 346 real cancer scans.
2. The Actors: They had 13 experts (some senior doctors, some trainees) read these scans on a computer screen.
3. The Tech: The screen had a special eye-tracker (like a high-tech webcam for eyes) that recorded where the doctor was looking 60 times every second.
4. The Result: They captured 9,030 unique paths of eyes moving from one part of the body to a tumor. They also recorded the doctors' thoughts: "I'm sure this is cancer" vs. "I'm not sure, but I'll check it."

This dataset is called GazeXPErT. It's not just a list of tumors; it's a movie of how experts think and search.

---

🧪 What Did They Do With It? (The Experiments)

The team tested if teaching AI to "watch" like a human helps it get smarter.

1. The "Super-Segmentation" Test

• The Goal: Draw a perfect outline around a tumor.
• The Test: They trained an AI to draw the outline.
  • AI alone: Got it right about 60% of the time.
  • AI + Eye Tracking: When the AI was told, "Hey, look where the human doctor looked first," its accuracy jumped to 68%.
• The Lesson: Even though the doctors only looked directly at the tumor for a tiny fraction of the time, their search pattern (looking at the lungs, then the liver, then back to the lungs) contained secret clues that helped the AI find the tumor better.

2. The "Gaze Correction" Test

• The Problem: Eye trackers can be a little bit "wobbly" or inaccurate.
• The Fix: They taught an AI to predict where the doctor intended to look based on the wobbly eye data and the image.
• The Result: The AI could "fix" the shaky eye data, making the target much more precise. It's like a GPS that knows you meant to turn left at the gas station, even if your phone signal was glitching and said you were in the next town.

3. The "Mind Reading" Test

• The Goal: Can the AI tell if the doctor is looking for a tumor (intentional) or just scanning the room (unintentional)?
• The Result: The AI got it right about 67% of the time. It's not perfect yet, but it proves that we can start teaching computers to understand a doctor's intent just by watching their eyes.

---

🌟 Why This Matters (The "So What?")

Imagine a future where you are a doctor reading a scan, and an AI assistant is sitting next to you.

• Without GazeXPErT: The AI just shouts, "There's a tumor here!" and you have to trust it blindly.
• With GazeXPErT: The AI says, "I see you are looking at the liver. Based on how you are searching, I think you might be worried about this spot. Here is a zoomed-in view of that area, and here is why I think it's suspicious."

This makes the AI a partner rather than a replacement. It builds trust because the AI is using the same "visual reasoning" that human experts use.

🚀 The Bottom Line

GazeXPErT is a bridge between human intuition and machine speed.

• It captures the secret sauce of how experts find cancer.
• It turns that "secret sauce" into data that AI can learn from.
• The goal is to create AI that doesn't just calculate answers, but understands how doctors think, making cancer diagnosis faster, more accurate, and trustworthy.

In short: They taught the computer to follow the doctor's eyes, so the computer can learn to think like a doctor.

Technical Summary

1. Problem Statement

The integration of Artificial Intelligence (AI) into oncologic imaging, specifically [18F]FDG-PET/CT, faces significant barriers to clinical adoption. While deep learning models exist for tumor segmentation, they often lack:

• Interpretability and Explainability: Clinicians cannot easily understand why a model made a specific decision.
• Robustness: Models struggle with out-of-distribution data and real-world variability.
• Workflow Integration: Current AI tools often act as standalone modules that do not align with the natural visual reasoning and search patterns of human experts, potentially increasing cognitive load rather than reducing it.

Existing datasets for PET/CT lack the human visual reasoning component. They provide static images and ground truth labels but do not capture the dynamic process of how experts search for, verify, and interpret lesions.

2. Methodology

A. Dataset Construction (GazeXPErT)

The authors constructed a 4D multimodal dataset (3D spatial + time) capturing expert eye-tracking data during routine clinical reads.

• Data Source: 346 unique whole-body FDG-PET/CT scans (98 lymphoma, 122 lung cancer, 126 melanoma) sampled from a public dataset (Tubingen).
• Participants: 13 independent human experts, including board-certified specialists and trainees (residents/fellows) from institutions in the US, Germany, Korea, and Japan.
• Hardware & Software:
  • Eye Tracker: Tobii Pro Spark (60 Hz sampling rate).
  • Platform: A custom Python-based viewer simulating a routine clinical read.
  • Interaction: Readers could adjust windowing, scroll through 2D axial slices, switch modalities (PET, CT, Fused), and view Maximum Intensity Projections (MIP).
• Annotation Protocol:
  • Task: Identify metabolically active lesions and classify them as "certain" (definite tumor) or "uncertain" (indeterminate, requiring follow-up).
  • Gaze-Assisted Annotation: A semi-automated system proposed bounding boxes based on the reader's last gaze location and PET intensity thresholds. Readers used hotkeys to accept, reject, or resize these boxes.
  • Output: 3,948 minutes of raw 60Hz eye-tracking data, resulting in 9,030 unique gaze-to-lesion trajectories.
• Data Format: Data is rendered in COCO-style format (JSON/CSV) and 3D NIFTI volumes (heatmaps) to facilitate machine learning applications.

B. Validation Experiments

The authors conducted three experiments to validate the utility of the dataset:

1. AI Tumor Segmentation (nnUNet):

   • Goal: Determine if incorporating expert gaze patterns improves segmentation accuracy.
   • Method: A 3D nnUNet model was trained with and without an additional input channel representing a gaze heatmap (aggregated gaze trajectories).
   • Result: The model with gaze attention achieved a DICE score of 0.6819, significantly outperforming the baseline without gaze (0.6008).

2. Dynamic Gaze Correction (Vision Transformer):

   • Goal: Improve the accuracy of low-cost (60 Hz) eye-tracking devices by using AI to predict the "true" target based on sequential gaze and image context.
   • Method: A Vision Transformer (ViT) was trained to predict a gaze point closer to the lesion than the raw last gaze point, using a sequence of up to 16 gaze coordinates and the corresponding image slice.
   • Result: The model achieved 74.95% accuracy in predicting a point closer to the lesion than the raw input. It reduced the average gaze angle error from 1.134° to 0.856°, bringing dynamic tracking performance into the range of static targeting tasks.

3. Human Intention Prediction:

   • Goal: Distinguish between "intentional" targeting (selecting a lesion) and "unintentional" searching (scanning).
   • Method: A binary classification task using a ViT architecture with focal loss, trained on gaze trajectories and slice images.
   • Result: The model achieved an Accuracy of 67.53% and an AUROC of 0.747, demonstrating that gaze trajectories contain sufficient signal to predict clinical intent, though this remains a challenging task.

3. Key Contributions

• GazeXPErT Dataset: The first large-scale, machine-learning-ready dataset that embeds expert visual reasoning patterns (gaze trajectories) into oncologic PET/CT analysis.
• Uncertainty Modeling: The dataset explicitly captures expert uncertainty ("certain" vs. "uncertain" labels), a critical but often unmodeled aspect of clinical decision-making.
• Technical Framework: A novel pipeline for converting raw eye-tracking data into 3D heatmaps and COCO-style trajectories, enabling diverse ML tasks (segmentation, grounding, intention prediction).
• Performance Gains: Empirical evidence that integrating human gaze into AI training pipelines significantly improves segmentation accuracy and enables the prediction of human intent.

4. Results Summary

• Segmentation: Incorporating gaze heatmaps improved the DICE score by ~13.5% (0.6008 → 0.6819).
• Gaze Correction: AI post-processing reduced gaze error by ~24%, making affordable 60Hz trackers viable for dynamic clinical tasks.
• Intention Prediction: The model successfully differentiated between searching and targeting with ~67.5% accuracy, proving the feasibility of "human-in-the-loop" intent recognition.
• Annotation Efficiency: The gaze-assisted semi-automated approach allowed for a median annotation time of 191 seconds per study and 0.81 seconds per 3D lesion, proving scalability.

5. Significance and Future Impact

• Interpretable AI: GazeXPErT allows for the development of AI models whose attention mechanisms are grounded in real expert behavior, moving beyond post-hoc saliency maps to "by-design" explainability.
• Human-AI Collaboration: The dataset supports the development of adaptive AI assistants that can anticipate radiologist intent, flag overlooked findings, or calibrate assistance based on the user's confidence level.
• Clinical Translation: By addressing the "black box" problem and demonstrating performance gains through human-AI synergy, this work paves the way for regulatory acceptance and clinical deployment of AI in high-volume oncologic imaging.
• Open Science: The dataset and code are open-sourced via the Stanford AIMI Center and Kaggle, fostering further research in visual grounding, causal reasoning, and gaze-rewarded reinforcement learning.

In conclusion, GazeXPErT bridges the gap between static medical imaging data and the dynamic cognitive processes of human experts, providing a foundational resource for the next generation of trustworthy, explainable, and collaborative medical AI.