Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs

This paper introduces a method that enhances medical Vision-Language Models by using sequential eye-tracking data as supervision to train dedicated gaze tokens, enabling the models to mimic radiologists' visual search patterns and achieve state-of-the-art performance in both in-domain and out-of-domain medical reasoning tasks.

The Gist

Imagine you are trying to teach a super-smart robot how to read a chest X-ray.

The Old Way: The "Blind" Reader
Currently, most AI models look at an X-ray, take a quick glance, and then immediately start writing a report in plain English. It's like asking a student to look at a complex map for one second and then describe the route without ever pointing to the landmarks. They might get the general idea, but they often miss the tiny details or guess wrong because they aren't actually "looking" at the specific spots where the trouble is. They are just guessing based on the text they've read before.

The New Idea: "Thinking with Gaze"
This paper introduces a new way to train these robots. Instead of just letting them guess, the researchers taught the AI to think with its eyes, just like a human doctor does.

Here is the analogy:

• The Human Doctor: When a radiologist looks at an X-ray, they don't just stare at the whole picture at once. They scan it in a specific order. They might look at the top left, then zoom in on a shadow in the middle, then check the bottom right. Their eyes move in a sequence (a path) to gather clues one by one.
• The Eye-Tracker: The researchers used special glasses that recorded exactly where real doctors looked and in what order. This created a "treasure map" of the doctor's attention.

How the AI Learned (The "Gaze Tokens")
The researchers gave the AI a special set of "thinking tools" called Gaze Tokens. Think of these as four sticky notes the AI has to fill out before it writes its final report.

1. The Training: The AI was shown an X-ray and the "treasure map" of where the human doctor looked first, second, third, and fourth.
2. The Task: The AI had to use its four sticky notes to point to the exact spots on the image the doctor looked at, in the exact same order.
3. The Result: By forcing the AI to "point" to the evidence in the right order, it learned to simulate the human thought process. It stopped guessing and started gathering evidence step-by-step, just like a doctor.

Why This Matters

• Better Accuracy: Because the AI is actually looking at the right spots in the right order, it got much better at spotting diseases (like pneumonia or fractures) than previous models.
• The "Zero-Shot" Superpower: The best part? This training made the AI smarter even on X-rays it had never seen before. It's like teaching a student how to study (the method of looking) rather than just memorizing the answers. Because it learned the process of finding evidence, it could apply that skill to new, unfamiliar hospitals and different types of scans.
• Trust: Since the AI has to "show its work" by pointing to the specific spots it looked at, doctors can trust it more. They can see why the AI made a decision, rather than just taking its word for it.

In a Nutshell
This paper is about teaching AI to stop "guessing" and start "hunting" for clues. By mimicking the eye movements of expert doctors, the AI learned to think visually, leading to smarter, more reliable, and more trustworthy medical diagnoses.

Technical Summary

Here is a detailed technical summary of the paper "Thinking with Gaze: Sequential Eye-Tracking as Visual Reasoning Supervision for Medical VLMs."

1. Problem Statement

Current Vision-Language Models (VLMs) typically process images as visual tokens but perform intermediate reasoning primarily in text space. This "text-only" reasoning chain-of-thought is suboptimal for inherently visual tasks like medical radiology, where critical diagnostic signals are often difficult to verbalize without information loss.

In contrast, radiologists diagnose via a sequential visual search process, revisiting suspicious regions and integrating evidence over time. Existing methods often treat attention as a static spatial map, failing to capture the temporal dynamics of how experts acquire evidence. The paper addresses the challenge of how to inject this time-ordered, human-like reasoning process into VLMs to improve both accuracy and interpretability in medical imaging.

2. Methodology

The authors propose a framework that uses eye-tracking data as a supervision signal to guide a VLM's reasoning process. The core innovation is the introduction of dedicated "gaze tokens" that force the model to predict image patches in the same temporal order a radiologist would scan them.

Key Components:

• Dataset: The method utilizes MIMIC-EYE, a dataset containing chest X-rays synchronized with radiologists' eye-tracking data and transcribed speech.
• Data Preprocessing:
  • Temporal Alignment: Audio transcripts are aligned with eye-gaze signals at the sentence level to handle rapid speech and gaze dropouts (e.g., blinks).
  • Gaze to Patch Indices: Continuous gaze heatmaps are discretized into fixed image patch grids compatible with the VLM's visual tokenizer. These are converted into a sequence of patch indices representing the radiologist's scanpath.
• Model Architecture:
  • Backbone: A pretrained VLM (specifically Qwen2.5-VL-7B-Instruct).
  • Fixed-Format Generation: The model is instructed to output a strict format: four special placeholder tokens (<st>1 to <st>4) followed by the final diagnostic answer (14 binary findings).
  • Gaze Tokens: The four <st> tokens act as latent "gaze tokens." Their hidden states are supervised to predict the specific image patch indices the radiologist looked at during that step.
  • Heads:
    1. Gaze Projection Head: A linear layer mapping the hidden state of each <st> token to a probability distribution over image patch IDs (Cross-Entropy loss).
    2. Classifier Head: A linear layer on the final token to predict the 14 radiographic findings (Binary Cross-Entropy loss).
• Two-Stage Training Strategy:
  1. Stage 1 (Gaze-Supervised Token Learning): The model learns to map the four gaze tokens to the correct patch indices using Cross-Entropy loss. Only lightweight LoRA adapters and the projection head are trained to preserve the backbone's knowledge.
  2. Stage 2 (Multi-label Classification): A classifier head is added to predict the 14 diagnostic labels. The model is trained with a combined loss of language modeling and classification, while maintaining the gaze token predictions.

3. Key Contributions

1. Gaze-Guided Reasoning Supervision: The paper introduces a lightweight mechanism that uses temporally ordered eye-gaze to supervise a small set of dedicated tokens. This explicitly trains the VLM to mimic the radiologist's step-by-step evidence acquisition rather than relying on static spatial attention.
2. State-of-the-Art Performance with Interpretability: The approach achieves the best performance on the MIMIC-EYE benchmark while providing gaze-linked patch evidence. This allows for case-level auditing, showing clinicians exactly which image regions the model "looked at" to reach a conclusion.
3. Enhanced Out-of-Domain Robustness: By learning human-like evidence acquisition patterns (sequential reasoning) rather than dataset-specific shortcuts, the model demonstrates superior generalization to external, out-of-distribution benchmarks.

4. Experimental Results

The method was evaluated on the MIMIC-EYE test set and three external zero-shot benchmarks (CheXpert, RSNA, SIIM-ACR).

• In-Domain Performance (MIMIC-EYE):
  • The proposed Original-Gaze model achieved an AUROC of 90.17, outperforming the Supervised Fine-Tuning (SFT) baseline (87.60) and other variants like Shuffled-Gaze (88.51) and Random-Gaze (86.45).
  • This confirms that preserving the temporal order of gaze is critical; simply using gaze as a static heatmap or randomizing the order yields lower performance.
• Zero-Shot Generalization:
  • On external datasets, the Original-Gaze model consistently outperformed all baselines.
  • CheXpert: 62.45% Accuracy / 61.73% F1.
  • RSNA: 76.61% Accuracy / 53.73% F1.
  • SIIM-ACR: 64.07% Accuracy / 61.89% F1.
  • The results indicate that the model learns transferable visual reasoning cues that generalize well under distribution shifts.

5. Significance

This work represents a significant shift in medical VLM research by moving beyond "thinking in text" to "thinking with images" via gaze supervision.

• Cognitive Alignment: It bridges the gap between AI reasoning and human expert cognition, proving that the sequence of visual attention is a powerful supervision signal.
• Clinical Trust: By generating a traceable "thinking process" (the sequence of patches looked at), the model offers a new level of interpretability that aligns with clinical workflows, potentially aiding in retrospective review and auditing.
• Robustness: The findings suggest that teaching models how to look (sequential evidence gathering) is more effective for generalization than simply teaching them what to look for (static labels).