🤖 AI

A World Model of Radiologist Reading for Medical Image Representation Learning

GazeWorld is a medical imaging world model that leverages radiologist eye-tracking data to autoregressively predict latent image representations, achieving state-of-the-art diagnostic accuracy and zero-shot performance by learning how experts read images rather than just what they conclude.

Original authors: Yiwei Li, Zihao Wu, Huaqin Zhao, Yifan Zhou, Chao Cao, Dajiang Zhu, Tianming Liu, Lin Zhao

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yiwei Li, Zihao Wu, Huaqin Zhao, Yifan Zhou, Chao Cao, Dajiang Zhu, Tianming Liu, Lin Zhao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer how to read an X-ray. Usually, we show the computer thousands of pictures and tell it, "This one has pneumonia, this one is clear." But this is like trying to teach someone to drive by only showing them the destination, without ever explaining how they got there. The computer might get lucky and guess right, but it doesn't actually understand the process of finding the problem.

This paper introduces a new way to teach computers, called GazeWorld. Instead of just looking at the final answer, GazeWorld tries to understand how a human expert (a radiologist) looks at the picture.

Here is the breakdown using simple analogies:

1. The Problem: The "Black Box" and the "Data Shortage"

Medical AI has two big headaches:

Not enough data: It's hard to get millions of labeled X-rays because patient privacy laws are strict and doctors are busy.
The "Black Box": Current AI models often just spit out a diagnosis ("It's pneumonia!") without showing their work. Doctors can't trust a machine if they don't know why it made that decision.

2. The Secret Ingredient: Eye-Tracking

Radiologists don't just stare at an X-ray randomly. They have a specific way of looking at it. They might look at the top left, then jump to the bottom right, then zoom in on a specific spot. This is called a scanpath.

Old way: Previous AI models treated this eye movement like a static "heat map" (a blurry red blob showing where the doctor looked). This throws away the order of the look. It's like knowing a detective visited a crime scene but not knowing the order in which they found the clues.
New way (GazeWorld): This model treats the X-ray as a world and the doctor's eye movements as a journey through that world.

3. How GazeWorld Works: The "Storyteller" and the "Imaginator"

The model learns in two simultaneous ways, like a student who is both reading a story and imagining the missing pages:

The Storyteller (Next-Fixation Prediction):
Imagine you are reading a book, but you only see the first few sentences. GazeWorld tries to guess what the next sentence will be based on the ones you've already read.
- In the AI's case, it looks at the first patch of the X-ray the doctor looked at, then the second, and tries to predict the latent representation (the "meaning") of the next patch the doctor will look at.
- By doing this, it learns the logic of the doctor's search. It learns that "If I see a shadow here, the next logical place to look is there."
The Imaginator (Spatial Completion):
What about the parts of the X-ray the doctor didn't look at? Maybe they skipped them because they looked normal, or because they were far away from the problem.
- GazeWorld has a second branch that acts like an imaginative artist. It takes the "story" of where the doctor did look and tries to "fill in the blanks" for the parts they skipped.
- It asks: "Based on the evidence the doctor gathered, what does this empty, unvisited corner of the image likely contain?"

4. The Result: A Smarter, More Trustworthy AI

The authors tested this on three major chest X-ray datasets (CheXpert, RSNA, and SIIM-ACR). Here is what happened:

Better Diagnosis: When they used the "frozen" (pre-trained) GazeWorld features to diagnose diseases, it beat every other existing method, even when they only gave it a tiny amount of labeled data (1% or 10%).
Zero-Shot Success: It performed the best even when it had to guess on new diseases without any specific training for them.
Better Eye-Tracking Prediction: They tested if the model could predict where a human would look next. Even though GazeWorld wasn't explicitly trained to predict eye movements, its internal "brain" was so good at understanding the reading process that a simple decoder could predict the doctor's eye path better than specialized models built just for that task.
Visual Proof: When they visualized where the AI was "looking" (using heatmaps), it focused much more tightly on the actual medical problems (like pneumonia spots) compared to other models, which were often scattered and confused.

The Bottom Line

GazeWorld doesn't just learn what a disease looks like; it learns how to look for it. By mimicking the step-by-step journey of a radiologist's eyes, it builds a smarter, more efficient, and more interpretable understanding of medical images. It proves that teaching AI how experts think is just as important as teaching them what they conclude.

Technical Summary: GazeWorld – A World Model of Radiologist Reading for Medical Image Representation Learning

1. Problem Statement

The clinical translation of medical imaging AI faces two primary challenges: the scarcity of large-scale, diverse labeled datasets due to privacy regulations and the high cost of expert annotation, and the lack of interpretability in current models, which often emit diagnostic labels without exposing the underlying evidence. While radiologist eye-tracking data offers a promising solution by encoding expert prior knowledge and the temporal sequence of evidence gathering, existing methods utilize this signal only partially. Current approaches typically treat gaze as a static spatial prior (e.g., heatmaps) or as an auxiliary prediction target decoupled from the diagnostic reasoning process. Consequently, the temporal structure of how experts search, compare, and accumulate evidence remains underutilized for improving diagnostic representations.

2. Methodology: GazeWorld

The authors propose GazeWorld, a medical imaging world model that reframes the radiologist's diagnostic process as a trajectory through a perceptual world defined by the image. Instead of treating gaze as a static map, GazeWorld models the reading process as an autoregressive sequence where the radiologist navigates the image, extracting local evidence at each fixation to inform the next region of interest.

Core Architecture

GazeWorld consists of three main components (illustrated in Figure 1 of the paper):

Patch Representations and Fixation Sequence:
- An online visual encoder ( $\phi$ ) maps a medical image $X$ into a set of patch-level semantic representations $Z$ .
- Radiologist fixation sequences $F$ are mapped to unique, ordered patch indices $S = (s_1, \dots, s_L)$ , preserving the temporal order of visits while merging repeated visits to the same patch.
Next-Fixation Prediction Branch (Autoregressive):
- Following the Joint-Embedding Predictive Architecture (JEPA) paradigm, the model predicts the latent representation of the next fixated patch based on all previously visited patches.
- A fixation embedder ( $\psi$ ) combines the visual token of a visited patch with spatial embeddings, temporal rank, and dwell duration.
- An autoregressive predictor ( $g_\theta$ ), implemented as a causal Transformer, processes the prefix of fixation tokens to predict the representation of the next patch ( $\hat{z}_{s_i}$ ).
- The prediction target is derived from a momentum target encoder ( $\bar{\phi}$ ), an exponential moving average (EMA) copy of the online encoder, ensuring stable representation space targets without backpropagating gradients through the target.
Spatial-Completion Branch:
- To address unvisited regions, which may be clinically informative (e.g., skipped because they are normal), a spatial-completion branch predicts representations for all unvisited patches in parallel.
- A lightweight cross-attention decoder uses the full gaze-context representation (output of the autoregressive predictor) to infer the representations of unvisited patches.

Pretraining Objective

The model is pretrained using a combined loss function:
$L_{pre} = L_{AR} + \lambda L_{SC}$
Where $L_{AR}$ is the autoregressive loss for next-fixation prediction and $L_{SC}$ is the spatial-completion loss for unvisited regions. This objective is self-supervised, requiring no diagnostic labels, only image-text pairs and gaze trajectories.

3. Key Contributions

The paper outlines three primary contributions:

Gaze World Model Formulation: The authors introduce a novel formulation of the radiologist's reading process as a trajectory through a perceptual world. This approach fully exploits the temporal structure of expert gaze as a self-supervised signal, moving beyond static heatmaps and decoupled scanpath prediction.
GazeWorld Architecture: They instantiate this formulation as GazeWorld, which uniquely combines gaze-ordered autoregressive prediction in latent space with a spatial-completion branch. This shapes the representation to capture both diagnostic content and the sequential structure of expert reasoning.
Dual Utility of Representations: The learned representations demonstrate that a single model can support both accurate diagnosis and interpretable scanpath decoding. A generic decoder trained on frozen GazeWorld features outperforms purpose-built scanpath predictors, indicating that the representation inherently encodes the sequential structure of expert visual search.

4. Experimental Results

The authors evaluated GazeWorld on chest radiograph diagnosis and scanpath prediction across three benchmarks: CheXpert, RSNA Pneumonia, and SIIM-ACR Pneumothorax.

Diagnostic Performance

Supervised Settings: Frozen GazeWorld features achieved state-of-the-art (SOTA) AUROC across all nine supervised settings (three datasets $\times$ three label fractions: 1%, 10%, 100%). Notably, under a controlled comparison with EGMA (matching backbone, initialization, and data), GazeWorld improved CheXpert 1% AUROC from 71.21 to 78.37.
Zero-Shot Performance: GazeWorld achieved the highest zero-shot accuracy on all three benchmarks and the highest F1 scores on CheXpert and SIIM-ACR, outperforming models like MedCLIP, EGMA, and GLoRIA.
Qualitative Analysis: Grad-CAM visualizations indicated that GazeWorld produces more concentrated activations over clinically relevant anatomy compared to baseline models.

Scanpath Prediction

Probing Experiment: A generic decoder trained on frozen GazeWorld features was evaluated on the GazeSearch benchmark (ChestSearch protocol).
Performance: Despite not being explicitly trained to predict gaze, the GazeWorld features enabled a decoder to outperform the purpose-built LogitGaze-Med by over 16% in ScanMatch and 22% in SED (Scanpath Euclidean Distance).
Implication: This suggests that the world-model pretraining successfully encodes the sequential structure of expert reading into the visual representation.

Ablation Studies

Ordering: Replacing expert scanpaths with raster-scan or random orders significantly degraded performance, confirming that the chronological order of evidence gathering is critical.
Components: The autoregressive (AR) component alone provided strong performance, but adding spatial completion further improved results. Spatial completion alone was weaker, suggesting it relies on the autoregressive context to be effective.
Backbones: The method consistently improved zero-shot performance when applied on top of different medical visual backbones (MGCA, EGMA, MedCLIP), indicating its generalizability.

5. Significance and Claims

The paper claims that GazeWorld demonstrates a promising pretraining paradigm for medical imaging AI by modeling how experts read, not just what they conclude. By treating the image as a world and the fixation sequence as a trajectory, the model learns to predict the next step in diagnostic reasoning.

The authors assert that this approach:

Reduces reliance on large labeled datasets by leveraging the rich, self-supervised signal of eye-tracking data.
Provides a basis for interpretability that static attention maps cannot offer, as the learned representations retain the sequential structure of expert reasoning.
Offers a complementary refinement objective for existing medical foundation models, injecting radiologist reading dynamics into pretrained encoders without requiring a complete retraining from scratch.

The study is modestly limited to chest radiographs and a single eye-tracking corpus (MIMIC-EYE), with the authors noting that future work should explore larger multi-reader datasets and broader imaging modalities.