Technical Summary: DeepGaze3.5-VL

Problem Statement

Modeling human visual attention over time (scanpath prediction) has historically relied on specialized architectures incorporating hand-crafted geometric priors, such as constrained Lévy exploration, saccadic biases, or inhibitory tagging. While these models successfully capture fixation sequences, their rigid structural biases limit extendability and flexible conditioning. Integrating task-specific instructions or adapting to distinct viewer identities traditionally requires custom, disjoint architectural additions. Furthermore, existing generative approaches (e.g., diffusion models, neural point processes) often train from scratch on gaze data alone, failing to leverage the rich sequential priors available in foundation models. A critical bottleneck remains in evaluation: continuous generative models cannot natively compute exact spatial probability densities, forcing reliance on heuristic sequence-alignment metrics (e.g., ScanMatch) rather than rigorous information-theoretic measures.

Methodology

The authors propose DeepGaze3.5-VL, a framework that reframes scanpath prediction purely as a discrete sequence modeling task using Large Vision-Language Models (LVLMs).

Core Formulation

• Autoregressive Token Prediction: Instead of regressing continuous coordinates, the model predicts gaze as a sequence of discrete tokens. Spatial coordinates $(x, y)$ are mapped to zero-padded two-digit integers (00–99) representing percentage positions along image dimensions. This ensures every fixation contributes exactly four tokens (two for $x$, two for $y$), guaranteeing uniform token length and fair probabilistic comparison.
• LVLM Integration: The system utilizes pre-trained LVLMs (specifically InternVL3.5-8B) fine-tuned via Low-Rank Adaptation (LoRA). The model encodes the input image $I$ and optional conditioning factors $C$ (e.g., task descriptions, subject IDs) via natural language prompts.
• Training Objective: The model is trained via standard next-token prediction on ground-truth scanpaths. The autoregressive structure allows for the exact computation of the log-likelihood of any valid fixation sequence:
  $$\log P(S \mid I, C) = \sum_{i=1}^{n} \log P((x_i, y_i) \mid I, C, (x_1, y_1), \ldots, (x_{i-1}, y_{i-1}))$$

Evaluation Strategy

• Information Gain (IG): The authors leverage the autoregressive alignment to compute exact Information Gain, defined as the improvement in log-likelihood over a center-bias baseline. Unlike continuous models, this formulation allows for the exact calculation of spatial probability densities without heuristics.
• Efficient Computation: To compute full spatial distributions (e.g., for AUC) across a $100 \times 100$ grid, the authors employ a prefix tree strategy. By caching Key-Value states at shared prefixes, they decompose the joint probability into a 4-phase expansion (marginal $x_1$, conditional $x_2$, conditional $y_1$, conditional $y_2$), requiring only 1,111 forward passes instead of 10,000. For IG specifically, only 4 forward passes per fixation are needed.

Key Contributions

1. Formulation as Autoregressive Token Prediction: By fine-tuning LVLMs to predict discrete coordinate tokens, the authors leverage pre-trained semantic and sequential priors. This approach achieves a state-of-the-art 2.18 bits of Information Gain on the MIT1003 dataset, a 46% improvement over the matched-backbone DeepGaze III baseline.
2. Exact and Efficient Evaluation: The framework exploits the autoregressive structure to decompose full spatial joint probabilities, enabling mathematically exact and rapid Information Gain computation at scale, resolving a major evaluation bottleneck for generative scanpath models.
3. Flexible Conditioning and Generalization: The model naturally absorbs diverse factors of variation through natural language prompting. It demonstrates strong out-of-distribution generalization (Leave-One-Dataset-Out) and can be conditioned on task objectives (free-viewing vs. visual search) or individual observer identities without architectural changes.
4. In-silico Intervention: The generative framework serves as a computational tool for controlled behavioral interventions, allowing for the simulation of counterfactual scenarios (e.g., varying fixation durations) that are difficult to conduct experimentally.

Results

• Performance: DeepGaze3.5-VL establishes new state-of-the-art performance across five diverse datasets (MIT1003, COCO-FreeView, CAT2000, DAEMONS, FiGrIm). On MIT1003, it achieves 2.18 bits IG compared to 1.49 bits for DeepGaze III (retrained with the same InternViT-6B backbone).
• Role of Representations: The performance gain is not solely due to better visual features. A control experiment showed that DeepGaze III with the superior InternViT-6B encoder still underperformed the VLM. Furthermore, a strong correlation ($\rho = 0.93$) was observed between Information Gain and the MMMU benchmark score (a measure of multimodal reasoning), suggesting that high-level semantic understanding drives scanpath prediction quality.
• Sequential vs. Spatial: Ablation studies reveal that while static spatial saliency drives early fixations, sequential dependencies (the history of the specific observer) become the dominant predictor for later fixations. The full model outperforms "history-invariant" and "spatial-only" variants, particularly in the later stages of a scanpath.
• Conditioning Effects:
  • Duration: Conditioning on fixation duration improves spatial prediction, particularly for short fixations (<100 ms), recovering known oculomotor phenomena like pre-programmed saccades.
  • Subject Identity: Providing subject IDs improves IG by +0.09 bits/fixation, allowing the model to learn distinct spatial priors per observer.

Significance and Claims

The paper claims that framing scanpath prediction as autoregressive token prediction bridges the gap between specialized gaze modeling and general foundation models. The authors assert that:

• Semantic Understanding is Key: The quality of learned multimodal representations (measured by reasoning benchmarks) is a primary driver of scanpath prediction, supporting cognitive theories that top-down task goals and semantic understanding drive visual exploration.
• Native Evaluation: The autoregressive formulation aligns scanpath prediction with the native training objective of foundation models, allowing Information Gain to be a direct manifestation of the next-token prediction objective, eliminating the need for surrogate metrics.
• Generative Utility: Beyond prediction, the framework acts as a powerful tool for "in-silico" behavioral interventions. The authors demonstrate that the model can recover known oculomotor phenomena (such as the modulation of spatial entropy by fixation duration) purely from data statistics, enabling controlled simulations without additional data collection.

The work suggests that the rigid, hand-crafted priors of previous generations are no longer necessary, as the sequential and semantic priors inherent in LVLMs are sufficient to model complex human gaze dynamics with greater flexibility and accuracy.