💻 computer science

Predicting Human Visual Attention on Words in Source Code

This paper introduces a novel computational model that significantly outperforms existing baselines and advanced AI systems in predicting human visual attention on source code by utilizing a specialized loss function to align neural network internal attention with empirical eye-tracking data across Java and C programming languages.

Original authors: Chia-Yi Su, Collin McMillan

Published 2026-07-17

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chia-Yi Su, Collin McMillan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to understand how a human brain works while they are reading a book. You can't just ask them, "What are you thinking about right now?" because they might not know, or they might lie. So, scientists use a special camera to watch their eyes. This is called eye-tracking. It's like having a tiny spotlight that follows where a person looks, showing us exactly which words they stare at, which ones they skip, and how long they pause. In the world of computer science, this is a big deal because software engineers spend their lives staring at source code—the text-based instructions that tell computers what to do. But code isn't just a story; it's a puzzle. Engineers don't read it like a novel; they jump back and forth, re-reading lines, and hunting for clues. Understanding where they look helps us build better tools, design clearer screens, and even teach computers to think more like humans.

Now, here is the tricky part: computers are usually really bad at guessing what humans are looking at. They might think a long, complicated word is the most important, while a human engineer knows the short, boring word next to it is actually the key to the whole program. This paper is about teaching a computer to stop guessing and start "seeing" like a human programmer. The researchers built a special kind of computer brain (a model) that learned from watching real humans read code. They didn't just tell the computer, "Here is the code, guess the next word." Instead, they gave it a secret cheat sheet: a map of where human eyes actually landed. They taught the computer to compare its own internal "gaze" with the human's real gaze, adjusting its thinking until the two matched up.

The main discovery is that this new computer model is surprisingly good at mimicking human attention. When the researchers tested it against three different groups of programmers (some reading Java code, some reading C code), the computer's predictions were much closer to what humans actually did than any previous computer model. In fact, on some tests, the computer was 64% and even 467% better at guessing where a human would look than the old standard methods. It didn't just get the numbers right; it learned the pattern of human thought.

But the researchers didn't stop there. They wanted to see if this "human-like" attention could help the computer do a harder job: predicting the exact sequence of eye movements, known as a scanpath. Imagine trying to predict the exact path a hiker will take through a forest, step by step. The computer, trained with human eye data, was able to predict these paths better than other famous AI models (like GPT-5 and Claude) and better than the previous best computer program on the reading task. While it also showed improvements on the writing task, the researchers noted that these specific gains were not statistically significant compared to the previous best computer program, though they still outperformed the commercial models. This suggests that when we teach computers to pay attention the way humans do, they don't just become better at reading; they start to understand the process of thinking itself.

However, the authors are careful not to say they have solved the mystery of the human mind. They admit their model is an "estimator" of what a group of people usually does, not a perfect copy of any single person's brain. It also has limits; it works best on smaller chunks of code and might get confused if the code is too huge to fit in its memory at once. But the results are a strong hint that by watching how our eyes move, we can teach machines to see the world a little more like we do.

Technical Summary: Predicting Human Visual Attention on Words in Source Code

Problem Statement
Understanding human visual attention during software engineering tasks is critical for optimizing user interface design, improving software engineering tools, and modeling human cognitive processes. While eye-tracking data has been used to study how programmers read source code, predicting this attention computationally remains challenging. Existing approaches fall into two categories: general computer vision models, which treat attention as spatial coordinates and ignore semantic information in code; and text-reading models, which assume eye-scan patterns not applicable to the complex, regression-heavy nature of code reading. Furthermore, while Large Language Models (LLMs) exhibit attention mechanisms, it has not been established whether their internal attention aligns with human visual attention during specific, task-oriented code reading. The gap this paper addresses is the lack of a computational model that predicts fine-grained visual attention (specific words in source code) by leveraging eye-tracking data to align machine attention with human cognitive patterns.

Methodology
The authors propose a computational model based on a pre-trained Transformer architecture (specifically the jam model, a 350M parameter GPT-2 style model) that is fine-tuned using actual eye-tracking data. The methodology consists of three primary components:

Prediction Target: The model predicts the Percent Total Gaze Time (ptgt) for each word in a source code snippet. Ptgt is defined as the total gaze time on a specific word (Region of Interest) divided by the total gaze time for the entire code artifact. This metric quantifies the relative importance or attention allocated to a specific token.
Input Representation: The input sequence comprises three parts: the source code tokens, the fixation tokens (words where the human eye stopped), and the tokens immediately adjacent to the fixation tokens.
Custom Loss Function: The core innovation is a novel loss function designed to align the model's internal attention with human attention during fine-tuning. The process involves:
- Attention Readout: Extracting attention matrices ( $Q$ and $K$ ) from the final Transformer layer (specifically averaging heads 10–16, based on prior findings that deeper layers align better with human attention).
- Machine Attention Score: Calculating a scalar attention score ( $\hat{a}$ ) by averaging attention probabilities between tokens adjacent to fixations and the fixation tokens themselves.
- Human Supervision: Computing the similarity between the machine's attention score ( $\hat{a}$ ) and the ground-truth human ptgt ( $a$ ).
- Loss Composition: The total loss ( $L$ ) combines the standard Categorical Cross-Entropy (CCE) loss for language modeling with two auxiliary terms: Mean Squared Error (MSE) to enforce absolute accuracy in gaze time magnitude, and a transformed Pearson Correlation loss ( $L_{corr}$ ) to enforce the relative structure of attention (ranking high vs. low attention tokens). The loss is dynamically rescaled to prevent the auxiliary terms from dominating the language modeling objective.

Key Contributions

Novel Loss Function: The design of a loss function that explicitly minimizes the divergence between human visual attention (ptgt) and the internal self-attention of a neural network, utilizing both MSE and Pearson correlation.
Fine-Grained Prediction: Moving beyond coarse-grained biometric or task-difficulty predictions to predict attention at the level of individual words (tokens) in source code.
Ablation Study: A systematic evaluation demonstrating the specific contribution of the Pearson correlation component to the model's performance.
Scanpath Prediction Extension: Applying the attention-aligned model to the task of predicting the sequence of fixations (scanpath), a task requiring the understanding of human thought processes.

Experimental Results
The model was evaluated on three distinct datasets involving Java and C programming languages and tasks including code documentation and bug localization (Smith, Wallace, and Rodeghero studies).

Attention Prediction (RQ1): The proposed model significantly outperformed baselines from software engineering (Bansal et al.) and computer vision (Tafasca et al.).
- In the Wallace study (Java), the model achieved a Pearson correlation of 0.45, compared to 0.27 and 0.21 for the baselines (a 64% improvement over the best baseline).
- In the Rodeghero study (Java), it achieved 0.48 vs. 0.41 and 0.42 (16% improvement).
- In the Smith study (C), it achieved 0.25 vs. 0.04 and 0.04 (467% improvement).
- The model's similarity to the aggregated human reference was comparable to, and in some metrics exceeded, the similarity between individual human participants, suggesting the model captures the "consensus" attention structure obscured by individual variability.
Ablation Study (RQ2): Removing the Pearson correlation loss resulted in lower performance across all metrics (Pearson, Spearman, Kendall, Cosine), confirming that the correlation term is essential for learning the correct ranking of attention.
Scanpath Prediction (RQ3): Using the Bansal et al. dataset for code summarization, the model outperformed the original Bansal baseline and commercial models (GPT-5 Nano and Claude-Haiku-4.5) in Normalized Levenshtein Distance (NLD), NDCG, Precision, and Recall. Notably, the model predicted scanpaths that aligned more closely with human programmers (e.g., focusing on method bodies rather than just signatures) compared to commercial models which exhibited "alien" thought processes.

Significance and Claims
The paper claims that a Transformer-based language model, when fine-tuned with a custom loss function incorporating human eye-tracking data, can effectively predict aggregate human visual attention on source code words. The authors emphasize that the model acts as an estimator of aggregate attentional structure rather than a surrogate for a specific human.

Key claims include:

Alignment: Machine attention, when guided by human data, can mimic the distribution of human attention on code words with a level of accuracy comparable to inter-human agreement.
Task Understanding: The model's improved performance on scanpath prediction suggests that aligning internal attention with human visual patterns enhances the model's capability to handle tasks requiring an understanding of human thought processes (e.g., code summarization).
Limitations: The authors modestly note that the model aggregates attention over time, losing temporal dynamics and regression patterns. They also caution that the model is not a complete account of human attention mechanisms and that results may not generalize to tasks requiring context windows larger than the model's capacity (1,024 tokens) or to languages/tasks not represented in the training data.

The work positions itself as a step forward in computational modeling of programmer cognition, offering a tool to better understand what information programmers need and how to design interfaces that align with human cognitive processes.

More like this