Technical Summary: EYEMULATOR

Problem Statement

Current Code Language Models (CodeLLMs) rely on "machine attention," a statistical mechanism that learns token importance through data correlations and treats context uniformly based on co-occurrence. In contrast, human developers employ distinct cognitive strategies characterized by selective attention; they fixate on semantically critical elements (e.g., control flow, method signatures) while skimming over syntactic boilerplate. Existing adaptation methods, such as instruction tuning or Retrieval-Augmented Generation (RAG), improve adherence to user intent or provide external context but fail to fundamentally alter the model's underlying attention mechanism. Consequently, state-of-the-art models often distribute attention uniformly across logic and boilerplate, missing the nuanced, selective focus that characterizes expert program comprehension.

Methodology

EYEMULATOR is a model-agnostic framework designed to inject human visual-attention priors into CodeLLM fine-tuning without requiring architectural changes or expensive pre-training from scratch. The approach operates in three stages:

1. Data Transformation and Artifact Distillation:
   Using the EyeTrans corpus (120Hz gaze recordings from 27 programmers reading/writing Java), the system identifies fixations (stable gaze points >100ms) and maps them to Abstract Syntax Tree (AST) leaf tokens. From this, two portable artifacts are distilled:

   • Semantic Salience Priors: A Bayesian approach (using Beta distributions) models the static importance of token types (e.g., variables vs. keywords) based on fixation density.
   • Sequential Transition Models: Bigram and trigram statistics capture the dynamic flow of expert reading (e.g., jumping from a variable definition to its usage).

2. Pseudo-Scan Path Generation:
   For standard training data lacking eye-tracking, EYEMULATOR synthesizes "pseudo-scan paths." It samples a saliency ratio and allocates attention quotas to token classes based on the distilled priors. These paths are constructed by greedily matching masked tokens into valid n-grams that satisfy line-span constraints, creating a training signal that mimics human visual selectivity.

3. Gaze-Informed Fine-Tuning:
   The model is fine-tuned using a composite loss function:

   • Weighted Supervised Fine-Tuning (SFT): Token-level cross-entropy loss is re-weighted. Weights combine a base importance term, an inverse frequency term (to upweight rare constructs), and the semantic salience probability derived from the priors.
   • Token-Level Direct Preference Optimization (DPO): The framework defines a "winning" trajectory as the generated pseudo-scan path (high salience) and a "losing" trajectory as the complement (low salience background). This loss explicitly rewards the model for assigning higher probability to semantically salient tokens prioritized by humans.

Key Contributions

1. A Portable Pipeline: A method for converting raw human gaze traces into reusable semantic salience and sequential transition priors over code tokens.
2. Model-Agnostic Objective: A fine-tuning objective that incorporates these priors through token-level weighting and preference alignment without modifying the base CodeLLM architecture.
3. Comprehensive Evaluation: An evaluation across six diverse backbones (ranging from 1B to 3B parameters), three CodeXGlue tasks (completion, translation, summarization), and two data regimes (low-data and full-data).

Results

The evaluation demonstrates that EYEMULATOR yields positive, quality-safe gains in all 36 evaluated model-task-setting cells:

• Structure-Preserving Tasks: The most significant improvements were observed in code completion and Java-to-C# translation. For instance, StarCoderBase-1B saw a +34.4% relative improvement in completion under full-data settings, and DeepSeek-Coder-1.3B achieved a +33.4% gain.
• Summarization: While more sensitive to decoding and tokenizer behavior, summarization tasks (Java-to-English) showed consistent positive gains under the METEOR metric after targeted tuning (e.g., +9.3% for Llama-3.2-1B).
• Robustness: The gains were consistent across different model sizes and data regimes, with seed-based robustness checks confirming stability.
• Component Analysis: Ablation studies confirmed that both semantic salience and sequential transition flow are complementary; removing either signal consistently reduced performance. Session-mode analysis indicated that reading-derived priors were particularly strong for translation, while writing-derived priors aided code completion.

Significance and Claims

The paper claims that EYEMULATOR successfully bridges the gap between statistical machine attention and selective human focus. By grounding CodeLLM training in cognitive visual patterns, the method enables models to process code with a semantic salience mirroring that of a developer.

The authors emphasize that the significance of this work lies in its ability to improve model performance through modular, model-agnostic signals rather than architectural overhauls. The results suggest that compact CodeLLMs benefit significantly when human selectivity is distilled into training-time priors. The paper posits that human attention is useful not as a single heuristic, but as a structured prior combining what developers inspect and how they move through code. This approach offers a pathway to enhance semantic grounding in CodeLLMs without the computational cost of training new architectures from scratch.