Evaluating Visual Prompts with Eye-Tracking Data for MLLM-Based Human Activity Recognition

1. Problem Statement

The paper addresses the challenges of integrating Large Language Models (LLMs) and Multimodal LLMs (MLLMs) into Internet of Things (IoT) applications, specifically Human Activity Recognition (HAR) using high-frequency, multi-dimensional sensor data like eye-tracking.

• Token Inefficiency: Directly feeding raw numerical time-series data (e.g., x,y coordinates at 30–1000 Hz) into LLMs results in an explosion of token counts, leading to prohibitive computational costs.
• Information Loss: LLM tokenizers are designed for natural language, not continuous numerical sequences. This leads to the loss of temporal correlations and the "lost in the middle" problem, where critical patterns in long sequences are overlooked.
• Limitations of Current Approaches: While specialized encoders or training foundation models on time-series data exist, they often require extensive retraining. The authors seek a training-free alternative that leverages the existing world knowledge and visual reasoning capabilities of MLLMs.

2. Methodology

The study proposes and systematically evaluates a Visual Prompting strategy, where eye-tracking sensor data is transformed into visualization images before being fed into an MLLM.

A. Visualizations Designed

The authors designed six distinct visualizations categorized into three types, comparing Raw Data vs. Feature-Based Data (extracted using the I-DT algorithm for fixations and saccades):

1. Timeline:
   • Raw: X and Y coordinates plotted as distinct colored lines over time.
   • Feature-based: Fixations shown as thick solid lines; saccades as dashed lines.
2. Heatmap:
   • Raw: Absolute gaze points overlaid on a grid.
   • Feature-based: Fixation counts or durations encoded via Gaussian interpolation.
3. Scanpath:
   • Raw: Gaze trajectories rendered as nodes (fixations) and paths (saccades).
   • Feature-based: Nodes sized by fixation duration with a temporal color gradient (light to dark blue).

B. Datasets

The approach was evaluated on three public eye-tracking datasets with varying characteristics:

• GazeBase: High-frequency (1000 Hz), screen-based, 6 activity classes (e.g., Reading, Video Viewing).
• SedentaryActivity: 30 Hz, screen-based, 8 complex computing tasks (e.g., Debug, Write).
• DesktopActivity: 30 Hz, wearable (egocentric) device, 6 activities.

C. Experimental Setup

• Model: OpenAI's gpt-5.1 (state-of-the-art MLLM).
• Settings: Zero-shot and One-shot learning.
• Variables:
  • Visualization type (6 variants).
  • Temporal window sizes (10s to 100s in 10s increments for specific datasets).
  • Baselines: Raw text (CSV coordinates) and Feature text (structured symbolic representation of fixations/saccades).
• Task: The model was prompted to classify the activity based on the visual or text input and provide a reasoning explanation.

3. Key Contributions

1. Systematic Evaluation of Visual Prompts: The first comprehensive study evaluating how different visualization techniques (Timeline, Heatmap, Scanpath) and feature extraction methods impact MLLM performance in HAR.
2. Token Efficiency Demonstration: Quantitative proof that visual prompting drastically reduces token consumption compared to textual representations of the same data.
3. Window Size Analysis: Investigation into how temporal context length affects recognition accuracy, revealing that optimal window sizes vary by dataset complexity.
4. Task-Specific Visualization Insights: Identification that no single visualization is universally superior; effectiveness depends on the specific behavioral patterns of the activity (e.g., spatial density vs. temporal sequence).

4. Key Results

A. Accuracy Performance

• Visual vs. Text: Visual prompts generally outperformed raw text baselines. In the GazeBase dataset (10s window), visual prompts (Heatmap/Timeline) achieved 0.811 accuracy, significantly beating raw text (0.539).
• Feature Text vs. Visual: Interestingly, Feature Text (structured symbolic representation) performed comparably to visual prompts in many cases, suggesting that explicit feature extraction is effective even without images. However, visual prompts often matched or exceeded feature text without requiring complex text formatting.
• Dataset Variance:
  • GazeBase: High accuracy (up to 0.811) with 10s windows.
  • SedentaryActivity: Lower accuracy (max ~0.53), requiring larger windows (60s+) to capture complex behaviors.
  • DesktopActivity: Lowest accuracy (~0.3), likely due to the noise introduced by egocentric wearable sensors and lack of screen context.
• Visualization Specifics:
  • Heatmaps generally performed best for SedentaryActivity (spatial attention distribution).
  • Scanpaths performed better for Reading tasks (temporal sequence importance).
  • Timelines showed peak performance at specific window sizes (e.g., 70s) but declined with overcrowding.

B. Token Efficiency & Scalability

• Massive Reduction: Visual prompts consumed significantly fewer tokens than text.
  • Raw Text: Required 2.3× to 21.8× more tokens than visual prompts for a 10s window.
  • Feature Text: Required 1.46× to 2.2× more tokens.
• Scalability: As window size increased (up to 100s), token usage for textual prompts scaled linearly (e.g., 83,901 tokens for 100s), whereas visual prompts remained constant (e.g., ~4,061 tokens) because the image resolution was fixed. This makes visual prompting highly scalable for long-duration sensor data.

5. Significance and Implications

• Training-Free IoT: The study validates a training-free approach for IoT, bypassing the need for massive labeled datasets and specialized model training.
• Bridging Sensor Data and Foundation Models: It establishes data visualization as a viable "intermediate representation" that allows MLLMs to reason over high-frequency sensor signals effectively.
• Context-Aware Agent Design: The findings suggest that autonomous agents in ubiquitous computing should dynamically select visualization types based on the target activity (e.g., using Heatmaps for spatial tasks and Scanpaths for sequential tasks).
• Cost-Effectiveness: By reducing token costs by an order of magnitude, visual prompting makes real-time, long-duration HAR feasible on commercial MLLM APIs.

Limitations & Future Work

• Accuracy Gap: While efficient, visual prompting accuracy is still lower than conventional, task-specific deep learning models.
• Dataset Specificity: Performance varied significantly across datasets, particularly failing on egocentric wearable data, indicating a need for better adaptation to different sensor modalities.
• Closed-Source Models: The study relied on a specific commercial MLLM, limiting reproducibility compared to open-source alternatives.
• Future Direction: The authors propose exploring real-world open-ended scenarios and combining multiple visualization techniques to further enhance reasoning capabilities.