Technical Summary: Known By Their Actions: Fingerprinting LLM Browser Agents via UI Traces

1. Problem Formulation

As Large Language Model (LLM) agents transition from research prototypes to production systems that browse the web on behalf of users, a critical security question arises: can websites passively identify the specific underlying model powering an agent? Current bot detection typically frames the problem as a binary classification (human vs. automated). However, as scripted automation is displaced by LLM agents, the threat landscape shifts. In classical security frameworks, target identification is the prerequisite for exploitation. If an adversary can determine which model is visiting a page, they can tailor attacks—such as selecting specific jailbreaks from a known corpus or reducing the search space for white-box adversarial attacks—based on known model-specific vulnerabilities.

This paper investigates whether the underlying foundation model of a browser agent can be inferred solely from passive, in-page User Interface (UI) traces. The authors posit that an agent's actions (clicks, scrolls, keypresses) and interaction timings encode a unique behavioral fingerprint distinct to the model, independent of browser headers or network-layer metadata which can be spoofed.

2. Methodology

Experimental Setup

The study evaluates 14 frontier multimodal LLMs, including proprietary models (GPT-5.4, Claude Opus 4.6, Gemini-3.1/Flash) and open-source models (Qwen3/3.5-VL series, GLM-4.6V, Gemma-4, UI-TARS-1.5, Seed-2.0-Lite).

• Environment: Agents operate within a standardized browser harness (Midscene.js) that translates model decisions into Playwright commands. This ensures that behavioral differences arise from the model itself rather than the execution environment.
• Tasks: Agents perform tasks across four live web environments: two information retrieval benchmarks (2WikiMultiHop, FRAMES on Wikipedia) and two shopping benchmarks (Webshop, Deepshop on Amazon).
• Data Collection: A lightweight JavaScript tracker is injected into the target pages. It passively logs client-side interaction events with millisecond resolution, capturing:
  • Action types (click, scroll, keypress, navigation).
  • Spatial coordinates (click X/Y, scroll depth).
  • Temporal dynamics (inter-event intervals, time to first action).
  • Structural patterns (link click ratios, key types).
• Trace Format: A session generates a trace $\tau = \{(s_t, u_t, \Delta t_t)\}_{t=1}^T$, where $s_t$ is the observation, $u_t$ is the action, and $\Delta t_t$ is the time elapsed since the previous action.

Classification Approach

The problem is formalized as a supervised classification task to predict the originating model $\hat{m} = f(\tau)$.

• Feature Extraction: 41 scalar features are extracted, categorized into temporal dynamics (e.g., inter-event interval statistics), scrolling behavior, click spatial distribution, and navigation volume.
• Models: Five classifier families are trained: Lasso Regression, Logistic Regression, Random Forest, XGBoost, and an LSTM network. XGBoost is highlighted as the primary performer.
• Evaluation Protocols:
  • Closed-Set Fingerprinting: The classifier is trained and tested on a fixed set of 14 known agents.
  • Open-Set Fingerprinting: A "leave-one-agent-out" (LOO) protocol is used to test the ability to distinguish known agents from an unknown agent (flagging the trace as unknown).

3. Key Results

High Accuracy in Closed-Set Identification

The study demonstrates that agents are highly identifiable from their action traces.

• Performance: The XGBoost classifier achieves Macro F1 scores up to 96% across the 14 models.
• Consistency: High performance is observed across all four datasets (2WikiMultiHop, FRAMES, Webshop, Deepshop). For instance, Seed-2-lite achieved 96.1% F1 on 2WikiMultiHop, and UI-TARS-1.5 achieved 92.1% on Webshop.
• Efficiency: Strong classifiers can be trained using fewer than one-third of the available training traces. Furthermore, model identity can be inferred early in an episode; Macro F1 rises sharply within the first 40% of observed actions, plateauing near full-trace performance.

Open-Set Detection

While open-set identification (distinguishing known from unknown) is more challenging, it remains feasible.

• Most agents achieve an AUROC > 0.60 in distinguishing known from unknown traces.
• Notably, there is a dissociation between closed-set and open-set performance: agents that are easiest to identify when their identity is known (e.g., Seed-2-lite) are not necessarily the easiest to distinguish from unknown agents in an open-set setting.

Feature Importance and Robustness

• Timing Dominance: Under normal conditions, temporal features (Inter-Event Intervals, time to first action) are the primary drivers of classification accuracy.
• Robustness to Perturbation: The authors tested the resilience of these fingerprints by injecting random delays (up to 5 seconds) between actions.
  • Classifiers trained on clean traces suffer a sharp performance drop when evaluated on delayed traces.
  • However, classifiers retrained on delayed traces largely recover performance. They shift reliance from pure timing to robust structural features, such as click-coordinate dispersion, structural key ratios, and link-click ratios. This indicates that while timing is the strongest signal, the behavioral "fingerprint" is not solely dependent on it.

Generalization

• Task Transfer: Single-task transfer (training on one task, testing on another within the same site) is weak. However, pooled-site training (combining traces from multiple tasks on the same website) recovers strong attribution performance.
• Cross-Site Transfer: Transfer between different websites (e.g., Wikipedia to Amazon) remains weak, suggesting fingerprints are conditioned on the specific interaction between the model, the task distribution, and the website interface.

4. Significance and Implications

The paper claims that this work establishes a new attack surface for LLM agents: passive model attribution via UI traces.

• Threat Model: The attack is passive and co-located (the adversary controls the page). It does not require network visibility or active probing.
• Exploitation Vector: Once an agent is identified, an adversary can:
  1. Deploy Targeted Prompt Injections: Select jailbreaks known to be effective against the specific identified model.
  2. Inflate Costs (Sponge Attacks): Serve content designed to maximize token consumption for high-cost frontier models (e.g., Claude Opus, GPT-5.4), effectively creating a denial-of-service against the user's inference budget.
  3. Conditional Access Control: Implement model-specific whitelisting or blacklisting, or serve poisoned content visible only to specific models while appearing benign to others.
• Defense Limitations: Standard browser normalizations, such as randomizing timing delays, are insufficient to protect against this attack if the adversary is willing to retrain their classifier on the perturbed data.

5. Conclusion

The authors conclude that the on-page behavior of LLM agents serves as a persistent, identifying signal comparable to the models themselves. The shift from "human vs. bot" detection to "which model" attribution represents a fundamental change in the threat landscape for deployed agents. The paper releases a labeled corpus of agent traces and a browser harness to facilitate further research into behavioral attribution and the development of countermeasures.

Note: The paper explicitly states that its current scope is limited to a single agent harness (Midscene.js), a closed set of 14 models, and open-set detection that is not yet perfect. Future work is needed to characterize harness-invariant signals and evaluate obfuscation defenses against adaptive adversaries.