🤖 AI

What Does It Take to Detect an AI Agent? Minimal Feature Sets for Behavioral Detection under Browser Automation

This paper proposes a three-class detection framework that overcomes the limitations of binary human-vs-bot classifiers by identifying AI agents through minimal, robust behavioral feature sets (specifically mouse_event_rate and teleport_click_ratio) that exploit inherent browser automation artifacts rather than reasoning patterns, achieving near-perfect detection accuracy even under sophisticated evasion attempts.

Original authors: Vishisht Choudhary, Lukas Schmidt, Anne Zoë Kenntner, Feras Skhab, Michel Osswald, Jens Ernstberger

Published 2026-07-30

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Vishisht Choudhary, Lukas Schmidt, Anne Zoë Kenntner, Feras Skhab, Michel Osswald, Jens Ernstberger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling city. For decades, the security guards at the gates had a very simple job: they only needed to spot two types of people. There were the Humans, who walked around with natural, slightly wobbly steps, stopped to think, and sometimes tripped over their own feet. Then there were the Bots, which were like robots on a strict schedule, marching in perfectly straight lines, never hesitating, and moving with mechanical precision. The guards built their entire system around this "Human vs. Robot" binary. If you walked like a human, you got in; if you marched like a robot, you got stopped.

But recently, a third type of visitor has started showing up: the AI Agent. Think of these as incredibly smart, super-intelligent tourists who can read maps, plan complex routes, and solve puzzles better than anyone else. However, they still need to use the city's sidewalks and crosswalks to get around. The problem is that these AI Agents are tricky. They are smart enough to act like humans when they need to, but they still have to use the city's automated subway system (browser automation) to move. The old security guards, trained only to spot "Human" or "Robot," are getting confused. They are letting the AI Agents slip through the gate, thinking they are just regular people, because the agents are too clever to look like the old-fashioned robots the guards were trained to catch.

This is the puzzle a team of researchers from the Technical University of Munich and Kontext Security decided to solve. They asked a simple but crucial question: Can we tell the difference between a human, a robot, and a smart AI agent just by watching how they move their mouse and click buttons? And if we can, what is the absolute minimum amount of information we need to do it?

The Great Identity Crisis

The researchers found that the old security systems were failing spectacularly. When they tested standard "Human vs. Bot" detectors on these new AI Agents, the results were messy. A common type of detector (called an MLP) let 39.1% of the AI Agents slip through, thinking they were humans. Another sophisticated detector (called SAINT) let 34.5% through. Even a Random Forest model missed 30.0% of them.

The team realized this wasn't just because the detectors were "dumb." It was a structural problem. The detectors were built with only two boxes to sort people into: "Human" or "Bot." When a smart AI Agent showed up, it didn't fit neatly into either box, so the detector just guessed. The researchers proved that if they simply added a third box labeled "Agent," the detectors became perfect, catching 100% of the agents every single time.

The Secret is in the "Ghost" Moves

So, how did they catch them? The team discovered that the AI Agents weren't being caught because of their "thinking" or their clever plans. Instead, they were being caught by the ghosts in the machine.

Imagine a human typing on a keyboard or moving a mouse. Their hand is a physical object. It has weight, it has inertia, and it makes tiny, messy, jittery movements that a camera or sensor can see. But when an AI Agent uses a browser automation tool (like Playwright), it's not moving a physical hand. It's sending digital commands. It's like a ghost pushing a button.

The researchers found that these digital ghosts leave a very specific, invisible signature:

No Jitter: When a human moves a mouse, it wobbles a little. When the AI clicks, it just appears at the target. There are no tiny, messy movements leading up to the click.
No "Wheel" Spins: If you scroll a webpage with a mouse wheel, it creates a specific stream of data. The AI just tells the page to scroll, skipping the wheel entirely.
Instant Clicks: A human takes a tiny fraction of a second to press and release a mouse button. The AI does it instantly, with zero variation.

The team built a "five-level evasion ladder" to test if the AI could hide. They tried everything:

Level 1: The AI just clicked normally.
Level 2: The AI tried to draw curved lines (like a human) using math formulas.
Level 3: The AI used a "GAN" (a type of AI that learns to copy human movements) to generate fake mouse paths.
Level 4: The AI added random "jitter" to look more human.
Level 5: The AI played back recordings of real humans moving their mice.

Even when the AI played back a recording of a real human (Level 5), it still got caught! Why? Because even though the path of the mouse looked human, the way the click happened was still a digital command. The AI couldn't fake the fact that it wasn't using a physical mouse wheel or that it didn't have the tiny, natural wobbles of a human hand. The "ghost" signature was still there.

The Magic Two-Feature Set

The most exciting part of the discovery was how little information was actually needed. The researchers tested thousands of different combinations of features to find the smallest set that could catch every single AI Agent.

They found that you don't need a super-complex system. You only need two simple things to spot the AI:

Mouse Event Rate: How often the mouse moves. (AI moves are often sparse or non-existent before a click).
Click Duration Standard Deviation: How much the time it takes to click varies. (AI clicks are perfectly consistent; human clicks vary).

With just these two clues, the system could identify 100% of the AI agents, even when they were trying to hide using the most advanced tricks. If they added three more features (like how fast the typing was or how much the scroll speed varied), the system could perfectly distinguish between Humans, Bots, and Agents all at once with a score of 0.991 (which is almost perfect).

What This Means

The paper concludes that catching AI Agents isn't about trying to figure out if they are "thinking" like a human. It's about noticing that they are using a different kind of "muscle" to move around the web. They are using digital commands instead of physical hands.

The researchers warn that if you try to build a detector using just one feature, you might get tricked. For example, one feature that looked perfect on its own actually made the detector guess "Agent" for everyone, including real humans, which is useless. But with the right two features, the detector is incredibly robust.

This discovery is a big deal for the future of the internet. As AI Agents become more common, websites will need to know who is browsing. This research shows that we don't need to build massive, complicated systems to spot them. We just need to look for the tiny, invisible footprints that digital ghosts leave behind when they try to walk like humans. And as long as they are using standard browser tools, those footprints will always be there.

Technical Summary: Minimal Feature Sets for Behavioral Detection of AI Agents under Browser Automation

1. Problem Statement

The security industry has long operated on a binary assumption: web traffic is either human or bot. This dichotomy has driven the development of behavioral biometrics, CAPTCHAs, and bot management platforms. However, the emergence of LLM-driven AI agents (e.g., Anthropic's Computer Use, OpenAI's Operator) has introduced a third, distinct traffic class. Unlike traditional bots that follow rigid scripts, these agents autonomously browse, reason, and execute multi-step tasks through real browsers using automation APIs like Playwright.

The core problem identified is that binary classifiers structurally cannot represent a three-class world. When forced to choose between "human" and "bot," binary detectors misroute AI agents. The paper demonstrates that this is not merely a modeling capacity issue but an architectural one: the label space lacks an "agent" class. Consequently, sophisticated binary detectors (MLP, SAINT) misclassify roughly 34–39% of real AI agents as "human," while models that do catch them (like XGBoost) often rely on brittle, coincidental features rather than robust agent-specific signals.

2. Methodology

Dataset Construction

The authors constructed a unified three-class dataset comprising:

Humans: 14,000+ sessions from CaptchaSolve30k, capturing mouse dynamics during CAPTCHA solving.
Bots: 5,000 synthetic samples generated from three archetypes (Headless, Scripted, Sophisticated) based on literature-grounded distributions. The "Sophisticated" archetype intentionally overlaps with human behavior to prevent trivial feature separation.
Agents: 1,025 real sessions where an LLM (Claude) drove a Chrome browser via Playwright across 10 diverse web tasks (forms, navigation, login, etc.).

Feature Engineering

The study extracted 17 continuous behavioral features (trajectory, click, timing, and agent-specific ratios) and 7 binary environment features. Key features included:

mouse_event_rate: Rate of raw mousemove events.
teleport_click_ratio: Fraction of clicks preceded by raw-event jumps.
click_duration_std: Standard deviation of click hold times.
inter_action_gap_ features: Designed to capture the "pause-think-act" cycle of LLMs.

Model Architecture & Evaluation

Models: The study evaluated MLP, Random Forest, XGBoost, and SAINT (a tabular transformer).
Three-Class Formulation: To address the structural gap, the authors adapted SAINT for three-class classification (Human, Bot, Agent).
Evasion Ladder: To test robustness, a five-level evasion ladder was constructed, escalating from:
1. None: Native Playwright API.
2. Rule-based: Bézier curves and human-sampled delays.
3. GAN: WGAN-GP generated trajectories.
4. GAN+: GAN trajectories with micro-jitter and Fitts' law timing.
5. Human Replay: Real human cursor trajectories from CaptchaSolve30k replayed via Playwright.
Feature Selection: An exhaustive search over 9,401 feature subsets ( $k=1$ to $5$), complemented by forward selection and backward elimination, was performed using Gradient Boosting Machines (GBM) for computational tractability.

3. Key Results

The Binary Detection Gap

Binary detectors fail to distinguish agents from humans effectively:

MLP: 39.1% of agents misclassified as human.
SAINT: 34.5% of agents misclassified as human.
Random Forest: 30.0% misclassified.
XGBoost: Achieved 100% detection but relied on brittle, coincidental features (e.g., missing scroll events) rather than agent-specific signals.

Resolution: Switching to a three-class formulation (Human, Bot, Agent) eliminated the gap entirely. A SAINT model trained on the three classes achieved F1 = 1.000 for agents across all 30 runs (3 model families × 10 seeds).

Robustness Against Evasion

The three-class detector maintained 100% agent recall across all five evasion levels, including the "Human Replay" level where real human trajectories were replayed via Playwright.

Critical Insight: Evasion at the trajectory level (GANs, replay) fails because the event-stream structure remains non-human. Automation APIs (Playwright) do not emit raw pointer-move streams or wheel-delta streams that physical devices produce.
Artifacts: The detection signal relies on "absence signatures": zero raw-event teleports before clicks, empty wheel-delta series, and empty pre-click mousemove tracks. These persist even when the cursor path is human-like.

Minimal Feature Sets

Through exhaustive search and backward elimination, the paper identified that a small subset of features is sufficient:

Two Features:
- {mouse_event_rate, teleport_click_ratio} (from exhaustive search): Achieves 100% agent recall with Agent Precision = 0.994 and Macro-F1 = 0.926.
- {mouse_event_rate, click_duration_std} (from backward elimination): Achieves 100% agent recall with Agent Precision = 1.000 and Macro-F1 = 0.904.
Five Features: Lift Macro-F1 to ≥0.99, enabling robust discrimination across all three classes.
Degenerate Case ( $k=1$ ): The single "perfect" feature (cursor_path_linearity) is a diagnostic trap; it forces the classifier to always predict "agent," resulting in low precision (0.33) and Macro-F1 (0.17).

Signal Origin

The discriminative signal is not an artifact of agent reasoning (the "pause-think-act" cycle) or task complexity.

Features designed to capture LLM inference gaps (e.g., inter_action_gap_variance) were removed by backward elimination without affecting recall.
The signal is an API execution artifact: the specific way browser automation APIs (Playwright/CDP) generate events (programmatic DOM calls) differs fundamentally from physical input devices.
Task Independence: Removing task-correlated features (typing, scrolling) did not impact agent detection, confirming the signal resides in mouse dynamics and event-stream presence.

4. Significance and Claims

The paper claims to be the first work to:

Treat LLM-driven browser agents as a distinct detection category separate from traditional bots.
Demonstrate that the "binary vs. agent" confusion is an architectural limitation of the label space, solvable by adding a third class.
Show that detection relies on low-level API artifacts (missing raw event streams) rather than high-level cognitive patterns, making it robust against trajectory-level evasion (GANs, human replay).
Provide a minimal feature set for practical deployment, proving that two behavioral features can isolate agents with unit precision.

Limitations and Scope:

The results are specific to CDP-mediated automation (Playwright, Puppeteer, Selenium with CDP).
The paper explicitly states that agents bypassing browser APIs entirely (e.g., via OS-level input simulation like xdotool or hardware emulators) would likely evade this detection method.
The bot class is synthetic; while agent detection is validated on real data, the macro-F1 scores for three-class discrimination depend on the synthetic bot distributions.
The "Human Replay" evasion level is limited to the Playwright API; it does not test OS-level injection.

In conclusion, the paper argues that detecting AI agents is currently feasible and robust because they operate through automation APIs that leave a distinct, non-human "absence signature" in the event stream. This signal is redundant, minimal (requiring only 2–5 features), and survives even the most sophisticated trajectory-based evasion attempts.