Modeling Distinct Human Interaction in Web Agents

1. Problem Statement

Despite rapid advancements in autonomous web agents powered by Large Language Models (LLMs), current systems lack a principled understanding of when and why humans intervene during task execution.

• The Gap: Agents often proceed autonomously past critical decision points (causing errors) or request unnecessary confirmations (causing user fatigue).
• The Challenge: Existing collaborative frameworks focus on specific interaction mechanisms (e.g., asking follow-up questions) rather than modeling the broader spectrum of human intervention patterns (e.g., mid-task takeover, error correction, or preference refinement).
• Goal: To develop agents that can proactively anticipate human intervention based on user interaction styles, thereby minimizing unnecessary interruptions while ensuring reliability.

2. Methodology

A. Data Collection: COWCORPUS

The authors introduce COWCORPUS, a novel dataset designed to capture human-agent collaboration dynamics.

• Scale: 400 real-user web navigation trajectories containing over 4,200 interleaved actions (2,748 agent steps, 1,476 human steps).
• Composition:
  • Standard Tasks: 10 tasks from the Mind2Web benchmark (covering domains like travel, shopping, and government services).
  • Free-form Tasks: 10 user-chosen tasks on arbitrary websites, capturing diverse real-world intents.
• Annotation: Step-level annotations mark intervention moments where users pause, resume, or override agent execution.

B. Analysis of Interaction Patterns

Through quantitative analysis of intervention frequency, intensity, timing, and handback rates, the authors identified four distinct user collaboration styles:

1. Takeover: Infrequent interventions, usually late in the task; users retain control once they step in (low handback rate).
2. Hands-on: Frequent, high-intensity interventions; users alternate control with the agent regularly.
3. Hands-off: Rarely intervenes; allows the agent to execute end-to-end.
4. Collaborative: Selective, modest interventions with a strong tendency to return control to the agent (high handback rate).

C. Problem Formulation

The task is formulated as a Partially Observable Markov Decision Process (POMDP) and a step-wise binary classification problem.

• Input: A multimodal observation $o_t$ (screenshot + accessibility tree), the agent's proposed action $\hat{a}_t$, and the history trajectory $\tau_{t-1}$.
• Output: A probability $p(y_t=1 | \dots)$ predicting if a human will intervene ($y_t=1$) or if the agent should continue ($y_t=0$).
• Modeling: Large Multimodal Models (LMMs) are fine-tuned via Supervised Fine-Tuning (SFT) to generate tokens <ask_user> or <agent_continue>.

D. Evaluation Metrics

• Step Accuracy & F1 Score: Standard classification metrics.
• Perfect Timing Score (PTS): A novel metric that penalizes predictions based on their temporal distance from the ground-truth intervention. It rewards correct timing and penalizes early false positives more heavily than late ones.

3. Key Contributions

1. COWCORPUS Dataset: The first large-scale corpus of real-user collaborative web trajectories with fine-grained intervention annotations, enabling the study of human-agent interaction styles.
2. Interaction Style Taxonomy: Empirical identification of four distinct user collaboration patterns (Takeover, Hands-on, Hands-off, Collaborative), demonstrating that human intervention is a structured behavioral signal rather than random noise.
3. Intervention-Aware Modeling:
   • Developed general intervention-aware models that outperform base LMs.
   • Developed style-conditioned models (fine-tuned on specific user clusters) to tailor agent behavior to individual user preferences.
4. Real-World Deployment: Integrated the intervention prediction model into a live web agent (PlowPilot) and validated it through a user study.

4. Experimental Results

A. Offline Prediction Performance

• Baseline Failure: Proprietary models (GPT-4o, Claude 4, Gemini 2.5) and zero-shot approaches struggle with temporal dynamics. For instance, GPT-4o achieved high accuracy on non-intervention steps (F1: 0.846) but failed to predict active interventions (F1: 0.198), resulting in a low PTS (0.147).
• Fine-Tuning Success: Fine-tuning open-weight models (Gemma 27B, LLaVA 8B) on COWCORPUS yielded significant gains.
  • Gemma 27B (SFT): Achieved a state-of-the-art PTS of 0.303, outperforming Claude 4 Sonnet (0.293).
  • Improvement: Intervention prediction accuracy improved by 61.4–63.4% over base LMs.
• Style Conditioning: Models trained on specific clusters generally performed best on their corresponding clusters, though data sparsity in the "Takeover" group limited performance there.

B. User Study (Live Deployment)

The authors deployed the intervention-aware model in PlowPilot (a Chrome extension) and compared it against a baseline agent (CowPilot) in a study with 4 participants performing 20 tasks each.

• User Satisfaction: The intervention-aware agent saw a 26.5% increase in user-rated usefulness compared to the baseline.
• Qualitative Feedback: Users reported fewer unnecessary interruptions, better timing of prompts, and a stronger sense of control without sacrificing the benefits of automation.
• Efficiency: While agent execution time remained similar, the reduction in "monitoring burden" (constant checking for errors) improved the overall user experience.

5. Significance and Impact

• Paradigm Shift: The work advocates shifting from optimizing for pure agent autonomy to designing adaptive agents that dynamically align with human collaboration styles.
• Efficiency vs. Reliability: It demonstrates that anticipating human intervention allows agents to reduce the "oversight burden" on users without compromising task reliability.
• Practical Application: The results suggest that smaller, fine-tuned models can outperform massive, generalist models in specific interaction tasks when trained on high-quality, domain-specific interaction traces.
• Future Direction: This approach paves the way for "truly collaborative partners" that know when to act independently and when to seek human guidance, a critical step for deploying AI agents in complex, real-world environments.