Technical Summary: cotomi Act

Problem Statement

Current computer-using agents (CUAs) face a dual limitation that prevents them from functioning as reliable organizational co-workers. First, while Large Language Models (LLMs) have improved in reasoning, they still struggle with the reliability required for long-horizon, multi-step web tasks, often falling short of human baselines on benchmarks like WebArena. Second, and more critically, existing agents lack organizational knowledge. They operate without awareness of tacit preferences, unwritten workflows, internal jargon, or the current state of pending tasks. This forces agents to either ask frequent clarifying questions (reducing efficiency) or guess silently (risking errors). Furthermore, much of the knowledge essential for work—how people actually operate—is rarely documented in static repositories, making Retrieval-Augmented Generation (RAG) alone insufficient.

Methodology

The paper presents cotomi Act, a browser-based agent that addresses these challenges through two integrated components: a robust Agent Scaffold for execution and a Behavior-to-Knowledge Pipeline for learning.

1. Agent Scaffold (Execution)

The scaffold manages the agent's ReAct (Reason-Act) loop, optimizing the context window and action space to handle complex web tasks. It employs four key mechanisms:

• Adaptive Lazy Observation: Instead of feeding the full accessibility tree at every step, the agent observes only viewport-visible elements by default, requesting the full tree only on demand. This reduces latency significantly (from 471s to 184s per task) while maintaining accuracy.
• Verbal-Diff-Based History Compression: Rather than retaining raw past observations which bloat the context window, the system replaces stale history with natural-language descriptions of changes (verbal diffs). This reduces token costs by 3.8× while preserving semantic context better than standard diff formats.
• Coarse-Grained Actions: The action space is abstracted to the highest level that still covers common interactions (e.g., scrollInto(element) instead of pixel-level scrolling). This reduces the number of steps required for a task, preventing context overflow.
• Test-Time Scaling: The system utilizes best-of-N action selection and task decomposition (spawning parallel sub-agents for independent sub-goals) to reduce variance and handle long-horizon tasks.

2. Behavior-to-Knowledge Pipeline (Learning)

To acquire organizational knowledge, cotomi Act passively observes user browsing behavior and abstracts it into structured artifacts.

• Capture: A background monitor records site-level actions, tab transitions, and screenshots, using diff-based compression to reduce noise.
• Abstraction: An agentic ETL (Extract-Transform-Load) pipeline, driven by an LLM, processes these events. It segments raw events into task-level episodes, summarizes them into structured records, and aggregates them into knowledge artifacts.
• Shared Knowledge Workspace: The resulting artifacts are stored in a shared workspace accessible to both the user and the agent. These include:
  • Activity Timelines: Calendar-based views of user activity.
  • Task Boards: A shared database where the agent proposes status updates and the user approves them.
  • Wiki Pages: Procedural notes, rules, and conventions distilled from behavior.
• Privacy: The system is strictly opt-in, with PII masking applied before artifacts are written to the workspace.

3. Human-Agent Collaboration

The workspace acts as a "boundary object." The agent consults these artifacts via a search_workspace tool when instructions are ambiguous or require context. Crucially, the loop is bidirectional: the user can inspect, edit, and approve the agent's derived knowledge, ensuring alignment with reality without explicit synchronization.

Key Contributions

1. cotomi Act System: A browser-based agent that achieves 80.4% success on the 179-task WebArena human-evaluation subset, exceeding the reported human baseline of 78.2%. This is achieved through the specific combination of adaptive lazy observation, verbal-diff history, coarse-grained actions, and test-time scaling.
2. Behavior-to-Knowledge Pipeline: A novel approach that passively observes user browsing to progressively abstract raw events into reusable knowledge artifacts (task boards, wikis, timelines) within a shared workspace, enabling persistent learning and transparent alignment.
3. Empirical Validation: A controlled proxy evaluation demonstrating that downstream task success improves as the agent accumulates more behavior-derived knowledge, validating the efficacy of learning from observation.

Results

• Execution Performance: On the WebArena human-evaluation subset, cotomi Act achieved an 80.4% success rate. The base scaffold (without test-time scaling) reached 76.5%, while adding best-of-N selection and task decomposition provided a further 3.9% gain. This performance surpasses the 78.2% human baseline and previous state-of-the-art agents like OpAgent (71.6%).
• Knowledge Impact: In a controlled study using WorkArena-L1, task success generally improved as domain coverage of behavioral knowledge increased.
  • With 0% coverage, the baseline success was ~51%.
  • With 100% coverage, success improved by up to +10 percentage points.
  • The study noted that while raw trajectories can degrade performance at low coverage due to noise, abstracted formats (scripts and insights) remained stable.
  • A specific case study in the service-catalog domain showed success jumping from ~75% to ~99% after the agent acquired 48 procedural scripts derived from user behavior.

Significance

The paper claims that cotomi Act demonstrates a practical path to organizational grounding for AI agents. By combining a scaffold capable of exceeding human performance on standardized benchmarks with a mechanism to learn tacit, unwritten workflows from everyday behavior, the system moves beyond being a "stateless executor" to becoming a "situated co-worker." The significance lies in the bidirectional loop: the agent learns from the user, but the user also curates the agent's knowledge base through a shared workspace, ensuring that the automation remains aligned with evolving organizational realities without requiring manual documentation.