🤖 AI

SaaS-Bench: Can Computer-Use Agents Leverage Real-World SaaS to Solve Professional Workflows?

The paper introduces SaaS-Bench, a comprehensive benchmark comprising 106 realistic tasks across 23 professional SaaS systems, which reveals that current computer-using agents struggle significantly with complex, long-horizon workflows, achieving less than 4% end-to-end success due to limitations in planning, state tracking, and error recovery.

Original authors: Kean Shi, Zihang Li, Tianyi Ma, Zengji Tu, Jialong Wu, Xinbo Xu, Qingyao Yang, Ruoyu Wu, Weichu Xie, Ming Wu, Jason Zeng, Michael Heinrich, Elvis Zhang, Liang Chen, Kuan Li, Baobao Chang

Published 2026-05-18

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kean Shi, Zihang Li, Tianyi Ma, Zengji Tu, Jialong Wu, Xinbo Xu, Qingyao Yang, Ruoyu Wu, Weichu Xie, Ming Wu, Jason Zeng, Michael Heinrich, Elvis Zhang, Liang Chen, Kuan Li, Baobao Chang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a new assistant to handle your most complicated workday. You don't just want them to answer emails; you want them to log into your bank, your project management software, your medical records, and your design tools to actually do the work.

This paper introduces a new "final exam" called SaaS-Bench to test how good AI assistants (called Computer-Using Agents) really are at this kind of real-world job.

Here is the breakdown of what they did and what they found, using simple analogies.

1. The Problem: The "Video Game" vs. The "Real Job"

Until now, most tests for AI assistants were like playing a video game. The levels were short, the rules were simple, and if you got stuck, the game just reset.

The Old Way: "Click the red button, then type 'Hello'." (Easy, short, isolated).
The Real World: "Go to the HR system to fire an employee, then go to the accounting system to calculate their final paycheck, then go to the CRM to reassign their clients, and make sure the dates match up perfectly." (Hard, long, messy).

The authors realized that passing the "video game" tests didn't mean an AI could handle a real job. So, they built a new test based on 23 real, deployable software systems (like real medical record systems, accounting tools, and project managers) that professionals actually use.

2. The Exam: SaaS-Bench

Think of SaaS-Bench as a massive, multi-day obstacle course.

The Course: It has 106 different tasks across six different "neighborhoods" of work (like Healthcare, Finance, and Software Engineering).
The Rules: The AI has to navigate these systems using a web browser, just like a human would. It can't cheat by looking at the database code; it has to click buttons and read screens.
The Difficulty: These aren't 5-minute tasks. The average task takes over 100 steps (clicks, types, and scrolls). It's like asking someone to bake a cake, write a recipe for it, mail the recipe to a friend, and then file the receipt for the flour, all in one go.

3. The Results: The AI Got Lost

The researchers tested the smartest AI models available (like the "top students" of the AI world) on this exam. The results were sobering:

The Score: Even the best AI model only finished less than 4% of the tasks completely from start to finish.
The "Almost" Trap: The AI was actually good at the beginning of the tasks. It could get 80% of the way there. It could fill out the first form, click the right buttons, and create the first document.
The Crash: But then, it would make a tiny mistake (like typing the wrong date), fail to notice the mistake, and then keep going down the wrong path. By the time it reached the end, the whole result was wrong.

The Analogy: Imagine a GPS that is great at telling you to "turn left" and "drive 5 miles." But if you accidentally miss a turn, the GPS doesn't say, "You're lost, let's recalculate." Instead, it confidently tells you to keep driving straight for another 50 miles until you run out of gas. The AI is confident, but it's often wrong.

4. Why Did They Fail? (The Four Big Hurdles)

The paper identifies four main reasons why the AI struggled, using some great metaphors:

The "House of Cards" Effect (Fragility): In these long tasks, every step depends on the one before it. If you mess up step 10, steps 11 through 100 are ruined. The AI is so good at step 10 that it thinks it's doing great, but that one small error collapses the whole structure.
The "Silent Poison" (Error Cascading): Sometimes the AI makes a mistake that looks correct on the surface.
- Example: The AI creates a customer named "Elena" instead of "Arcturus Digital." The screen shows "Elena (Arcturus)," so the AI thinks, "Great, I did it!" But because the name is technically wrong, every single financial transaction that follows is attached to the wrong person. The AI never realizes it poisoned the well.
The "Confident Liar" (Self-Deception): The AI has a "memory" where it tells itself what it did. Sometimes, it sees a mistake, tries to fix it, but then forgets to check if the fix worked. It then writes a report saying, "I fixed it!" even though the screen still shows the error. It's like a student who realizes they made a math error, tries to erase it, but then just writes the answer down anyway and says, "I'm done."
The "Coin Flip" (Unreliability): If you ask the same AI to do the same task three times, it might get a perfect score on the first try, fail completely on the second, and do okay on the third. It's not consistent. This means you can't trust it to do a job reliably today, even if it worked yesterday.

5. The Bottom Line

The paper asks: "Can AI agents use real-world software to solve professional work?"

The answer is: Not yet.

While these AI models are incredibly smart at talking and understanding language, they are currently terrible at long, complex, real-world execution. They get lost in the details, they don't double-check their work, and they can't recover when they make a mistake.

The authors built this test (SaaS-Bench) not to say AI is useless, but to show us exactly where it breaks so we can fix it. Until the AI can handle a 100-step workflow without getting confused or confident about the wrong answer, it's not ready to take over your professional job.

Technical Summary: SaaS-Bench

Problem Statement

Computer-Using Agents (CUAs) have evolved from passive text generators to active executors capable of interacting with graphical user interfaces (GUIs) and web browsers. However, existing benchmarks for evaluating these agents suffer from three critical limitations that lead to a systematic overestimation of their real-world capabilities:

Limited Application Complexity: Many benchmarks rely on simplified, static, or synthetic web environments that lack the full frontend-backend logic, dynamic state dependencies, and business constraints of production software.
Isolated Workflows: Tasks are often confined to single applications with short horizons, failing to capture the cross-application coordination, domain-specific knowledge, and structured multi-step processes inherent in professional work.
Short-Horizon Dependencies: Existing tasks rarely require the extended interaction sequences (often exceeding 100 steps) necessary to test an agent's planning, state tracking, and error recovery capabilities over time.

Consequently, there is a lack of rigorous evaluation frameworks to determine if CUAs can effectively operate in realistic, high-value Software-as-a-Service (SaaS) environments.

Methodology

Benchmark Construction (SaaS-Bench)

The authors introduce SaaS-Bench, a benchmark designed to evaluate CUAs in realistic, deployable SaaS environments. The construction process adheres to three core principles:

Realistic and Deployable Environments:
- The benchmark is built upon 23 real, open-source SaaS systems (e.g., OpenEMR, OnlyOffice, BigCapital) rather than toy websites.
- These systems are containerized via Docker, ensuring full frontend-backend logic, persistent database states, and user authentication.
- Environments are reset to predefined initial states before each task to ensure reproducibility and prevent state contamination.
- Data is semantically populated using a combination of LLM-generated realistic data and imported open-source datasets to ensure meaningful business context.
Professional, Cross-Application, and Multimodal Workflows:
- Tasks are organized across six professional domains: Software Engineering, Business Operations, Healthcare Administration, Team Collaboration, Agriculture, and Media Creation.
- The benchmark includes 106 tasks grounded in real-world workflows.
- 93.4% of tasks involve at least two applications, with 50% requiring coordination across three applications.
- The set includes 74 text-only tasks and 32 multimodal tasks (involving images, documents, or media), primarily in Agriculture and Media domains.
Long-Horizon Execution:
- Tasks are designed with an average of over 100 interaction steps.
- A four-stage pipeline (Task Seed $\rightarrow$ Synthesis Loop $\rightarrow$ Static Check $\rightarrow$ Execution Check) ensures tasks are professional, executable, and verifiable. This involves an iterative "Builder-Challenger-Refiner" loop where LLMs generate candidates and human experts review them for ambiguity, completeness, and verifiability.

Evaluation Protocol

Execution Framework: Agents interact exclusively through browser-based UI operations (navigation, clicking, typing) using the browser-use framework. Direct access to databases, APIs, or file systems is prohibited.
Verification: Each task is decomposed into weighted verification checkpoints. Three verification methods are employed:
- State-Check: Verifies objective system states (DB records, API responses).
- Content-Check: Validates textual outputs via string matching.
- LLM-Judge: Evaluates open-ended outputs like report quality.
Metrics:
- Resolved Score: Binary (1 or 0); 1 only if all checkpoints pass.
- Checkpoint Score: A weighted partial score reflecting progress, allowing for the measurement of intermediate success in long-horizon tasks.

Key Results

Performance Overview

Experiments were conducted on seven frontier models (including Claude Opus 4.6, GPT-5.4 High, Qwen 3.6 Plus, and others). The results indicate that current agents struggle significantly in realistic SaaS workflows:

Low End-to-End Completion: Even the strongest model, Claude Opus 4.6, achieved a Resolved Score of only 1.9% (completing fewer than 4% of tasks end-to-end).
Checkpoint Scores: The overall Checkpoint Score for the best model was 43.2%, indicating that while agents can often complete intermediate steps, they fail to maintain coherence to finish the entire workflow.
Domain Variance: Performance varied by domain, with "Teamwork" tasks generally being easier than "Business" and "Healthcare" tasks, which involve stricter numerical constraints and domain-specific procedures.

Analysis of Failure Modes

The paper identifies four fundamental failure modes explaining the low resolved scores:

Fragility of Long-Horizon Completion: Due to the multiplicative nature of success probabilities ( $p^N$ ), even high per-checkpoint accuracy (e.g., 95%) results in near-zero resolved scores for tasks with 10–20 checkpoints.
Error Cascading in Multi-Application Workflows: A single semantic error (e.g., misclassifying a customer as "individual" instead of "company") can propagate silently through a dependency chain, causing downstream checks to fail even if the agent believes the task is complete.
Unreliable Self-Verification: Agents often fail to re-verify their actions after corrections. They may recognize an error, attempt a fix, and then proceed to the next subtask without confirming the fix was successful, leading to "silent" failures.
High Run-to-Run Variance: Performance is highly unstable. The same model can achieve a score of 0.80 in one run and 0.10 in another on the same task. This suggests that current evaluations relying on single-run metrics (pass@1) are misleading; pass@k metrics reveal that while agents can occasionally find a successful trajectory, consistency remains a major challenge.

Significance and Claims

The paper claims that SaaS-Bench provides the first rigorous, realistic testbed for evaluating CUAs in professional, cross-application, long-horizon settings. Its significance lies in:

Exposing Capability Gaps: It demonstrates that despite advances in LLMs, current agents are not yet capable of reliably solving professional workflows in real-world SaaS environments, completing fewer than 4% of tasks end-to-end.
Redefining Evaluation Standards: It argues that short-horizon, single-application benchmarks are insufficient. The field must shift toward evaluating agents on their ability to maintain state, recover from errors, and coordinate across multiple systems over long sequences.
Identifying Structural Limitations: The results highlight that the primary bottlenecks are not just knowledge retrieval but planning, state tracking, and error recovery. The paper suggests that current agent architectures lack the closed-loop verification mechanisms necessary to handle the fragility of long-horizon tasks.

The authors conclude that while SaaS-Bench reveals significant limitations, it serves as a necessary foundation for future research into more reliable and practically deployable Computer-Using Agents.