Technical Summary: DSAgentBench

Problem Statement

Real-world data science involves complex, long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation. These tasks require the coordinated use of diverse tools—including notebooks, IDEs, terminals, browsers, and databases—within a functioning operating system. While recent advances in large language models (LLMs) have demonstrated capabilities in generating code or performing isolated analytical tasks, existing benchmarks fail to evaluate whether agents can autonomously execute complete end-to-end data-science workflows in realistic computing environments.

Current benchmarks fall into two categories: those that evaluate code generation in isolation (e.g., DS-1000, DSEval) without requiring system interaction, and those that evaluate general computer control (e.g., OSWorld, WebArena) without assessing domain-specific analytical reasoning. Consequently, there is a lack of evaluation frameworks that test an agent's ability to navigate file systems, manage dependencies, interpret errors, and refine analyses based on intermediate outputs within a real OS environment.

Methodology

Benchmark Construction (DSAgentBench)

The authors introduce DSAgentBench, the first benchmark designed to evaluate autonomous data-science workflows inside real operating systems. The benchmark consists of 275 diverse, human-authored tasks covering the entire data-science lifecycle.

• Task Formulation: Tasks are defined as tuples $(C_i, I_i, V_i)$, where $C_i$ is the initial system configuration (datasets, file structure, installed libraries), $I_i$ is a natural language instruction, and $V_i$ is a deterministic Python evaluator.
• Data Sourcing: Datasets are drawn from heterogeneous real-world sources, including Kaggle, OpenML, GitHub, SQLite databases, and web APIs. The data includes tabular (95.3%), image (3.6%), and text (1.1%) modalities.
• Task Categories: Tasks are organized into six capability categories: Data Acquisition, Exploratory Data Analysis (EDA), Feature Engineering, Modeling, Evaluation/Deployment, and Visualization/Reporting.
• Evaluation Protocol: Unlike benchmarks that score code execution, DSAgentBench employs deterministic evaluators that verify analytical correctness, visual output quality, and model performance. A task is considered successful only if the agent's final outputs satisfy the evaluator's criteria (score $\ge 0.95$).
• Construction Pipeline: The benchmark was built through a three-stage process: sourcing datasets, collaborative task/evaluator design by human experts (with LLMs used only for refinement), and dual-annotator verification to ensure reproducibility and technical correctness.

Environment Architecture

DSAgentBench extends the OSWorld framework to create a realistic execution environment:

• OS: Ubuntu with pre-installed Python and data-science libraries.
• Tools: Visual Studio Code, Jupyter Notebook, Chrome, and access to Kaggle/OpenML APIs.
• Observation Space: Agents receive either a 1920×1080 screenshot or a hybrid Screenshot + Accessibility Tree (A11y) modality, which provides structured UI metadata (roles, bounding boxes, interaction states).
• Action Space: Agents interact via GUI actions (mouse clicks, keyboard input) and meta-actions (WAIT, DONE, FAIL). The environment captures state transitions after each action.

Models Evaluated

The authors evaluated 15 closed- and open-source agents, including:

• Closed-source: GPT-4o, GPT-5 (and mini), O4-mini, Claude Sonnet 4/4.5/4.6, Gemini 2.5 Pro, and OpenAI's Computer Agent.
• Open-source: UI-TARS (2B/7B), GUI-OWL-7B, OpenCUA-72B, and hybrid models (Jedi paired with GPT-4o).

Key Results

Overall Performance

The experiments reveal a substantial capability gap between current agentic systems and the demands of real data-science workflows:

• Strongest Agent: Claude-4.6-Sonnet achieved the highest performance with 56.70% task success under the Screenshot + A11y Tree setting.
• Other Closed-Source Models: Performance dropped significantly for other models, with GPT-5 at 29.81% and others (GPT-4o, Gemini-2.5-Pro) hovering around 20%.
• Open-Source Models: All open-source agents achieved less than 1% success, frequently failing at tool orchestration, OS grounding, and multi-step reasoning.
• Human Baseline: Human experts achieved an 85.09% success rate, highlighting the remaining gap even for the strongest AI agents.

Ablation and Error Analysis

• Task Complexity: Performance degrades monotonically with difficulty. "Hard" tasks (5+ steps) remain the most challenging. Multi-stage workflows (56.7% of tasks) are significantly harder than single-stage tasks due to the need for state maintenance and error recovery.
• Tool Usage: Tasks executed in Jupyter Notebooks performed better than those in VS Code, largely due to fewer terminal and environment-related failures.
• Observation Modality: Adding A11y tree information generally improved performance, suggesting structured UI metadata aids grounding, though gains varied by model.
• Failure Modes:
  • Open-source agents failed almost exclusively (97–98%) due to grounding errors (inability to align instructions with the desktop state).
  • Stronger closed-source agents exhibited mixed failures, including terminal errors, code generation flaws, and reasoning deficits.
  • Temporal Structure: Open-source and weaker models often failed early (unable to open terminals), while stronger models tended to fail in later stages after prolonged, ineffective exploration.
• Budget Sensitivity: Increasing the interaction budget from 15 to 50 steps yielded only marginal gains (24.54% $\to$ 25.81%), indicating that failures are not primarily due to step limits but rather fundamental issues in planning and reasoning.

Significance and Claims

The paper positions DSAgentBench as a foundational resource for developing grounded, verifiable, and autonomous data-science agents. Its primary contributions are:

1. First Real-OS Benchmark: It is the first benchmark to evaluate autonomous data-science workflows inside a functioning operating system, covering the full lifecycle from data acquisition to validation.
2. Execution-Based Evaluation: It shifts the evaluation paradigm from "code correctness" to "analytical correctness," requiring agents to produce valid artifacts (visualizations, models, reports) verified by deterministic scripts.
3. Revealing the Gap: The results expose a significant limitation in current agentic systems, demonstrating that even the strongest models struggle with the coordination of tools, long-horizon reasoning, and OS grounding required for real-world data science.
4. Future Direction: By identifying specific failure modes (grounding, planning, tool orchestration), the benchmark provides a clear roadmap for future research aimed at building agents capable of performing real-world data science tasks.

The authors release the benchmark at https://github.com/vis-nlp/DSAgentBench to facilitate further research in this domain.