ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows

1. Problem Statement

While Large Language Models (LLMs) and Vision-Language Models (VLMs) have shown promise in natural language processing and simple coding tasks, their ability to function as autonomous "AI co-scientists" in complex, real-world scientific workflows remains unproven. Existing benchmarks often focus on static question answering, single-step coding, or simplified web interactions. They fail to capture the dynamic, visually rich, and multi-modal nature of scientific research, which requires:

• Interacting with specialized, complex scientific software (e.g., molecular modeling, astronomical simulation).
• Navigating both Graphical User Interfaces (GUI) and Command-Line Interfaces (CLI).
• Performing long-horizon planning where actions have state-dependent consequences.
• Integrating domain-specific knowledge with tool manipulation.

The paper identifies a critical gap: there is no realistic environment to rigorously evaluate whether current computer-using agents can autonomously execute end-to-end scientific discovery tasks.

2. Methodology: The SCIENCEBOARD Framework

The authors introduce SCIENCEBOARD, a comprehensive infrastructure comprising a realistic environment and a rigorous benchmark.

A. The Environment

• Infrastructure: Built on an Ubuntu Virtual Machine (VM) to ensure isolation and reproducibility.
• Scientific Software Suite: The environment integrates six open-source, professional scientific applications across diverse domains:
  1. Biochemistry: UCSF ChimeraX (molecular visualization, AlphaFold integration).
  2. Algebra: KAlgebra (symbolic math, plotting).
  3. Theorem Proving: Lean 4 (formal logic, proof verification).
  4. Geographic Information Systems (GIS): GrassGIS (raster/vector processing, hydrology).
  5. Astronomy: Celestia (3D space simulation, orbital mechanics).
  6. Scientific Documentation: TeXstudio (LaTeX editing, report generation).
• State Monitoring: To enable rigorous evaluation, the authors modified these applications to inject lightweight HTTP servers. These servers expose the internal runtime states of the software (e.g., selected atoms, layer names, simulation parameters) via APIs, allowing the system to verify task completion beyond simple visual inspection.
• Interaction Modes: Agents can interact via:
  • GUI: Mouse/keyboard actions (clicks, drags, typing) based on screenshots and accessibility trees (a11ytree).
  • CLI: Executing shell commands or application-specific scripts.
  • Hybrid: Switching between GUI and CLI as needed.

B. The Benchmark

• Task Curation: A dataset of 169 high-quality tasks curated by human experts with domain backgrounds.
• Annotation Pipeline: Involves tutorial learning, task drafting, formalization (using LLMs to standardize prompts), configuration (initializing VM states), and rigorous validation (cross-checking by multiple annotators).
• Task Characteristics:
  • Difficulty: Categorized into Easy (53.8%), Medium (28.4%), Hard (16.6%), and Open Problems (1.2%).
  • Complexity: Tasks require multi-step reasoning, visual grounding, spatial understanding, and domain knowledge.
  • Evaluation: Uses a fine-grained, state-based evaluation script that checks internal VM states, I/O correctness, and numerical tolerances rather than just final text output.

C. Experimental Setup

The authors evaluated a wide range of State-of-the-Art (SOTA) models, including:

• Proprietary Models: GPT-4o, GPT-5, Claude-3.7-Sonnet, Gemini-2.0/2.5-Pro, o3-mini.
• Open-Source VLMs/LLMs: Qwen2.5-VL, InternVL3, QvQ, GPT-oss.
• GUI Action Models: OS-Atlas, UGround, UI-TARS, GUI-Actor.
• Observation Settings: Tested under four conditions: Screenshot only, a11ytree (text-only), Screenshot + a11ytree, and Set-of-Marks (SoM).

3. Key Contributions

1. First Realistic Scientific Agent Environment: SCIENCEBOARD is the first benchmark to simulate end-to-end scientific workflows using professional, complex software with dynamic visual and textual interfaces.
2. State-Based Evaluation Mechanism: Unlike previous works relying on text matching, SCIENCEBOARD validates tasks by querying the internal state of scientific applications, ensuring high-fidelity assessment of complex operations (e.g., verifying a specific protein structure is loaded).
3. Comprehensive Benchmark: A curated set of 169 tasks spanning six scientific domains, covering diverse interaction types (GUI, CLI, Hybrid) and difficulty levels.
4. Empirical Analysis of Limitations: A systematic analysis revealing that even the most advanced agents struggle significantly with scientific workflows, highlighting specific bottlenecks in grounding, domain knowledge, and long-horizon planning.

4. Results

The evaluation yielded sobering results regarding the current state of AI in science:

• Low Overall Success Rates: Even the best-performing models (e.g., GPT-5, Gemini-2.5-Pro) achieved an overall success rate of only ~20% in the most favorable settings. Open-source models performed significantly worse (<12%).
• Domain Disparities:
  • Stronger Performance: Agents performed moderately well in Algebra and Biochemistry (often supporting both CLI and GUI).
  • Weak Performance: Performance dropped drastically in GIS and Astronomy, where tasks rely heavily on fine-grained GUI grounding and complex spatial reasoning.
• Observation Modality Impact:
  • The Screenshot + a11ytree combination generally yielded the best results, leveraging both visual context and structured element data.
  • Set-of-Marks (SoM) sometimes degraded performance due to visual noise in dense scientific interfaces (e.g., star charts).
• Human vs. Agent Gap: Human annotators achieved an average success rate of ~60%, highlighting a massive gap between current AI capabilities and human proficiency in scientific workflows.
• Scaling Limits: Increasing inference-time compute (e.g., using GPT-5's native reasoning mode or higher token limits) provided marginal improvements but did not fundamentally solve the core failures.
• Modular Approaches: Separating Planning (using a strong LLM like GPT-4o) from Action/Grounding (using specialized GUI models) significantly improved performance, suggesting a path forward for future agent architectures.

5. Significance and Future Directions

• Realistic Benchmarking: SCIENCEBOARD sets a new standard for evaluating AI agents, moving beyond toy problems to realistic, high-stakes scientific scenarios.
• Identifying Bottlenecks: The paper demonstrates that the primary limitation of current agents is not just reasoning depth, but general computer-using capabilities (accurate perception of dense UIs) and domain-specific knowledge.
• Design Principles: The findings suggest that future "AI Co-scientists" must be:
  • Hybrid: Capable of seamlessly switching between GUI and CLI.
  • Modular: Separating high-level planning from low-level grounding.
  • Knowledge-Enhanced: Equipped with specialized domain knowledge and the ability to retrieve it dynamically.
• Path to Physical Labs: The authors posit that mastering these digital workflows is a necessary precursor to deploying agents in physical laboratories (robotics), enabling a "lab-in-the-loop" future.

In conclusion, while autonomous agents hold transformative potential for scientific discovery, SCIENCEBOARD reveals that current technology is still in its infancy. Significant advancements in grounding, domain adaptation, and multi-agent collaboration are required before agents can reliably assist scientists in complex research pipelines.