OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents

1. Problem Statement

The rapid advancement of Multimodal Large Language Models (MLLMs) has enabled the development of OS Agents—autonomous systems capable of navigating Graphical User Interfaces (GUIs) on computers and mobile devices to execute complex tasks. While existing research focuses heavily on task completion rates, there is a critical lack of rigorous evaluation frameworks addressing four other vital dimensions required for deploying these agents as trustworthy daily partners:

• Safety: Current benchmarks focus on narrow scenarios (e.g., pop-ups) and fail to cover diverse environmental and human-induced hazards.
• Performance: Existing datasets often contain low-value trajectories, annotation errors, or tasks that are too simple or impossibly difficult, leading to biased evaluations.
• Efficiency: Most benchmarks measure efficiency solely by the number of steps, ignoring temporal latency and token consumption (which directly correlates to operational costs).
• Robustness: Current evaluations typically test robustness in a single modality (visual or textual), failing to assess how agents handle cross-modal disturbances.

2. Methodology: The OS-SPEAR Framework

The authors propose OS-SPEAR, a comprehensive toolkit designed to systematically evaluate OS agents across the four dimensions mentioned above. The framework consists of four specialized subsets and an automated analysis tool.

A. The Four Specialized Subsets

1. S-subset (Safety):
   • Goal: Evaluate agent resilience against unsafe factors.
   • Construction: Aggregates three types of threats:
     • Environmental Distractions: Intentionally injected factors (e.g., persuasive pop-ups) to mislead the agent.
     • Real-world Anomalies: Naturally occurring issues (e.g., network drops, unexpected system dialogs).
     • Adversarial Misleading: Maliciously crafted inputs designed to induce unsafe actions.
2. P-subset (Performance):
   • Goal: Curate high-quality, high-value evaluation trajectories.
   • Construction: Uses a two-stage filtering process:
     • Trajectory Value Estimation: Small-scale agents filter out trivial tasks; large-scale agents vote on remaining tasks to label them as Easy, Medium, or Hard. Tasks too difficult even for large models (likely annotation errors) are discarded.
     • Stratified Sampling: Ensures a balanced distribution of difficulty levels (approx. 4:3:3 ratio) to create a robust test set.
3. E-subset (Efficiency):
   • Goal: Measure economic and temporal efficiency.
   • Metrics: Tracks inference time and token consumption.
   • Cost Calculation: Adopts a user-centric billing model where the total cost is calculated as Input Tokens + (3 × Output Tokens), reflecting the higher resource intensity of generation tasks.
4. R-subset (Robustness):
   • Goal: Assess stability under perturbations across both visual and textual modalities.
   • Visual Perturbations: Includes masking, zooming (cropping), and Gaussian noise at varying intensities (30%, 50%, 70%).
   • Textual Perturbations: Includes state conflicts (contradictory status), bad memory (shuffled history), bad knowledge (scrambled instructions), and irrelevant memory/knowledge.

B. Automated Analysis Tool

OS-SPEAR includes a multi-agent system (four expert agents + one integrated agent) that processes evaluation logs to generate human-readable diagnostic reports. The experts analyze specific dimensions (Safety, Performance, Efficiency, Robustness), while the integrated agent synthesizes these into a holistic report with rankings and risk assessments.

3. Key Contributions

1. Systematic Audit: Identified critical gaps in existing benchmarks, specifically the lack of standardized metrics for efficiency, cross-modal robustness, and diverse safety scenarios.
2. OS-SPEAR Toolkit: Introduced a unified framework with four specialized subsets (S, P, E, R) and a multi-agent analysis tool for generating diagnostic reports.
3. Large-Scale Evaluation: Conducted an extensive benchmarking study on 22 popular OS agents, including both general-purpose MLLMs (e.g., Qwen, GLM) and specialized OS agents (e.g., UI-TARS, GUI-Owl).
4. Empirical Insights: Uncovered critical trade-offs and vulnerabilities in the current state of OS agents, providing a quantitative baseline for future development.

4. Experimental Results & Key Findings

The evaluation of 22 agents yielded several significant insights:

• Efficiency vs. Safety/Robustness Trade-off: Models optimized for low inference costs (e.g., smaller parameter models) often exhibit significantly lower safety and robustness scores. Efficiency-oriented designs tend to omit critical contextual reasoning, making them fragile under adversarial conditions.
• Specialized vs. General Models: Specialized OS agents (trained specifically for GUI tasks) consistently outperform general-purpose models in overall rankings. Domain adaptation improves grounding, operation logic understanding, and static OS knowledge.
• Model Scaling Limits:
  • Larger models generally show better safety performance.
  • However, within the same model family, simply increasing parameter size does not guarantee a higher overall rank. The performance gains from scaling are often offset by increased inference costs and diminishing returns in task success rates.
• Modality-Specific Vulnerabilities:
  • Visual: Agents are highly sensitive to the completeness of visual inputs. Masking or zooming (disrupting layout) causes significant performance drops, whereas Gaussian noise has a lesser impact.
  • Textual: Agents are surprisingly insensitive to temporal causality in memory (shuffled history) but are highly vulnerable to conflicting or irrelevant textual information, which can easily mislead action generation.
• Real-world Anomalies: Agents struggle more with naturally occurring anomalies (e.g., unexpected pop-ups) than with artificially injected adversarial attacks, as the former are harder to distinguish from normal environment noise.

5. Significance

OS-SPEAR addresses a critical gap in the transition of MLLMs from passive text generators to active, trustworthy behavioral executors. By providing a standardized, multi-dimensional evaluation framework, it:

• Moves the field beyond simple task completion metrics to include safety, cost, and reliability.
• Offers a diagnostic tool for developers to identify specific weaknesses (e.g., visual fragility vs. textual confusion).
• Establishes a quantitative baseline that highlights the current limitations of OS agents, guiding the development of the next generation of safe, efficient, and robust autonomous systems for real-world deployment.

The dataset and code are publicly available to facilitate further research in this domain.