Safe, or Simply Incapable? Rethinking Safety Evaluation for Phone-Use Agents

Technical Summary: Safe, or Simply Incapable? Rethinking Safety Evaluation for Phone-Use Agents

Problem Statement
Current safety evaluations for phone-use agents suffer from a fundamental ambiguity: a harmless outcome does not necessarily indicate safe behavior. When an agent avoids causing harm at a risky moment, it may be due to safe judgment (recognizing the risk and choosing the correct safe action) or inability to act (failing to understand the interface, locate controls, or execute any relevant action). Existing benchmarks often conflate these distinct causes under aggregate metrics like "task success," "refusal," or "final harmful outcome." This conflation leads to two critical issues:

1. Overestimation of weak models: Agents that fail to act are incorrectly credited with safety because no harm occurred.
2. Misdiagnosis of strong models: Agents that understand the interface but make poor safety choices are not distinguished from those that simply cannot act.

As agents transition from passive assistants to systems capable of directly operating smartphones, one wrong tap can expose private data, authorize payments, or grant permissions. Evaluating whether an agent makes the right choice at the moment a safety boundary becomes action-relevant is distinct from evaluating whether it can complete a task.

Methodology: PHONESAFETY
To address this, the authors introduce PHONESAFETY, a benchmark designed to evaluate agents at safety-critical moments rather than across full task episodes.

• Data Construction: The benchmark consists of 700 safety-critical moments drawn from 4,512 real-device trajectories across 130+ Chinese mobile apps. These trajectories cover diverse scenarios including native apps, mini-programs, and cross-app workflows.
• Evaluation Unit: The unit of analysis is the specific decision point where the next action can directly determine the safety of the interaction. The model is provided with the user instruction, interaction history, and current screen context, then asked to predict the next phone action.
• Three-Way Decomposition: Instead of a binary safe/unsafe label, outcomes are categorized into three mutually exclusive classes:
  1. Safe Action: The model engages with the decision and chooses the safe side (e.g., asking for confirmation, refusing a harmful instruction, closing a deceptive popup).
  2. Unsafe Action: The model engages with the decision but crosses the safety boundary (e.g., proceeding with payment without confirmation, granting unnecessary permissions).
  3. Failing to Do Anything Useful (Capability Failure): The model fails to realize either the safe or unsafe behavior. This includes tapping irrelevant areas, leaving the page, or producing malformed actions. This category is treated as an operational signal of inability to act rather than a claim about deep model internals.
• Metrics: The authors report the Safe-action rate, Unsafe-action rate, and Capability-Failure Rate (CFR). They also track $1 - \text{CFR}$ (the rate of producing any relevant action) as a proxy for general operational capability.

Key Results
The paper evaluates eight representative phone-use agents (including Gemini 3.1 Pro, Claude Opus 4.6, and AutoGLM 9B) and reveals two primary patterns:

1. General Capability Does Not Predict Safe Choices:
   There is no reliable correlation between a model's general phone-use performance and its safety judgment at critical moments.

   • Evidence: Models ranking high on general tasks (e.g., Seed 2.0 Pro) do not necessarily have the highest safe-action rates. Conversely, models with lower general scores sometimes exhibit safer decision-making when the boundary is explicit.
   • Implication: High operational capability does not guarantee safe behavior; dedicated safety evaluation is required.

2. Distinct Failure Modes:
   Failures are not monolithic; they split into two distinct patterns based on the scenario:

   • Unsafe Choices: In scenarios requiring simple refusal (e.g., "Safety" family), failures are predominantly unsafe actions. The model understands the interface but chooses the wrong side of the decision (e.g., proceeding with a harmful instruction).
   • Inability to Act: In visually and operationally demanding scenarios (e.g., "Trap Resistance," "Permission Minimization"), failures are predominantly failures to do anything useful. The model cannot ground the interface or execute the necessary action.
   • Correlation: The rate of "failing to do anything useful" (CFR) strongly correlates with general phone-use capability ($\rho = 0.922$), whereas safe-action rates do not.

3. Protocol Invariance of Inability:
   When the evaluation protocol (defining what constitutes a safe vs. unsafe action) is varied, the Safe-action rate shifts significantly. However, the Capability-Failure Rate (CFR) remains stable ($\Delta \text{CFR} = 0.0$). This confirms that "failing to do anything useful" is a signal of operational inability rather than a sensitivity to specific safety definitions.

Significance and Claims
The paper argues that for action-taking agents, harmlessness is not sufficient evidence of safety. A system should not receive safety credit merely because no harm occurred if that outcome was the result of an inability to act.

• Core Contribution: The introduction of a three-way evaluation framework that separates unsafe judgment from inability to act.
• Methodological Shift: Moving from episode-level outcomes to safety-critical moments allows for the isolation of the specific decision point where safety is determined.
• Practical Implication: Safety evaluation must distinguish between a model that chooses poorly and a model that cannot act. Treating "no relevant action" as a safety success is a methodological error that obscures the true nature of agent risks.

The study is limited to offline evaluation within a Chinese mobile ecosystem, but the authors posit that the underlying principle—that harmless outcomes must be interpreted rather than merely observed—applies broadly to agents capable of acting in the real world.