Technical Summary: Detecting and Correcting Misaligned Actions in Computer-Use Agents

1. Problem Definition

Computer-use agents (CUAs) have demonstrated significant progress in automating digital tasks but frequently produce misaligned actions—behaviors that deviate from the user's original intent. These deviations arise from two primary sources:

1. External Attacks: Malicious instructions embedded in the environment (e.g., indirect prompt injections in web content) that redirect the agent.
2. Internal Limitations: Erroneous reasoning, grounding failures, or misinterpretation of task boundaries that lead to unintended or harmful behaviors without adversarial input.

Existing safety research primarily focuses on policy violations or safety risks (e.g., data leaks, unauthorized access) and relies on predefined policies or known attack patterns. However, this safety-centric paradigm leaves a critical gap: agents may perform actions that are technically permissible and non-malicious yet still unjustifiably deviate from user intent (e.g., deleting a source file after exporting a PDF, or opening unrelated applications). Such behaviors erode user trust and degrade task efficiency but are not captured by traditional safety guardrails.

This work defines action alignment as whether a proposed action can be justified as advancing the user's authentic intent. An action is considered misaligned if it:

• Responds to external directives (e.g., injected instructions) rather than the user's instruction.
• Results in unauthorized or undesired consequences.
• Cannot be reasonably interpreted as contributing to the intended task.

The core problem addressed is misaligned action detection: determining whether a proposed action is misaligned before execution, without access to the actual consequences.

2. Methodology

2.1 MISACTBENCH: A Comprehensive Benchmark

To enable systematic evaluation, the authors constructed MISACTBENCH, a benchmark containing 2,264 human-annotated, action-level alignment labels across 558 realistic trajectories. The benchmark covers three categories of misaligned actions:

1. Malicious Instruction Following: Actions complying with external malicious directives (e.g., indirect prompt injections).
2. Harmful Unintended Behavior: Actions causing harm due to internal limitations (e.g., reasoning errors) rather than attacks.
3. Other Task-Irrelevant Behavior: Actions irrelevant to the task that degrade efficiency (e.g., opening unrelated apps).

Data Collection Strategy:

• External Attacks: Trajectories were collected by running diverse CUAs (including GPT-4o, Claude Sonnet 3.5/3.7, and specialized CUA models) on adversarial benchmarks (OS-Harm, DoomArena, RedTeamCUA, RiOSWorld) where attacks succeeded.
• Internal Limitations: Since internal failures are rare in benign execution, the authors employed a controlled synthesis procedure. They took benign trajectories from OSWorld, identified suitable steps to inject unintended subgoals (e.g., deleting a file after extraction), executed these subgoals as standalone tasks, and merged the resulting actions back into the benign trajectory.
• Annotation: A rigorous two-phase human annotation process was used, achieving a Fleiss' Kappa score of 0.84.

2.2 DEACTION: Runtime Guardrail

The authors propose DEACTION, a practical, plug-and-play runtime guardrail that detects and corrects misaligned actions before execution. It operates independently of the agent's internal parameters.

Architecture:
DEACTION employs a two-stage detection pipeline conditioned on a narrative summary of the interaction history (to reduce token consumption and latency compared to raw screenshots).

1. Stage 1: Fast Check: A lightweight LLM assessment determines if an action is "clearly aligned" (e.g., opening settings to adjust configurations). If yes, the action bypasses further analysis to minimize latency.
2. Stage 2: Systematic Analysis: For ambiguous actions, a deeper, structured analysis is performed via four components:
   • Injection Analysis: Inspects the screenshot for malicious/deceptive instructions.
   • Action Understanding: Interprets the semantics of the low-level action based on the current state.
   • Outcome Prediction: Anticipates state changes and long-term effects.
   • Misalignment Analysis: Integrates signals to determine if the action aligns with the user's objective.

Iterative Correction:
Instead of merely blocking misaligned actions, DEACTION provides structured feedback to the agent, explaining the identified issues (e.g., detected injections, predicted negative outcomes). The agent is prompted to revise its action, and the loop continues until an aligned action is proposed or a retry limit is reached.

3. Key Results

3.1 Offline Evaluation (MISACTBENCH)

• Performance: DEACTION outperforms baselines (Task Shield, InferAct) by over 15% absolute in F1 score across various backbone models (Qwen3-VL, Claude Sonnet 4.5, GPT-5.1).
• Precision vs. Recall: While baselines often achieve high recall with low precision (frequent false positives), DEACTION maintains strong recall while significantly improving precision (up to 89.9%), resulting in more reliable detection.
• History Representation: Using narrative summaries for history context outperformed using raw screenshots or actions alone, achieving better performance with lower latency (11s vs. 18s per step) and token consumption.
• Latency-Performance Trade-off: The two-stage design reduces latency compared to running systematic analysis on every step, while maintaining performance comparable to full systematic analysis.

3.2 Online Evaluation

• Adversarial Settings (RedTeamCUA): DEACTION reduced the Attack Success Rate (ASR) by over 90% across different CUAs while improving or preserving the Utility under Attack (UA).
• Benign Settings (OSWorld): DEACTION preserved or improved task success rates, demonstrating it does not over-constrain normal behavior.
• Efficiency: The guardrail added manageable overhead (approx. 25% of per-step execution time). The two-stage routing allowed 45% of actions to bypass deep reasoning.
• Correction Efficacy: Of the misaligned actions flagged, 78% were successfully corrected to aligned behavior through iterative feedback, with 62% fixed in a single revision.

4. Contributions

The paper makes three primary contributions:

1. Intent-Centric Perspective: It frames CUA deviations as an action misalignment problem rather than solely a safety/policy violation problem, identifying three common categories of misaligned actions in real-world deployments.
2. MISACTBENCH: The introduction of a comprehensive benchmark with 2,264 human-annotated, action-level alignment labels covering both external attacks and internal limitations.
3. DEACTION: A practical, universal guardrail that proactively detects misaligned actions before execution and iteratively corrects them via structured feedback, validated in both adversarial and benign environments.

5. Significance and Claims

The authors position this work as the first systematic study of misaligned action detection in CUAs. They claim that DEACTION serves as a plug-and-play protection mechanism that enhances the reliability and robustness of computer-use agents in open-ended, real-world environments.

The significance lies in addressing the gap between "safety" (avoiding harm) and "alignment" (faithfully executing user intent). By detecting and correcting actions that are technically permissible but intent-deviating, the work aims to reduce safety risks, unnecessary side effects, and execution failures. The authors assert that their approach supports the deployment of more trustworthy and controllable machine learning systems without requiring changes to the underlying agent architecture.

Limitations Acknowledged:
The paper notes that well-disguised malicious instructions can still bypass the fast check, and outcome prediction remains challenging due to current LLM world-modeling limitations. Additionally, the synthesis of internal failures, while controlled and validated, may not exhaust all possible internal failure modes. Future work is suggested to focus on more robust injection detection, fine-grained action understanding, and accurate intent-level reasoning.