SABER: A Stealthy Agentic Black-Box Attack Framework for Vision-Language-Action Models

1. Problem Statement

Vision-Language-Action (VLA) models enable robots to execute tasks based on natural language instructions and visual observations. However, the reliance on text inputs creates a critical vulnerability: small perturbations to the instruction can cause significant physical failures, such as task failure, inefficient execution, or safety constraint violations.

Existing adversarial attacks on VLAs suffer from three main limitations:

1. Lack of Automation: Most rely on manual perturbation design or fixed heuristics.
2. High Cost & Low Stealth: Methods often use expensive GPT-based searches or require excessive instruction rewrites, making them impractical and easily detectable in real-world settings.
3. Poor Generalization: They struggle to adapt across diverse tasks, unseen objectives, and different frozen VLA policies without gradient access.

The paper addresses the need for a general-purpose, automated, black-box attacker that can generate minimal, plausible instruction edits to induce targeted behavioral degradation in frozen VLA models.

2. Methodology: SABER Framework

SABER (Stealthy Agentic Black-box Attack Framework for VLA) treats the attack generation as a sequential decision-making problem solved by a ReAct-style agent trained via Group Relative Policy Optimization (GRPO).

A. Core Components

• Attack Agent: A single LLM-based agent (Qwen2.5-3B-Instruct) trained to act as a "Red-Team" attacker. It operates without access to the target VLA's gradients.
• Perturbation Toolbox: The agent uses three distinct tool families to edit instructions, following a FIND $\rightarrow$ APPLY workflow:
  1. Character-level: Typo-style edits (insertion, deletion, substitution, transposition) to simulate OCR errors or typos (e.g., "pick" $\rightarrow$ "plck").
  2. Token-level: Word or subword replacements/removals (e.g., "top" $\rightarrow$ "semi-top").
  3. Prompt-level: Injection of clauses, verification steps, or extra constraints (e.g., "verify the drawer is open first").
• Training Objective: The agent optimizes a reward function $J_{atk}$ that balances adversarial effectiveness against stealth:
  $$J_{atk}(\psi) = \mathbb{E}[R_O(\delta; \tau) - \lambda P_{stealth}(\delta)]$$
  Where $R_O$ is the reward for achieving a specific objective (Task Failure, Action Inflation, or Constraint Violation), and $P_{stealth}$ penalizes excessive tool calls and character edits.

B. Training Pipeline

The training follows a two-stage process:

1. Cold-Start (SFT): The agent is initially fine-tuned via Supervised Fine-Tuning (SFT) on a small dataset of tool-use trajectories generated by a frozen GPT model. This ensures the agent learns valid tool-calling syntax and the FIND $\rightarrow$ APPLY protocol before reinforcement learning begins.
2. Agentic RL (GRPO): The agent is optimized using GRPO. In each step, multiple attack rollouts are generated for the same scenario. The agent learns from the relative advantage of these rollouts based on the final outcome (success/failure, action length, constraint violations) compared to a clean baseline.

3. Key Contributions

1. Formulation of Black-Box VLA Attacks: The authors formulate instruction-only attacks as a constrained optimization problem over robot behavioral objectives under bounded edit budgets, moving beyond simple text-output attacks to physical behavior attacks.
2. Agentic Attack Approach: They propose a novel framework where a single GRPO-trained agent adaptively composes character, token, and prompt-level perturbations. This eliminates the need for model-specific redesign or gradient access.
3. Comprehensive Evaluation: The framework is evaluated on the LIBERO benchmark across six state-of-the-art VLA models (including $\pi_0$, $\pi_0.5$, X-VLA, GR00T-N1.5, DeepThinkVLA, and InternVLA-M1).

4. Experimental Results

SABER was tested against three attack objectives: Task Failure, Action Inflation (unnecessarily long sequences), and Constraint Violation.

• Performance Degradation: Across six VLA models, SABER achieved:
  • 20.6% reduction in task success rate.
  • 55% increase in action-sequence length.
  • 33% increase in constraint violations.
• Efficiency & Stealth: Compared to strong GPT-based baselines (using the same tool interface), SABER was significantly more efficient:
  • 21.1% fewer tool calls.
  • 54.7% fewer character edits.
  • This demonstrates that the RL-trained agent learns "high-leverage" perturbations that achieve maximum damage with minimal changes.
• Ablation Studies:
  • Cold-Start Necessity: Removing the SFT cold-start stage resulted in a 5.8% drop in Attack Success Rate (ASR). Without SFT, the agent converged to low-cost but ineffective behaviors, failing to discover valid attack patterns.
  • Strategy Evolution: Early training relied on broad prompt-level edits, while the final policy shifted toward selective token-level edits that preserved stealth while maintaining effectiveness.

5. Significance and Impact

• Red-Teaming Tool: SABER provides a scalable, automated tool for stress-testing robotic foundation models before deployment, uncovering failure modes (like unsafe behavior or inefficiency) that static prompt attacks miss.
• Vulnerability Insight: The results prove that small, plausible instruction edits are sufficient to substantially degrade robot execution, highlighting the fragility of the language interface in embodied AI.
• Methodological Advance: It demonstrates the efficacy of Agentic RL for black-box security evaluation, showing that an agent can learn to navigate discrete action spaces (tool calls) to optimize complex, delayed rewards (physical robot outcomes) without gradient access.

Limitations & Future Work:
Currently, SABER is limited to text-only perturbations in simulation. Future work aims to extend attacks to multimodal settings (perturbing vision and language jointly) and target the intermediate reasoning processes (chain-of-thought) of the VLA models.