If you're waiting for a sign... that might not be it! Mitigating Trust Boundary Confusion from Visual Injections on Vision-Language Agentic Systems

1. Problem Statement: Trust Boundary Confusion

The paper addresses a critical security vulnerability in Vision-Language Agentic Systems (VLAS)—embodied AI systems (e.g., autonomous robots, drones) that use Large Vision-Language Models (LVLMs) to perceive environments and execute actions.

• The Core Issue: In real-world environments, agents must interpret in-band signals (e.g., traffic lights, warning labels) that act as legitimate safety constraints. However, these same visual channels can be exploited by attackers to inject misleading instructions (e.g., a "Turn Right" sign on a cliff edge).
• Trust Boundary Confusion: The authors define this as the agent's inability to distinguish between trusted user intent ($I_u$) and untrusted environmental payloads ($I_v$).
  • Fail-Closed: The agent ignores all visual cues (including legitimate safety warnings) to strictly follow user text, leading to unsafe outcomes (e.g., ignoring a "Do Not Enter" sign).
  • Fail-Open: The agent prioritizes visual injections over user intent, leading to hijacked behavior (e.g., following a malicious sign).
• The Gap: Previous research focused on text-only prompt injections or visual perception errors (VQA). This paper highlights the decision-making level ambiguity where agents must choose which source of authority to trust.

2. Methodology & Evaluation Framework

A. Dual-Intent Dataset & Evaluation

The authors constructed a Dual-Intent Evaluation Framework to systematically test this confusion across two domains:

1. Image Editing (Digital): 2,500 test cases involving copyright warnings (helpful) vs. unsafe editing goals (misleading).
2. Embodied Manipulation (Physical): 400 test cases involving safety constraints (e.g., "microwave is broken") vs. harmful physical actions.

They evaluated 7 representative LVLMs (including GPT-4o, GPT-5, Gemini, Claude, Qwen, InternVL, DeepSeek) using two attack vectors:

• Structure-based Injection: Overlaying text/symbols that mimic natural signage (e.g., "Ignore previous instructions," "Task Completed").
• Noise-based Injection: Using gradient-optimized adversarial perturbations (white-box) to force the model's latent space to align with the attack.

B. Key Research Questions (RQs)

• RQ1: Does OCR capability dictate action planning?
• RQ2: How do agents resolve conflicts between user intent and environmental cues?
• RQ3: Can structural or noise-based enhancements break the "modality laziness" (ignoring visual text)?
• RQ4: Can a new defense preserve helpful cues while neutralizing attacks?

3. Key Findings & Results

A. Vulnerability Analysis

• Modality Laziness: Despite strong OCR capabilities, many models (especially open-source ones) exhibit "modality laziness," ignoring visual text entirely and blindly following user text prompts (Fail-Closed).
• Asymmetric Sensitivity: Models with better spatial perception (e.g., Gemini, Claude) are more susceptible to Fail-Open attacks. They prioritize malicious visual injections over helpful safety cues.
• Attack Effectiveness:
  • Structure-based: Semantic enhancements (e.g., adding "WARNING!!!" or "Ignore previous instructions" to the visual text) significantly increased attack success rates, particularly for "Goal Changing" attacks.
  • Noise-based: White-box gradient optimization achieved near-perfect attack success (>94%), proving that visual signals can mathematically override trusted text instructions. However, these attacks showed poor cross-model transferability.

B. The Proposed Solution: Decoupled Multi-Agent Defense

To solve this, the authors propose a Multi-Agent Mitigation Framework that separates perception from decision-making, creating a dynamic trust boundary. The system consists of three specialized planes:

1. Signaling Plane (Observation Agent): An LVLM dedicated solely to extracting all in-band visual signals (text, symbols, watermarks) without filtering. This forces the system to "see" everything, preventing Fail-Closed behavior.
2. Trust Plane (Judgment Agent): A cognitive core that uses In-Context Learning (ICL) to adjudicate the intent of the extracted signals. It determines if a visual cue is a Helpful Constraint (legitimate safety) or a Misleading Payload (attack).
3. Execution Plane (LVLM Agent): Generates the final action plan based only on the Judgment Agent's routing decision.
   • If Helpful: The agent overrides user intent to follow the safety cue.
   • If Misleading: The agent ignores the visual cue and follows the user intent.

4. Performance Results

The proposed framework demonstrated superior performance compared to state-of-the-art baselines (OCR filtering, JPEG purification, simple LVLM classifiers):

• Mitigation of Misleading Attacks:
  • Reduced the success rate of misleading injections from high vulnerability (up to 45% in some models) to < 3% across all tested architectures.
  • Effectively neutralized both structure-based and noise-based attacks.
• Preservation of Utility (Helpful Cues):
  • Maintained > 95% success rate for legitimate safety cues (e.g., obeying a "Do Not Touch" warning).
  • Outperformed "Fail-Closed" defenses that blindly filter all visual text.
• Generalization: The framework is model-agnostic, showing consistent robustness across proprietary (GPT-4o, Gemini) and open-source (Qwen, InternVL) models.
• Sim-to-Real Validation: The framework was tested in physical deployment scenarios (robotic arm manipulation, urban driving simulation, UAV landing), confirming that the mitigation transfers from simulation to real-world hardware.

5. Significance and Contributions

1. Formalization of Trust Boundary Confusion: The paper introduces a new security paradigm for embodied AI, moving beyond simple prompt injection to the complex conflict between user intent and environmental authority.
2. Dual-Intent Dataset: Provides the first benchmark specifically designed to evaluate the trade-off between following helpful safety cues and resisting malicious visual hijacking.
3. Architectural Innovation: Proposes a decoupled multi-agent architecture that solves the "Fail-Closed vs. Fail-Open" dilemma. Instead of filtering inputs, it reasons about intent, allowing agents to be both safe and useful.
4. Real-World Impact: Demonstrates that current embodied AI systems are vulnerable to environmental hijacking but that this can be mitigated through architectural changes rather than just model fine-tuning.

Conclusion

The paper concludes that as LVLMs become more capable of spatial reasoning, they become more vulnerable to visual hijacking. The solution lies not in suppressing vision, but in architectural safeguards that explicitly separate signal extraction from trust adjudication. This approach ensures that embodied agents can navigate the complex, multi-intent signaling of the real world without compromising safety or utility.