Technical Summary: IMMERSEDPRIVACY

Problem Statement

As Vision-Language Models (VLMs) evolve from digital conversationalists into physically embodied assistants operating in intimate spaces (homes, hospitals, offices), their role as the "sole arbiter" of privacy violations becomes critical. Unlike digital models constrained to text generation, embodied agents possess the physical agency to observe, infer, and manipulate privacy-sensitive artifacts. However, current benchmarks for privacy awareness remain limited to unimodal, text-based representations (e.g., Planning Domain Definition Language or descriptive text). These static simulations fail to capture the multimodal, spatially distributed, and context-dependent nature of real-world privacy, where sensitive information is conveyed through visual clutter and acoustic cues rather than explicit labels. Consequently, there is a significant gap between simulated evaluations and the actual performance of VLMs in physical environments.

Methodology: IMMERSEDPRIVACY

To bridge this gap, the authors present IMMERSEDPRIVACY, an interactive audio-visual evaluation framework built upon a highly customized Unity-based simulator (extending VirtualHome). The framework simulates realistic physical environments across four themes: home, office, restaurant, and public venues. It evaluates physically grounded privacy awareness through three progressive tiers, each targeting a specific competency:

1. Tier 1: Perceptual Sensitivity Grounding

   • Goal: Evaluate the ability to parse complex, cluttered environments and identify inherently sensitive items (e.g., medical records, social security cards) without explicit textual hints.
   • Protocols:
     • Single-Turn: Models view a scene from multiple viewpoints and list sensitive items in one shot.
     • Multiple-Turn: Models engage in up to three rounds of progressively closer views (overview, mid-range, close-up), simulating active exploration. They can request closer views or commit to an answer.
   • Metrics: Sensitive Ratio (precision), Identified Ratio (recall), and the number of identified items.

2. Tier 2: Dynamic Socio-Contextual Adaptation

   • Goal: Evaluate the ability to uphold socio-contextual norms in shifting physical environments. Appropriateness depends on the social state (e.g., a meeting in progress vs. an empty room), signaled by visual cues (people at a table) and audio cues (overlapping speech vs. silence).
   • Protocols:
     • Rating Mode: Models rate candidate actions on a 1–5 scale.
     • Selection Mode: Models select the single most appropriate action from a candidate set.
   • Metrics: Mean Absolute Error (MAE) against human ratings and Selection Accuracy (SA).

3. Tier 3: History-Conditioned Inferential Adherence

   • Goal: Evaluate the ability to manage conflicts between explicit task commands and implicit privacy constraints inferred from prior multimodal observation history.
   • Protocol: Models observe a scripted history (video + dialogue) establishing a privacy boundary (e.g., a character hiding a gift). They are then given a benign but conflicting command (e.g., "move everything on the desk"). They must plan actions that complete the task while respecting the inferred boundary.
   • Metrics: Task Completeness Rate (TCR), Privacy Preservation Rate (PPR), and Exact Match (EM) for selecting the correct subset of actions.

The study evaluates 12 state-of-the-art models (including Gemini, GPT, Qwen, and InternVL families) across over 400 procedurally generated scenarios.

Key Results

The evaluation reveals consistent deficits across all models, highlighting a gap between linguistic knowledge and physical execution:

• Perceptual Fragility (Tier 1): All models exhibit monotonic performance decay as scene complexity (number of distractors) increases. In cluttered scenes, models fail to recognize sensitive items visually in over 40% of cases. While the "Multiple-Turn" protocol (providing close-up views) significantly improves recall (Identified Ratio), the Sensitive Ratio (precision) often degrades, indicating models over-flag benign distractors when cluttered. Human evaluations confirm the scenes are visually solvable, attributing failures to model perception limits rather than benchmark ambiguity.
• Social-Context Gap (Tier 2): No model exceeds 65% selection accuracy. Models struggle to distinguish between appropriate and inappropriate actions based on shifting social states. Error analysis reveals three failure modes: miscalibrated intervention intensity (acting regardless of context), spatial grounding failures (misinterpreting position), and unnecessary social conservatism (over-cautiousness).
• Command Override (Tier 3): When explicit commands conflict with inferred privacy constraints, models default to task completion. The best-performing model, gemini-3.1-pro, achieves a balanced Exact Match (selecting correct actions without violating privacy) in only 51% of cases. Most other models show a strong bias toward executing the explicit command, often selecting the privacy-violating action.
• Audio Modality: Ablation studies replacing audio with textual descriptions show no significant performance degradation, indicating that the failure is not due to an inability to process raw audio, but rather a fundamental misalignment in interpreting social context and history.

Significance and Claims

The paper claims that current VLMs in the physical world suffer from perceptual fragility and fail to let their knowledge of privacy cues govern their situated behavior. The authors argue that privacy-aware embodied behavior requires capabilities that current models lack at every stage of the pipeline:

1. Robust Multimodal Perception: The ability to identify sensitive objects amidst visual clutter.
2. Context-Sensitive Judgment: The ability to interpret shifting social norms via multimodal cues.
3. Constraint-Respecting Action Planning: The ability to prioritize inferred privacy boundaries over explicit user commands.

The authors conclude that existing text-based safety training is insufficient for real-world deployment. They emphasize the need for "physically grounded alignment strategies" and dedicated perceptual safeguards that extend beyond textual representations to address the unique vulnerabilities of embodied agents. The work provides a foundational diagnostic framework (IMMERSEDPRIVACY) to expose these specific failure modes, which are invisible to traditional text-only benchmarks.