Do Not Leave a Gap: Hallucination-Free Object Concealment in Vision-Language Models

1. Problem Statement

Vision-Language Models (VLMs) are increasingly used for image captioning and visual reasoning but are vulnerable to adversarial attacks designed to hide specific objects (Region of Interest or ROI) for privacy purposes.

• The Limitation of Prior Work: Existing object concealment methods (e.g., masking, patch erasing, or attention suppression) work by removing or nullifying visual features associated with the target object.
• The Consequence: This suppression creates a semantic discontinuity or a "representational gap" in the visual encoder. When the language decoder encounters this missing evidence, it attempts to compensate by inferring content based on learned priors, leading to hallucinations (inventing plausible but incorrect objects or attributes).
• The Core Hypothesis: The authors argue that hallucination is not an inevitable side effect of object removal, but a direct consequence of the semantic gap created by suppression-based attacks.

2. Methodology: Background-Consistent Re-encoding (BCR)

The authors propose Background-Consistent Re-encoding (BCR), a novel attack framework that conceals objects by re-encoding their visual representations rather than removing them. The goal is to make the target object statistically and semantically indistinguishable from the surrounding background, thereby preserving the continuity of the visual feature manifold.

Key Technical Components

BCR operates via a pixel-level optimization framework that modifies the input image $x$ to produce an adversarial image $x_{adv}$ while keeping the VLM weights frozen. The optimization minimizes a composite loss function across multiple layers of the vision transformer:

1. Statistical Alignment Loss ($\mathcal{L}_{stat}$):

   • Aligns the first-order (mean) and second-order (standard deviation) statistics of the ROI tokens with those of the background tokens.
   • Ensures the distribution of the hidden object matches the background, preventing detectable anomalies.

2. Dictionary Projection Loss ($\mathcal{L}_{dict}$):

   • Enforces semantic consistency by projecting ROI features onto the "dictionary" of background features.
   • ROI tokens are reconstructed as convex combinations of background tokens, ensuring they lie within the background feature span.

3. Background Preservation Loss ($\mathcal{L}_{pres}$):

   • Explicitly penalizes changes to non-ROI (background) features.
   • Ensures global scene semantics remain stable and prevents collateral semantic drift.

4. Total Variation Regularization ($\mathcal{L}_{tv}$):

   • Encourages spatially smooth perturbations to ensure the adversarial image remains visually coherent to human observers.

Optimization Strategy:
The attack is applied to intermediate layers of the vision encoder (e.g., layers 22–25 in ViT) to enforce consistency across the hierarchy. The language model remains frozen and is used only for evaluation.

3. Key Contributions

• Novel Paradigm: Introduced BCR, the first object concealment method that preserves token structure and attention flow, avoiding the representational voids that trigger hallucinations.
• Theoretical Insight: Demonstrated that hallucination in VLMs under concealment attacks is driven by representational discontinuity, not merely the absence of an object.
• Evaluation Framework: Designed specific metrics to measure Grounded Hallucination Rate (using external grounding models like GLIP to verify if new objects are visually supported) and Global Preservation.
• Comprehensive Experiments: Validated the approach on state-of-the-art models (LLaVA-1.5, BLIP-2, InstructBLIP) and datasets (ImageNet, COCO).

4. Experimental Results

The authors compared BCR against suppression-based baselines (e.g., VIP, Masking, PRM) across three models.

• Hallucination Reduction: BCR significantly reduces grounded hallucinations.
  • Example: On InstructBLIP with ImageNet, BCR reduced the grounded hallucination rate from 0.43 (VIP) to 0.19.
  • Across all models, BCR reduced hallucinations by up to 3× compared to attention-suppression attacks.
• Global Preservation: BCR preserves non-target objects much better.
  • On ImageNet/InstructBLIP, Global Preservation increased from 0.57 (VIP) to 0.81 (BCR).
  • BCR maintained up to 86% of non-target objects in captions.
• Concealment Success: BCR achieved concealment rates comparable to suppression methods (e.g., ~92–98% success in hiding the target object).
• Perceptual Fidelity: BCR produced adversarial images with higher Structural Similarity (SSIM) and lower Learned Perceptual Image Patch Similarity (LPIPS) compared to baselines, meaning the visual artifacts were less noticeable to humans.

Qualitative Findings:

• Suppression Methods (VIP/Masking): Often caused the model to hallucinate unrelated objects (e.g., describing a park scene as an abstract painting or inventing a fire hydrant) to fill the "gap."
• BCR: Successfully hid the target object (e.g., a person) while the model continued to describe the scene accurately (e.g., mentioning pigeons and benches) without inventing new entities. In some cases, the object was semantically substituted with a background-consistent entity (e.g., a person replaced by a sea urchin in a marine context) rather than a hallucinated object.

5. Significance and Conclusion

This work fundamentally shifts the understanding of adversarial attacks on generative multimodal models.

• Insight: It proves that maintaining semantic continuity in the visual representation is crucial for preventing hallucinations. Simply removing information is insufficient; the representation must remain coherent.
• Implication: BCR offers a more robust method for visual privacy that protects sensitive data without degrading the model's ability to understand the rest of the scene or inducing false outputs.
• Future Direction: The paper suggests that future adversarial analysis and robustness training for VLMs should focus on preserving representational continuity rather than just blocking specific features.

In summary, Guesmi and Shafique demonstrate that by "filling the gap" with statistically consistent background data rather than leaving a void, one can effectively hide objects in VLMs without triggering the model's tendency to hallucinate.