Technical Summary: CloudWeb – Atmospheric Retrieval Hijacking in Remote Sensing Vision-Language RAG

Problem Statement

Multimodal Retrieval-Augmented Generation (RAG) systems increasingly rely on vision-language retrievers to ground visual queries in external textual evidence. While prior adversarial research has focused on manipulating the retrieval corpus (knowledge poisoning) or targeting end-task predictions in vision-language models (VLMs), the evidence retrieval stage of remote sensing multimodal RAG remains largely unexplored.

Specifically, there is a gap in understanding whether visually plausible, natural-looking atmospheric changes to a query image can systematically hijack the retrieval process before generation begins. If a query image is mapped to an incorrect region of the shared vision-language embedding space, the system may retrieve misleading weather-related evidence (e.g., "cloudy," "foggy," "hazy") instead of scene-consistent evidence. This compromises the grounding of the final generated response, causing the system to produce "evidence-grounded hallucinations" where the output appears supported by retrieved context, even though that context was hijacked by the input perturbation.

Methodology: CloudWeb

The authors propose CloudWeb, an input-space atmospheric retrieval hijacking attack. Unlike traditional adversarial attacks that add imperceptible pixel noise or manipulate the database, CloudWeb modifies only the input image by overlaying parameterized cloud-, haze-, fog-, and mist-like patterns. The attack assumes a threat model where the adversary has offline access to the retriever's encoders for optimization but does not modify the retriever, generator, vector database, or retrieval logic at deployment.

1. Atmospheric Parameterization

CloudWeb restricts perturbations to a low-dimensional, naturalistic atmospheric family rather than arbitrary pixel noise. The perturbation is parameterized by $\theta = (\rho, \alpha, \beta, \sigma, r)$, controlling:

• Atmospheric severity ($\rho$) and cloud opacity ($\alpha$).
• Haze strength ($\beta$) and edge softness ($\sigma$).
• Stochastic texture seed ($r$).

The attack generates a smooth low-frequency cloud mask $M_\theta$ and an opacity map $A_\theta$. It applies a global haze transformation and blends it with the original image, ensuring the perturbation retains natural atmospheric structures (e.g., soft occlusion, reduced contrast) rather than high-frequency noise.

2. Retrieval-Oriented Objective

CloudWeb optimizes the retrieval behavior directly, not classification logits. The total loss function $L(\theta)$ combines five terms:

• Target Attraction ($L_{tar}$): Pulls the adversarial image embedding toward a set of target atmospheric evidence texts ($T_q$).
• Source Suppression ($L_{src}$): Reduces similarity to the original scene evidence ($S_q$).
• Rank Separation ($L_{rank}$): Enforces a margin to ensure target evidence ranks higher than source evidence.
• Naturalness Regularization ($L_{nat}$): Uses Total Variation and Gaussian smoothing to ensure the perturbation looks like natural atmospheric phenomena.
• Coverage Constraint ($L_{area}$): Prevents the attack from succeeding solely through excessive spatial coverage.

The optimization is performed using Differential Evolution with local refinement to find the optimal parameters $\theta$.

Experimental Setup

The authors evaluated CloudWeb on a seven-dataset remote sensing RAG benchmark covering scene classification, image captioning, change captioning, disaster imagery, and visual question answering (VQA). The datasets include NWPU-RESISC45, RSICD, RSVQA-LR, LEVIR-CC, FloodNet, RSIVQA-UCM, and RSIVQA-Sydney.

The evaluation involved:

• Five CLIP-style retrievers: GeoRSCLIP, RemoteCLIP, OpenAI CLIP (ViT-B/32, ViT-L/14), and OpenCLIP.
• Six downstream VLMs: LLaVA-1.5/1.6, InstructBLIP, Qwen2.5-VL, GeoChat, and H2RSVLM.
• Metrics:
  • Retrieval: $T@k$ (Top-$k$ Changed) for generic disruption and $W@k$ (Weather@k) for targeted weather-evidence hijacking.
  • Generation: Weather Hallucination Rate (WHR) and Attack Success Rate (ASR) on a subset of 303 cases where retrieval hijacking was successful.

Key Results

1. Retrieval-Stage Hijacking

CloudWeb consistently outperformed clean retrieval, handcrafted atmospheric baselines (e.g., Gaussian Blur, Brightness Haze), and random noise perturbations.

• On GeoRSCLIP ViT-B/32, the targeted weather evidence injection rate ($W@5$) increased from 0.71% (clean) to 43.29% (CloudWeb).
• The attack successfully redirected top-ranked evidence from scene-consistent descriptions (e.g., "residential area," "school campus") to weather-related descriptions (e.g., "cloudy low-visibility," "foggy overpass").
• The vulnerability was not retriever-specific; similar gains were observed across all five CLIP-style models, though GeoRSCLIP and OpenAI CLIP ViT-L/14 showed the highest susceptibility.

2. Downstream Generation Impact

The hijacking of retrieval evidence propagated to the final generation stage.

• On the Qwen2.5-VL generator, CloudWeb induced 172 newly weather-related responses out of 303 cases, achieving a Weather Hallucination Rate (WHR) of 61.39%.
• Clean inputs led models to describe scene-consistent content (e.g., farmlands, rivers), while adversarial inputs triggered descriptions involving fog, clouds, and low visibility, even when the underlying image remained largely recognizable.
• This confirms that retrieval-stage hijacking can steer VLM outputs toward semantically shifted, weather-hallucinated responses.

3. Mechanism and Robustness Analysis

• Attention Shift: Class Activation Map (CAM) visualizations revealed that CloudWeb shifts the retriever's attention from source-relevant structures (roads, buildings) to atmospheric regions (clouds, haze), effectively re-grounding the query in the embedding space.
• Robustness: The attack remained effective under common post-processing transformations, including JPEG compression, resizing, and mild Gaussian blur.
• Ablation: The study confirmed that both the atmospheric structure (opacity/severity) and the retrieval-oriented optimization are critical. Removing the target attraction or rank separation terms significantly degraded performance.

Significance and Claims

The paper claims to present the first study of retrieval-stage atmospheric evidence hijacking for remote sensing multimodal RAG. Its primary contributions are:

1. New Failure Mode: It identifies a practical failure mode where natural-looking input-space atmospheric changes systematically compromise evidence retrieval before generation begins. This challenges the assumption that RAG robustness only requires securing the generator or the database.
2. Input-Only Threat: The attack demonstrates that an adversary can hijack the entire RAG pipeline by modifying only the input image, without needing access to or modification of the retriever, generator, or knowledge base.
3. Semantic Redirection: Unlike generic adversarial noise that degrades performance, CloudWeb acts as a "semantic carrier," redirecting the query embedding toward a specific atmospheric semantic region (weather) that is strongly associated with the perturbation in vision-language embedding spaces.

The authors conclude that robust remote sensing RAG systems require retrieval-aware defenses (e.g., atmospheric risk screening, consistency-aware retrieval, and scene-weather disentangled reranking) rather than relying solely on generator-side safeguards. They emphasize that the risk lies in targeted evidence hijacking, not just generic neighbor disruption.