On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces

Technical Summary: Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces

Problem Statement
Deep neural networks, particularly modern Vision-Language Models (VLMs), are known to be vulnerable to adversarial perturbations that alter model predictions while remaining visually imperceptible. While existing research has extensively analyzed this vulnerability through input-space geometry, decision-boundary properties, feature robustness, and input–output Jacobians, the spectral behavior of intermediate linear transformations within these networks remains largely unexplored. Specifically, there is a lack of understanding regarding how adversarial signals propagate through the singular-vector subspaces of the linear layers (e.g., in self-attention, feed-forward networks, and fusion modules) that constitute the core of transformer-based VLMs. This paper investigates whether the "bottom" singular-vector subspaces—directions associated with near-zero singular values that attenuate information—constitute a previously overlooked attack surface.

Methodology
The authors propose a framework grounded in the analogy between adversarial propagation and the instability of ill-posed inverse problems. In inverse problems, reconstruction fails along near-null singular directions; similarly, the authors hypothesize that aligning intermediate representations with the bottom singular-vector subspaces of linear transformations in VLMs leads to information attenuation and semantic degradation.

To test this, the paper introduces the Spectral Subspace Guided Representation Attack (SSGRA), a white-box untargeted attack. The methodology involves:

1. Spectral Decomposition: Performing Singular Value Decomposition (SVD) on the weight matrices of intermediate linear transformations within the VLM.
2. Alignment Measure: Defining a metric, $\Psi_k$, to quantify the energy of token representations projected onto specific singular subspaces (top-$k$ vs. bottom-$k$).
3. Optimization Objective: SSGRA formulates an optimization problem that combines two terms:
   • A representation discrepancy term (based on Blockwise Similarity Attack) to disrupt feature hierarchies.
   • A spectral guidance term that explicitly maximizes the alignment of adversarial intermediate representations with the bottom-$s$ singular-vector subspaces of selected linear layers.
4. Layer-wise Probing: The authors employ a probing mechanism to analyze how adversarial optimization naturally shifts representations toward bottom singular subspaces, even without explicit spectral guidance, and to compare the spectral characteristics of different VLMs (Qwen2.5-VL, LLaVA-1.5, and Gemma-3).

Key Contributions

• Identification of a New Attack Surface: The paper identifies the bottom singular-vector subspaces of intermediate linear transformations as a critical, previously overlooked spectral attack surface in transformer-based VLMs.
• SSGRA Attack: The authors propose SSGRA, a spectral-guided white-box attack that leverages this insight. It demonstrates improved effectiveness over existing state-of-the-art baselines (including feature-space attacks like FDA, DRA, BSA, and output-level attacks like CE and EGA).
• Empirical Validation of Spectral Dynamics: Through layer-wise probing, the study shows that untargeted adversarial optimization naturally tends to increase the alignment of intermediate representations with information-attenuating (bottom) singular subspaces.
• Spectral Characterization of VLMs: The authors provide a spectral analysis of three major VLMs, revealing that models with a higher proportion of near-null singular directions (e.g., Qwen2.5-VL) exhibit greater vulnerability to SSGRA compared to models with fewer such directions (e.g., Gemma-3).

Experimental Results
The experiments were conducted on ImageNet with three state-of-the-art VLMs: Gemma-3 (4B), Qwen2.5-VL (7B), and LLaVA-1.5 (7B).

• Attack Effectiveness: SSGRA consistently outperformed all baselines. On Qwen2.5-VL, it achieved an additional 7.90–19.74% relative degradation in BERTScore F1 and 30.50–97.93% in ROUGE-L F1 compared to the strongest baseline. On LLaVA-1.5 and Gemma-3, it also achieved superior performance, with improvements scaling with the perturbation budget.
• Qualitative Analysis: Visual inspection of adversarial examples showed that while baseline attacks often preserved the general semantic content (with minor hallucinations), SSGRA frequently induced completely meaningless or unrelated predictions (e.g., describing a car crash instead of an ambulance).
• Spectral Correlation: The results confirmed a correlation between the density of near-null singular directions and vulnerability. Qwen2.5-VL, having the highest proportion of near-null singular values, was the most vulnerable, while Gemma-3, with the lowest proportion, was the most robust.
• Computational Cost: SSGRA maintained a computational complexity (measured in FLOPs) comparable to other optimization-based attacks, with no significant overhead.

Significance and Claims
The paper claims that its primary significance lies in shifting the perspective of adversarial vulnerability from input-space properties to the spectral structure of intermediate linear transformations. By demonstrating that adversarial optimization naturally exploits near-null singular directions, the authors argue that robustness strategies should not only focus on constraining large singular values (via spectral normalization) but must also address the control of near-null singular directions.

The authors explicitly state that their work isolates the spectral mechanisms of vulnerability through untargeted white-box attacks. They modestly note that the applicability of this framework to transfer-based and black-box settings has not yet been investigated and remains a direction for future work. The findings suggest that explicitly controlling the alignment of representations with bottom singular-vector subspaces offers a complementary strategy for improving the adversarial robustness of VLMs.