Technical Summary: Feature-Space Smoothing for Certified Robustness of Deep Representations

1. Problem Statement

Modern deep learning models, particularly Multimodal Large Language Models (MLLMs) like GPT-4 and Gemini, are highly effective but vulnerable to adversarial attacks. These attacks inject subtle, imperceptible perturbations into clean inputs to induce erroneous predictions by distorting feature representations. While empirical defenses (e.g., adversarial training) exist, they lack formal robustness guarantees and often fail against stronger, adaptive adversaries. Furthermore, existing certified defense methods (e.g., Randomized Smoothing) are primarily designed for classification tasks with discrete outputs, making them difficult to apply to general feature representations, regression tasks, or the complex decoding processes of MLLMs. Additionally, retraining large pre-trained models to improve robustness is computationally prohibitive.

The core challenge addressed is developing a defense framework that provides certified robustness at the feature representation level, remains generalizable across diverse models and tasks (including MLLMs), and avoids the need for costly model retraining or fine-tuning.

2. Methodology

The proposed solution consists of two main components: Feature-space Smoothing (FS) and the Gaussian Smoothness Booster (GSB).

2.1 Feature-space Smoothing (FS)

FS is a general framework that transforms a standard feature encoder $f_e$ into a smoothed variant $\hat{f}_e$ via Gaussian smoothing.

• Mechanism: For an input $x$, the smoothed encoder outputs the expectation of feature representations over a Gaussian distribution: $\hat{f}_e(x) = \mathbb{E}_{\epsilon \sim \mathcal{N}(0, I)}[f_e(x + \epsilon)]$.
• Theoretical Guarantee: The authors establish a Feature Cosine Similarity Bound (FCSB). They prove that under $\ell_2$-bounded perturbations, the cosine similarity between the clean feature and the adversarial smoothed feature is guaranteed to be above a specific lower bound.
• Certified Radius: This bound allows for the derivation of a certified radius $R$. If the perturbation magnitude is within $R$, the cosine similarity between clean and adversarial features is guaranteed to remain above a threshold (e.g., 0.5).
• Prediction-wise Extension: The framework extends this feature-level guarantee to prediction-level certification. For tasks where predictions can be represented as normalized vectors (e.g., retrieval, prototype-based classification), the FCSB translates into a guaranteed bound on the cosine similarity between the adversarial prediction and the target prediction.

2.2 Gaussian Smoothness Booster (GSB)

The theoretical bounds of FS depend on the intrinsic Gaussian robustness score of the underlying encoder. Pre-trained encoders often have limited Gaussian robustness, leading to suboptimal certified radii. To address this without retraining the backbone, the authors propose GSB, a plug-and-play module consisting of:

1. Gaussian Denoiser ($\mathcal{P}$): A lightweight module (e.g., U-Net) placed before the feature encoder. It is trained to reconstruct clean inputs from Gaussian-corrupted inputs, mitigating noise before feature extraction.
2. Residual Smoothness Mapper ($\mathcal{M}$): A module placed after feature extraction. It refines the purified features to enhance robustness while preserving the original feature distribution. It uses a residual learning approach to ensure stability on clean inputs.

Training Strategy: GSB is trained in a self-supervised manner using a utility-robustness trade-off loss. The backbone encoder remains frozen. The denoiser minimizes reconstruction loss and feature consistency loss, while the mapper minimizes feature consistency loss and statistical regularization (to preserve mean and variance) alongside an identity loss for clean inputs.

3. Key Contributions

• Feature-space Smoothing (FS): A general defense framework that converts any feature encoder into a smoothed variant with a provable lower bound (FCSB) on the cosine similarity between clean and adversarial features under $\ell_2$ perturbations.
• Prediction-wise Certification: Theoretical extension showing that feature-level FCSB guarantees can be propagated to prediction-level robustness for tasks involving normalized vector outputs (e.g., Top-1 retrieval, classification).
• Gaussian Smoothness Booster (GSB): A plug-and-play module that enhances the Gaussian robustness score of existing encoders without fine-tuning the backbone, thereby improving the certified radius.
• Systematic Evaluation: Comprehensive experiments across diverse vision encoders (CLIP, SigLIP 2, EVA-CLIP) and MLLMs (LLaVA, OpenFlamingo) demonstrating that FS provides non-trivial certified robustness and significantly improves empirical performance against strong white-box attacks.

4. Experimental Results

4.1 Certified Robustness

• Feature-wise: On ImageNet, integrating GSB with FS significantly increased the average certified radius ($R$) for FCSB $\geq 0.5$. For example, with $\sigma=0.5$, the average radius for CLIP increased from 0.101 to 0.455, and for SigLIP 2 from 0.134 to 0.514.
• Prediction-wise: On ImageNet classification, FS achieved certified accuracy comparable to standard Randomized Smoothing (RS) but without requiring a separate smoothed classifier. For SigLIP 2, FS achieved 64.0% certified accuracy at $\epsilon=1.00$, comparable to RS (67.6%).
• Retrieval: On MS COCO zero-shot retrieval, FS provided non-trivial certified accuracy, though margins were tighter than in classification due to the dense nature of text embeddings.

4.2 Empirical Robustness against White-Box Attacks

The framework was tested against state-of-the-art white-box attacks (PGD, APGD, M-Attack, FOA, AttackVLM) on MLLMs (LLaVA-1.5-7B, OpenFlamingo-9B) across image captioning, classification, and VQA tasks.

• Image Captioning: Under the strong FOA attack, vanilla LLaVA accuracy dropped to 1%. With FS, accuracy rose to 65%, and the Attack Success Rate (ASR) dropped from 94% to 6%. FS also improved the robustness of adversarially trained baselines (FARE, TeCoA) without additional fine-tuning.
• Image Classification: FS increased LLaVA accuracy from 3.8% to 87.2% under FOA attacks, reducing ASR from 81% to 0.2%.
• VQA: FS preserved clean accuracy (54.0%) while improving Gaussian-noisy accuracy and reducing ASR to near zero, whereas adversarial training baselines (FARE, TeCoA) suffered significant clean performance degradation.

4.3 Efficiency

Feature-wise smoothing incurs lower inference overhead than whole-model smoothing. With $n_0=8$ Gaussian samples, the inference time for LLaVA increased by only ~0.36 seconds per image, demonstrating practical viability.

5. Significance and Claims

The paper claims that Feature-space Smoothing (FS) offers a significant advancement by shifting the focus of certified robustness from task-specific output spaces to the feature representation level. This allows for:

1. Generalizability: The method applies to diverse downstream tasks (classification, retrieval, generation) without modifying the prediction head or retraining the backbone.
2. Plug-and-Play Integration: Through GSB, existing large-scale pre-trained models (including MLLMs) can be made robust without the computational cost of full model retraining or alignment.
3. Provable Guarantees: It provides mathematically provable lower bounds on feature similarity, which translate to prediction stability, offering a level of trust that empirical defenses lack.

The authors acknowledge limitations, noting that current certification is restricted to $\ell_2$-bounded perturbations and relies on cosine-similarity-based decision rules, which may not cover all open-ended MLLM decoding behaviors. Future work aims to extend the framework to other threat models (e.g., $\ell_\infty$, semantic attacks) and broader generation tasks.