NutVLM: A Self-Adaptive Defense Framework against Full-Dimension Attacks for Vision Language Models in Autonomous Driving

1. Problem Statement

Vision Language Models (VLMs) are increasingly integrated into Autonomous Driving (AD) systems to enhance perception and semantic reasoning. However, this integration expands the attack surface, making AD systems vulnerable to full-dimension adversarial threats:

• Local Attacks: Physical patches or high-intensity noise applied to specific regions (e.g., traffic signs) to hijack attention or cause misclassification.
• Global Attacks: Imperceptible noise distributed across the entire image to disrupt cross-modal alignment and multimodal reasoning.
• Limitations of Existing Defenses: Current methods struggle to balance robustness with clean-sample performance. Many rely on computationally expensive adversarial training (which degrades clean accuracy) or simple inference-time detection that lacks real-time purification capabilities. Furthermore, few defenses address both local and global threats simultaneously within a unified framework suitable for SOTA AD platforms.

2. Methodology: The NutVLM Framework

NutVLM is a self-adaptive online defense framework designed to secure the entire perception-decision lifecycle without requiring full-model retraining. It operates through a dual-branch architecture triggered by a unified detection module:

A. NutNet++ (Unified Detection & Purification)

NutNet++ serves as a "sentinel" that performs three-way classification to distinguish between:

1. Benign Samples
2. Local Attacks (Physical patches)
3. Global Attacks (Diffused noise)

It utilizes three core metrics derived from reconstruction error patterns:

• Anomaly Magnitude ($M_{anom}$): Quantifies overall reconstruction error intensity using a Conditional Value-at-Risk (CVaR) strategy to handle temporal jitter.
• Energy Entropy ($H_{energy}$): Measures spatial dispersion. Global attacks exhibit high entropy (diffused), while local attacks show low entropy (concentrated).
• Local Concentration ($C_{local}$): Quantifies the spatial aggregation of perturbations to identify contiguous attack regions.

DualGate Decision Logic:

• Stage I (Detection): Uses a "Soft-Recall" mechanism with an enhanced concentration indicator ($C_{enh}$) to detect attacks with low magnitude but high structural anomalies, reducing false negatives.
• Stage II (Classification): Separates detected attacks into Local or Global based on $C_{enh}$ thresholds.

Action:

• If Local: Applies Pixel-level Grayscale Masking to neutralize the patch while preserving the global context.
• If Global: Triggers the EAPT module for instruction correction.

B. Expert-guided Adversarial Prompt Tuning (EAPT)

For global attacks where the visual input is corrupted but the text prompt is clean, NutVLM performs parameter-free instruction correction:

• Semantic Verification: A frozen CLIP model acts as an "Expert" to calculate a Semantic Verification Score ($V_{sem}$). If the alignment between the perturbed image and the original prompt drops below a threshold, EAPT is activated.
• Latent Optimization: EAPT optimizes a latent embedding ($e_{opt}$) to restore image-text consistency. It minimizes a joint loss function comprising:
  • Consistency Loss ($L_{con}$): Ensures the embedding matches the true visual context verified by the Expert.
  • Drift Loss ($L_{drift}$): Prevents the optimized prompt from deviating too far from the original driving task intent.
• Suffix Projection: The optimized latent vector is projected into the VLM's vocabulary to generate a "Corrective Driving Prompt" (Suffix $S$). This suffix acts as a semantic anchor (e.g., "please delete remaining car," "attention gain") to refocus the VLM's attention on safe driving actions without retraining the model.

3. Key Contributions

1. Comprehensive Framework: Proposes NutVLM, the first self-adaptive framework capable of mitigating both local (patch) and global (noise) adversarial threats in AD VLMs simultaneously.
2. NutNet++ Sentinel: Introduces a novel three-way classification mechanism that outperforms binary detectors by distinguishing between clean, local, and global threats using statistical energy distribution metrics.
3. EAPT Mechanism: Develops a real-time, parameter-free defense strategy that corrects adversarial hallucinations via gradient-based latent optimization and discrete prompt projection, avoiding the high costs of full-model fine-tuning.
4. Robust Validation: Extensive evaluation across multiple VLM backbones (InstructBlip, LLaVA, MiniGPT-v4) and datasets (Dolphins, CADA, APRICOT, DAmageNet).

4. Experimental Results

The framework was evaluated on the Dolphins benchmark and the CADA dataset under various attack intensities (Level 0 to 4).

• Overall Performance: NutVLM achieved a 4.89% improvement in overall metrics (Accuracy, Language Score, GPT Score) compared to baselines under adversarial conditions.
• Global Attack Defense: Under the strongest CADA Level 4 global attacks, NutVLM maintained a Final Score of 41.74%, significantly outperforming standard VLMs and other defense baselines (e.g., NRP, AAA, Bit-Red).
• Local Attack Defense: Against patch attacks, NutVLM achieved a GPT Score of 56.31%, a 3.83% improvement over the baseline, successfully neutralizing physical adversarial patches.
• Detection Efficacy: NutNet++ achieved an F1-score of 73.94% and a D-Acc of 71.89% on AD-specific datasets, significantly outperforming the original NutNet baseline.
• Efficiency: The framework maintains high inference speed. The EAPT module introduces negligible latency, operating at 0.44 FPS (compared to 0.46 FPS baseline), making it viable for real-time AD applications.
• Generalization: The defense proved effective across diverse VLM backbones (InstructBlip, LLaVA, MiniGPT-v4), demonstrating architecture-agnostic robustness.

5. Significance

• Safety-Critical Application: NutVLM addresses a critical gap in securing autonomous driving systems against sophisticated, full-dimension adversarial attacks that could lead to catastrophic navigation failures.
• Efficiency vs. Robustness Trade-off: By utilizing inference-time purification and prompt tuning rather than heavy adversarial training, NutVLM achieves high robustness without sacrificing clean-sample performance or requiring expensive retraining.
• Scalability: The framework is designed to be compatible with state-of-the-art AD platforms, offering a scalable security solution for the future of intelligent transportation systems.
• Methodological Innovation: The separation of defense strategies (visual masking for local threats vs. semantic correction for global threats) provides a nuanced approach to handling different attack modalities, setting a new standard for VLM security research.