🤖 machine learning

REVIVE: A Multi-Modal Framework for Vandalism Detection and Recovery in Autonomous Vehicles

The REVIVE framework addresses vandalism-induced occlusion attacks in autonomous vehicles by integrating multi-stage detection, segmentation, and type-aware recovery techniques—including Stable Diffusion inpainting and direct pixel replacement—to restore camera-stream utility and significantly improve downstream object detection performance.

Original authors: Abdullah Tariq Choudhry, Tapadhir Das

Published 2026-07-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abdullah Tariq Choudhry, Tapadhir Das

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a self-driving car as a highly intelligent robot that "sees" the world almost entirely through its camera eyes. These cameras are its most important sense, helping it spot pedestrians, read traffic lights, and stay in its lane.

However, just like a human, a robot can have its vision blocked. If someone sprays paint on the lens, slaps a sticker over it, or if mud splatters across the glass, the car goes blind in that spot. This is what the paper calls a "Vandalism Attack."

The authors of this paper, Abdullah Tariq Choudhry and Tapadhir Das, built a system called REVIVE (Recovery and Enhancement of Vandalized Images for Vision Excellence). Think of REVIVE not as a magic eraser that fixes everything instantly, but as a smart, multi-step first-aid kit for the car's camera.

Here is how REVIVE works, broken down into simple steps:

1. The Triage Nurse (Detection)

First, the system acts like a triage nurse. It looks at the camera image and asks a simple question: "Is this camera lens dirty, or is it clean?"

If the image is clean, the car keeps driving normally.
If the system detects vandalism (like paint or mud), it flags the image for treatment.

2. The Specialist Doctor (Classification)

Once the problem is identified, REVIVE needs to know what kind of injury it is dealing with. It sorts the vandalism into five specific categories, much like a doctor diagnosing different types of wounds:

Random: Like random splatters of paint or debris (noise).
Center-out: A square sticker right in the middle of the view.
Top-bottom: Paint or a cover blocking the top half of the view.
Bottom-top: Mud or dirt smeared across the bottom.
Targeted: A specific patch covering a particular object.

3. The Surgeon (Localization)

Now that the system knows the "diagnosis," it needs to know exactly where to operate. It uses a specialized tool (an AI model) to draw a precise outline around the dirty or blocked area, separating the "bad" pixels from the "good" ones.

4. The Treatment Plan (Recovery)

This is the most clever part. REVIVE doesn't use the same fix for every problem. It chooses a treatment based on the diagnosis:

For Random Splatters (The "Dustbuster"): If the damage is just random noise (like tiny paint splatters), the system uses a simple, fast filter to smooth it out. It's like using a dustbuster to clean up crumbs.
For Big Blocks (The "AI Artist"): If a large, solid block covers the view (like a sticker in the center), the system tries to use a powerful AI artist (Stable Diffusion) to "paint" what it thinks should be there. However, the paper warns that this artist can sometimes "hallucinate" (make things up). So, this method is treated as a slow, background process, not something the car relies on for immediate steering.
The "Time-Travel" Fix (Direct Pixel Replacement): The paper found the best way to fix the image is actually the simplest: if the car has a clean picture of the exact same spot from a split second ago (and the car hasn't moved much), it just copies that clean picture over the dirty one. This is like looking at a photo of your face before you spilled coffee on your shirt and pasting it over the stain. It works perfectly, but only if the car is stopped or moving very slowly.

5. The Quality Control Inspector (The Gate)

This is the most important safety feature. After the system tries to fix the image, a "Quality Gate" checks the work.

The Rule: The fixed image is only allowed to be used if it helps the car see better than it did with the dirty image.
The Safety Net: If the AI artist makes a mistake or the "Time-Travel" fix is blurry because the car moved too fast, the Quality Gate says, "Nope, this isn't good enough." Instead of sending a bad image to the car's brain, it sends a warning signal: "I can't see clearly, fall back to safety mode."

What Did They Find?

The researchers tested this on 500 pairs of images (clean vs. vandalized).

Without help: When the camera was vandalized, the car's ability to spot objects (like people or cars) dropped significantly (about 59% success rate).
With the "Time-Travel" fix: If they had a clean reference frame, the success rate jumped back up to nearly 97%.
With the AI Artist: The AI could make the image look pretty, but it didn't always help the car actually see the objects correctly. Sometimes it looked good but was dangerous.
The Verdict: The system works best when it uses the right tool for the right job and has a strict inspector to ensure no bad fixes get through.

The Bottom Line

REVIVE isn't a magic wand that instantly cleans a dirty windshield. It is a smart framework that:

Detects the dirt.
Figures out what kind of dirt it is.
Applies the fastest, safest fix possible.
Crucially, refuses to use a fix if it makes the situation worse than leaving the image alone.

The paper concludes that while we can't always perfectly restore a vandalized camera, we can build a system that ensures the car never makes a decision based on a "hallucinated" or worse-than-original image.

Technical Summary: REVIVE Framework for Vandalism Recovery in Autonomous Vehicles

1. Problem Statement

Autonomous Vehicles (AVs) rely heavily on camera-based sensors for semantic perception, including object detection, lane recognition, and traffic sign interpretation. However, these systems are vulnerable to Vandalism-Induced Occlusion Attacks (VOAs), where physical obstructions (e.g., spray paint, stickers, mud, or debris) block or distort the camera lens. Unlike digital adversarial perturbations, physical vandalism persists until the sensor is cleaned or replaced, directly compromising the utility of the camera stream.

While existing research addresses VOA detection or relies on sensor redundancy (LiDAR, radar), there is a critical gap in automated image-level recovery. Current restoration methods often treat all corrupted regions uniformly, failing to account for the varying spatial structures of different vandalism types. Furthermore, many recovery techniques prioritize image similarity metrics over the preservation of downstream perception tasks, potentially hallucinating safety-critical content or failing to restore the evidence required for object detection.

2. Methodology: The REVIVE Framework

The paper proposes REVIVE (Recovery and Enhancement of Vandalized Images for Vision Excellence), a multi-modal, end-to-end pipeline designed to detect, classify, localize, and recover vandalized camera frames. The framework employs a hierarchical, type-aware recovery strategy that routes different occlusion patterns to the most appropriate restoration branch.

Pipeline Architecture

The pipeline processes an input image $I$ through four distinct stages:

Binary Vandalism Detection:
- A lightweight Convolutional Neural Network (CNN) with three convolutional blocks, global average pooling, and a fully connected head distinguishes between clean and vandalized frames.
- Purpose: To enable early termination for clean images, ensuring computational efficiency.
- Loss: Binary Cross-Entropy (BCE).
Multi-Class VOA Classification:
- If vandalism is detected, a classifier identifies the specific pattern of occlusion into five categories based on spatial characteristics:
  - Random: Scattered, noise-like occlusions (e.g., spray splatters).
  - Center-out: Square patches at the image center (e.g., targeted stickers).
  - Top-bottom: Horizontal bands from the upper region.
  - Bottom-top: Horizontal bands from the lower region (e.g., mud).
  - Targeted: Randomly positioned square patches in specific regions.
- Loss: Categorical Cross-Entropy (CCE).
VOA Segmentation Localization:
- An EfficientNet-based U-Net architecture generates a binary mask delineating the vandalized regions.
- Type-Aware Conditioning: The VOA class label from Stage 2 is converted into a feature vector and integrated into the segmentation network (e.g., concatenated with encoder feature maps) to adapt the model's focus to the expected spatial structure of the specific attack.
- Loss: Weighted sum of Binary Cross-Entropy and Dice Loss.
Type-Aware Image Recovery:
The framework selects a recovery strategy based on the identified VOA type:
- Statistical-Parametric Median Filtering: Applied to Random VOAs. Uses adaptive thresholding and multi-scale morphological clustering to suppress noise-like artifacts without semantic synthesis.
- BLIP-Guided Stable Diffusion: Applied to Structured VOAs (Center-out, Top-bottom, etc.). Uses BLIP to generate descriptive prompts from scene context, guiding Stable Diffusion to synthesize semantically plausible content for large occluded regions. This branch is treated as asynchronous due to high latency and hallucination risks.
- Direct Pixel Replacement: An upper-bound reference method used when a temporally recent, spatially aligned clean reference frame is available. It replaces vandalized pixels directly with corresponding pixels from the reference frame.

Quality Gate

A critical component of REVIVE is a post-recovery quality gate that filters candidates before they reach the downstream perception stack. A recovered frame is accepted only if:

It does not suppress safety-relevant detections (Recall $\ge$ unrecovered baseline).
It does not introduce excessive false positives.
It meets a structural consistency floor (SSIM $\ge$ 0.5).
If a candidate fails, the system raises a low-confidence (degraded-mode) signal rather than forwarding a potentially harmful reconstructed image.

3. Key Contributions

REVIVE Framework: An end-to-end pipeline coupling multi-class VOA classification with type-aware recovery branch selection, moving beyond uniform restoration approaches.
Threat Model Definition: A specific scope for image-level recovery within an AV perception stack, distinguishing between image restoration and full vehicle-level planning recovery.
Evaluation Metrics: A comprehensive evaluation using both image quality metrics (SSIM, PSNR) and, crucially, downstream object detection consistency (YOLOv8l recall, precision, F1-score).
Reference-Based Upper Bound Analysis: A rigorous comparison of direct pixel replacement against classical (Telea, Navier-Stokes) and learned (LaMa, Stable Diffusion) inpainting, highlighting the conditions under which generative recovery is unsuitable for real-time blocking.
Quality Gate Implementation: An evaluation of a reference-available quality gate demonstrating that filtering candidates prevents the pipeline from degrading performance below the unrecovered baseline.

4. Experimental Results

The framework was evaluated on 5,000 images from the BDD100K dataset (with synthetic VOAs) and 500 tracked clean/vandalized image pairs.

Detection and Segmentation Performance

Binary Detection: Achieved 97% overall accuracy with precision and recall >95%.
Multi-Class Classification: Achieved 99% overall accuracy across all five VOA types.
Segmentation: The EfficientNet-U-Net achieved a mean Dice coefficient of 0.91 for structured VOAs and 0.87 for random VOAs. Mean IoU scores exceeded 85% across all types.

Recovery Performance (Downstream Object Detection)

Using YOLOv8l for evaluation on 500 tracked pairs:

Unrecovered VOAs: Reduced detection recall to 0.588 and mAP50 to 0.635.
Direct Pixel Replacement (Aligned Reference): Restored recall to 0.967, F1-score to 0.970, and mAP50 to 0.979 with near-perfect image fidelity (SSIM 0.988, PSNR 52.8dB). Note: This is a reference-based upper bound, not a general deployment guarantee.
LaMa Inpainting: Improved recall to 0.667 and mAP50 to 0.680, outperforming unrecovered frames but remaining slower than lightweight candidates.
Stable Diffusion: Showed variable performance (SSIM 0.667–0.867) and lower aggregate detection recovery (Recall 0.339, mAP50 0.162) for structured VOAs, confirming its unsuitability as a blocking real-time step.
Median Filtering: Fast (0.26ms) but resulted in low detection recall (0.214) for random VOAs, indicating that visual noise suppression does not guarantee perception recovery.

Quality Gate Impact

Without the quality gate, type-aware routing degraded per-image recall to 0.304.
With the quality gate, recall returned to 0.608, ensuring the forwarded stream is never worse than the unrecovered frame.
The gate accepted 93.8% of aligned pixel-replacement candidates and 86.4% of LaMa candidates, while rejecting most candidates from branches that failed to restore detection evidence (e.g., only 17.8% of Stable Diffusion candidates).

Latency

The online (non-generative) REVIVE preprocessing pipeline (classification, segmentation, recovery, gating) adds 22.7 ms per frame, fitting within a 30 FPS (33 ms) budget.
Stable Diffusion requires ~8.6 seconds per image, confirming its role as an asynchronous branch.

5. Significance and Claims

The paper positions REVIVE not as a universal solution that blindly restores all images, but as a structured framework for selecting and validating recovery strategies.

Practical Utility: The primary contribution is the demonstration that recovery must be judged by downstream object detection consistency, not just image similarity. A visually plausible reconstruction that fails to restore detection evidence is unsafe.
Safety Mechanism: The quality gate is central to the framework's safety claim. It ensures that the system never forwards a reconstructed frame that degrades perception below the level of the unrecovered (vandalized) frame.
Modest Scope: The authors explicitly state that REVIVE addresses image-level recovery to support object detection, not full vehicle-level planning or control. They acknowledge that direct pixel replacement is a theoretical upper bound dependent on aligned references, which is rarely available in dynamic driving conditions.
Future Work: The paper identifies limitations, including the use of synthetic masks, the lack of real-world vandalism datasets, and the need for reference-free quality gating and multi-view sensor integration for future deployment.

In summary, REVIVE provides a practical, type-aware pipeline that integrates detection, classification, and recovery with a safety-critical quality gate, ensuring that any recovery attempt either improves or maintains the utility of the camera stream for autonomous driving perception.