NIC-RobustBench: A Comprehensive Open-Source Toolkit for Neural Image Compression and Robustness Analysis

This paper introduces **NIC-RobustBench**, an open-source framework designed to evaluate the adversarial robustness of neural image compression models and their impact on downstream tasks, addressing the current gap in benchmarks that primarily focus only on rate-distortion performance.

The Gist

Imagine you have a super-smart, high-tech photo album that can shrink your massive photo collection down to a tiny size so you can send it over the internet instantly. This is Neural Image Compression (NIC). It's like a magical wizard that learns how to fold a giant blanket into a tiny square without losing the pattern.

But here's the catch: this wizard is a bit fragile. If someone whispers a tiny, almost invisible secret into the photo before the wizard folds it, the wizard might get confused and unfold a completely different, messy blanket. Or, the wizard might decide to use way more space than necessary, defeating the whole purpose of compression.

This paper introduces NIC-RobustBench, which is essentially a "Stress Test Gym" for these photo-wizards.

The Problem: The "Fragile Wizard"

In the past, researchers only tested these wizards to see how small they could make a photo (efficiency). They didn't ask: "What happens if a hacker tries to trick you?"

The authors realized that these compression tools are vulnerable. A tiny, carefully crafted "glitch" (an adversarial attack) added to an image can cause:

1. Total Collapse: The decompressed image looks like a nightmare of artifacts.
2. Bloat: The file size explodes, making it useless for fast transmission.
3. Downstream Failure: If this compressed image is then fed into a self-driving car or a security camera, the car might not see a pedestrian, or the camera might miss a face.

The Solution: The "Stress Test Gym" (NIC-RobustBench)

The authors built an open-source toolkit (a gym) where researchers can throw all kinds of "tricks" at these compression models to see how tough they are.

Think of it like a car crash test, but for digital images. Instead of crashing cars into walls, they "crash" images into compression algorithms using 8 different types of "attacks" and see which models survive.

What's inside the gym?

• The Attackers (8 types): These are the "bad guys" trying to break the compression. Some try to make the image look ugly; others try to make the file size huge.
• The Defenders (9 types): These are the "bodyguards" trying to protect the image. Some are simple tricks like flipping the image upside down or rotating it before compression (so the attacker's trick doesn't work). Others are complex AI "cleaners" that try to scrub the noise out.
• The Models (10+ types): A huge variety of compression algorithms, from old-school ones to the brand new JPEG AI standard.

What Did They Discover? (The "Training Results")

After running thousands of tests, they found some surprising things:

1. The "Big Brains" are Fragile: The most advanced, complex models (like HiFiC and CDC) that produce the best quality images are actually the most easily tricked. They are like a master chef who gets confused if you whisper a tiny wrong ingredient name. Simpler, smaller models are surprisingly tougher.
2. Generative Models are Weak: Models that try to "imagine" missing details (Generative models) are very vulnerable. If you trick them slightly, they hallucinate weird, scary artifacts.
3. Compression is a Filter: Interestingly, models that compress images heavily (making them very small) are actually more robust. Why? Because they act like a sieve, filtering out the tiny "noise" the hackers try to inject.
4. The "Flip" Defense Works: Sometimes the simplest defense is the best. Just flipping an image horizontally or rotating it before compressing it can confuse the attackers enough to save the day.
5. Complex Defenses Can Backfire: Some fancy AI defenses that try to "clean" the image before compressing it actually make the final picture worse or the file size bigger. It's like trying to clean a muddy shoe with a wet sponge; you just end up with a wet, muddy mess.

Why Should You Care?

You might think, "I just want to send a JPEG to my grandma." But in the future, everything will use these compression tools:

• Self-driving cars compressing video feeds.
• Video calls for telemedicine.
• Satellite imagery for weather forecasting.

If a hacker can trick the compression algorithm, they could make a self-driving car "see" a stop sign where there is none, or make a security camera miss an intruder.

NIC-RobustBench is the first tool that helps engineers build compression systems that are not just efficient, but also unbreakable. It ensures that when we shrink our digital world down to fit in our pockets, we don't accidentally leave the front door wide open to hackers.

Technical Summary

1. Problem Statement

Neural Image Compression (NIC) has emerged as a superior alternative to traditional codecs (like JPEG) in terms of rate-distortion (RD) performance. However, NIC models inherit the vulnerability of deep neural networks to adversarial attacks.

• The Risk: Small, imperceptible perturbations added to an input image can cause NIC models to produce severe reconstruction artifacts or drastically increase the bitrate.
• The Gap: Existing benchmarks focus almost exclusively on RD performance in clean (non-adversarial) scenarios. While some studies have explored specific attacks on individual codecs, there is no unified framework to evaluate the robustness of diverse NIC architectures against a wide range of attacks and defenses, nor to assess the downstream impact of these attacks on computer vision pipelines.
• The Challenge: Evaluating NIC robustness is more complex than in classification tasks because adversarial perturbations affect both the reconstruction quality and the bitrate (compression efficiency) simultaneously.

2. Methodology: NIC-RobustBench Framework

The authors introduce NIC-RobustBench, an open-source, modular, and scalable framework designed to systematically benchmark NIC robustness.

A. Framework Architecture

The framework is built as a Python library with standardized classes for:

• Codecs: Integration of various NIC models.
• Attacks: Implementation of adversarial generation algorithms.
• Defenses: Pre-processing and post-processing strategies.
• Evaluation: Metrics for image quality, bitrate, and downstream task performance.
• Reproducibility: Uses Docker containers and YAML configuration files to ensure experiment consistency.

B. Components of the Benchmark

1. NIC Models (10+ Architectures): The benchmark covers a diverse set of models, including:

   • Generative: HiFiC, CDC, QRes-VAE.
   • Discriminative/Transformers: ELIC, LIC-TCM, Cheng2020 (with attention), MBT-2018.
   • Standard: JPEG AI (v7.1), Balle et al. (factorized/hyperprior).
   • Variants: Multiple compression levels (bitrates) for each model.

2. Adversarial Attacks (8 Types, 6 Objectives):

   • White-box attacks: I-FGSM, PGD, FTDA (specifically designed for NIC), MADC (Maximum Differentiation Competition) with various proxy metrics (L2, L∞), and SSAH (frequency-based).
   • Black-box attacks: NES and Square Attack (evaluated but found ineffective for NIC).
   • Optimization Objectives:
     • Reconstruction Loss: Maximize distortion between original and decoded adversarial image.
     • Bitrate Increase: Maximize the Bits Per Pixel (BPP) of the compressed adversarial image.
     • Source Reconstruction: Maximize difference between the original source and the decoded adversarial image.

3. Defense Strategies (9 Methods):

   • Reversible Transformations: Flip, Random Roll, Random Rotate, Random Color Reorder, and Ensembles (mixing these transforms).
   • Neural Purification: DiffPure (diffusion-based), DISCO, and MPRNet (image restoration).
   • Note: The framework focuses on transformation-based defenses to avoid the prohibitive cost of retraining large NIC models.

4. Evaluation Metrics:

   • Quality Metrics: PSNR, MSE, MS-SSIM, and VMAF (Video Multi-method Assessment Fusion).
   • Robustness Scores:
     • $\Delta$-score: Measures how much the attack distortion is amplified or suppressed by the codec ($FR(x, x') - FR(C(x), C(x'))$).
     • $\delta$-score: Measures the quality loss introduced by the attack on the decompressed image ($FR(x, C(x)) - FR(x', C(x'))$).
   • Downstream Tasks: Evaluation of classification (ResNet50), object detection (YOLO11), and depth estimation (Depth Anything V2) on attacked images.

5. Datasets: KODAK, Cityscapes, NIPS 2017, ImageNet, and CLIC.

3. Key Contributions

1. First Large-Scale Benchmark: The first comprehensive adversarial robustness benchmark for NIC, covering the largest collection of models (including the new JPEG AI standard) and attack/defense combinations.
2. Scalable Open-Source Library: A modular framework allowing researchers to easily integrate new codecs, attacks, and defenses.
3. Comprehensive Empirical Study: Extensive experiments across 5 datasets, 10 model families, 8 attacks, and 9 defenses.
4. Defense Implementation: Systematic evaluation of both reversible transformations and neural purification methods specifically for compression pipelines.

4. Key Results & Insights

A. Model Vulnerability

• Generative Codecs are Most Vulnerable: Models with generative priors (HiFiC, CDC, QRes-VAE) exhibit the highest susceptibility to attacks. Their reliance on global latent representations means small perturbations shift the entire latent space, causing catastrophic reconstruction failures.
• Model Size Correlation: There is a strong positive correlation (Spearman $\rho \approx 0.72$) between model parameter count and vulnerability. Larger models, while better at clean RD performance, have more pathways for attacks to exploit.
• Compression Rate Effect: Within a model family, lower bitrate (higher compression) variants are more robust. They act as low-pass filters, suppressing high-frequency adversarial perturbations.

B. Attack Efficiency

• Objective Matters: Attacks targeting Reconstruction Loss cause the most severe degradation in image quality. Attacks targeting Bitrate (BPP) shift the RD curve to the right (higher file size) but may preserve visual quality better.
• Attack Variability: FTDA and MADC-$L_\infty$ are generally the strongest attacks, but effectiveness varies by architecture. For instance, FTDA performs poorly on Cheng2020 (anchor) but excels on QRes-VAE, likely due to gradient quantization differences.
• Black-Box Inefficiency: Query-based black-box attacks (NES, Square Attack) were found to be largely ineffective against NIC, producing negligible degradation compared to white-box attacks.

C. Defense Effectiveness

• Best Defenses: DiffPure and DISCO (purification methods) showed the most consistent improvement in robustness, effectively removing off-manifold noise.
• Geometric Transformations: Simple Flip operations were surprisingly effective because they are lossless and align with model invariances. However, Random Rotate/Roll often degraded clean image quality due to interpolation artifacts, limiting their net robustness gain.
• Trade-offs: While defenses improve robustness, they often introduce new artifacts or increase bitrate, sometimes resulting in a net loss of Rate-Distortion efficiency compared to the undefended model on clean data.

D. Downstream Impact

• Detection is Most Affected: Adversarial attacks on NIC significantly degrade object detection performance (YOLO11).
• Classification/Depth are Robust: Image classification and depth estimation tasks showed relatively minor performance drops, suggesting that the artifacts introduced by NIC attacks are less detrimental to these specific tasks than to detection.

5. Significance

• Safety Critical: As NIC (specifically JPEG AI) moves toward standardization and deployment in real-world pipelines, understanding its failure modes is crucial. A compromised compression layer can silently break downstream AI systems (e.g., autonomous driving perception).
• Research Direction: The paper highlights that defenses designed for classification (like adversarial training) may not translate directly to compression. It calls for compression-specific defense mechanisms that account for the unique interplay between bitrate, quantization, and reconstruction.
• Community Resource: By releasing the code and benchmarks, the authors enable the community to rigorously test new NIC architectures for robustness before deployment, shifting the field from "efficiency-only" to "efficiency + robustness."

In conclusion, NIC-RobustBench establishes that while NIC offers superior compression, it introduces significant security vulnerabilities. The framework provides the necessary tools to identify these weaknesses and develop robust, secure compression standards for the future of digital media.