Technical Summary: SEAMS – Sufficiency Saliency via MSE-Preservation Soft-Masks

Problem Statement

Current visual explanation methods in computer vision predominantly rely on sensitivity-based approaches (e.g., gradient attribution, Integrated Gradients, Layer-wise Relevance Propagation). These methods estimate how sensitive a model's output is to infinitesimal input perturbations. However, the authors argue that sensitivity and sufficiency are fundamentally distinct concepts. A pixel may have a high attribution score due to high sensitivity (small changes cause large output shifts) yet be redundantly encoded elsewhere in the image. Conversely, pixels with low gradients may be strictly necessary to preserve a prediction.

Existing perturbation-based methods that attempt to address sufficiency often suffer from specific limitations: they may require auxiliary distractor datasets, rely on architecture-specific attribution mechanisms, or necessitate complex differentiable top-$k$ relaxations and sorting operators to select discrete subsets of pixels. The central problem addressed is how to identify the minimal, compact subset of image pixels that is sufficient to preserve a specific model output (such as a class probability, CLS embedding, or token representation) without these auxiliary dependencies.

Methodology: SEAMS

The authors propose SEAMS (Sufficiency Saliency via MSE-Preservation Soft-Masks), a post-hoc, model-agnostic framework that directly optimizes a continuous soft mask to preserve a target representation.

Core Framework

Given a frozen differentiable vision model $g$ and an input image $x$, the goal is to find a soft mask $M \in [0, 1]^{H \times W}$ such that the model output computed from a masked composite image $\tilde{x}$ remains close to the original output $y^\star = g(x)$.

Key Components

1. Mask Parameterization:
   Instead of optimizing the mask directly, SEAMS optimizes unconstrained logits $\ell$. These are transformed into a valid mask via a normalise-and-clip parameterization:

   • $u = \text{softplus}(\ell)$ ensures non-negativity.
   • $\bar{u} = \alpha \frac{u}{\text{mean}(u)}$ enforces a learnable sparsity budget $\alpha \in (0, 1]$.
   • $M = \text{clip}(\bar{u}, 0, 1)$ constrains values to $[0, 1]$.
     This approach avoids explicit subset selection, differentiable sorting, Gumbel sampling, or top-$k$ relaxations. The normalization step creates competition among pixels for the budget, naturally driving the mask toward sparsity (near-binary values) without explicit binarization mechanisms.

2. Three-Way Image Composite:
   To evaluate the mask, the authors construct a composite image $\tilde{x}$ using three sources generated entirely from the query image $x$:

   • Original ($x$): Preserved regions where $M \approx 1$.
   • Self-Augmented Distractor ($x_{dist}$): Generated via heavyAug (strong geometric/photometric transforms, swirl, color shifts, cutout). Regions where $M \approx 0$ are replaced by this to disrupt weak texture correlations.
   • Blurred Context ($x_{blur}$): A heavily blurred version of $x$. Regions with intermediate mask values or where the budget is exceeded are filled with this to preserve global scene structure and reduce optimization artifacts.
     The composite is defined as $\tilde{x} = M \odot x + M'' \odot x_{dist} + N \odot x_{blur}$, where coefficients sum to 1.

3. Optimization Objective:
   The mask and budget are optimized to minimize a preservation loss:
   $$\mathcal{L}(\ell, \alpha) = \mathbb{E}_{\mathcal{A}} \left[ \| g(\mathcal{A}(\tilde{x})) - y^\star \|_2^2 \right] + \lambda_\alpha \alpha$$

   • Preservation Term: Minimizes the Mean Squared Error (MSE) between the target representation of the augmented composite and the fixed reference target $y^\star$. Augmentations $\mathcal{A}$ (moderate crops, flips, jitter) are applied to the composite to ensure robustness, while $y^\star$ remains fixed.
   • Sparsity Term: Penalizes the budget $\alpha$ to encourage compact masks.

4. Modular Targets:
   The framework is agnostic to the target $g(x)$. By changing only the preserved output, the same pipeline generates:

   • Class-conditioned explanations: Preserving a specific class probability.
   • Global representation masks: Preserving the CLS embedding.
   • Token-level explanations: Preserving dense patch-token representations.

Key Contributions

1. Sufficiency-Based Formulation: The paper introduces a saliency formulation that directly optimizes for preservation fidelity (sufficiency) rather than local sensitivity.
2. Simplified Optimization Framework: SEAMS requires no auxiliary distractor datasets, no architecture-specific attribution rules, and no differentiable subset-selection operators (like top-$k$ relaxations). It relies solely on a soft-mask optimization with a learnable budget and self-generated distractors.
3. Unified Multi-Target Pipeline: A single optimization procedure generates diverse explanation types (object-level, class-specific, token-level) simply by altering the preserved target representation.
4. Architecture-Dependent Insights: The method reveals that different architectures (e.g., ViT vs. ConvNeXt) often rely on distinct sets of sufficient pixels to achieve similar preservation fidelity, a nuance difficult to capture with sensitivity-based methods.

Experimental Results

The authors evaluated SEAMS on ImageNet-1k (ViT-S/16, ConvNeXt-Tiny, ViT-B/16, DeiT-III-B/16) and a private medical dataset (Retinopathy of Prematurity).

• Faithfulness (Insertion/Deletion): SEAMS achieves competitive and often superior Insertion and Deletion AUC scores compared to gradient-based methods (IxG, SmoothGrad, Integrated Gradients) and other perturbation-based methods (DAVE). Notably, SEAMS demonstrates high pixel-deletion faithfulness: the target class probability remains stable until a large fraction of pixels (up to 70-80%) are removed, dropping sharply only when the compact sufficient region is occluded. This indicates the method successfully isolates a minimal sufficient subset.
• Cross-Architecture Generalization: When applied to different backbones (ViT vs. ConvNeXt) with identical hyperparameters, SEAMS produces semantically meaningful masks. However, quantitative analysis (IoU, cosine similarity) shows that the specific pixels selected differ significantly between architectures, confirming that visual sufficiency is architecture-dependent.
• Multi-Target Flexibility: Experiments demonstrate that changing the target from class probability to CLS embedding or patch tokens yields distinct, interpretable masks that highlight different semantic aspects of the image.
• Medical Imaging Transfer: Without modifying hyperparameters or the optimization pipeline, SEAMS successfully identified clinically relevant retinal structures (vascular trajectory, demarcation lines) in Retinopathy of Prematurity (ROP) fundus images, demonstrating transferability beyond natural images.
• Qualitative Quality: Visual comparisons show SEAMS produces sharper, more object-aligned, and spatially coherent explanations with fewer patch-grid artifacts than prior techniques. The masks are notably sparser and more binary than competing methods.

Significance and Claims

The paper claims that SEAMS provides a general, model-agnostic framework for representation-oriented visual explanation. Its primary significance lies in shifting the paradigm from "what changes the output?" (sensitivity) to "what is enough to keep the output?" (sufficiency).

The authors emphasize that SEAMS:

• Requires no external data or complex architectural modifications.
• Generates compact, interpretable, and stable masks.
• Reveals that different neural architectures may rely on different visual evidence to reach the same conclusion, a finding made visible through sufficiency-based analysis.
• Is applicable across scales (from 224px to 518px) and domains (natural images to medical imaging).

The paper modestly acknowledges limitations, noting that SEAMS is computationally more expensive than gradient-based methods due to iterative optimization and that the current evaluation focuses exclusively on image encoders. Future work is suggested for video, multimodal, and generative architectures.