Extremal Contours: Gradient-driven contours for compact visual attribution

1. Problem Statement

Current methods for explaining deep neural network (DNN) decisions in computer vision face significant challenges regarding faithfulness, compactness, and stability:

• Dense Perturbation Masks: Methods like Extremal Perturbations or Smooth Masks optimize dense pixel-level masks to preserve or delete image regions. While effective at finding causal regions, these masks often result in fragmented, noisy, or multi-component outputs. They lack topological guarantees, requiring heavy post-processing or regularization to become interpretable.
• Gradient-Based Saliency: Methods like Integrated Gradients or Grad-CAM are computationally efficient but often produce diffuse, unstructured heatmaps that highlight irrelevant background noise and fail sanity checks for faithfulness.
• The Core Gap: There is a need for an explanation method that produces compact, coherent, and topologically simple regions (e.g., a single connected shape) without sacrificing the causal fidelity of perturbation-based approaches, while avoiding the instability of learning millions of pixel parameters.

2. Methodology: Extremal Contours

The authors propose a training-free, gradient-driven method that replaces dense pixel masks with smooth, tunable contours.

A. Parametric Representation

Instead of optimizing every pixel, the method parameterizes a star-convex region using a truncated Fourier series.

• Center: A learnable center point $c \in \mathbb{R}^2$.
• Radial Function: The boundary radius $\hat{r}(\theta)$ at angle $\theta$ is defined as:
  $$\hat{r}(\theta) = r_0 + \Re\left( \sum_{k=1}^{K} w_k e^{ik\theta} \right)$$
  where $w_k$ are complex coefficients.
• Mask Generation: For a pixel $p$ at polar coordinates $(\rho_p, \theta_p)$ relative to $c$, the mask value $m(p)$ is a smooth sigmoid function of the difference between the pixel's radius and the contour radius:
  $$m(p) = \frac{1}{1 + \exp(\tau \cdot [\hat{r}(\theta_p) - \rho_p])}$$
  This reduces the number of free parameters from millions (pixels) to $2K+3$, drastically lowering dimensionality.

B. Optimization Objective

The contour is optimized to satisfy an extremal perturbation objective using the classifier's gradients. The total loss $L$ consists of three terms:

1. Extremal Loss ($L_{extremal}$): Encourages the "preserved" variant (keeping the region, blurring the rest) to retain the original embedding, while the "deleted" variant (blurring the region, keeping the rest) suppresses it.
   $$L_{extremal} = -\cos(e_p, e_o) + \cos(e_d, e_o)$$
   where $e_o, e_p, e_d$ are embeddings of the original, preserved, and deleted images.
2. Area Regularization ($\lambda_a \alpha_r$): Penalizes large areas to ensure compactness. The weight $\lambda_a$ is adaptively tuned based on the fidelity of the preserved region. If the preserved region fails to maintain the embedding, the area penalty is relaxed; once fidelity is achieved, the penalty tightens to find the smallest sufficient region.
3. Spectral Regularization ($\lambda_r L_{spec}$): Penalizes high-frequency oscillations in the Fourier coefficients to enforce smooth boundaries:
   $$L_{spec} = \sum_{k=1}^{K} k^2 |w_k|^2$$

C. Multi-Object Extension

The framework extends to multiple objects by optimizing $N$ independent contours simultaneously. The final mask is the pixel-wise maximum of individual masks, with the loss summing the area and spectral penalties of all contours.

3. Key Contributions

1. Topological Guarantees: By construction, the method produces single, simply connected, smooth masks, eliminating the need for post-processing to remove noise or fragmentation.
2. Dimensionality Reduction: Reduces learnable parameters by orders of magnitude (from $H \times W$ to $\approx 2K$), leading to faster convergence and stability without dataset-level optimization.
3. Robustness to Adversarial Artifacts: The low-dimensional, smooth constraint prevents the optimizer from exploiting unconstrained degrees of freedom to create "adversarial" masks that satisfy the loss mathematically but are semantically meaningless.
4. Explicit Area Control: Introduces a mechanism to generate fidelity-area profiles, allowing users to visualize the trade-off between the size of the explanation and the model's confidence.
5. Model Agnosticism: The method works effectively on both supervised (ResNet-50) and self-supervised (DINO) models.

4. Experimental Results

The method was evaluated on ImageNet and COCO datasets using ResNet-50 and DINO ViT-B/16 models.

• Qualitative Performance:
  • Produces clean, single-contour boundaries that tightly enclose the object of interest.
  • Unlike gradient maps (diffuse) or dense masks (fragmented), Extremal Contours provide a structured, interpretable representation.
• Quantitative Performance:
  • Relevance Mass & Rank: Achieves competitive or superior scores compared to baselines (Smooth Mask, Grad-CAM++, Integrated Gradients).
  • Complexity & Sparseness: Delivers highly sparse explanations with lower entropy than dense methods.
  • Faithfulness: Crucially, on self-supervised DINO models, where many baselines fail (showing negative faithfulness correlations), Extremal Contours maintain positive faithfulness correlations and improve relevance mass by >15%.
• Robustness:
  • The method is stable across different initialization points (center $c$).
  • It is robust to the spectral regularization weight $\lambda_r$, consistently converging to the correct object regardless of whether the resulting shape is circular or irregular.

5. Significance and Future Directions

• Interpretability: The shift from pixel-level heatmaps to geometric contours offers a more human-intuitive way to understand model reasoning, particularly for safety-critical domains like medical imaging (CT, MRI) where clinicians need clear, editable boundaries rather than noisy heatmaps.
• Efficiency vs. Fidelity: It resolves the trade-off between the computational efficiency of gradient methods and the causal fidelity of perturbation methods, offering the best of both worlds.
• Limitations:
  • Star-Convex Constraint: The method cannot naturally represent objects with deep concavities or holes (non-star-convex shapes).
  • Binary Nature: It produces binary masks rather than fine-grained attribution maps, which may reduce granularity for fine-grained classification tasks.
  • Runtime: Requires iterative optimization rather than a single forward/backward pass, making it slower than direct gradient methods.
• Future Work: The authors suggest extending the parameterization to handle non-star-convex shapes, applying the method to object detection and segmentation tasks, and utilizing the contours as interactive tools for radiologists to refine annotations.

In summary, Extremal Contours presents a novel, mathematically grounded approach to XAI that prioritizes geometric coherence and topological simplicity, proving that restricting the search space to smooth contours yields more reliable and interpretable explanations than unconstrained dense pixel optimization.