Consistent Evidence, Robust Recognition: Faithful Attribution Regularization under Geometric Transformations

Technical Summary: Consistent Evidence, Robust Recognition

Problem Statement
Attribution methods are widely used to identify the input evidence underlying deep neural network predictions. However, their potential to actively improve model behavior remains underexplored. A critical issue is attribution inconsistency under label-preserving geometric transformations (e.g., flipping or rotation). While the predicted class may remain unchanged, the model often shifts its reliance from the original object evidence to different or irrelevant regions. This indicates a sensitivity to geometric transformations that undermines robustness.

Existing approaches to attribution regularization typically rely on self-supervised consistency objectives derived from gradient-based maps (e.g., Grad-CAM). A fundamental limitation of these methods is faithfulness: gradient-based maps do not always accurately reflect the evidence driving the model's decision. Consequently, aligning these maps ensures consistency in the explanation output but does not guarantee consistency in the underlying decision process. A model may achieve explanation-level consistency while still relying on transformation-sensitive evidence. The central challenge addressed by this work is how to convert faithful, decision-linked attribution into an effective training signal to enforce consistent evidence reliance without human annotations.

Methodology
The authors propose an annotation-free attribution regularization framework based on submodular search over image regions. The method operates in two stages:

1. Query Stage (Generating Supervision):
   Instead of using gradient-based heatmaps, the framework employs a search-based attribution method (specifically LIMA) to extract compact, class-discriminative evidence. The process greedily selects image regions that maximize a submodular score function, which combines consistency, confidence, and collaboration metrics. The search terminates when a consistency threshold is reached.

   • Filtering: Only samples that are correctly classified, have high confidence, and result in a compact attribution area are retained. This ensures the supervision signal is derived from reliable model behavior.
   • Supervision Sequence: For retained samples, the ordered sequence of selected regions and the stopping condition constitute the "search-derived supervision."

2. Training Stage (Submodular Ranking Loss):
   To regularize the model, the framework aligns the attribution search trajectory of a transformed image with the spatially corresponding trajectory of the original image. Because the search process is discrete and sequential, a differentiable surrogate is required. The authors introduce the Submodular Ranking Loss, composed of two terms:

   • Selection-Ranking Loss ($L_{SelR}$): At each step of the greedy search on the transformed image, this term enforces that the spatially corresponding target region (from the original sequence) ranks higher than all remaining non-target candidates.
   • Selection-Truncation Loss ($L_{SelT}$): This term encourages the accumulated evidence of the transformed trajectory to satisfy the same stopping criterion as the original query sequence at the target terminal step.

The total objective combines the standard classification loss with the Submodular Ranking Loss, weighted by a hyperparameter $\lambda$. The framework is implemented with optimizations such as constrained search spaces and sparse regularization to manage computational overhead.

Key Contributions

1. Faithful Attribution Prerequisite: The work formulates attribution regularization around the principle that supervision must faithfully represent the evidence underlying the prediction, moving beyond simple map consistency.
2. Annotation-Free Framework: A novel framework that derives compact supervision sequences from filtered submodular searches, eliminating the need for human-provided relevance annotations.
3. Submodular Ranking Loss: A differentiable loss function that aligns candidate selection along transformed trajectories and enforces stopping criteria, providing a surrogate for the otherwise discrete evidence-selection process.
4. Empirical Validation: Extensive experiments demonstrating improved attribution stability and faithfulness, alongside stronger robustness to transformed inputs, with minimal impact on clean-image accuracy.

Experimental Results
The method was evaluated on ImageNet-100 and ImageNet-1K using ViT-B/16, ViT-L/16, ResNet-50, and ConvNeXt-B architectures.

• Attribution Quality (ImageNet-100, ViT-B/16):
  • Attribution stability increased from 0.14 to 0.27.
  • Insertion score improved by 52.1%.
  • Deletion score reduced by 62.2%.
  • Classification accuracy dropped by only 0.28 percentage points.
• Robustness (ImageNet-1K):
  • Mean transformed-input accuracy improved by 0.55 (ResNet-50) and 0.33 (ConvNeXt-B) percentage points.
  • Clean-accuracy reduction was limited to 0.30 points.
• Efficiency: The method achieves training throughput comparable to CGC and faster than CLIP-AFT, with significantly lower GPU memory requirements than second-order gradient methods.

Significance and Claims
The paper claims that faithful attribution promotes consistent evidence reliance without materially compromising predictive performance. By shifting the focus from aligning potentially unfaithful gradient maps to aligning the actual evidence-selection process derived from submodular search, the method addresses the root cause of transformation sensitivity. The results suggest that enforcing equivariance on the decision-relevant evidence, rather than just the explanation output, leads to more robust models that rely on spatially corresponding evidence across geometric transformations. The authors note that the code will be released soon to facilitate further research.