On Pitfalls of $\textit{RemOve-And-Retrain}$: Data Processing Inequality Perspective

Technical Summary: On Pitfalls of RemOve-And-Retrain: A Data Processing Inequality Perspective

Problem Statement

The RemOve-And-Retrain (ROAR) benchmark is a widely adopted protocol for evaluating feature attribution methods in mechanistic interpretability. The core premise of ROAR is that if an attribution method correctly identifies features critical to a model's decision, removing those features and retraining the model should result in a significant drop in accuracy. However, the validity of ROAR from an information-theoretic perspective remains underexplored.

This paper questions the reliability of ROAR as a metric for determining whether an attribution map genuinely carries information about a model's decision function. The authors hypothesize that ROAR scores can be artificially improved through model- and data-agnostic post-processing of attribution maps. Such post-processing, by the Data Processing Inequality (DPI), cannot add information about the decision function, yet it may still yield better ROAR scores. This suggests that a superior ROAR ranking may reflect biases in the benchmark's sensitivity to specific mask geometries (e.g., spatial blurriness) rather than the true informativeness of the attribution method.

Methodology

The authors employ a combination of theoretical analysis using structural causal models and empirical validation on real-world datasets.

1. Theoretical Framework (Data Processing Inequality):

   • The authors formalize the ROAR data-generation process using a structural causal model where the attribution map $A$ is generated from the input $X$ and a model-side variable $Z$ (representing the decision function and explainer identity).
   • They introduce a post-processing function $k(\cdot)$ that transforms the attribution map $A$ into $\tilde{A}$ without accessing $X$, $Y$, or $Z$ directly (only via $A$).
   • Theorem 3.1 establishes that for any such agnostic post-processing, the conditional mutual information $I(Z; \tilde{A} | X)$ is less than or equal to $I(Z; A | X)$. This confirms that post-processing cannot increase the information the map holds about the model.
   • Theorem 3.2 provides a counterexample demonstrating that it is possible to construct a post-processing $k$ such that the mutual information between the modified input and the label, $I(\tilde{X}'_t; Y)$, is strictly lower than $I(X'_t; Y)$, even though $I(Z; \tilde{A} | X) = 0$. In the context of ROAR, a lower $I(\cdot; Y)$ corresponds to a lower re-trained accuracy, which is interpreted as a "better" score. This proves that a ROAR improvement does not necessitate that the attribution is more informative about the model.

2. Empirical Instantiation:

   • To test the hypothesis that ROAR rewards specific mask shapes, the authors apply two simple, agnostic post-processing functions to attribution maps: Gaussian smoothing and Max-pooling.
   • These operations are chosen because they tend to produce spatially coherent, "blurry," or block-like masks, analogous to the "BlockRandom" baseline which removes structured content more effectively than "PixelRandom."
   • The experiments utilize the ROAR and ROAD (RemOve-And-Retrain with Drop) protocols on three datasets: CIFAR-10, SVHN, and CUB-200.
   • Various attribution methods are evaluated, including Input-Gradient, Grad*Input, Integrated Gradients, SmoothGrad, VarGrad, and Grad-CAM, both in their original and squared forms.

Key Results

• Theoretical Finding: The authors prove that agnostic post-processing can strictly improve ROAR scores (by lowering re-trained accuracy) while simultaneously reducing or eliminating the information the attribution map carries about the model's decision function.
• Empirical Findings:
  • Experiments show a consistent association between the "blurriness" of the mask and improved ROAR/ROAD performance.
  • Applying Gaussian or Max-pooling post-processing to attribution maps resulted in lower re-trained accuracies (better ROAR scores) in the vast majority of cases. Specifically, in the ROAR benchmark, Max-pooling lowered accuracy in 74/81 comparisons, and Gaussian smoothing in 76/81. Similar trends were observed in the ROAD variant.
  • The results indicate that the benchmark is sensitive to the spatial structure of the mask (preferring block-like or blurry removals) rather than solely the fidelity of the attribution to the model's internal logic.

Key Contributions

1. Theoretical Proof: The paper provides a formal proof, grounded in the Data Processing Inequality, that model/data-agnostic post-processing can improve ROAR scores without adding information about the decision function.
2. Structural Causal Model: The authors construct a formal counterexample and a structural causal model of the ROAR data-generation process to isolate the failure modes of the benchmark.
3. Identification of Blurriness Bias: The study reveals a persistent bias in ROAR (and ROAD) metrics toward spatially blurry masks, demonstrating that these metrics can be optimized by transformations that do not enhance mechanistic understanding.
4. Practical Guidelines: The authors offer guidelines for more cautious benchmarking of interpretability methods, urging the community to consider the geometric properties of masks when interpreting ROAR results.

Significance and Claims

The paper claims that an improved ROAR ranking is not, by itself, evidence that an attribution map carries more information about the model. Instead, such improvements may simply reflect how the mask-generation pipeline interacts with the data distribution, specifically favoring masks that remove structured content efficiently.

The significance of this work lies in its cautionary stance on the validation of mechanistic understanding. The authors argue that without addressing these pitfalls, researchers cannot reliably distinguish between methods that genuinely reveal decision-relevant structure and those that merely produce visually appealing but uninformative saliency maps. The findings suggest that current benchmarks may inadvertently reward specific mask geometries over true attribution fidelity, necessitating a more nuanced approach to evaluating feature attribution methods in safety-relevant model auditing and circuit discovery.