Technical Summary: Rule of Thumb: Explaining Artificial Intelligence Systems using Partial Information

Problem Statement

Explainable Artificial Intelligence (XAI) faces critical limitations in three emerging application areas: zero-shot classification with Large Language Models (LLMs), auditing opaque proprietary AI systems, and using AI for scientific discovery. Existing methods, primarily based on sensitivity analysis (e.g., SHAP, LIME, Integrated Gradients), rely on perturbing input features to observe changes in model output. This approach encounters fundamental barriers:

1. Inaccessibility: Sensitivity-based methods often require access to model weights (gradients) or the ability to query the model with synthetically perturbed data. This is impossible for closed-source LLMs accessed only via API and prohibitively expensive due to the high number of required API calls (e.g., 500–5000 calls per explanation).
2. The Rashomon Effect in Auditing: When auditing proprietary systems, researchers often build "mimic" models to approximate the target system. However, different mimic models with similar accuracy can possess different internal mechanisms, leading to contradictory explanations (the Rashomon effect). This undermines the validity of audits based on mimic models.
3. Scientific Validity: Sensitivity analysis assumes that model behavior on out-of-distribution (synthetic) data reflects real-world causality. This can lead to spurious hypotheses, where uninformative features appear important simply because the model reacts arbitrarily to synthetic perturbations.

Methodology: Rule of Thumb (ROT)

The authors propose "Rule of Thumb" (ROT), a novel XAI formulation that defines feature importance based on predictiveness rather than sensitivity. Instead of asking how altering a feature changes the output, ROT asks how knowing a specific feature value updates the prediction of the AI system's behavior.

Formulation

ROT models the AI system's output $C(x)$ as a sum of additive functions representing the information provided by individual feature values. For a dataset $X$ and feature set $\mathcal{J}$, the model estimates:
$$C(x) \approx F\left(\sum_{j \in J} f_{\theta_j}(x_j) + G\right)$$
where:

• $F$ is a sigmoid (classification) or identity (regression) function.
• $f_{\theta_j}(x_j)$ is a learned function representing the importance of feature $j$ taking value $x_j$.
• $G$ is a global bias term.
• The estimation holds for any subset of features $J \subseteq \mathcal{J}$.

Optimization

The objective is to find optimal weights $\theta$ that minimize the loss between the true model output $C(x)$ and the estimate derived from random subsets of features. The loss function $L$ is marginalized over all datapoints and all possible feature subsets:
$$\min_{\theta} L = \min_{\theta} \sum_{J \subseteq \mathcal{J}} \sum_{x \in X} p^{|J|} (1-p)^{(|\mathcal{J}|-|J|)} \cdot \ell\left[C(x), F\left(\sum_{j \in J} f_{\theta_j}(x_j) + G\right)\right]$$
Crucially, this optimization is efficiently solved using dropout applied to feature importances during training. This allows the model to be fitted on the entire unperturbed dataset without requiring additional model queries for each explanation. Once fitted, the feature importance for a specific datapoint is simply the value of $f_{\theta_j}(x_j)$.

Key Distinctions

• No Perturbation: ROT does not require generating synthetic data or perturbing inputs.
• Non-Input Features: ROT can incorporate external, non-input features (e.g., demographic attributes not passed to the model) into the explanation by fitting on extended datasets, allowing for the detection of bias even if the model claims not to use those features.
• Model Agnostic: It requires only the ability to observe predictions on existing data, not access to weights or the ability to query new inputs.

Key Contributions and Results

1. Zero-Shot Classification with LLMs

• Experiment: The authors applied ROT to zero-shot image classification (GPT-4o-mini API) and text classification (judicial case outcomes and movie reviews).
• Results: ROT explanations aligned closely with human annotations (weighted AUROC of 0.77 for judicial cases, comparable to SHAP's 0.74).
• Efficiency: While the first ROT explanation required training (approx. 1160 seconds for the judicial dataset), subsequent explanations were virtually instantaneous (<0.1 ms). In contrast, SHAP required ~1320 seconds per explanation. ROT enabled explanations for LLMs where sensitivity-based methods were computationally infeasible.

2. Auditing Proprietary Black-Box Systems

• Experiment: The authors audited Amazon's e-commerce recommendation system, replicating a study by The Markup that used a Random Forest mimic model.
• Results:
  • Rashomon Effect: Different mimic models (varying hyperparameters or algorithms) produced contradictory SHAP/LIME explanations regarding feature importance (e.g., whether "product reviews" or "brand is amazon" was most important).
  • ROT Advantage: ROT provided consistent explanations without any mimic model. It identified "brand is amazon" as the most important feature but also uniquely highlighted "sold by amazon" as a significant factor, revealing potential self-preferencing behavior that mimic-based audits missed.

3. Scientific Discovery

• Experiment: The authors tested ROT on diabetes prediction, recidivism, and loan approval datasets, specifically engineering scenarios where certain features were either uninformative or masked by adversarial "foil" features.
• Results:
  • Uninformative Features: When "age" was mathematically decoupled from diabetes outcomes, SHAP and LIME still assigned it importance (generating spurious hypotheses), whereas ROT correctly identified it as unimportant.
  • Adversarial Robustness: In adversarial setups designed to mislead explainers into prioritizing "foil" features over sensitive ones (e.g., race or gender), SHAP and LIME failed (recovering the sensitive feature <5% of the time). ROT maintained a success rate >89%, correctly identifying the true sensitive features.

4. Computational Efficiency and Environmental Impact

• ROT is orders of magnitude faster than SHAP and LIME for large-scale explanations. The paper notes that after the initial fit, ROT can compute millions of explanations in the time it takes SHAP to compute one, significantly reducing the environmental footprint of XAI.

Significance and Claims

The paper claims that ROT addresses long-standing validity and practicality issues in XAI by shifting the paradigm from sensitivity analysis to feature predictiveness.

• Regulatory Compliance: ROT is positioned as a tool that meets the requirements of the EU AI Act (e.g., Article 15 and 74) for auditing high-risk systems where regulators may have access to datasets but not full model access or the ability to query new inputs.
• Scientific Rigor: By avoiding assumptions about causal mechanisms and out-of-distribution behavior, ROT offers a more robust baseline for generating scientific hypotheses compared to sensitivity-based methods.
• Practical Deployment: ROT is presented as a drop-in replacement for existing XAI codebases that is model-agnostic, faster, and capable of explaining systems (like API-only LLMs) that were previously unexplainable.

The authors emphasize that ROT does not claim to solve causal inference but provides a reliable method for understanding model behavior based on observed data predictiveness, making it uniquely suited for the constraints of modern, proprietary AI systems.