LAMP: Extracting Local Decision Surfaces From Large Language Models

Technical Summary: LAMP (Local Attribution Mapping Probe)

Title: LAMP: Extracting Local Decision Surfaces From Large Language Models
Authors: Ryan Chen, Youngmin Ko, Catherine Cho, et al. (Northwestern, Yale, Mayo Clinic)
Venue: AISTATS 2026

1. The Problem: The "Faithfulness" Gap in LLM Rationales

Large Language Models (LLMs) are increasingly used to provide free-text rationales (Chain-of-Thought) to explain their predictions. However, recent research suggests these rationales are often unfaithful: models may provide a logical-sounding explanation that does not actually reflect the underlying decision process (e.g., rationalizing a biased prediction or relying on irrelevant cues).

Existing interpretability methods like LIME or SHAP often require access to model gradients (white-box) or operate in the high-dimensional token space, which lacks semantic interpretability. There is a need for a black-box, lightweight, and semantically meaningful way to audit whether a model’s stated reasons actually "steer" its decisions.

2. Methodology: The LAMP Framework

The authors propose LAMP (Local Attribution Mapping Probe), a method that treats a model’s self-reported explanations as a coordinate system to map its local decision surface. Unlike methods that ask "is this explanation true?", LAMP asks "does the model behave consistently with the factors it claims are important?"

The LAMP workflow consists of four key steps:

1. Explanation Extraction: The model is prompted to classify an input $x$ and simultaneously output a list of explanatory factors $\{f_i\}$ and their associated importance weights $\{w_i\}$.
2. Stochastic Perturbation: To probe the local geometry, LAMP perturbs the weight vector $w_0$ by adding a stochastic jitter $\delta$. The model is re-queried with these updated weights to observe how the predicted probability $P(x=c)$ changes.
3. Dimensionality Reduction (Meta-Aggregation): Because LLMs can generate long, redundant lists of factors, LAMP uses a "meta-aggregation" step. It prompts the model to consolidate the raw factors into a smaller set (e.g., $n=5$) of aggregate themes. This ensures the resulting surrogate model is stable and interpretable.
4. Local Linear Surrogate Fitting: LAMP fits a linear model $\hat{y} = X\hat{\beta}$ where $X$ is the matrix of perturbed weights and $y$ is the resulting probabilities. The coefficients $\hat{\beta}$ represent the "dials" of the decision surface, indicating how much a change in a specific factor influences the output.

Mathematical Innovation: The authors derive an optimal perturbation radius ($\delta^*$) using a second-order Taylor expansion. This ensures the perturbations remain within a neighborhood where the decision surface is sufficiently linear, minimizing the Mean Squared Error (MSE) caused by local curvature.

3. Key Contributions

• Model-Agnostic Auditing: LAMP requires no access to gradients, logits, or internal activations, making it ideal for auditing proprietary models (like GPT-4 or Claude).
• Semantic Interpretability: By operating in the "explanation space" (themes/factors) rather than the "token space" (individual words), the resulting importance scores are human-readable.
• Theoretical Grounding: The paper provides a mathematical framework for determining the optimal scale of perturbations based on local curvature (Hessian) and sample size.
• Counterfactual Validation: The authors demonstrate that perturbations in the "factor space" can be translated into "text space" via LLM-based rewriting, proving the method captures real behavioral shifts.

4. Experimental Results

The authors evaluated LAMP across sentiment analysis (IMDB), controversial topic detection (Pseudo-Harmful), safety auditing (HateBenchSet), and clinical gastroenterology case files.

• Predictive Accuracy: The linear surrogates achieved high $R^2$ values (explained variance), particularly in the "middle" (uncertain) regions of the probability distribution.
• Consistency: LAMP surrogates successfully predicted the model's output on rewritten, counterfactual text. In many cases, LAMP outperformed LIME and was competitive with SHAP, while maintaining significantly lower variance.
• Expert Alignment: In the medical domain, the surrogate coefficients aligned with the directional expectations of gastroenterologists. Notably, the model's factors often aligned more closely with individual physician judgments than physicians did with each other.
• Comparison to White-Box Methods: While Integrated Gradients (IG) performs well on open-source models (Qwen), LAMP provides comparable performance in a black-box setting, proving its utility for closed-source APIs.

5. Significance and Implications

LAMP shifts the interpretability paradigm from adjudicating faithfulness (which is difficult) to characterizing behavioral consistency (which is measurable).

Practical Applications:

• High-Stakes Auditing: In medicine or law, practitioners can use LAMP to see if a model's diagnosis is actually driven by the clinical symptoms it claims to be looking at.
• Safety and Bias Detection: It provides a tool to detect if a model is "hiding" its reliance on protected attributes or controversial cues behind benign-sounding rationales.
• Human-AI Collaboration: By providing a "map" of how to nudge a model's decision, LAMP enables more interactive and controllable AI systems.