💻 computer science

Not All Explanations Simulate Equally: Comparing Verbalized Feature Attributions and Self-Generated Rationales

This paper evaluates and compares the simulatability of verbalized feature attributions and self-generated rationales for question-answering models, demonstrating that explanation format, granularity, and source significantly influence an LLM judge's ability to predict model behavior in counterfactual settings.

Original authors: Pingjun Hong, Benjamin Roth

Published 2026-06-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pingjun Hong, Benjamin Roth

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to guess what a very smart, but sometimes quirky, robot will say next. You have a list of questions, and the robot gives you answers. Now, imagine you change one word in the question (a "counterfactual" change) and ask: Can you predict what the robot will say to this new question?

This paper is about testing different types of "hints" (explanations) to see which ones actually help a human (or another AI) make that prediction correctly. The authors call this process simulatability. Think of it like a "crystal ball test": Does the explanation give you enough information to see the future of the robot's behavior?

The researchers compared two main types of hints:

1. The "Highlighter" vs. The "Storyteller"

Type A: The Highlighter (Verbalized Feature Attributions)
Imagine the robot reads a long article and answers a question. A "Highlighter" explanation points to the specific sentences or words in the text that the robot used to make its decision.

The Experiment: The researchers took these highlighted parts and turned them into sentences.
- Sentence-level: "The robot relied on this whole paragraph."
- Token-level: "The robot relied on the word 'apple'."
- Rewritten: They asked a super-smart AI to rewrite the highlight into a smooth, flowing sentence.

Type B: The Storyteller (Self-Generated Rationales)
This is when the robot is asked to explain its own thinking before or after giving the answer.

Post-hoc: "Here is my answer. By the way, here is a short reason why."
Chain-of-Thought (CoT): "Let me think step-by-step... First I looked at X, then I considered Y, so my answer is Z."

2. The Results: What Actually Worked?

The researchers used a "Judge" (another powerful AI) to try and predict the robot's answer to the new, changed question. They gave the Judge the original answer and one of the "hints" listed above.

Here is what they found, using simple analogies:

The "Whole Paragraph" Wins (Sentence-level):
When the hint pointed to a whole sentence or paragraph, the Judge got much better at predicting the robot's next move. It was like being given a full chapter of a book to understand the plot.
- Analogy: If you tell someone, "The character is angry because of the letter he just read," they can guess what the character will do next.
The "Single Word" Fails (Token-level):
When the hint only pointed to single words (like "apple" or "1805"), the Judge got confused or did worse than before.
- Analogy: If you just say, "The character is angry because of the word 'letter'," it's too vague. You don't know what the letter said. The context was missing.
Fluency is a Trap (LLM Rewriting):
The researchers tried taking the "Highlighter" notes and asking a super-smart AI to rewrite them into beautiful, smooth English. They thought this would help. It didn't.
- Analogy: It's like taking a rough map and having an artist redraw it with beautiful colors and fancy fonts. If the map still only shows a single street instead of the whole neighborhood, the pretty font doesn't help you navigate. The information mattered more than the style.
The "Step-by-Step" Guide is King (Chain-of-Thought):
The absolute best hints were the Chain-of-Thought explanations. When the robot explained its reasoning step-by-step, the Judge could predict the robot's future answers with incredible accuracy.
- Analogy: This is like the robot showing you its scratchpad. Instead of just saying "The answer is 42," it says, "I added 20 and 20, then subtracted 18, so I got 22, then added 20 again..." You can follow the logic perfectly, so you know exactly what it will do next.

3. The Big Takeaway

The paper concludes that not all explanations are created equal.

Context is King: Giving a chunk of the original text (a sentence) is much better than giving a tiny fragment (a word).
Logic > Polish: A rough, step-by-step logical explanation (Chain-of-Thought) is far more useful for predicting behavior than a smooth, rewritten summary of a highlight.
The "Why" Matters: To understand how a model will behave in a new situation, you need to see its reasoning process, not just the evidence it looked at.

In short: If you want to understand a robot's brain, don't just show me the words it highlighted. Show me the story it told itself to get there. That's the only way to truly predict what it will do next.

Technical Summary: Not All Explanations Simulate Equally

Problem Statement

Natural-language explanations for Large Language Models (LLMs) are often treated as a unified interface for understanding model behavior. However, these explanations originate from diverse sources—ranging from input-feature attributions (identifying important tokens or sentences) to self-generated rationales (such as post-hoc justifications or Chain-of-Thought reasoning). Despite sharing the label "explanation," these objects differ fundamentally in source, granularity, format, and intended use.

Prior evaluation protocols often assess explanations along type-specific dimensions, such as faithfulness to the decision process, plausibility to human readers, or agreement with human rationales. This paper argues that these type-specific metrics make it difficult to compare disparate explanation formats on a common ground. The central challenge addressed is: How can we systematically compare different types of LLM explanations to determine which ones genuinely support the understanding and prediction of model behavior?

Methodology

The authors propose a unified evaluation framework based on counterfactual simulatability. Rather than asking if an explanation is plausible in isolation, this framework asks whether the explanation helps a predictor accurately anticipate the model's behavior under new, perturbed inputs.

Experimental Setup

Task Formulation: The evaluation is framed as a counterfactual simulation problem. Given an original input $x = (c, q)$ (context and question) and a model answer $y = M(x)$ , a counterfactual input $x_{cf} = (c, q_{cf})$ is constructed using the same context but a modified question. The target model $M$ produces a true counterfactual answer $y_{cf}$ .
Simulation Protocol: An LLM judge ( $P$ $P$ ) acts as a predictor to guess $y_{cf}$ $y_{c f}$ .
- Phase 1 (Baseline): The predictor sees $x$ , $y$ , and $x_{cf}$ but no explanation.
- Phase 2 (Explanation): The predictor sees the same inputs plus an explanation $e$ for the original answer $y$ .
- Metric: Simulatability is measured as the accuracy gain ( $\Delta$ ) in predicting $y_{cf}$ when the explanation is provided: $\Delta = \text{Score}(2) - \text{Score}(1)$ . A positive $\Delta$ indicates the explanation improves predictive understanding.
Explanation Types Compared:
- Verbalized Feature Attributions: Using the MExGen framework, the authors identify influential input spans (sentences or tokens) and convert them into natural language. They test two verbalization strategies:
  - Template-based: Deterministic insertion of top-ranked evidence into fixed templates (Sentence-level, Token-level, and Hybrid).
  - LLM-rewriter: An LLM rewrites the selected attribution features into fluent text without adding new information.
- Self-Generated Rationales: Explanations generated by the target QA models themselves via prompting:
  - Post-hoc: The model answers first, then provides a justification.
  - Chain-of-Thought (CoT): The model reasons step-by-step before answering.
Models and Data: Experiments were conducted on the SQuAD 2.0 dataset using three instruction-tuned target models (Llama-3-8B, Qwen2.5-7B, Mistral-7B) and GPT-5-mini as the primary judge. Robustness was checked with DeepSeek-R1 and a teacher-student training paradigm.

Key Contributions

Unified Evaluation Criterion: The paper applies counterfactual simulatability as a single metric to compare attribution-based explanations and self-generated rationales, enabling direct comparison across formats that were previously evaluated in isolation.
Controlled Comparison of Verbalization: The study contrasts template-based verbalizations with LLM-rewritten versions of the same attribution data, isolating the effects of evidence granularity (sentence vs. token) from linguistic fluency.
Granularity and Source Analysis: The work systematically analyzes how explanation source, verbalization strategy, and feature granularity affect simulatability across multiple models.

Results

The study yields several distinct findings regarding what makes an explanation "useful" for simulation:

Granularity Matters: Sentence-level attributions provide the strongest simulatability gains among attribution-based methods. Token-level attributions (even when highly ranked) often yield near-zero or negative gains, suggesting that isolated words lack the necessary context for predicting model behavior. Hybrid approaches (sentence + tokens) perform better than tokens alone but worse than pure sentence-level attributions.
Fluency $\neq$ Usefulness: LLM-rewritten explanations, while more fluent than template-based ones, do not significantly improve simulatability. In some cases, they perform worse. This suggests that the informational content (the evidence selected) drives simulatability more than the linguistic form.
Self-Generated Rationales Vary: There is a stark contrast between post-hoc explanations and CoT rationales. Post-hoc explanations offer modest improvements, whereas CoT rationales provide the largest gains in counterfactual prediction across all models and metrics.
Cross-Model Transfer: CoT rationales contain substantial task-relevant information that generalizes across different models (cross-model transfer), though they also encode model-specific signals.
Evaluation Paradigm Divergence: While CoT rationales show consistent positive gains in both LLM-judge and teacher-student evaluations, attribution-based explanations show a discrepancy. They perform well in the judge-based (in-context) setting but show weak or negative gains in the teacher-student (training) setting, suggesting that attribution signals are useful for immediate prediction but may not be easily internalized as learnable supervision.

Significance and Claims

The paper claims that not all natural-language explanations serve the same role in supporting the understanding of model behavior. The significance of these findings lies in shifting the evaluation focus from "plausibility" or "faithfulness" in isolation to functional utility: does the explanation actually help a user (or an automated agent) predict how the model will behave on new inputs?

The authors conclude that:

Explanation format is critical: Sentence-level evidence is superior to token-level evidence for simulation.
Source matters: CoT rationales are currently the most effective explanation type for anticipating model behavior, outperforming both post-hoc justifications and verbalized attributions.
Evaluation must be context-aware: The utility of an explanation depends on whether it is used for immediate in-context prediction (where CoT and sentence attributions excel) or for training internal models (where CoT remains robust, but attributions may not transfer).

Ultimately, the work provides empirical guidance for selecting explanation formats in QA settings, cautioning against the assumption that all natural-language explanations are equally effective for simulation.