Technical Summary: "LLMs Should Not Yet Be Credited with Decision Explanation"

Problem Statement

The paper addresses a critical methodological issue in the emerging field of using Large Language Models (LLMs) for human decision modeling. While LLMs are increasingly utilized as synthetic participants, behavioral surrogates, and theory-generation tools, there is a tendency to conflate three distinct scientific claims: decision prediction, rationale generation, and decision explanation.

The core problem is that current literature often treats accurate behavioral prediction, the generation of plausible natural-language rationales, and outcome-conditioned reasoning traces as sufficient evidence that an LLM explains why humans decide as they do. The author argues that this conflation risks a premature redefinition of explanatory progress. Specifically, it allows persuasive text (narration) to be mistaken for cognitive evidence (tracking the actual decision-generating structure), thereby misinterpreting progress in prediction as progress in explanation.

Methodology and Framework

The paper does not propose new experimental data or a specific algorithm. Instead, it employs a conceptual and methodological framework to re-evaluate existing evidence. The methodology consists of:

1. Differentiation of Claims: The author establishes a "credit ladder" that separates three distinct claims, each carrying a different evidential burden:

   • Decision Prediction: The model captures patterns in what people choose.
   • Rationale Generation: The model produces coherent, plausible, or psychologically literate accounts of choices.
   • Decision Explanation: The model's account is warranted as tracking the relevant structure that generated, constrained, or causally shaped the decision.

2. Evidence Re-evaluation: The paper systematically reviews common evidence patterns in current LLM decision research (behavioral fit, plausible rationales, and prediction-useful traces) to determine which claim they actually support. It argues that behavioral fit supports prediction, plausible language supports rationale generation, and even prediction-useful traces often fail to distinguish explanation from "prediction-supportive rationalization."

3. Proposal of a "Bridge Standard": To move from weaker claims (prediction/hypothesis generation) to the stronger claim of decision explanation, the paper proposes a four-part standard that future work must satisfy.

Key Contributions

1. The Credit Ladder and Evidential Burdens

The primary contribution is the formal distinction between prediction, narration, and explanation. The paper demonstrates that:

• Behavioral fit (e.g., accuracy in predicting choices) does not identify the mechanism producing the choice; multiple theories can fit the same data.
• Plausible rationales (e.g., fluent text explaining a choice) do not establish source tracking; LLMs can generate psychologically literate text that matches an outcome without tracking the actual decision process.
• Prediction-useful traces (e.g., Chain-of-Thought) are stronger but still insufficient. A trace may be faithful to the model's internal computation or optimized for a reward signal without being faithful to the human decision-generating structure.

2. The Bridge Standard for Decision Explanation

The paper proposes a minimal standard required to credit an LLM with decision explanation. This standard requires four conditions:

• Specify the Explanatory Target: The claim must define exactly what structure is being tracked (e.g., a specific parameter like loss sensitivity, a rule like lexicographic choice, a process like attention, or a contextual factor like framing). Without a specific target, the claim is untestable.
• Discriminate Against Weaker Rationalizers: The evidence must distinguish the proposed account from weaker alternatives, such as:
  • Predictor-only models.
  • Predictor-plus-narrator systems (where the narrator is given the outcome and generates a post-hoc reason).
  • Outcome-conditioned rationale generators.
  • Protocol-retrieval narrators (reproducing textbook explanations).
  • Judge-optimized generators (selected for plausibility rather than truth).
• Use Target-Appropriate Validation: Validation must be sensitive to the specific target, not just the final choice. This includes process measures (e.g., reaction times, eye-tracking), interventions (e.g., counterfactual manipulations of reference points), or independent construct measurements. The evidence must show that the account predicts how decisions change when the alleged structure changes.
• Bound the Scope: Explanatory credit must be scoped to the specific task, population, domain, and level of analysis tested. Generalizing from a specific risky-choice task to "human decision explanation" is unwarranted without broader evidence.

3. Reporting Pattern

The paper introduces a reporting template (Table 2) to make the evidential upgrade inspectable. Researchers must explicitly report the claim level, the explanatory target, the target-appropriate evidence, the weaker alternatives considered, the discrimination results, and the scope of the credit.

Results and Findings

The paper's "results" are analytical conclusions regarding the state of the field:

• Current Evidence is Over-Credited: Most existing studies provide evidence for decision prediction and rationale generation, and occasionally for explanatory hypothesis generation. However, they rarely provide the "bridge evidence" necessary to support the stronger claim of decision explanation.
• The Gap is Methodological, Not Necessarily Technical: The failure to explain is not because LLMs cannot explain, but because current evaluation designs do not isolate explanation from rationalization.
• Hypothesis Generation is Valuable: LLMs are acknowledged as powerful tools for generating candidate mechanisms (e.g., suggesting that "loss sensitivity" might be a factor), but this remains a hypothesis generation claim until tested against the bridge standard.

Significance and Claims

The paper claims that adopting this position is crucial for the scientific integrity of human decision modeling. Its significance lies in:

1. Preventing Premature Redefinition: It stops the field from redefining "explanation" as merely "persuasive narration" or "accurate prediction," which would dilute the scientific value of explanatory claims.
2. Calibrating Credit: It advocates for a principle of "credit calibration": LLMs should be credited for the strongest claim their evidence warrants, and no stronger. This allows LLMs to be valued for what they currently do well (predicting, narrating, hypothesizing) without falsely attributing capabilities they have not yet demonstrated.
3. Improving Future Research: By enforcing the bridge standard, the paper argues that LLM-based research will shift toward sharper, more diagnostic tests of human behavior. This transforms LLMs from "persuasive storytellers" into reliable instruments for discovering, testing, and communicating genuine explanations of human behavior.

The paper concludes modestly, noting that it offers a standard for warranting claims rather than a complete theory of explanation. It does not prescribe universal metrics but insists that the burden of reporting and the selection of diagnostics must match the specific explanatory target.