🤖 AI

Superficial Beliefs in LLM Decision-Making

This paper demonstrates that large language models exhibit "superficial beliefs" in decision-making, where their choices are systematically driven by underlying attribute priorities that they can only partially and imperfectly articulate in their self-reported reasons.

Original authors: Gabriel Freedman, Francesca Toni

Published 2026-06-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Gabriel Freedman, Francesca Toni

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to figure out how a very smart, but slightly mysterious, robot makes decisions. You ask it to choose between two options, like "Drug A" or "Drug B." The robot picks one and then tells you why it picked it.

The big question this paper asks is: Is the robot actually thinking about the reasons it gives, or is it just making up a story after the fact?

Here is a simple breakdown of what the researchers did and what they found, using some everyday analogies.

The Experiment: The "Taste Test"

The researchers set up a game where the robot had to choose between two profiles (like two different job candidates or two medicines). Each profile had four "stats" (like Safety, Cost, Speed, and Quality), rated as Low, Medium, or High.

They asked the robot to do two things for every choice:

Pick a winner.
Explain its choice by naming the single most important stat that led to the decision.

To see if the robot was telling the truth, the researchers used a "secret decoder ring" (a mathematical model). They watched the robot make hundreds of choices and calculated a hidden "preference score" for each stat based on what the robot actually picked. This is like watching a person eat a thousand meals and guessing their favorite food based on what they order, rather than asking them what they like.

The Discovery: The "Split Personality"

The results were a mix of good news and confusing news.

1. The Robot Does Have a Pattern (The "Hidden Logic")
The "secret decoder ring" worked surprisingly well. When the researchers used the robot's past choices to predict its future choices, they were right about 80% of the time.

Analogy: It's like watching a friend order coffee for a month. Even if they don't say it out loud, you can predict they will order a latte tomorrow because they always do. The robot isn't just guessing randomly; it has a hidden, consistent logic.

2. The Robot's "Story" Doesn't Match the "Logic" (The "Superficial Belief")
Here is the twist: When the researchers compared the robot's actual hidden logic with the reasons the robot gave out loud, they didn't match perfectly.

The robot's hidden logic said, "I chose Drug A because of Safety."
But when asked, the robot often said, "I chose Drug A because of Efficacy."
Analogy: Imagine you are driving a car. Your hands are steering left because there is a pothole (the hidden logic). But if someone asks, "Why are you turning left?" you might say, "Because I want to see the view" (the story). You aren't lying maliciously; you just don't have full access to the real reason your hands moved.

The researchers call this "Superficial Belief." The robot behaves as if it has deep beliefs and priorities, but it only has a "superficial" (surface-level) ability to explain them. It can make the right choice, but it can't always articulate the real driver behind that choice.

The "Scorecard" Test

The researchers tried a second way to get the truth. Instead of asking the robot to pick a winner, they asked it to give a "score" (0 to 1) for how important each stat was.

Result: This method was more consistent (the robot gave similar scores every time), but it still didn't perfectly match the hidden logic derived from the choices. It was like asking the robot to rate its hunger; it gave consistent numbers, but those numbers still didn't perfectly explain which sandwich it actually wanted to eat.

The "Control" Check

To make sure the robot wasn't just picking words at random, they added "fake" stats that didn't matter at all (like "Packaging Symmetry" for a medicine).

Result: The robot almost never picked these fake stats as the reason. This proves the robot is paying attention to the real data; it just isn't always good at explaining which real data point mattered most.

The Bottom Line

The paper concludes that Large Language Models (LLMs) are not just random noise machines, nor are they fully transparent thinkers.

They are not random: They follow a hidden, predictable pattern when making decisions.
They are not fully transparent: Their spoken explanations are only a partial, imperfect reflection of that hidden pattern.

The Metaphor: Think of the LLM as a very talented chef who can cook a perfect meal every time (the decision). But if you ask the chef, "What was the secret ingredient?" they might guess "Salt" when the real secret was actually "A specific type of pepper." The chef knows how to make the dish work, but they don't have a clear, verbal map of exactly how their own brain is doing it. They have a "superficial belief" in their own cooking process.

Technical Summary: Superficial Beliefs in LLM Decision-Making

Problem Statement

This paper investigates whether Large Language Models (LLMs) possess a systematic underlying decision structure or merely imitate the language of belief when making choices. Specifically, it addresses the gap between an LLM's outward behavior (its choices) and its explicit self-reports (its stated reasons). The authors ask: Do LLMs exhibit stable, belief-like priorities over attributes in binary decision settings, and if so, are these priorities accessible through direct verbal elicitation?

Drawing on a "superficialist" perspective (Schwitzgebel, 2025), the paper posits that belief attribution should depend on stable patterns in behavior rather than internal architecture. The core hypothesis is that while LLM choices may be systematic enough to support a weak form of "superficial belief," the explicit reasons provided by the models may only imperfectly track the actual drivers of those choices.

Methodology

Benchmark Design

The authors constructed a synthetic benchmark for binary decision-making involving:

Structure: 400 training and 100 test source problems, each consisting of two options (profiles) defined by four graded attributes (Low, Medium, High).
Themes: The base attributes are instantiated across three main themes (Drugs, Policy, Software) and two control themes (replacing one attribute with irrelevant ones: Packaging Symmetry or Label Border Thickness).
Objectives: Each theme includes a specific optimization goal (e.g., "Optimise 5-year overall patient outcome").
Variants: Each problem is rendered in four prompt-order variants to test robustness against attribute ordering.
Sampling: Each prompt is sampled three times to measure consistency.

Experimental Setup

The study evaluates four model families (GPT-5-mini, GPT-5-nano, Qwen3-14B, Ministral-3-14B) across two settings: Non-Thinking (NT) and Thinking (T).

Three distinct evaluation streams are compared:

Direct Report: The model is prompted to choose an option and state the single most important attribute for that choice.
Score-Based Judge: The model is prompted to assign a decisiveness score $[0, 1]$ to each of the four attributes. These scores are signed based on the attribute difference direction and aggregated (signed-sum) to infer a choice and a "driver" (the attribute with the largest absolute signed contribution).
Behavioural Model: A binomial logistic regression is fitted to the training set choices. The model uses the visible attribute differences as predictors to learn global weights ( $\beta$ ). On the test set, this model predicts the choice and identifies the "revealed driver"—the attribute that contributes most strongly to the model's actual observed choice.

Evaluation Metrics

The primary comparison is between the explicitly elicited drivers (from Direct Report and Score-Based Judge) and the behaviorally inferred driver (from the Logistic Regression). Agreement is measured at two levels:

Choice Level: Does the elicited method predict the same option as the behavioral model?
Attribute Level: Does the elicited method identify the same "driver" attribute as the behavioral model?

Key Contributions

Synthetic Benchmark: Introduction of a controlled binary decision benchmark with graded attributes, multiple themes, and control variants to isolate decision structures from semantic noise.
Operational Methodology: A framework for reconstructing local decision priorities via behavioral modeling and comparing them against explicit elicitation methods (Direct Report and Score-Based Judge).
Empirical Findings: A systematic demonstration that while LLM choices are highly predictable and structured, the explicit reasons provided by the models (and even score-based judges) only partially align with the behaviorally inferred drivers.

Results

Choice Prediction

The behavioral model predicts held-out choices with high accuracy across all themes and models.

Aggregate Accuracy: 80.4% agreement between the behavioral model's predicted choice and the model's observed choice.
Score-Based Judge: Achieves 71.7% agreement with the behavioral model's choice target.
Control Themes: Irrelevant attributes are selected rarely (<2%), confirming that the models are not choosing randomly but are responding to semantically meaningful structures.

Attribute Recovery (The Core Finding)

There is a significant dissociation between the choice and the stated reason:

Direct Report: The attribute named by the model matches the behavioral model's revealed driver only 61.0% of the time.
Score-Based Judge: Matches the revealed driver 61.3% of the time.
Conclusion: Both explicit methods provide only partial access to the decision structure. The model's stated "most important attribute" often does not align with the attribute that mathematically drives its choice in the behavioral model.

Robustness and Perturbation

Reproducibility: The Score-Based Judge is more reproducible across prompt variants than Direct Reports, yet it does not recover the behavioral driver more accurately.
Intervention Analysis: Targeted occlusion (removing or neutralizing an attribute) shows that attributes ranked higher by the behavioral model cause larger disruptions in both choices and stated attributes. This confirms the behavioral model is recovering a structure relevant to the model's actual decision process, even if the model cannot explicitly articulate it.

Significance and Claims

The paper argues for a "weak, decision-local form of superficial belief" in LLMs.

Structured but Opaque: LLM decisions are not arbitrary; they exhibit a systematic, probabilistic local structure over attributes that allows for accurate prediction. However, this structure is not fully transparent to the model's own explicit reasoning.
Limitations of Self-Report: Explicit elicitation (both direct and score-based) fails to fully recover the underlying decision drivers. The models behave as if guided by specific priorities, but their verbal access to these priorities is limited and imperfect.
Modest Claims on Belief: The authors do not claim LLMs possess "fully articulated belief" in the philosophical sense. Instead, the evidence supports a view where behavior displays stable patterns (justifying a superficial attribution of belief) but lacks the internal coherence or self-knowledge required for stronger claims.
Reproducibility vs. Alignment: The study highlights that higher reproducibility in output (e.g., via score-based judges) does not necessarily imply closer alignment with the underlying decision structure.

The paper concludes that while LLMs display "superficial beliefs" in the form of stable behavioral dispositions, these are distinct from the explicit rationales they generate, suggesting a gap between doing and saying in current LLM architectures.