🤖 AI

What Do You Think I Think? Accounting for Human Beliefs Using Second-Order Theory of Mind

This paper presents a second-order Theory of Mind framework using I-POMDPs that enables an agent to detect and adaptively address human cognitive biases and erroneous beliefs about the agent, thereby improving interaction quality and feedback utility as demonstrated in a user study.

Original authors: Patrick Callaghan, Reid Simmons, Henny Admoni

Published 2026-05-14

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Patrick Callaghan, Reid Simmons, Henny Admoni

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot how to sort a deck of special cards. You have a secret rule in your head (e.g., "Red cards go in Bin 1, Blue cards go in Bin 2"). The robot doesn't know the rule yet; it has to learn it by watching you sort the cards and listening to what you say.

The problem is that you might be making mistakes in how you think the robot thinks.

The Core Problem: The "Mind-Reading" Gap

In this paper, the researchers are trying to solve a specific communication breakdown. Humans often have "cognitive biases"—mental shortcuts that make us assume things about others that aren't true.

The Scenario: You might think, "The robot is smart; it must know that Red means Bin 1 because that's the most obvious pattern."
The Reality: The robot is actually confused and thinks, "I have no idea what Red means yet."
The Result: You stop teaching because you think the robot knows, or you teach in a way that confuses the robot because you're assuming it understands your shortcuts.

The Solution: The "Second-Order" Mind

The researchers built a robot with a special mental superpower called Second-Order Theory of Mind (ToM-2).

To understand this, think of it like a game of "I know that you know that I know":

Zero-Order Robot (The Basic Bot): It only looks at the cards you put in the bins. It thinks, "Okay, you put a Red card here. I will update my rule." It has no idea what you are thinking.
Second-Order Robot (The Mind-Reader): This robot doesn't just look at the cards; it simulates a model of your brain. It asks itself: "What does the human think I know right now?"

If the robot realizes, "Oh, the human thinks I know the rule is about Color, but I actually think it's about Shape," it can spot the disconnect.

How the Robot Fixes the Conversation

The paper describes a game where the robot has to guess a sorting rule. The robot uses its "mind-reading" ability to detect two specific human habits (biases):

Representativeness Heuristic: You might focus only on the cards that look like the rule you have in mind and ignore the ones that prove other rules wrong.
Confirmation Bias: You might only pay attention to the robot's feedback if it confirms what you already believe.

When the robot detects these habits, it changes its strategy. Instead of just saying, "I'm not sure," it gives you a two-part nudge:

Part A (The Nudge): "It seems like you want me to think the rule is about Shape, but..."
Part B (The Correction): "...it's still possible the rule is about Color."

By explicitly stating, "I know what you think I'm thinking, but I'm actually thinking something else," the robot breaks the human's mental loop and forces them to reconsider their teaching strategy.

The Experiment: Teaching Cards

The researchers tested this with 30 real people.

The Setup: People used a computer to teach a virtual agent to sort cards with different colors, shapes, and numbers.
The Comparison:
- Group A taught a "Basic Bot" (Zero-Order) that couldn't read minds.
- Group B taught the "Mind-Reading Bot" (Second-Order) that could detect biases.

What Happened?

Better Teaching: People taught the Mind-Reading Bot much more efficiently. Because the bot corrected their misunderstandings, the humans stopped making redundant moves and gave better examples.
More Information: The cards people chose to show the Mind-Reading Bot contained more useful information. The humans were subconsciously adjusting their teaching style because the bot was "talking back" in a way that made sense of their confusion.
The "Unconscious" Benefit: Interestingly, when asked afterwards, the humans didn't say, "Wow, that bot was amazing at reading my mind!" They just felt the feedback was more useful. They didn't necessarily realize why the interaction was smoother; they just knew it worked better.

The Bottom Line

This paper shows that if an AI can model what you think it knows (and realize when you are wrong about that), it can give you better feedback. This helps you teach it faster and more accurately, even if you don't fully realize the AI is doing the heavy lifting to fix your mental blind spots.

Note: The researchers tested this in a simple card-sorting game. They admit that real life is much more complex, and their robot only looked at two specific types of human biases, but the core idea—that modeling human misconceptions improves teamwork—worked in their experiment.

Technical Summary: Accounting for Human Beliefs Using Second-Order Theory of Mind

Problem Statement
Successful human-agent interaction relies on the human possessing an accurate model of the agent's capabilities and beliefs. However, humans often employ cognitive biases and heuristics (CBH), such as the Representativeness Heuristic and Confirmation Bias, which can lead to erroneous beliefs about what an agent knows or how it reasons. These discrepancies between an agent's actual knowledge and a human's perception of that knowledge can hinder interaction efficiency. While prior work has explored endowing agents with Theory of Mind (ToM) or second-order Theory of Mind (ToM-2), this work specifically addresses the challenge of modeling how CBH influences a human's inference of an agent's beliefs, enabling the agent to detect these errors and adapt its feedback accordingly.

Methodology
The authors propose a framework utilizing an Interactive Partially Observable Markov Decision Process (I-POMDP) to implement a second-order Theory of Mind (ToM-2). In this architecture, the agent maintains a nested model of the human, where the human is modeled as a ToM-1 agent (a first-order reasoner).

Domain: The system is evaluated in a turn-based card-sorting task. A "rule" dictates how multi-featured cards (varying in Color, Fill, Number, Shape) are sorted into two bins. The human teacher demonstrates the rule by sorting cards, and the agent provides feedback on its current belief regarding the rule features.
CBH Modeling: The I-POMDP framework allows the agent to maintain beliefs over multiple human models ( $m_{H,1}$ $m_{H, 1}$ ) that encode specific CBH mechanisms.
- Representativeness Heuristic (RH): Operationalized via "Local Thinking" (LT), where the agent approximates that the human focuses on semantically similar features while neglecting counterfactual examples necessary to eliminate other rules. This is modeled by augmenting the observation function ( $\Omega$ ) with a weighted cosine similarity.
- Confirmation Bias (CB): Modeled as a tendency for the human to expect agent feedback that aligns with their prior beliefs about the rule, rather than feedback that logically eliminates alternatives.
Adaptive Feedback Generation: The agent utilizes its ToM-2 beliefs to detect discrepancies between its actual knowledge and the human's approximation of that knowledge. Based on this detection, the agent generates two types of feedback:
1. Confidence Statements (CS): Standard feedback conveying uncertainty or certainty about a rule feature (e.g., "I think the Class is Shape").
2. Understanding Statements (US): Feedback designed to "nudge" the human out of CB-induced reasoning by explicitly revealing what the agent knows about a feature the human intends to teach, while highlighting potential discrepancies (e.g., "It seems like you want me to know the Class is Shape, but it's still possible that the Class is Fill").
  The agent defaults to CS for information gain but switches to a (US+CS) tandem when it detects erroneous beliefs caused by CBH.

Key Contributions

Novel Framework: A ToM-2 architecture based on I-POMDPs that explicitly models the influence of CBH on a human's beliefs about an agent's beliefs, rather than just modeling the human's actions.
Detection and Adaptation: A mechanism for the agent to detect when a human is relying on specific heuristics (RH and CB) and adaptively generate feedback (US) to correct the human's reasoning process.
Empirical Validation: An in-person user study demonstrating that this approach improves the informativeness of human teaching actions and is perceived as more useful by participants.

Results
The study involved 30 participants teaching a virtual agent under three conditions: ToM-0 (no ToM), ToM-0 Random (providing extra feedback randomly), and ToM-2 (adaptive feedback based on CBH detection).

Objective Metrics:
- Information Gain: The ToM-2 learner elicited significantly more informative teaching actions (higher relative information gain) compared to both baselines, both after one-statement and two-statement feedback.
- Efficiency: The ToM-2 learner showed a marginal but significant reduction in the number of teacher actions required to successfully end a session compared to the standard ToM-0 baseline.
- Detection: Qualitative simulations showed the ToM-2 learner could identify the correct rule and the teacher's underlying model (including CBH), whereas the ToM-0 learner could not.
Subjective Metrics:
- Participants rated the feedback from the ToM-2 learner as significantly more useful than the baselines.
- However, there was no statistically significant difference in participants' self-reported confidence in the agent's understanding, their teaching confidence, or the pleasantness of the experience.
- Participants often reported finding the learners indistinguishable or equally easy to teach, suggesting the benefits of the ToM-2 model may be subconscious.

Significance and Limitations
The paper claims that enabling an agent to model the causes of human belief errors (specifically CBH) allows for more effective interaction by generating targeted feedback that aids human reasoning. The results suggest that while humans may not be consciously aware of the agent's advanced modeling, the agent's ability to account for CBH leads to objectively more efficient and informative interactions.

The authors acknowledge several limitations:

The proof-of-concept domain (card sorting) is simpler than real-world human-agent interactions.
The study explored only two specific biases (RH and CB) out of the many that influence human behavior.
The computational complexity of I-POMDPs remains a barrier to scaling this approach to more complex tasks and human models, as the authors utilized the framework strictly for tracking beliefs rather than solving for optimal policies in the full nested space.