Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind

1. Problem Statement

Large Language Models (LLMs) and LLM-based agents have advanced significantly but struggle when user instructions are imprecise or based on erroneous subjective beliefs.

• Epistemic Divergence: A critical gap exists between the user's subjective belief (influenced by limited observations, personal profiles, and domain knowledge) and the objective true state of the environment.
• Limitation of Current ToM: Existing Theory of Mind (ToM) evaluations for LLMs focus on isolated belief inference (e.g., "What does the user think?") rather than the functional utility of ToM in real-world interactions. Current models often fail to detect why a user holds a wrong belief and consequently prioritize following erroneous instructions over correcting the user's mental model, leading to task failure.
• Research Questions:
  1. Can ToM be evaluated via its utility in practical scenarios rather than as an isolated cognitive task?
  2. How can ToM-based reasoning be integrated to improve downstream task performance?

2. Methodology

A. Formal Framework: Bayesian Theory of Mind

The authors formalize ToM as a mechanism for epistemic divergence detection and resolution. They model the interaction as a partially observable process:

• Environment: Defined by a true latent state ($s^*$) and transition dynamics.
• User: Operates with a cognitive state $\Psi_u = \langle g, b \rangle$, where $g$ is the explicit goal and $b$ is a latent belief distribution over the state space. Due to limited observability, $b$ often diverges from $s^*$. The user generates an interaction trajectory $\tau$ based on $b$, which may deviate from the "gold" trajectory.
• Agent: Observes the user's trajectory, instructions, and partial observations. The agent must infer the user's latent belief ($\hat{b}$) and profile ($P$) to bridge the gap between $b$ and $s^*$. The agent's goal is to synchronize its understanding with the user to correct misconceptions rather than blindly executing instructions.

B. Benchmark: SynchToM

To evaluate this framework, the authors introduce SynchToM, a benchmark designed to operationalize ToM in practical scenarios.

• Data Generation Pipeline:
  1. Scenario Synthesis: Converts raw data from four domains (Software Engineering, User Preference, Education, Culture) into ToM scenarios. This involves generating a "cognitive gap" where the user's belief ($b$) mismatches the true state ($s^*$).
  2. Trajectory Construction: Simulates multi-turn interactions (up to 10 turns) where the user acts based on their flawed belief, creating a trajectory that reveals their mental state.
  3. Quality Control: An LLM-based validator ensures data fidelity across five dimensions: behavioral faithfulness, realism, action consistency, inferability, and cognitive soundness.
• Dataset Statistics: 390 high-fidelity test instances and 6,522 training instances.
• Evaluation Dimensions:
  1. Latent Belief Explanation: Does the model identify the user's misconception and its cause?
  2. User Profile Modeling: Can the model infer the user's background/preferences shaping the belief?
  3. Problem Resolution: Does the model provide a correct solution that addresses the root cause (not just the symptom)?

C. Enhancement Strategies

To bridge the gap identified in evaluations, the authors propose two methods:

1. Reinforcement Learning (RL): Training models (specifically Qwen3-8B) on the SynchToM trajectory data using a reward function that penalizes failure to align beliefs and rewards correct resolution. Special tokens (<ToM Belief>, <ToM Profile>) guide the model to focus on cognitive gaps.
2. Training-Free Multi-Agent Collaboration: A framework where multiple models vote on the best latent belief and profile. The "backbone" model uses these optimized mental states to improve its own resolution performance.

3. Key Contributions

1. Formalization of Functional ToM: Shifted the definition of ToM from a static reasoning skill to a dynamic, interaction-level mechanism for detecting and resolving epistemic divergence between user beliefs and objective reality.
2. SynchToM Benchmark: Introduced a comprehensive benchmark with interaction trajectories across four diverse domains, revealing a critical "ToM-Resolution Linkage": accurate mental state attribution is a causal prerequisite for successful task resolution.
3. Empirical Validation of Training: Demonstrated that RL training on trajectory-based ToM data significantly improves downstream task performance, proving that ToM capabilities are trainable and transferable.
4. Ablation Studies: Provided causal evidence that misaligned mental state modeling leads to resolution failure, while extended interaction trajectories (more turns) stabilize belief reconciliation.

4. Key Results

• Model Performance Gaps: Evaluation of 11 leading models (including GPT-5, Gemini-3, Claude, and open-source models) revealed that proprietary models do not always outperform smaller open-source models in ToM tasks. For instance, Kimi-K2 and MiniMax-M2.1 sometimes outperformed Claude-Sonnet-4.5.
• The "Belief Bottleneck": While reasoning effort (e.g., Chain of Thought) improved solution accuracy, Latent Belief tracking remained the bottleneck. Models often solved the task by guessing or following instructions without truly understanding why the user was wrong.
• Domain Sensitivity: Models performed better in Software Engineering and Culture scenarios (where world knowledge is clearer) compared to Preference and Education (more subjective/variable).
• Impact of Ground Truth: Providing models with Ground Truth (GT) beliefs or profiles significantly boosted resolution scores. Conversely, shuffling beliefs and profiles (creating mismatches) caused consistent performance degradation, proving the causal link between accurate ToM and task success.
• Training Success: The RL-trained model showed consistent improvements in the Solution dimension (up to +10-15% in some domains) and better robustness in multi-turn settings compared to the base model.
• Case Study (Culture): In a scenario where a user tried to dissolve sesame paste in water (mistaking it for a soup base), a lower-performing model (InternLM) tried to "fix" the physics (add emulsifiers), while a high-performing model (GPT-5) recognized the epistemic error (the ingredient is a dipping sauce, not a stock) and redirected the user.

5. Significance

• Paradigm Shift: The paper argues that ToM should not be treated as a standalone cognitive test but as an essential interaction mechanism for AI agents.
• Practical Utility: It demonstrates that agents capable of "reading between the lines" (inferring latent beliefs) are more effective at solving real-world problems where user instructions are flawed.
• Future Directions: The work establishes a foundation for developing more collaborative, resilient, and human-aligned AI agents that can actively manage epistemic gaps rather than passively following instructions. It highlights that belief calibration is a critical component of the next generation of user-agent interaction.