Theory of Mind in Action: The Instruction Inference Task in Dynamic Human-Agent Collaboration

1. Problem Statement

Successful human-agent collaboration often relies on the agent's ability to interpret instructions that are incomplete or ambiguous. In natural communication, humans frequently rely on Theory of Mind (ToM)—the ability to model another entity's mental states (beliefs, intentions, goals)—to infer unspoken intentions from context.

• The Gap: Traditional autonomous agents often fail in these scenarios because they interpret instructions literally, lacking the pragmatic context required to infer latent goals.
• The Challenge: While Large Language Models (LLMs) show emergent ToM capabilities in static benchmarks (e.g., false-belief tests), it remains unclear if they can apply these capabilities in dynamic, goal-oriented collaborative environments where they must act on ambiguous instructions to achieve a shared objective.
• Core Question: Can LLM-based agents effectively exercise ToM to infer a human principal's intended goal from unclear instructions and execute optimal collaborative actions?

2. Methodology

A. The Task: Instruction Inference

The authors introduce a novel task called Instruction Inference, designed to evaluate ToM in a dynamic setting.

• Environment: The task takes place in the "Doors, Keys, and Gems" grid world. An agent (Tomcat) must assist a principal (simulated human) in collecting a gem by retrieving keys and unlocking doors.
• Instruction Types: Based on Grice's Cooperative Principle, instructions are categorized into:
  • Clear: No maxim violations.
  • Incomplete: Violation of the Quantity maxim (missing critical info, e.g., "Get the key" when multiple exist).
  • Ambiguous: Violation of the Manner maxim (multiple interpretations, e.g., "Get that" while pointing).
• Goal: The agent must infer the principal's latent goal (which gem to retrieve) based on the instruction, the principal's movement, and the grid configuration, then generate a plan and execute actions.

B. The Agent: Tomcat

The authors developed Tomcat, an LLM-based agent with two distinct prompting variants:

1. Commonsense Prompt (CP): Relies on general commonsense knowledge and two detailed problem walkthroughs (demonstrations) to understand the task. It does not include specific reasoning examples for the current scenario.
2. Few-shot Chain-of-Thought (Fs-CoT): Uses seven specific input-output exemplars demonstrating structured reasoning (identifying instruction type, inferring intent, generating a plan, and listing actions). This variant leverages in-context learning to guide the reasoning process.

C. Experimental Setup

• Models: Tomcat was implemented on three leading LLMs: GPT-4o, DeepSeek-R1, and Gemma-3-27B.
• Human Baseline: A study was conducted with 52 human participants (STEM students) who performed the same task under conditions identical to the CP variant (using the same information and objectives).
• Dataset: 20 unique grid-instruction scenarios derived from 16 grid configurations, varying in spatial obstruction (low, moderate, high) and ambiguity types (deictic vs. implicit goal inference).
• Metrics:
  • Intent Accuracy: Correctly identifying the principal's goal.
  • Action Feasibility/Optimality: Whether actions are executable and efficient.
  • Plan Feasibility/Optimality: Whether the generated plan (including principal's future steps) is executable and logically ordered.
  • Instruction Accuracy: Correctly classifying the instruction type (Clear vs. Unclear).

3. Key Contributions

1. Instruction Inference Task: A novel benchmark that moves beyond static narrative ToM tests to evaluate ToM in a dynamic, collaborative, goal-driven environment with incomplete/ambiguous instructions.
2. Tomcat Framework: An LLM-based agent architecture designed to reason about pragmatic context and principal actions. The study systematically compares Fs-CoT (structured reasoning) vs. CP (commonsense) prompting strategies.
3. Empirical Human-LLM Comparison: A rigorous evaluation comparing three LLMs against 52 human participants, providing a benchmark for human-level ToM performance in collaborative tasks.
4. Insights on Prompting & Model Architecture: Analysis of how different LLM architectures (RLHF-aligned vs. Self-Discovery vs. Distillation) respond to few-shot reasoning exemplars in ToM tasks.

4. Key Results

A. Performance Comparison

• Fs-CoT vs. CP: Across all models, the Fs-CoT variant consistently outperformed the CP variant.
  • GPT-4o (Fs-CoT): Achieved performance comparable to human participants in Intent Accuracy (85% vs. 80.4%) and Plan Optimality (75% vs. 71.2%). It significantly outperformed its CP counterpart.
  • DeepSeek-R1 (Fs-CoT): Also achieved performance comparable to humans, showing substantial gains over its CP variant.
  • Gemma-3-27B (Fs-CoT): Showed significant improvement over its CP variant (e.g., 500% gain in Plan Optimality) but still lagged behind human participants, likely due to smaller model capacity limiting complex reasoning.

B. Statistical Significance

• GPT-4o & DeepSeek-R1: Statistical tests (McNemar's, Chi-square, Kruskal-Wallis) confirmed that the Fs-CoT variants performed on par with humans, while CP variants generally underperformed humans significantly.
• Gemma-3-27B: Both variants performed significantly worse than humans, suggesting that model size is a critical bottleneck for ToM reasoning in this domain.

C. Qualitative Findings

• Reasoning Depth: Fs-CoT enabled agents to infer latent goals (e.g., realizing that "Get the red key" implies needing a second key to unlock a subsequent door) and generate multi-step plans. CP variants often failed to look beyond the immediate instruction, treating tasks as isolated actions.
• Instruction Identification: While both variants had high recall for unclear instructions, Fs-CoT had better precision, reducing false positives in classifying clear instructions as ambiguous.

5. Significance and Implications

• ToM in Action: The study demonstrates that ToM is not just a static reasoning capability but a functional requirement for effective human-agent collaboration. Agents must model the principal's future steps to act correctly.
• The Power of Structured Reasoning: The results strongly suggest that Few-shot Chain-of-Thought (Fs-CoT) prompting is a highly effective strategy for unlocking ToM capabilities in LLMs, bridging the gap between literal instruction following and pragmatic understanding.
• Model Architecture Matters: The varying success of GPT-4o, DeepSeek-R1, and Gemma-3-27B highlights that model capacity and training objectives (e.g., RLHF alignment vs. self-discovery) significantly influence ToM performance.
  • DeepSeek-R1 showed that self-directed reasoning models can perform well even without extensive few-shot examples, though Fs-CoT still helped.
  • GPT-4o benefited most from the structural alignment provided by Fs-CoT.
• Future Directions: The paper argues for moving LLMs from "transactional chatbots" to "relational cognitive agents" capable of maintaining persistent mental models of users. It also challenges the notion that few-shot prompting always degrades performance in reasoning-heavy models (specifically regarding DeepSeek-R1).

In conclusion, the paper establishes that with the right prompting strategies (Fs-CoT) and sufficient model capacity, LLMs can achieve human-level Theory of Mind capabilities in dynamic collaborative tasks, enabling them to interpret ambiguous instructions and act as effective partners.