Technical Summary: MindClaw

Problem Statement

Current Theory of Mind (ToM) benchmarks have advanced mental-state recognition in text and multimodal domains but largely treat ToM as an offline question-answering or label-prediction task. These existing formulations fail to test whether an embodied agent can maintain a connection to a changing environment, update actor-specific beliefs over time, decide when reasoning is necessary, and intervene only when assistance is truly useful.

In real-world human-centered environments, an assistant that acts by default can be disruptive or harmful. The core challenge is not merely inferring a human's beliefs, desires, and intentions (the Belief–Desire–Intention or BDI framework), but determining the precise timing for intervention. An agent must distinguish between scenarios where a human is progressing normally (requiring no action) and scenarios where a human is acting under a false belief or hidden goal (requiring a minimal, helpful intervention). Existing robot-centric frameworks like MindPower have moved toward action generation but still operate primarily as static, final-stage decision processes rather than real-time, closed-loop cognitive control systems.

Methodology

The authors propose MindClaw, a framework that extends robot-centric ToM reasoning from static hierarchy completion to a real-time, closed-loop setting focused on precision intervention. The system is designed to remain connected to an executable environment (e.g., VirtualHome, ThreeDWorld, or static video replays) and decide dynamically whether to update memory, reason, act, or remain silent.

Architecture

MindClaw consists of three primary layers:

1. Input Interface: Accepts multi-source inputs including static videos, live simulator streams, and human-control events, normalizing them into a common window representation.
2. Claw Layer (Control): The core control mechanism that maintains the online state and orchestrates the interaction loop. It contains:
   • Engine: Manages the runtime loop, ensuring observation precedes trigger decisions.
   • Adapter: Converts diverse input sources into a canonical format for the reasoning layer.
   • Memory: Stores a belief table separating visually grounded facts from actor-specific beliefs (e.g., actor_beliefs[actor][object] → location). This allows the system to detect mismatches between reality and an actor's mental state.
   • Trigger: An embodied cognitive skill that decides the next internal operation ($z_t$) based on the current observation, memory, and candidate slots.
3. Reasoning Layer (Model-Based Services): Provides three modular capabilities:
   • Observation: Extracts structured descriptions of the scene.
   • Mental Reasoning: Invoked only when the trigger decides, this module infers if an actor is acting under a false belief or hidden goal and determines if intervention is warranted.
   • Action Generation: Converts reasoning results into executable robot actions or a "none" (noop) command.

The Trigger as an Embodied Cognitive Skill

The central innovation is formulating the trigger as a skill that selects intermediate cognitive operations rather than mapping video directly to robot actions. The operation space includes:

• Updating visual facts or actor beliefs.
• Running mental reasoning.
• Running action generation.
• Noop: Explicitly choosing to do nothing.

The trigger is implemented as a skill-conditioned policy ($\pi_\theta$) that factors the decision into operation selection (what to do) and slot grounding (who/what/where).

Skill Collection and Inference

To optimize the trigger, the authors collect "embodied cognitive skills" from paired correct and incorrect trajectories. A strong LLM summarizes these into reusable decision patterns (rules), categorized as:

• Strongly Preferred ($S^+$): Deterministic conditions (e.g., regex matches) that force a specific operation.
• Preferred ($S^\pm$): Conditions that suggest an operation but require model judgment.
• Negative ($S^-$): Conditions that rule out an operation.

During inference, the system first applies deterministic matching for strong rules. If no strong rule applies, a skill-augmented trigger model predicts the operation using the remaining preferred and negative conditions.

Key Contributions

1. Closed-Loop Extension: The work extends robot-centric ToM reasoning from final-stage decision generation to a real-time closed-loop setting where the agent maintains state and acts back into a changing environment.
2. Precision Intervention: It establishes precision intervention as the central objective: the robot must generate helpful actions only when needed and remain silent otherwise, avoiding disruptive or unnecessary help.
3. Cognitive Trigger Skill: The trigger is formulated as an embodied cognitive skill that performs intermediate operation selection over observation, belief memory, and scene candidates. This enables explicit control over the timing of memory updates, reasoning, and action generation.

Experimental Results

The framework was evaluated on the MindPower Benchmark, which includes 590 video-text examples involving false-belief correction and implicit goal inference.

• Baselines: Direct Video-Language Models (VLMs) struggled significantly with task awareness (Task Accuracy, TA) and intervention calibration (Precision Intervention Accuracy, PIA). Many baselines achieved near-zero TA, indicating a failure to identify relevant mental-state contexts.
• MindClaw Performance: MindClaw achieved the best overall performance, significantly outperforming direct baselines in both TA (14.36% vs. <2% for most baselines) and PIA (36.63% vs. <20% for most baselines).
• Closed-Loop Satisfaction: MindClaw maintained a 100% Closed-loop / Action Satisfaction (CS) score, demonstrating that improved timing and precision did not come at the cost of action validity.
• Ablation Studies:
  • Belief Table: Removing the explicit belief table caused a drastic performance drop (e.g., Qwen3-4B TA fell from 9.51% to 1.64%), confirming the necessity of actor-specific memory.
  • Skill Guidance: Removing the skill-based rules reduced accuracy significantly. Combining supervised fine-tuning with skill guidance yielded the highest non-noop accuracy (64.14%), proving that skills provide essential structured intermediate knowledge for decision-making.

Significance and Claims

The paper claims that MindClaw demonstrates the critical importance of trigger-skill optimization for closed-loop embodied ToM assistance. By decoupling internal cognitive control (the trigger) from external robot behavior, the system can effectively manage the trade-off between helpfulness and intrusiveness.

The authors emphasize that real embodied assistance is not equivalent to always producing an action. The ability to remain silent when a human is aligned with the environment, and to intervene precisely when a mismatch exists, is a fundamental capability for useful interactive helpers. MindClaw provides a framework where the agent explicitly reasons about when to reason and when to act, moving beyond simple mental-state recognition to calibrated, precision intervention.