Technical Summary: Belief Without Behavior

Problem Statement

Effective social interaction requires agents to translate mental state inferences (Theory of Mind, or ToM) into coordinated behavioral signals across verbal and non-verbal channels simultaneously. While existing benchmarks evaluate ToM reasoning (often via static question-answering) and embodied physical task completion in isolation, they fail to measure the critical gap between social inference and social action. Current frameworks either treat agents as passive observers of mental states or reduce strategic interaction to language and abstract actions, neglecting the integration of spatial trajectories, gaze, and facial expression required for "embodied social intelligence." Consequently, it remains an open empirical question whether current AI systems can translate belief inferences about others into coordinated multimodal actions.

Methodology: The MOSAIC Benchmark

The authors introduce MOSAIC (Multimodal Orchestration of Social Action, Inference, and Communication), a controlled benchmark designed to close this assessment gap.

• Environment & Mechanics: Two embodied agents interact in a virtual 3D environment (Unity) containing two boxes, one hiding a reward. The interaction spans 10 rounds with strict alternation.
  • Roles: The "Subject" (Marie) knows the reward location; the "Participant" must infer it.
  • Modes: In Cooperative mode, the Subject guides the Participant to the reward. In Competitive mode, the Subject strategically misleads the Participant.
  • ToM Constraints: The Subject operates under two conditions: ToM-0 (acting on own preferences without modeling the other's beliefs) and ToM-1 (explicitly modeling and manipulating the Participant's beliefs).
  • Modalities: Agents generate verbal statements, spatial trajectories, gaze direction, and facial expressions. Verbal exchanges occur in rounds 3, 6, and 9; other rounds are non-verbal.
• Models Evaluated: 13 models were tested, including 11 Vision-Language Models (VLMs) from families like LLaVA, Qwen-VL, InternVL, and MiniCPM-V, a text-only LLM baseline (Llama 3.1), and PCM-LLM, a hybrid cognitive architecture with an explicit structured ToM module used as an architectural reference.
• Metrics:
  • TOCS (ToM Outcome Conformance Score): Measures if trial outcomes match ToM-theoretic predictions (e.g., successful deception in Competitive-ToM-1).
  • Signal-Level Metrics: Facial Expressivity Score (FES), Trajectory Alignment Score (TAS), Gaze Alignment Score (GAS), and Signal Sensitivity Score (SSS) to analyze the production and decoding of non-verbal signals.

Key Contributions

1. A Novel Benchmark Framework: MOSAIC is the first framework to simultaneously require active ToM deployment, multichannel embodied communication (verbal + non-verbal), and controlled causal manipulation across cooperative and competitive scenarios.
2. Systematic Characterization of Failure Modes: The study identifies two sequential bottlenecks in current open-source VLMs:
   • Production Bottleneck: Most VLMs fail to produce directionally coherent non-verbal signals (trajectory and gaze).
   • Decoding Bottleneck: Even when signals are present, participant agents fail to extract and act upon them.
   • The structured reference architecture, PCM-LLM, successfully clears both bottlenecks, suggesting that explicit belief-action coupling is a sufficient ingredient for these tasks.
3. Modality Ablation Evidence: The study provides evidence that current VLMs do not meaningfully integrate visual social signals. In some models (e.g., InternVL-8b), removing visual input improved directional signal quality, suggesting visual content acts as attentional interference rather than an informative channel. In others (e.g., MiniCPM-8b), behavior was driven primarily by language-based priors regardless of visual input.

Results

• Performance Gap: Across 13 models and 200 trials per model, VLMs failed to produce behaviors consistent with expected outcomes under ToM-order constraints. Imposing explicit ToM constraints produced no reliable behavioral change aligned with the specified reasoning level.
• PCM-LLM Superiority: PCM-LLM achieved high TOCS across all conditions with low uncertainty and positional bias. It demonstrated the ability to modulate signaling strategies (e.g., inverting gaze and trajectory in Competitive-ToM-1 to deceive), confirming that the observed failures in VLMs are architectural rather than inherent to the task.
• Fine-tuning Limitations: Action-level fine-tuning (imitation learning) on VLMs using PCM-LLM demonstrations failed to produce genuine strategic improvement. While it altered TOCS scores, it did not improve cross-channel signal coherence, suggesting that trial-level communicative capacity cannot be acquired through timestep-level supervision alone.
• Signal Decoding Failure: Even in cases where a subject model produced coherent signals (e.g., InternVL-14b), the participant model failed to decode them (SSS near zero), indicating a systemic inability to translate non-verbal cues into belief updates and actions.

Significance and Claims

The paper argues that the gap between belief inference and social action constitutes a distinct and measurable limitation of current VLM architectures. The authors claim that:

• Social reasoning expressed in language does not reliably propagate into the coordinated behavioral outputs required for ToM-theoretic predictions.
• The failure is not merely a lack of reasoning capability but a failure of embodied integration: the inability to couple belief states with multimodal action selection.
• The success of PCM-LLM suggests that explicit, structured belief-action coupling is a necessary architectural component for embodied social intelligence, a capability currently missing in end-to-end neural VLMs.
• The benchmark provides a tool to investigate whether this gap is a fundamental architectural constraint or can be reduced through alternative training objectives, such as reinforcement learning with trial-level outcome signals.

The authors remain modest regarding the scope, acknowledging that the current scenario is a single interaction topology and that the study lacks a human performance baseline due to methodological challenges in recording physiological signals in the virtual environment. They posit that the documented failure modes may not generalize to all embodied social tasks but represent a critical limitation in the current generation of open-source models up to 15B parameters.