Inferring World Belief States in Dynamic Real-World Environments

1. Problem Statement

The paper addresses the challenge of enabling a robot to accurately infer a human teammate's belief state (defined as Level 1 Situation Awareness) in a dynamic, 3D, and partially observable environment (e.g., a household).

• Core Objective: To estimate what a human knows about the locations and existence of objects in their environment without relying on privileged information (like wearable sensors or direct communication).
• Key Challenges:
  1. Partial Observability: The robot has a limited field of view (mounted camera) and cannot see the entire workspace simultaneously.
  2. Dynamic Environments: Objects can be moved, and people can occlude them.
  3. Object Permanence: The system must remember objects even when they are out of sight or relocated.
  4. Portability: The inferred belief state must be an explicit, structured data format usable for downstream tasks (e.g., Theory of Mind reasoning).
• Gap in Literature: Previous work on explicit belief state modeling was limited to abstract 2D simulations or required privileged data. No existing software package supports semantic mapping with object permanence in dynamic 3D environments.

2. Methodology

The authors propose a system based on Recursive Theory of Mind, where the robot maintains its own belief state ($\beta_{robot}$) and uses it to infer the human's belief state ($\hat{\beta}_{human}$).

A. System Architecture

The system consists of two main subsystems:

1. Perception Stack:
   • Input: RGB/D camera streams and robot localization.
   • Object Detection: Uses OWLv2 for open-vocabulary detection (allowing specific class distinctions like "blue cup" vs. "red cup") and SAM2 for segmentation.
   • 3D Localization: Projects 2D detections into global 3D coordinates using depth data and the robot's pose.
   • Human Tracking: Uses RTMPose and MMPose to estimate human skeleton keypoints in 3D. Gaze direction is approximated by the normal vector between hip and shoulder joints.
   • False Positive Reduction: Cross-references RGB and depth detections with a 30% bounding box overlap threshold.
2. Belief State Algorithm:
   • Representation: A semantic map containing a set of objects $\{o\}$ with properties (location, class).
   • Update Mechanism: When the robot observes objects, it updates $\beta_{robot}$.
   • Inference Logic: When a human is visible, the robot estimates the human's trajectory (using A* pathfinding) and gaze region. It then "raycasts" from the human's perspective through the robot's map to determine which objects the human would have seen.
   • Resolution: Uses a minimum cost matching algorithm (L2-norm distance) to map observed objects to the human's existing belief state, handling object permanence and class interchangeability.

B. Evaluation Metric: SMCC

The authors introduce SMCC (Summed Minimum Cost by Class) to measure the distance between two belief states.

• It treats objects of the same class as interchangeable (e.g., any spoon is valid for a "spoon" slot).
• It calculates the minimum distance required to map objects from set A to set B.
• Figures of Merit:
  1. Inference Error: $SMCC(\hat{\beta}_{human}, \beta_{human})$ (Robot's estimate vs. Human's actual state).
  2. Human Error: $SMCC(\beta_{human}, \beta_{true})$ (Human's state vs. Ground Truth).
  3. Robot Error: $SMCC(\beta_{robot}, \beta_{true})$ (Robot's map vs. Ground Truth).

3. Experimental Setup & Scenarios

• Simulation: Used VirtualHome for quantitative evaluation.
  • Scenario: "Parents are Out!" – A person leaves a tidy house; objects are shuffled (simulating a party); the person returns, walks through to assess the mess, and prepares for an activity.
  • Duration: ~60 seconds of agent action.
• Real-World Deployment: Tested on a Stretch RE2 robot in a physical environment to verify portability.
• Downstream Task (Active Assistance): The robot identifies objects the human is unaware of that are relevant to a specific task (e.g., cooking) and notifies the user. This was evaluated using Large Language Models (LLMs) like Qwen2.5-32B and a deBERTa-v3 classifier.

4. Key Results

• Perception Accuracy: The perception stack using off-the-shelf models (OWLv2, SAM2) performed nearly identically to ground-truth perception in simulation. The mean inference error was approximately 0.35m, with ablation studies showing that errors were primarily due to the simulator's limitations (e.g., lack of precise camera pose) rather than the perception models.
• Inference Performance:
  • The robot successfully tracked the human's belief state.
  • The inference error ($\hat{\beta}_{human}$ vs. $\beta_{human}$) was significantly lower than the human's own error relative to the true world state, indicating the robot could accurately model the human's "blind spots."
  • Simulator limitations (wavering camera poses) accounted for roughly one-third of the total error.
• Active Assistance:
  • All semantic reasoning methods outperformed random selection.
  • LLM Performance: The "Single Chain-of-Thought" (CoT) prompt yielded the highest Precision (0.77), which is critical for user trust (minimizing false alarms).
  • Efficiency: The deBERTa model offered comparable performance to LLMs at a lower computational cost.

5. Key Contributions

1. First 3D Dynamic Implementation: The first approach to inferring human belief states in a realistic 3D, partially observable, and dynamic environment without privileged data.
2. Explicit Belief State Representation: A portable, explainable data structure (semantic map) that supports object permanence and can be used for downstream reasoning.
3. New Evaluation Metric: The introduction of SMCC, a class-aware distance metric specifically designed for evaluating situation awareness in dynamic environments.
4. Downstream Application: Demonstrated a practical use case where the inferred belief state enables active assistance, allowing the robot to proactively inform humans about missing or moved objects relevant to their tasks.
5. Open Source: Release of the codebase, data, and a Python library for the belief state algorithm.

6. Significance and Future Work

• Significance: This work bridges the gap between cognitive engineering (mental models) and robotics. It enables robots to move beyond simple task execution to collaborative teamwork by understanding what their human partners know and don't know. This is foundational for Theory of Mind in Human-Robot Interaction (HRI).
• Limitations: The current system assumes the human has perfect perception once an object is in their field of view (binary "seen/not seen"). It relies on a static initial map and assumes the robot is notified of object additions/removals.
• Future Directions:
  • Developing more robust simulators with better APIs for navigation and object manipulation.
  • Creating datasets of actual human situation awareness responses to train better human models.
  • Personalizing perception models using individual gaze data.
  • Exploring continuous dynamic domains (e.g., "Clean Up" scenarios where objects move constantly).

In conclusion, the paper presents a robust framework for robots to "read" a human's mind regarding their environment, facilitating more fluent, proactive, and safe human-robot collaboration in complex real-world settings.