Technical Summary: WorldLines and ObsMem

Problem Statement

Existing benchmarks for embodied agents and long-term memory are fundamentally misaligned with the requirements of real-world, long-horizon household assistance. Current long-term memory benchmarks (e.g., LoCoMo, LongMemEval) focus primarily on text-centric retrieval and dialogue consistency, often decoupling memory from physical state transitions, action feedback, and executable constraints. Conversely, embodied benchmarks (e.g., ALFWorld, Habitat 3.0, PARTNR) typically evaluate short-horizon tasks where world states do not persist across interactions or are fully observable.

This gap leaves a critical evaluation void: Can embodied agents maintain a persistent, partially observable world state over extended periods and utilize that state for downstream question answering and planning? Real-world service requests often depend on evolving routines, object states, device settings, and historical events that span multiple days. Agents must distinguish between direct observations, reported information, and unobserved state changes, a capability that flat text-retrieval systems struggle to support.

Methodology

1. WorldLines Benchmark

The authors introduce WorldLines, a project-driven benchmark designed to evaluate long-horizon stateful embodied agents in dynamic, partially observable environments.

• Construction Pipeline: WorldLines generates extended household traces by simulating "long-term projects" (e.g., meal preparation, device coordination) across multiple days. The pipeline involves:
  • World Grounding: Utilizing Habitat/HSSD scenes with curated interactive objects and controllable devices.
  • Project Planning: Defining multi-day life processes with specific participants, spaces, and temporal constraints.
  • Closed-Loop Simulation: Generating cross-day event streams (dialogue, human activity, robot actions, device operations) where a deterministic executor validates affordances and updates structured state trajectories.
  • Carry-Forward Memory: Extracting notes on state changes, preferences, and unresolved plans to ensure continuity between days.
• Task Families: The benchmark converts these traces into evidence-linked samples for two tasks:
  • Memory QA: Evaluates the recovery of historical events, state changes (including overwritten states), preferences, and routines.
  • Embodied Task Planning: Tests the ability to generate state-consistent plans or next-step decisions based on visible history and executable constraints.
• Evaluation Protocol: Each sample includes a context cutoff ($c_i$). The agent observes only the history prior to the cutoff ($H_{<c_i}$). Ground truth states and evidence chains are used solely for evaluation to prevent information leakage.

2. ObsMem: Observer-Grounded Memory Framework

To address the limitations of flat text memories, the authors propose ObsMem, a memory framework that organizes information into typed views based on observation provenance and semantic role.

• Observer-Grounded Ingestion: Events are gated by an "observer gate" to distinguish between what the robot directly observed ($e^{obs}$) and what was merely reported by others ($e^{rep}$). This distinction is preserved throughout the memory lifecycle.
• Typed Memory Tracks:
  • Event Track: An append-only log of historical evidence (what happened).
  • State Track: Maintains both a current snapshot and a history of state facts (e.g., object locations), allowing the system to track overwritten states.
  • Belief Track: Records the robot's epistemic confidence (fresh, stale, uncertain, contradicted) for each fact, accounting for partial observability and intervening events.
  • Commitment Track: Stores future obligations, promises, and schedules.
• Query-Time Retrieval: Instead of a single similarity search, ObsMem composes evidence based on the query intent. It routes queries to specific views (e.g., State Track for "where is X now?", Event Track for "who said X?") and synthesizes a typed evidence bundle that considers state, belief, and historical support simultaneously.

Key Contributions

1. WorldLines Benchmark: A novel benchmark for long-horizon stateful embodied agents that integrates Memory QA and Embodied Task Planning within dynamic, partially observable household environments.
2. Project-Driven Trace Generation: A pipeline that transforms grounded worlds and long-term activity threads into evidence-linked evaluation samples, ensuring that world states persist and evolve across interactions.
3. ObsMem Framework: An observer-grounded memory architecture that separates event evidence, structured state trails, and belief records, enabling robust decision-making under partial observability.

Experimental Results

The authors evaluated ObsMem against several baselines (Mem0, A-mem, MemoryOS, GraphMem) on 310 Memory QA samples.

• Overall Performance: ObsMem achieved the highest Judge score (0.713) and Perfect Rate (69%), outperforming the strongest baseline (A-mem) by 0.138 in Judge score.
• Evidence Recovery: While baselines achieved high Session Any@5 (retrieving the correct coarse temporal context), they struggled with Event R@5 (identifying specific state-changing events). ObsMem improved Event R@5 by 0.159 over Mem0, demonstrating superior capability in recovering precise embodied evidence.
• State Reasoning: ObsMem showed significant gains in StateMultiHop and StateSingleHop questions, which require tracking mutable states through event transitions. The StateMH-E metric (event-level recall for multi-hop state questions) showed a gap of 0.188 over the strongest baseline.
• Ablation Studies: Removing the Belief view, State view, or Episode consolidation resulted in performance drops, confirming that the separation of state, belief, and evidence is critical for handling partial observability and state updates.

Significance and Claims

The paper claims that WorldLines reveals persistent challenges in existing systems, specifically regarding partial observability, overwritten world states, and the translation of long-term memory into embodied plans.

The authors assert that ObsMem offers a stronger reference architecture for this setting by treating memory as an online process that turns interaction streams into typed evidence. The core significance lies in the demonstration that long-horizon embodied agents require more than flat text-retrieval; they need mechanisms to distinguish observed from reported information, current from historical states, and known from uncertain facts. ObsMem provides this structure, enabling both evidence-grounded question answering and state-aware embodied planning. The work does not claim to solve all embodied AI challenges but provides a specific benchmark and framework to evaluate and improve long-term state maintenance in dynamic household settings.