RoboStream: Weaving Spatio-Temporal Reasoning with Memory in Vision-Language Models for Robotics

1. Problem Statement

Current Vision-Language Models (VLMs) have shown promise in robotic manipulation for short-horizon tasks but struggle significantly with long-horizon planning. The paper identifies two critical representation deficits in existing VLM-based planners:

• Ungrounded Spatial Perception: Current systems treat each step as an isolated observation-to-action mapping. They rely on raw pixels to infer scene geometry at every decision step without persistent geometric anchoring. This leads to "spatial hallucinations" where small perceptual errors compound over time, causing precondition violations and cascading failures.
• Untracked Causal History: Existing methods lack a record of how actions modify the environment. When objects are occluded or displaced, planners lose track of them because they cannot reason about state transitions that occurred in previous steps. They fail to maintain "object permanence" under occlusion.

The core challenge is that humans maintain a persistent mental model integrating spatial relations and causal consequences, whereas current robots must "reinvent" the world state from scratch at every step.

2. Methodology: RoboStream

RoboStream is a training-free framework designed to weave spatio-temporal reasoning and persistent memory directly into the VLM planning loop. It operates through three hierarchical stages:

A. Spatio-Temporal Grounding (STF-Tokens)

To solve the spatial grounding issue, RoboStream introduces Spatio-Temporal Fusion Tokens (STF-Tokens).

• Mechanism: Instead of feeding raw image patches to the VLM, the system extracts per-object visual evidence and binds it to explicit 3D geometric attributes.
• Components: For each object, it computes:
  • Visual Evidence: Filtered patch tokens from the RGB image.
  • 3D Centroid: The geometric center derived from the point cloud.
  • Gaussian Shape: A vector representing the object's spatial extent (mean, standard deviation, and min/max bounds along x, y, z axes) to capture shape more robustly than bounding boxes.
• Function: These elements are serialized into a single token sequence. This shifts the VLM's reasoning from pixel-level reconstruction to structured, object-instance references, suppressing spatial hallucinations.

B. Causal Memory Tracking (CSTG)

To solve the causal history issue, RoboStream constructs a 4D Causal Spatio-Temporal Graph (CSTG).

• Structure: The graph maintains persistent object identities and a sliding-window buffer of STF-Tokens.
• Causal Log: It records semantic events (e.g., "object moved," "object occluded," "subtask completed") alongside their timestamps and causal sources.
• Function: This allows the system to track objects even when they are visually occluded. If an object is hidden, the planner can infer its state based on the last known pose and the causal chain of actions, preventing "visual amnesia."

C. VLM-Based Reasoning and Planning

The framework integrates these components into the planning loop:

1. Context Assembly: The VLM receives the current RGB-D observation, the goal specification, and the structured CSTG (containing current states and historical causal logs).
2. Chain-of-Thought Verification: The VLM performs a logical check: reviewing historical states, verifying action preconditions against the 3D geometry of STF-Tokens, and resolving conflicts caused by occlusion.
3. Action Generation: The VLM outputs a semantic action directive (e.g., "move the blue cube").
4. Deterministic Instantiation: A separate module resolves the semantic directive into precise 6-DoF coordinates by querying the geometry stored in the STF-Tokens, ensuring physical precision without requiring the VLM to output raw coordinates directly.

3. Key Contributions

1. RoboStream Framework: A novel, training-free approach that eliminates the need for fine-tuning VLMs on robotic data while significantly improving long-horizon performance.
2. Dual Mechanism Design:
   • STF-Tokens: Convert pixel-level inference into deterministic spatial references by binding visual appearance to 3D geometry.
   • CSTG: Enables causal inference of occluded states and prevents error accumulation by maintaining a persistent record of action-triggered state transitions.
3. Decoupled Planning-Execution: The architecture separates high-level semantic reasoning (VLM) from low-level coordinate extraction (deterministic geometry resolution), balancing generalization with precision.

4. Experimental Results

RoboStream was evaluated across five benchmarks spanning simulation and real-world settings:

• Real-World Manipulation (Franka Robot):
  • Tasks: Block building, disassembly, and "Hide and Restore" (full occlusion).
  • Performance: On hard block-building tasks, RoboStream-235B achieved 44.4% success, compared to 11.1% for SoFar and VoxPoser.
  • Occlusion: In "Hide and Restore" tasks, RoboStream-235B achieved 88.9% success, while baselines failed completely (0%).
• Long-Horizon Simulation (RLBench):
  • Performance: Achieved a 90.5% average success rate on long-horizon tasks with the 235B model, significantly outperforming SoFar (28.0%) and VoxPoser (26.5%).
  • Scaling: Performance scaled with model capacity, but even the 8B variant outperformed larger baselines, proving the efficacy of the memory/grounding architecture.
• Zero-Shot Generalization (SIMPLER):
  • Outperformed state-of-the-art baselines (including RT-2-X and SoFar) on cross-embodiment tasks (Google Robot and WidowX) without any fine-tuning.
• Spatial Reasoning (6-DoF Benchmarks):
  • Achieved state-of-the-art results on Open6DOR V2 and 6-DoF SpatialBench, demonstrating superior ability to reason about position and orientation compared to image-text alignment baselines.

Ablation Study: Removing the CSTG caused success rates to collapse from 90.5% to 14.5%, highlighting that causal memory is the foundational driver for long-horizon tasks. Removing STF-Tokens dropped performance to 79.5%, showing that geometric grounding is essential for physical precision.

5. Significance

This work addresses a fundamental bottleneck in embodied AI: the inability of current VLMs to maintain a coherent, persistent model of the world over time.

• Paradigm Shift: It moves beyond "reactive" planning (step-by-step pixel matching) to "proactive" planning with persistent state tracking.
• Efficiency: By being training-free, it offers a scalable solution that leverages existing large VLMs without the massive data and compute costs of fine-tuning.
• Reliability: It demonstrates that spatio-temporal reasoning and causal memory are more critical for robust long-horizon manipulation than simply increasing model parameter scale. This paves the way for reliable robots capable of complex, multi-step tasks in dynamic, open-world environments.