Technical Summary: SERF – Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation

1. Problem Formulation

Long-horizon mobile manipulation in large, dynamic environments requires a robot to continually reason about three coupled questions: its current localization, changes in the environment, and its progress toward a task. While Vision-Language-Action (VLA) models have shown promise in mapping observations to actions, they typically lack explicit mechanisms for maintaining information beyond the current observation window. This limitation is critical for mobile manipulation, where decisions depend on long-term context that evolves as the robot interacts with the world.

Existing approaches fall into two categories with distinct limitations:

1. Modular systems using structured scene representations (e.g., 3D scene graphs) often abstract away dense geometry and robot-environment interaction cues necessary for continuous visuomotor control.
2. Policy-embedded memory (e.g., history tokens, retrieval buffers) often leaves spatial grounding implicit, forcing the policy to infer relevance. Furthermore, existing feature-map approaches typically represent only static environments, failing to update as the robot interacts with objects, and often omit the robot body itself from the memory representation.

The authors propose a solution that addresses these gaps by creating a persistent, spatially grounded memory that evolves with robot interaction and explicitly represents the spatial relationships between the robot and its environment.

2. Methodology

The core contribution is the Spatiotemporal Environment and Robot Feature (SERF) map, a 4D feature mapping formulation that represents both the environment and the articulated robot body as neural points in a shared latent space.

A. Spatiotemporal Feature Mapping (Problem 1)

The map $m_\tau$ is parameterized using neural points in 3D space. Each point carries a learnable latent feature vector and an instance label.

• Representation: The map consists of two sets of points:
  • Environment Points ($P^e_\tau$): 3D points lifted from RGB-D observations using a Vision Foundation Model (VFM) like DINOv3. These points are voxelized and assigned instance labels.
  • Robot Points ($P^r_\tau$): Surface points sampled from the robot's kinematic model (e.g., URDF). Their positions are updated via forward kinematics based on the robot's proprioceptive state.
• Shared Latent Space: Both sets of points are decoded by a shared MLP decoder to reconstruct VFM embeddings. This encourages the robot and environment features to inhabit a common latent space.
• Training Objective: The system is trained offline using a reconstruction loss (cosine similarity) to match decoded features with VFM embeddings. Additionally, contrastive losses ($L_{inter}$ and $L_{intra}$) are used to align features within the same category/part and separate different categories/parts, inducing semantic structure.
• Online Updates: During execution, the latent features and decoder remain fixed. Only point coordinates are updated:
  • Environment: Object motion is tracked via 3D keypoint correspondences between consecutive frames, estimating an object-level SE(3) transform to rigidly move the associated neural points.
  • Robot: Point positions are updated using forward kinematics from the current joint states.

B. Map-Conditioned VLA Policy (Problem 2)

The authors design a policy $\pi_\phi$ that conditions action predictions on the SERF map.

• Map Tokenizer: A subset of the neural points is filtered (retaining robot body and task-relevant objects) and encoded using a Point Transformer backbone.
• Token Extraction: Eight parallel heads extract map tokens across multiple reference frames and spatial scales:
  1. Robot-base tokens: Three tokens at increasing radii (1m, 2m, 4m) for local context.
  2. End-effector tokens: Two tokens around the grippers for grasp reasoning.
  3. Robot-only token: Summarizes the robot's configuration.
  4. Environment-only token: Summarizes the current environment state.
  5. Global token: Aggregates all points for scene-level reasoning.
• Policy Architecture: The map tokens are projected and concatenated with image features, proprioceptive state tokens, and task embeddings. This combined input is fed into a VLA model (based on $\pi_{0.5}$) trained via conditional flow matching to predict action chunks. The VLM backbone and vision encoder are frozen, with LoRA layers inserted into the action expert and map tokenizer for fine-tuning.

3. Key Contributions

1. SERF Map: A spatiotemporal feature map representing the robot body and evolving environment in a shared latent space, enabling persistent memory that updates online.
2. Map-Conditioned Policy: A VLA policy design that utilizes map tokens from multiple reference frames (robot-base, end-effector, global) to provide both local and global spatial context for action prediction.
3. Empirical Validation: Demonstration that the SERF policy outperforms image-only baselines on long-horizon mobile manipulation tasks, specifically improving task progress, trajectory efficiency, robustness to scene shifts, and failure recovery.

4. Experimental Results

The method was evaluated on BEHAVIOR-1K, a benchmark for household bimanual mobile manipulation involving navigation and multi-object rearrangement.

• Task Progress: The full SERF policy achieved the highest task progress across three evaluated tasks (Collecting Toys, Putting Shoes on Rack, Assembling Gift Baskets), improving average progress from 44.0% (image-only baseline) to 58.7%.
• Trajectory Efficiency: SERF followed more direct trajectories and reached subgoals faster compared to image-only policies, which often stalled when objects left the field of view.
• Scene-Configuration Generalization: In out-of-distribution (OOD) tests involving moved goals, additional objects, and unvisited regions, SERF significantly outperformed the image-only baseline. For example, in unvisited regions, task progress increased from 28.0% to 51.0%.
• Failure Recovery: In scenarios where objects were dropped during transport, SERF successfully re-localized and re-grasped the objects with a 95% success rate (vs. 65% for the baseline) and reduced average recovery time from 24.3s to 20.5s.
• Ablation: The study confirmed that temporal map updates (SERF vs. static SBP) and the inclusion of robot neural points (SERF vs. SERF-env) both contributed positively to performance.

5. Significance and Claims

The paper claims that long-horizon mobile manipulation requires coherent spatiotemporal reasoning that current VLA models lack. By introducing a shared latent representation of the robot and environment, SERF enables two forms of reasoning:

1. Allocentric Reasoning: Maintaining persistent spatial memory to track object and goal locations globally.
2. Egocentric Reasoning: Explicitly embedding the robot body to support decisions based on distance, orientation, and reachability.

The authors state that SERF provides a structured context for neural policies, allowing them to recover from failures and generalize to new scene configurations more effectively than image-only approaches.

6. Limitations

The authors acknowledge several limitations:

• Simulation Dependence: The current implementation relies on privileged instance labels from the simulator and a prior map learned before execution. Real-world deployment would require feed-forward encoders and segmentation foundation models (e.g., SAM 2).
• Rigid Motion Assumption: Map updates assume object-level rigid motion (SE(3)), limiting the system's ability to handle articulated or deformable objects.
• Tokenization: The current map tokenizer uses manually specified spatial subsets rather than task-conditioned semantic separation.
• Real-World Validation: Experiments are currently limited to simulation; real-world transfer remains to be assessed.