💻 computer science

Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action

The paper introduces SOMA, a spatial memory framework that enhances Vision-Language-Action models' ability to perform out-of-vision manipulation by constructing, refining, and retrieving persistent 3D spatial representations from multi-view observations, thereby enabling robust reasoning and faster task execution even when target objects are initially invisible.

Original authors: Pengteng Li, Weiyu Guo, He Zhang, Tiefu Cai, Xiao He, Yandong Guo, Hui Xiong

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Pengteng Li, Weiyu Guo, He Zhang, Tiefu Cai, Xiao He, Yandong Guo, Hui Xiong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to pick up a specific red cup from a table, but every time you reach for it, you accidentally knock it out of sight. A standard robot camera works like a human who only trusts what they can see right now. If the cup disappears behind a bowl, the robot panics, stops, and says, "I can't find it!" It has no memory of where the cup was a second ago.

This paper introduces SOMA (Spatial Memory for Out-of-Vision Manipulation), a new "brain" for robots that solves this problem by giving them a persistent mental map of the room.

Here is how SOMA works, broken down into simple concepts:

1. The Problem: The "Goldfish Memory" Robot

Most current robots are like goldfish with a 7-second memory. They only know about objects currently in their camera's view.

The Scenario: A human asks a robot, "Pick up the pink cup and put it in the basket."
The Failure: If the cup is hidden behind a box, or if the robot moves its head and the cup goes out of frame, the robot freezes. It doesn't know the cup still exists; it just sees empty space. It gets "stuck" because it lacks a global understanding of the room.

2. The Solution: The Robot's "Mental Map"

SOMA gives the robot a spatial memory, similar to how you remember where you left your keys even if you can't see them right now. It does this in three steps:

Step A: The "Sweep" (Building the Map)

Before the robot tries to grab anything, if it can't see the target, it uses its head-mounted camera to scan the room like a security guard doing a 360-degree turn.

Analogy: Imagine you are looking for a lost coin in a dark room. You don't just stare at one spot; you sweep your flashlight around the whole room to see where everything is.
What happens: The robot takes these different angles and stitches them together into a single, unified "mental map" that includes both what the objects are (a pink cup) and where they are (3D coordinates).

Step B: The "Update" (Keeping the Map Fresh)

As the robot moves and the scene changes (objects move, get covered, or appear), SOMA doesn't just throw away the old map. It updates it.

Analogy: Think of a weather map. If a storm moves from the north to the south, you don't throw away the map; you just update the storm's location.
What happens: If the robot sees the cup move, it adjusts the cup's location on the mental map. If the cup gets hidden behind a box, the map remembers, "The cup is behind the box," so the robot doesn't forget it exists.

Step C: The "Search" (Using the Map to Act)

When the robot needs to grab the cup, it doesn't blindly wander around looking for it. It queries its memory.

Analogy: Instead of wandering a house looking for your phone, you remember, "I last saw it on the kitchen counter," and you walk straight there.
What happens: The robot asks its memory, "Where is the pink cup?" The memory says, "It's to the left, behind the basket." The robot then moves its head directly to that spot, grabs the cup, and places it in the basket—often in just one try.

3. The Results: From "Stuck" to "Smooth"

The researchers tested this on real robots with real tasks (like picking up cups and moving them to baskets).

Without SOMA: The robot would often get stuck, spin its head around randomly trying to find the object, and fail to grab it.
With SOMA: The robot was much faster. It knew exactly where to look, moved its head less, and grabbed the object almost immediately (like a "one-shot" grab). It succeeded even when the object was completely invisible at the start.

Summary

Think of SOMA as giving a robot a GPS and a diary combined.

Standard Robots only have eyes; if they can't see it, it doesn't exist.
SOMA Robots have eyes and a memory. They can "see" objects that are currently hidden because they remember where they are in the 3D space of the room.

This allows robots to perform complex tasks in messy, real-world environments where objects are constantly moving in and out of sight, making them much more reliable helpers.

Technical Summary: SOMA – Spatial Memory for Out-of-Vision Manipulation in Vision-Language-Action

1. Problem Statement

Current Vision-Language-Action (VLA) models typically operate under a "view-bound" assumption, where task-relevant objects are presumed to be visible within the robot's current camera field of view at the moment of decision-making. Most existing VLAs rely on static camera setups or single viewpoints, leading to brittle, reactive behaviors when targets become occluded or move outside the camera's frustum. While some recent approaches attempt to compensate for out-of-view perception using internal spatial reasoning or learned priors, these methods are inherently brittle; they lack persistent perceptual grounding and fail rapidly once targets are fully outside the observable field. This limitation prevents robust manipulation in scenarios requiring multi-step planning, dual-arm coordination, or interaction with objects that are temporarily invisible.

2. Methodology: The SOMA Framework

The authors propose SOMA (Spatial memory framework for Out-of-Vision Manipulation), a VLA framework designed to enable reasoning and action beyond the current visual frustum. SOMA integrates a movable head-mounted camera with a global, persistent spatial memory. The framework operates through three core components:

A. Spatial Memory Construction

Before manipulation begins, if the target object is not localized in the current view, the robot initiates an active scanning phase using a movable head camera.

Process: The system aggregates angular-wise observations from a scanning video sequence into a unified spatial–semantic representation.
Pipeline: It employs a multi-level visual abstraction pipeline combining:
1. Geometry Priors: Using VGGT to estimate camera poses and coarse 3D scene geometry.
2. Semantic Perception: Using YOLO for object localization and categorization.
3. Visual Features: Using DINOv3 for high-dimensional appearance embeddings.
Fusion: Detected 2D bounding boxes are lifted into a global 3D coordinate system. Instances are associated and fused across views using appearance-geometry similarity to resolve duplicates and ambiguities, forming an overview scene memory ( $M_0$ ) consisting of unified triplets (appearance, 3D spatial encoding, category).

B. Dynamic Memory Refinement

During manipulation, the scene evolves as objects move or become occluded.

Mechanism: New observations from the head camera are processed and fused into the existing memory ( $M_0$ ) to produce a refined current memory ( $\hat{M}_t$ ).
Adaptive Fusion: The system performs instance-level association between new observations and existing memory entries. It uses a similarity-aware fusion mechanism that computes a semantic similarity score and a dynamic fusion score. These scores determine an adaptive update coefficient, allowing the memory to balance new visual evidence with historical data via a temporal exponential moving average (EMA). This ensures global consistency while accommodating scene changes.

C. Contextual Memory Retrieval

To guide action selection, the framework retrieves relevant spatial cues based on the user instruction.

Alignment: Instruction-grounded vision–language embeddings from the VLM serve as queries, while the refined spatial memory acts as the key-value bank.
Integration: A cross-attention module selectively activates instruction-relevant regions within the memory. These retrieved, memory-enhanced features are injected into the DiT (Diffusion Transformer) blocks as a spatial–semantic prior, modulating token interactions to generate precise, grounded manipulation actions even when the target is temporarily out of view.

3. Key Contributions

SOMA Framework: The introduction of a spatial memory framework for VLA models that shifts the paradigm from reactive, view-bound exploration to memory-guided manipulation, enabling operation beyond the current camera view.
Memory-Centric Architecture: A novel architecture comprising Spatial Memory Construction, Dynamic Memory Refinement, and Contextual Memory Retrieval. This design maintains a persistent, object-centric spatial representation that supports cross-view consistency and instruction-aware action selection under limited viewpoint coverage.
Empirical Validation: Real-world experiments demonstrating that SOMA not only improves task success rates but also induces qualitatively different manipulation behaviors, such as faster target localization, reduced head exploration, and near one-shot grasping under partial observability.

4. Experimental Results

Real-World Out-of-Vision (OOV) Tasks

The authors evaluated SOMA on five self-designed real-world OOV pick-and-place tasks, ranging from single-object manipulation to sequential dual-object and dual-arm coordination scenarios where targets were initially invisible.

Performance: SOMA achieved the highest success rates across all five tasks compared to baselines including GR00T-N1.5, StarVLA, and SpatialVLA.
Behavioral Metrics: Compared to GR00T-N1.5, SOMA demonstrated significant improvements in:
- First-Fixation Time: Reduced by 40–59%.
- Head Search Path Length: Reduced by 40–57%.
- Viewpoint Correction Count: Reduced by 42–57%.
- Grasp Attempt Count: Reduced by 40–57%, indicating near one-shot grasping behavior.
- Time-to-Grasp: Reduced by 41–60%.
Ablation: Experiments confirmed that persistent spatial memory is the dominant factor in performance gains, outperforming strategies that rely solely on scanning without memory or memory without refinement.

Simulation Benchmarks

RoboCasa Tabletop GR1: SOMA achieved a state-of-the-art average success rate of 52.0% (with 300 demos) and 49.3% (full data), outperforming Diffusion Policy, GR00T-N1.5, and StarVLA. Notably, SOMA showed strong sample efficiency, achieving high success rates with only 30 demonstrations per task.
SimplerEnv: On the Fractal suite, SOMA achieved the best success rate (63.2% in Visual Matching and 52.5% in Variant Aggregation), surpassing other SOTA models like RoboVLM and OpenVLA.

5. Significance and Claims

The paper claims that SOMA fundamentally addresses the limitation of view-bound perception in current VLA paradigms. By integrating multi-view scene exploration with a persistent, queryable spatial–semantic memory, SOMA enables robots to retain and reuse spatial evidence beyond instantaneous visual input.

The authors emphasize that this memory-centric design allows for robust reasoning and manipulation when task-relevant objects are temporarily outside the field of view, without relying on brittle, view-dependent behaviors. Furthermore, the framework's effectiveness extends to conventional fully observable settings (as shown in RoboCasa and SimplerEnv), suggesting that structured spatial memory provides a beneficial inductive bias for general manipulation tasks, improving sample efficiency and stability even when objects are visible. The work highlights that spatial reasoning alone, without grounded perceptual evidence retained in memory, is insufficient for robust manipulation in dynamic or partially observable environments.