Technical Summary: SlotVLA – Towards Modeling of Object-Relation Representations in Robotic Manipulation

Problem Statement

Recent advances in Vision-Language-Action (VLA) models have improved robotic manipulation by integrating language conditions with visual cues. However, current state-of-the-art architectures (e.g., OpenVLA, $\pi^0$) rely on dense visual embeddings generated by pretrained encoders (e.g., DINOv2, SigLIP). These encoders produce hundreds of visual tokens (typically 256–512) per frame, leading to prohibitive computational costs as token counts increase. Furthermore, dense embeddings often entangle object features with background cues and redundant information, limiting interpretability and obscuring task-relevant signals for the action decoder.

While object-centric learning in computer vision offers disentangled representations, directly applying it to robotics is suboptimal. Purely object-centric approaches often fail to capture essential relational cues, such as gripper-object interactions, which are critical for embodied manipulation. Existing methods either ignore these relations or rely on external annotations (bounding boxes/keypoints) that limit flexibility. There is a need for a framework that models both objects and their interactions using a compact, token-efficient representation that supports multitask reasoning.

Methodology: SlotVLA

The authors propose SlotVLA, a framework designed to transform dense visual embeddings into compact, structured object-relation representations. The approach consists of three main components:

1. Task-Aware Object-Centric Encoder

This module compresses dense visual tokens into a small set of object-centric slots ($S_t$) guided by language-based task filtering.

• Slot Attention with Temporal Consistency: The encoder employs slot attention (using a GRU) to map dense visual patches into learnable slots. A "slot carryover" mechanism initializes slots randomly at $t=0$ and propagates them from the previous timestep to ensure consistent object identity across frames.
• Task-Aware Slot Filtering: Not all detected objects are relevant to a specific task. A bidirectional cross-attention (BCA) mechanism computes relevance scores between the object slots and the task description embeddings. A Top-$k$ selection retains only the most relevant slots (typically 2–4), filtering out background and irrelevant objects.

2. Relation-Centric Encoder

To address the lack of relational modeling in standard object-centric approaches, SlotVLA introduces a Relation-Centric Encoder that generates relation tokens ($R_t$).

• Interaction Modeling: Learnable relation queries integrate information from both the dense visual patches ($V_t$) and the filtered object slots ($S_t$) via a Cross-Attention Block (CAB).
• Output: This process explicitly captures interactions, such as gripper-object relationships and object-background context, producing a set of relation tokens that complement the object slots.

3. Action Decoding

The final representation combines the task-aware object slots ($S_t$), relation tokens ($R_t$), language embeddings ($P$), and robot proprioception ($o_t$).

• LLM Integration: These multimodal inputs are fed into a Large Language Model (LLM) decoder equipped with LoRA (Low-Rank Adaptation) for action decoding.
• Token Efficiency: The total number of tokens ($N_S + N_R$) is significantly smaller than the dense input ($N$), enabling efficient downstream reasoning.

Training Strategy

The model is trained in two stages:

1. Stage 1 (Encoder Training): Supervises the Task-Aware Object-Centric Encoder using bounding box, objectness, and mask losses (Hungarian matching), along with a tracking loss for temporal consistency and a class-imbalanced loss for task relevance.
2. Stage 2 (Fine-tuning): The encoder parameters are frozen. The Relation-Centric Encoder and Action Decoder are fine-tuned jointly using cross-entropy loss on action logits.

Key Contributions

1. LIBERO+ Dataset

The authors introduce LIBERO+, a fine-grained benchmark extending the existing LIBERO suite.

• Annotations: Unlike the original LIBERO, LIBERO+ provides explicit object-level supervision, including 2D bounding boxes, pixel-level object masks, and instance-level temporal IDs across RGB-D modalities.
• Task Relevance: It includes task-specific naturalistic nouns to identify active, task-relevant objects, enabling the evaluation of object-relation reasoning.
• Subsets: The dataset comprises four subsets (LIBERO-Goal, LIBERO-Object, LIBERO-Spatial, LIBERO-Long) with varying object counts and layout complexities.

2. SlotVLA Framework

SlotVLA is the first VLA framework to combine object-centric slots with relation-centric tokens for robotic manipulation.

• It achieves a compact representation by filtering for task relevance and explicitly modeling interactions.
• It reduces the visual token count by an order of magnitude compared to dense baselines while maintaining competitive performance.

Experimental Results

Experiments were conducted on the LIBERO+ benchmark, comparing SlotVLA variants against the dense-token baseline OpenVLA.

• Token Efficiency: SlotVLA reduces token counts from 256 (OpenVLA) to approximately 4–28 tokens, resulting in a 3–4$\times$ reduction in GFLOPs.
• Performance on Simple Tasks: On subsets with simple layouts and few objects (LIBERO-Goal and LIBERO-Object), SlotVLA achieves competitive or superior success rates compared to OpenVLA.
  • LIBERO-Goal: OpenVLA (0.77) vs. SlotVLA-ORC (0.86).
  • LIBERO-Object: OpenVLA (0.70) vs. SlotVLA-ORC (0.91).
• Performance on Complex Tasks: On subsets with complex spatial layouts and high object counts (LIBERO-Spatial and LIBERO-Long), performance degrades.
  • LIBERO-Long: OpenVLA (0.56) vs. SlotVLA-ORC (0.31). The authors attribute this to the limitation of having too few slots (e.g., 4) to represent scenes with up to 29 objects, where dense baselines capture more global context.
• Ablation Studies:
  • Task-Aware Filtering: Filtering irrelevant slots is crucial; without it, performance drops significantly, especially in complex environments.
  • Temporal Consistency: Removing temporal consistency causes a sharp performance drop (e.g., ORC drops from 0.86 to 0.40 on LIBERO-Goal), highlighting the necessity of stable slot identities.
  • Slot Count: Increasing the number of slots beyond the optimal small number (4) often introduces noise or dilutes relational grounding, rather than improving performance.

Significance and Claims

The paper claims that object-relation-centric tokenization offers a promising balance of efficiency, interpretability, and effectiveness for multitask robotic manipulation.

• Efficiency: By drastically reducing the number of visual tokens, SlotVLA enables robotic control under limited computational budgets.
• Interpretability: The slot-based approach disentangles entities and filters for task relevance, making the model's reasoning process more transparent compared to dense, entangled embeddings.
• Limitations: The authors modestly acknowledge that while effective for simple layouts, the approach struggles to scale to highly cluttered, long-horizon tasks where the number of objects exceeds the slot capacity. They note that the reliance on supervised bounding boxes and masks for training may limit immediate applicability in real-world settings without such annotations.

In conclusion, the authors posit that LIBERO+ and SlotVLA provide a foundational step toward structured, efficient, and explainable embodied AI, shifting the paradigm from dense perception to compact, relation-aware reasoning.