Technical Summary: MARVL

Problem Statement

Designing dense reward functions is critical for efficient Reinforcement Learning (RL) in robotic manipulation, yet manual engineering remains a bottleneck that limits scalability and automation. While Vision-Language Models (VLMs) offer a promising path to automate reward design by grounding rewards in natural language, existing approaches face three fundamental challenges:

1. Weak Spatial Grounding: Off-the-shelf VLMs are highly sensitive to camera perspectives. Embeddings often cluster by viewpoint rather than task identity, causing rewards to track incidental observational shifts instead of actual agent progress.
2. Lack of Progress Awareness: VLM-based similarity scores frequently exhibit excessive volatility and fail to reflect incremental task progress. They are easily distracted by irrelevant visual factors, leading to non-monotonic reward signals that hinder policy learning.
3. Semantic Misalignment: VLMs often misinterpret task instructions, resulting in unreliable reward signals where high similarity scores are triggered by spurious visual-text matches unrelated to the task goal.

These limitations suggest that current VLM-guided RL is less robust than initially believed, often failing to provide the stable feedback required for complex manipulation tasks.

Methodology: MARVL

To address these issues, the authors propose MARVL (Multi-stAge guidance for Robotic manipulation via Vision-Language models), a plug-and-play framework that refines VLM rewards through three key components:

1. Scene-View Decomposition (Fine-tuning)

To mitigate view-sensitivity, MARVL employs a dual-encoder-decoder architecture initialized from CLIP. It disentangles the visual representation into two codes:

• Scene Representation ($z_s$): Captures view-invariant, task-critical semantics (e.g., robot and object states).
• View Representation ($z_v$): Captures nuisance variations like camera pose.
  The model is fine-tuned using a reconstruction objective based on latent swapping (cross-compositional reconstruction) and consistency losses. This forces the scene encoder to isolate spatial semantics from perspective shifts, ensuring the reward signal is grounded in the physical scene rather than the camera angle.

2. Multi-Stage Decomposition and Task Direction Projection (TDP)

To bridge the gap between high-level instructions and low-level control, long-horizon tasks are decomposed into a sequence of subtasks with intermediate goal states.

• Task Direction Projection: Within each subtask, MARVL defines a "task direction" vector in the joint visual-semantic embedding space (the displacement between start and goal embeddings). A projection operator is applied to the current observation's embedding to preserve the component aligned with this task direction while attenuating orthogonal noise.
• Mechanism: This geometric filtering transforms volatile similarity scores into monotonic signals that correlate with task progress, effectively suppressing background variations and restoring signal-to-noise ratio without assuming linear physical dynamics.

3. Confidence-Thresholded Shaping (CTS)

To prevent agents from accumulating trivial positive scores due to generic semantic similarity (reward hacking), MARVL introduces a gating mechanism.

• Thresholding: A threshold $\theta$ is set based on the $m$-quantile of scores collected from random exploration (representing background noise).
• Soft Gating: A soft gating function zeros out low-magnitude noise while amplifying high-confidence matches. This ensures that only semantically relevant states receive significant reward signals, providing a decisive gradient for policy learning.

Key Contributions

1. Analysis of VLM Limitations: The paper systematically identifies and analyzes three critical failure modes in existing VLM-reward paradigms: weak spatial grounding, lack of progress sensitivity, and semantic misalignment.
2. MARVL Framework: The authors propose a novel framework integrating Scene-View Decomposition, Task Direction Projection, and Confidence-Thresholded Shaping to transform raw VLM similarities into stable, progress-sensitive reward signals.
3. Empirical Superiority: Extensive experiments demonstrate that MARVL consistently outperforms prior VLM-based methods (including FuRL, LIV, and Relay) and achieves performance parity with, or exceeds, hand-crafted "Oracle" dense rewards on sparse-reward manipulation tasks.

Experimental Results

The authors evaluated MARVL on the Meta-World benchmark (8 manipulation tasks) and the Panda-Gym benchmark for cross-domain generalization.

• Performance: MARVL achieved higher success rates and faster convergence than all VLM-reward baselines. On tasks like Button Press and Window Close, it matched the performance of the Oracle (which uses privileged, hand-crafted dense rewards). On exploration-heavy tasks like Push and Door Open, MARVL even surpassed the Oracle.
• Ablation Studies:
  • Removing Scene-View Decomposition degraded performance, confirming the necessity of view-invariant representations.
  • Removing Task Direction Projection or Multi-Stage Decomposition preserved final success but significantly slowed convergence, highlighting improved sample efficiency.
  • Removing Confidence-Thresholded Shaping led to training instability and performance collapse in certain tasks due to spurious signals.
• Generalization:
  • Cross-Domain: MARVL maintained high success rates on Panda-Gym (different robot embodiment and visual style) without target-domain adaptation, whereas non-fine-tuned variants failed.
  • Viewpoint Robustness: Performance remained consistent across diverse camera configurations.
  • Algorithm Agnosticism: The framework functioned effectively with different RL backbones (SAC and TD3).

Significance and Claims

The paper claims that the limitations of prior VLM-based rewards stem less from a lack of semantic knowledge in foundation models and more from a lack of structural alignment. By explicitly grounding representations and structuring the reward signal through multi-stage guidance and geometric projection, MARVL demonstrates that modest adaptation can substantially improve the practicality of VLM-based reinforcement learning.

The authors note that while MARVL is validated in simulation, extending it to strictly zero-shot, real-world deployment remains an objective for future work. Additionally, scaling the framework to longer-horizon and multi-object manipulation tasks is identified as a promising direction. The work suggests that VLMs can serve as powerful priors for robotic control, provided their representations are explicitly structured to handle spatial and semantic consistency.