Technical Summary: Automating Potential-based Reward Shaping with Vision Language Model Guidance

Problem Statement

Reinforcement Learning (RL) agents face significant challenges in environments with sparse rewards, where feedback is only provided upon task completion. This lack of intermediate guidance hinders exploration and makes it difficult to attribute success to specific parts of a trajectory. While Reward Shaping (adding auxiliary signals to guide exploration) is a common solution, arbitrary shaping often leads to reward hacking, where agents exploit the auxiliary signals to maximize reward without solving the intended task.

Potential-Based Reward Shaping (PBRS) offers a theoretically sound alternative by adding the difference of a potential function $\Phi(s)$ between successive states to the reward. PBRS guarantees the preservation of the optimal policy set of the original task. However, defining an effective potential function typically requires extensive expert knowledge and manual engineering, limiting its scalability.

Recent approaches using Vision Language Models (VLMs) to learn reward functions directly (e.g., RL-VLM-F) reduce the need for human design but rely on the VLM's ability to perfectly approximate the true reward function. This imposes a strict requirement on label accuracy; errors in VLM output can lead to sub-optimal policies. Furthermore, querying large, state-of-the-art VLMs repeatedly during training is computationally prohibitive.

Methodology: VLM-PBRS

The authors propose VLM-PBRS, a framework that automates the construction of a potential function for PBRS using VLM-generated preference labels. The core innovation lies in leveraging the policy invariance property of PBRS to relax the accuracy requirements on the VLM, allowing the use of smaller, more efficient models.

Core Components

1. Preference-Based Learning Loop:

   • The system operates in an iterative loop involving policy improvement, data collection, and VLM querying.
   • Data Collection: The agent interacts with the environment, collecting trajectories and image observations.
   • VLM Querying: Pairs of images ($\sigma_i, \sigma_j$) are sampled from the buffer. A prompt is constructed containing the image pair and a natural language description of the goal. The VLM is queried to determine which image better achieves the goal.
   • Prompt Structure: The prompt asks the VLM to identify common elements, describe differences relative to the goal, and output a machine-readable label ($y \in \{-1, 0, 1\}$) indicating preference.
   • Label Extraction: Unlike prior works (e.g., RL-VLM-F) that use a second LLM call to extract labels from verbose VLM outputs, VLM-PBRS extracts the label directly from a single VLM call to improve computational efficiency.

2. Potential Function Construction:

   • A preference model $r_\psi$ is trained on the collected VLM labels using a loss function (Cross-Entropy or Mean Absolute Error).
   • The trained model is transformed into a potential function $\Phi(s)$:
     $$\Phi(s) = \begin{cases} 0 & \text{if } s \text{ is terminal} \\ \lambda \sigma(r_\psi(s)) & \text{otherwise} \end{cases}$$
     where $\sigma$ is a sigmoid activation and $\lambda$ is a scaling factor (set to 0.9) to ensure the shaped reward remains positive and bounded below the goal reward.

3. Reward Shaping:

   • The shaped reward $R'$ is computed as $R'(s, a, s') = R(s, a, s') + (\gamma \Phi(s') - \Phi(s))$.
   • This shaped reward guides the RL agent (using SAC) during exploration. The replay buffer is recomputed with updated shaping rewards as the potential function improves.

Key Design Choices

• Small VLMs: The framework utilizes smaller, open-weight VLMs (e.g., Ovis2-16B, Qwen3-VL-8B) which are computationally efficient. The authors argue that while these models may produce less accurate labels than larger models, the PBRS framework is robust to such noise.
• Single-Step Labeling: By avoiding a second LLM call for label generation, the method significantly reduces computational overhead.

Key Contributions

1. First Application of VLM Preference Learning for PBRS: The paper introduces the first framework to synthesize a potential function for PBRS directly from VLM preference labels, requiring only a textual goal description and visual observations.
2. Principled Use of Small VLMs: The authors establish that the policy invariance guarantee of PBRS relaxes the strict label accuracy requirements of direct reward modeling. This enables the use of smaller, computationally efficient VLMs without compromising the optimality of the learned policy, even if the labels are noisy.
3. Empirical Validation: The method is validated in Meta-World and Franka Kitchen environments, demonstrating improved sample efficiency and robustness to reward hacking compared to sparse rewards and naive VLM reward learning.

Results

The authors evaluated VLM-PBRS against four baselines: Sparse Rewards, Human-Designed Dense Rewards, and RL-VLM-F (direct reward learning).

• Sample Efficiency: VLM-PBRS consistently accelerated learning compared to the sparse reward baseline across all tested tasks (e.g., button-press, window-open, microwave, light-switch).
• Comparison to RL-VLM-F: While RL-VLM-F improved over sparse rewards in some tasks, it often converged to sub-optimal policies in others (e.g., button-press, top-burner) due to VLM labeling errors creating incorrect incentives. In contrast, VLM-PBRS maintained optimal policy convergence even with noisy labels, as the PBRS property prevents the agent from being permanently misled.
• Performance vs. Human Design: In the button-press task, VLM-PBRS with a VLM accuracy of ~59% outperformed the human-designed dense reward baseline, which suffered from reward hacking.
• Label Accuracy Sensitivity:
  • Experiments with an oracle labeler showed that sample efficiency improvements correlate with label accuracy.
  • Even with accuracy levels below chance (0.0), VLM-PBRS did not degrade performance below the sparse baseline, relying on the underlying RL exploration.
  • In tasks where VLM accuracy was near random chance (e.g., drawer-open), VLM-PBRS showed slight improvements over sparse rewards but failed to match human-designed dense rewards.
• Ablation Studies:
  • Batch Size: Using 20 labels per VLM query batch was found to be sufficient, balancing performance and computational cost.
  • Loss Function: Cross-Entropy and Mean Absolute Error (MAE) losses performed similarly; Cross-Entropy was selected as the default.

Significance and Claims

The paper claims that VLM-PBRS provides a practical, scalable pathway for automating reward shaping in resource-constrained settings. By decoupling the need for perfect VLM reasoning from the guarantee of policy optimality, the method allows for the use of smaller, cheaper VLMs.

The authors emphasize that their approach:

• Preserves Optimality: Unlike direct reward learning, it does not risk learning sub-optimal policies due to VLM errors.
• Reduces Engineering Burden: It eliminates the need for expert-designed potential functions or dense reward terms, requiring only a high-level text description.
• Improves Robustness: It is inherently resilient to reward hacking, a common failure mode in arbitrary reward shaping.

The work suggests that as multi-modal foundation models improve, the generality and benefits of VLM-PBRS will naturally increase, offering a viable alternative to hand-crafted reward engineering for complex robotic tasks.