🤖 machine learning

Prompt-Driven Exploration

The paper introduces Prompt-Driven Exploration (PDE), a reinforcement learning strategy that leverages a vision-language model to iteratively refine natural language prompts based on rollout analysis, thereby enabling effective global exploration and learning from sparse rewards without relying on action-space noise.

Original authors: Sunshine Jiang, John Marangola, David Zhang, Raghuram Kowdeed, Ruiyang Luo, Nitish Dashora, Richard Li, Pulkit Agrawal, Zhang-Wei Hong

Published 2026-07-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Sunshine Jiang, John Marangola, David Zhang, Raghuram Kowdeed, Ruiyang Luo, Nitish Dashora, Richard Li, Pulkit Agrawal, Zhang-Wei Hong

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to do a tricky job, like closing a microwave door or stacking blocks. You give the robot a simple instruction, like "close the microwave." But the robot is confused. Instead of pushing the door, it grabs a nearby mug and tries to shove that into the microwave. It fails, over and over again.

In the world of robot learning, this is a common headache. Usually, when a robot gets stuck, scientists try to fix it by shaking things up a little bit. They add "noise"—tiny, random jitters to the robot's arm movements. It's like telling the robot, "Hey, maybe wiggle your hand a tiny bit to the left or right."

The Problem with Wiggling
The paper argues that this "wiggling" approach is often useless. If the robot is holding the wrong object (the mug) and trying to do the wrong thing, a tiny wiggle won't help. The robot needs a giant leap in thinking, not a tiny shuffle. It needs to realize, "Wait, I shouldn't be holding the mug; I should be pushing the door." But random jitters can't create that kind of big, global change. They only produce small, local mistakes.

The New Idea: Changing the Script
Enter Prompt-Driven Exploration (PDE). Instead of shaking the robot's arm, the researchers decided to shake up the words they give the robot.

Think of the robot like a very literal actor in a play. If the script says "grab the mug," the actor grabs the mug. If the actor keeps failing, you don't just tell them to wiggle their fingers; you rewrite the script.

Old Script: "Close the microwave." (Result: Robot grabs mug).
New Script: "Push the microwave door until it clicks." (Result: Robot pushes the door).

The paper shows that by simply changing the prompt (the instruction), the robot's behavior changes completely, often solving the problem without needing to retrain its brain at all.

The "Smart Editor" Assistant
Here is the clever part: How do you know which new script to write? You can't just guess. The researchers used a special "Smart Editor" (a Vision-Language Model) to watch the robot fail.

The robot tries the task with a weird instruction.
It fails.
The Smart Editor watches the video of the failure, figures out why it failed (e.g., "Oh, it grabbed the mug because the prompt didn't say to ignore it"), and then writes a brand new, better instruction.
The robot tries again with the new instruction.

This process is like a detective solving a mystery. The editor looks at the clues (the failed video), diagnoses the mistake, and rewrites the clues to guide the detective (the robot) to the right solution.

What the Paper Proves (and What It Doesn't)
The researchers tested this on many different tasks, from stacking blocks to closing drawers.

The Results: In simulations, PDE helped robots learn tasks that were impossible for standard methods. For example, on a set of "Hard" tasks where the robot started with a 0% success rate, standard methods (like random wiggling) stayed stuck near zero. PDE, however, managed to find a path to success, eventually reaching high success rates (like 98% on the microwave task).
Real-World Proof: They even tried this on a real robot in a real lab. With just 64 real-world attempts (about one hour of robot time), the PDE method improved success rates from roughly 13% to 35%. Standard methods only got to about 20% after 128 attempts (twice as much time).
What It's Not: The paper explicitly rules out the idea that just having many different ways to say the same thing is enough. They tested this by giving the robot random rephrasings (like "shut the appliance" instead of "close the microwave") without the Smart Editor's help. Those random changes barely helped. The magic wasn't in the variety of words; it was in the smart selection of the right words based on what went wrong.

Why This Matters
This approach suggests that for big, smart models, the best way to explore new behaviors isn't by messing with their muscles (actions), but by messing with their context (the prompt). It's like realizing that to get a better performance from an actor, you don't need to train them to move their hands differently; you just need to give them a better line to read.

The paper suggests this works because the "Smart Editor" acts like a guide, constantly updating a list of the best instructions to try. It's a way of exploring the "space of ideas" rather than the "space of movements," and it seems to be a much faster way to teach robots when they are starting from scratch.

Technical Summary: Prompt-Driven Exploration (PDE)

1. Problem Statement

Reinforcement Learning (RL) post-training for foundation models, particularly Vision-Language-Action (VLA) models, is bottlenecked by exploration. Standard RL methods rely on injecting stochasticity into the action space (e.g., action noise or parameter noise) to discover new behaviors. However, the authors argue that such noise only induces local perturbations, resulting in rollouts that remain close to the policy's current behavior.

This limitation is critical when fine-tuning weak initial policies (e.g., those with near-zero success rates on manipulation tasks). Escaping a suboptimal policy often requires global perturbations that alter the entire trajectory and strategy, which step-wise action noise cannot achieve. Furthermore, in sparse-reward settings where the initial policy rarely succeeds, reward signals are too sparse to guide the selection of useful exploratory actions.

2. Methodology: Prompt-Driven Exploration (PDE)

The paper proposes Prompt-Driven Exploration (PDE), a strategy that shifts the axis of exploration from the action space to the prompt space. Since foundation models (LLMs and VLAs) are conditioned on natural language prompts, modifying the prompt induces global changes in the policy's behavior without requiring weight updates.

Core Mechanism

PDE treats the prompt distribution as a proxy for the policy distribution. It implements a form of posterior sampling at the prompt level:

Implicit Posterior Sampling: Instead of maintaining a Bayesian posterior over neural network parameters (which is intractable for large models), PDE maintains a distribution over prompts, $\rho(\cdot | g, H)$ , where $g$ is the task and $H$ is the interaction history.
VLM as an Amortized Posterior Update: A Vision-Language Model (VLM) acts as a supervisor. It analyzes rollout videos generated by the current policy under various prompts, diagnoses failure modes, and rewrites the prompts to elicit better behavior. This process updates the prompt distribution $\rho$ to concentrate on prompts that yield successful rollouts.
Two-Stage Training Loop:
- Prompt Discovery: The policy weights are frozen. The VLM samples candidate prompts, executes rollouts, and refines the prompt pool based on observed trajectories and rewards.
- Policy Optimization: Once a pool of effective prompts is identified, the policy is fine-tuned using Proximal Policy Optimization (PPO).

Key Technical Components

Mixture Sampling: During RL training, prompts are sampled from a mixture of the canonical task prompt ( $p_g$ ) and the discovered exploratory prompts ( $\rho$ ). The mixture weight $\alpha$ is adaptive, increasing as the policy improves on the canonical prompt.
Mixed Backpropagation: To ensure improvements from exploratory prompts transfer to the canonical prompt, the PPO loss is modified. The advantage ratio uses the average log-probability of the action under both the exploratory prompt and the canonical prompt. This couples the learning signals, ensuring the policy learns behaviors that are robust across prompt variations.
Prompt Refinement: The VLM does not just generate random paraphrases; it reasons over specific failure modes (e.g., "the robot grasped the wrong object") to generate targeted instructions (e.g., "grasp the green container, not the mug").

3. Key Contributions

A New Exploration Axis: The paper introduces prompt-space exploration as a viable alternative to action-space noise, capable of inducing global behavioral changes that action noise cannot reach.
Zero-Shot to Non-Zero Reward: PDE enables RL to bootstrap learning from policies with zero initial success rates. By discovering prompts that unlock non-zero reward trajectories, PDE provides the necessary learning signal for RL to proceed, whereas standard methods fail to escape the zero-reward basin.
Sample Efficiency: The method significantly improves sample efficiency across manipulation and reasoning tasks, requiring far fewer environment interactions to reach high success rates compared to baselines.
Generalization: The approach is demonstrated to be effective not only for VLA control but also for LLM-based reasoning and coding tasks, suggesting that prompt-space exploration is a general mechanism for foundation model fine-tuning.

4. Experimental Results

The authors evaluate PDE on several benchmarks:

LIBERO and LIBERO-PRO:
- On "Hard" tasks (initial success rate < 10%), PDE achieves a 60% relative improvement over standard action noise.
- In the "Close the Microwave" case study, a policy with 0% success under the canonical prompt is guided by PDE to discover prompts (e.g., "push the microwave door until it shuts") that yield non-zero success, eventually reaching ~100% success on the canonical prompt.
- PDE outperforms baselines including Action Noise, Parameter Noise, Random Network Distillation (RND), and dense reward models (Robometer).
ManiSkill:
- PDE improves success rates across all difficulty tiers (Hard, Medium, Easy) and generalization axes (Visual, Semantic, Execution).
- On hard tasks where action noise fails to make progress, PDE successfully bootstraps learning.
Real-World Manipulation:
- Using a Franka FR3 arm, PDE demonstrates superior sample efficiency in physical environments. After 128 real-world rollouts, PPO+PDE achieved a 45% success rate, more than double the 20% achieved by standard PPO with action noise.
LLM Tasks (LiveCodeBench & AIME 2026):
- In coding and mathematical reasoning tasks, PDE improves early sample efficiency, reaching higher accuracy levels in fewer training steps compared to standard RL fine-tuning (RLOO and GRPO).

5. Significance and Claims

The paper claims that for foundation-model policies, effective exploration does not necessarily require perturbing the model's internal parameters or actions. Instead, effective exploration can come from the context (the prompt) that conditions the policy.

The authors position PDE as a solution to the "sparse reward" and "weak initialization" problems in RL fine-tuning. By leveraging the pre-trained language prior of VLMs to reason about failures and rewrite instructions, PDE realizes a classical RL framework (posterior sampling) in a tractable, high-level space. The work suggests that for models where the rollout follows from a prompt, modifying the prompt is a more powerful mechanism for global exploration than step-wise noise.

The paper concludes modestly, stating that PDE "enables RL to learn successful policies even from zero-reward starts" and "improves sample efficiency more broadly," without claiming it solves all exploration challenges in RL. The method is presented as a complementary strategy that changes what the policy is asked to do, rather than how the policy is updated.