Technical Summary: Residual Flow Steering (RFS) for Dexterous Manipulation

Problem Statement

Imitation learning, particularly using expressive generative models like diffusion and flow matching, has proven effective for bootstrapping policies in high-dimensional dexterous manipulation. However, policies pretrained solely on human demonstrations often exhibit limited generalization to unseen scenarios and require fine-tuning to achieve robust, production-level performance.

Existing adaptation methods face a trade-off:

• Supervised Fine-Tuning: Requires high-quality, curated expert data, which is expensive to collect.
• Reinforcement Learning (RL) with Generative Models: While RL offers a principled framework for behavior improvement via reward optimization, standard RL fine-tuning methods often rely on closed-form likelihoods or reparameterization tricks. These are incompatible with complex generative architectures (diffusion/flow models). Furthermore, existing policy modulation techniques struggle to balance global exploration with local precision:
  • Residual RL enables local action corrections but fails to induce necessary global behavioral changes.
  • Diffusion Steering (input modulation) allows global modulation but offers limited fine-grained control in off-distribution states.

The core challenge is adapting pretrained generative policies to new tasks and real-world dynamics while preserving the global exploration benefits of pretraining and enabling rapid correction of local execution errors.

Methodology: Residual Flow Steering (RFS)

The authors propose Residual Flow Steering (RFS), a data-efficient reinforcement learning framework that unifies input and output modulation to adapt pretrained flow-matching policies.

Unified Policy Modulation

RFS views policy adaptation as modulating a pretrained base policy $\pi_{base}$ (represented by a flow-matching velocity field $v_\theta$) without updating its underlying parameters. It jointly optimizes two components:

1. Latent Noise Steering (Input Modulation): A policy $\pi_H$ learns to output an initial latent noise vector $a_0$ conditioned on the state $s$. This steers the generative model's global behavior, allowing the policy to shift modes and explore beyond the demonstration manifold.
2. Residual Action Correction (Output Modulation): A policy $\pi_r$ learns a residual action $a_r$ that is added to the output of the base policy. This enables precise, local refinements to correct execution errors or handle off-manifold requirements.

The final action is computed as:
$$a = a_{base} + a_r$$
where $a_{base}$ is generated by integrating the flow dynamics from the steered latent noise $a_0$.

Implementation Pipeline

The framework is instantiated through a simulation-to-reality pipeline:

1. Simulation Pretraining: A base flow-matching policy is trained on a small set of VR-teleoperated demonstrations.
2. Online RL in Simulation: The RFS policy ($\pi_{RFS}$) is fine-tuned using online PPO to optimize grasp success and stability. Privileged state information (e.g., object poses) is used here to generate high-quality demonstrations.
3. Distillation: The RFS-enhanced policy is distilled into a visuomotor policy conditioned on point clouds, removing the need for privileged state information for deployment.
4. Real-World Offline Fine-Tuning: To bridge the sim-to-real gap, the distilled policy is adapted using a small dataset of human-corrected transitions.
   • Data Collection: The base policy executes rollouts; humans intervene with corrective actions ($a_{human}$).
   • Residual Definition: The residual is defined as $a_r = a_{human} - a_{base}$.
   • Training: The RFS policy is trained using Offline RL (specifically TD3+BC). The critic is conditioned on the final executed action ($a = a_{base} + a_r$) rather than the decomposed components, which the authors found critical for stability. The actor maximizes the critic value while applying a behavior cloning regularizer on the residual actions.

Key Contributions

1. RFS Framework: Introduction of Residual Flow Steering, a unified framework that jointly optimizes latent noise (for global adaptation) and residual actions (for local refinement) to adapt pretrained flow-matching policies.
2. Effective Simulation Pretraining: Demonstration that RFS enables robust simulation pretraining from human demonstrations, successfully capturing complex multi-finger coordination behaviors.
3. Data-Efficient Real-World Adaptation: Validation that RFS supports real-world fine-tuning via offline RL, outperforming bootstrapped imitation-learning baselines and standard residual-only or steering-only methods.
4. Comprehensive Ablations: Extensive experiments identifying the necessity of both input and output modulation components for high performance in both simulation and real-world settings.

Experimental Results

Simulation Performance

RFS was evaluated on six dexterous manipulation tasks: Grasping, Pick & Place, Packing, Push-to-Grasp, Stacking, and Pouring.

• Baselines: RFS was compared against Diffusion/Flow RL fine-tuning (DPPO, ReinFlow), Offline-to-Online RL (IQL, AWAC, Flow Q-Learning), and RL with Demonstrations (RLPD, IBRL).
• Ablations: Compared against Residual RL only (Policy Decorator, ResiP) and Diffusion Steering only (DSRL).
• Outcome: RFS achieved the highest success rates across all tasks, with an average success rate of 0.87.
  • Notably, RFS significantly outperformed the base policy and all baselines on long-horizon (Packing) and high-precision (Stacking, Pouring) tasks where other methods collapsed or struggled.
  • For example, on Stacking, RFS achieved 0.95 success, compared to 0.15 for the base policy and 0.135 for DSRL.

Real-World Performance

Experiments were conducted on a Franka arm with a LEAP hand, transferring policies trained in simulation to real-world grasping and pick-and-place tasks with both seen and unseen objects.

• Setup: Fine-tuning was performed using only 50 human-corrected demonstrations (offline RL).
• Baselines: Zero-shot transfer, Behavior Cloning (BC) fine-tuning, and Co-training.
• Outcome:
  • Seen Objects: RFS achieved 90.0% success in Pick-and-Place and 80.0% in Grasping, outperforming Co-training (60.0% / 83.3%) and BC (40.0% / 73.3%).
  • Unseen Objects: RFS demonstrated superior generalization, achieving 74.0% success in Grasping and 48.0% in Pick-and-Place, significantly higher than the zero-shot baseline (43.3% / 30.0%) and other offline RL variants.
• Critic Design: The authors found that conditioning the critic on the final executed action ($a_{base} + a_r$) was essential; conditioning on decomposed components failed to produce consistent results.

Significance and Claims

The paper claims that RFS provides a principled solution to the "adaptation bottleneck" in generative imitation learning. By decoupling global exploration (via latent noise) from local refinement (via residuals), RFS allows policies to:

1. Preserve Pretraining: Retain the expressive structure and multimodal capabilities of the pretrained flow-matching policy.
2. Adapt Efficiently: Achieve robust performance with minimal real-world interaction data (offline fine-tuning).
3. Generalize: Successfully transfer from simulation to the real world, handling novel objects and dynamic conditions that cause zero-shot policies to fail.

The authors conclude that while RFS is highly effective, current limitations include a reliance on point-cloud observations (lacking high-level semantic reasoning) and a restriction to offline fine-tuning, which limits responsiveness to dynamically changing conditions during deployment.