Flow-based Policy Adaptation without Policy Updates

Technical Summary: GLOVES – Flow-based Policy Adaptation without Policy Updates

Problem Statement
Modern robotic systems increasingly rely on agents with prior knowledge, such as human operators, imitation learning (IL) policies, reinforcement learning (RL) policies, and pretrained vision-language-action (VLA) models. While these agents can infer task intent, their actions are often noisy, biased, delayed, or misaligned with expert behavior, particularly when deployed in zero-shot or domain-shifted scenarios. Standard solutions involve finetuning agents on task-specific data; however, this is often infeasible or undesirable due to the agent being a human, the model being a black-box API, the difficulty of collecting large-scale demonstrations, or the prohibitive cost of updating large models. The paper addresses the challenge of adapting an agent's actions at execution time without updating the agent itself, specifically for image-conditioned action-chunk adaptation.

Methodology: GLOVES
The authors propose GLOVES (a family of Flow-based adaptation methods), which corrects non-expert actions by transporting them toward an expert action distribution using flow matching. The framework operates as a lightweight shared-control module that preserves agent intent while improving task success.

1. Core Mechanism (Flow Matching):
   GLOVES learns a conditional flow $v_\theta(x, t | c)$ that maps a tractable source distribution (Gaussian noise) to a target expert distribution conditioned on visual observations $c$. During training, it learns an expert flow from limited expert demonstrations. When available, it can also learn an agent flow from imperfect agent actions.

2. Selective Intervention via OOD Gate:
   A key innovation is the use of the expert flow itself as a natural out-of-distribution (OOD) detector. Instead of training a separate uncertainty model, the system evaluates a proposed action chunk $a_u$ by reversing the expert flow to estimate its latent representation $\hat{z} = F^{-1}_\theta(a_u)$. The score $s(a_u) = \|\hat{z}\|^2_2$ serves as a non-conformity metric.

   • If the score indicates the action is in-distribution (consistent with the expert prior), it is executed unchanged.
   • If the action is OOD, the system intervenes to correct it.
     This mechanism avoids unnecessary editing of valid actions and reduces computational overhead.

3. Adaptation Variants:
   The paper introduces three specific methods within the GLOVES family for refining actions:

   • FPAS (Flow-Prior Action Sampling): Uses the OOD score to guide a local sampling-based search around the agent's command. It samples perturbed candidates, penalizes those with high OOD scores or those deviating too far from the original action, and iteratively refines the chunk.
   • FEEG (Flow Editing with Energy Guidance): Directly guides the learned expert flow to generate an action that is expert-like but consistent with the user's intent. It employs an energy function $E(x) = \frac{1}{2}\|x - a_u\|^2$ to regularize the flow trajectory, balancing fidelity to the agent with conformity to the expert distribution.
   • IFAE (Inversion-Free Action-Chunk Editing): Inspired by FlowAlign, this method avoids the "noise-then-denoise" inversion step common in diffusion-based editors. It uses coupled flow trajectories between the agent and expert flows to directly transport the agent's action chunk toward the expert distribution, eliminating the need to select a task-dependent "assistance depth" hyperparameter.

Key Contributions

• Policy Adaptation without Updates: GLOVES enables the adaptation of black-box agents (including humans and VLAs) at execution time without modifying the underlying policy parameters.
• Unified Framework: It unifies policy adaptation and shared autonomy within a single action-correction framework.
• Source-Consistent Correction: Unlike methods that require selecting a partial-diffusion fidelity-conformity tradeoff hyperparameter, GLOVES (specifically IFAE) performs source-to-expert transport that remains consistent with the agent's proposal without explicit inversion.
• Built-in OOD Detection: The method utilizes the learned expert flow for both selective intervention (deciding when to assist) and the correction itself, removing the need for separate anomaly detectors.
• Data Efficiency: The system requires only a small number of expert demonstrations or reusable successful skill segments to train the adapter.

Experimental Results
The authors evaluated GLOVES across four simulation tasks (Slalom, CAN placement, Keypad entry, Charger Plug) and two real-world robot tasks (Charger Insertion, Cup Serving).

• Stochastic Wrapper Agents: In simulation, GLOVES variants achieved the best success rate in 13 out of 16 stochastic-wrapper settings (covering noisy, delayed, slow, and biased agents).
• VLA Agents: For finetuned VLA agents, GLOVES improved success rates by up to 29.03% in simulation.
• Real-World Performance: On real robots, GLOVES consistently outperformed baselines (DDPM-based and Consistency Model-based shared autonomy methods) and improved the performance of finetuned VLAs.
• Efficiency: The OOD gate successfully filtered out valid actions, ensuring corrections were only applied when necessary, thereby reducing inference costs.

Significance and Claims
The paper claims that GLOVES provides a lightweight, robust framework for improving imperfect agents without the computational and data costs of finetuning. By leveraging flow matching, the method offers a natural mechanism for selective intervention that balances the preservation of agent intent with adherence to expert behavior. The authors emphasize that their approach is largely agnostic to the nature of the agent (human or AI) and does not require access to an expert policy, reward design, or knowledge of system dynamics. The results suggest that action-level adaptation via flow transport is an effective alternative to full autonomy or model retraining, particularly in scenarios where agent updates are restricted or data is scarce.