🤖 machine learning

VGFM: Expressive Robot Policies via Dense Value Guidance in Flow Matching

The paper proposes Value-Guided Flow Matching (VGFM), a scalable offline reinforcement learning framework that integrates dense value-based guidance into expressive flow-matching policies for robotic control without requiring backpropagation through time or additional algorithmic overhead, achieving strong performance across diverse locomotion and manipulation tasks.

Original authors: Prajwal Koirala, Mark Campbell

Published 2026-09-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Prajwal Koirala, Mark Campbell

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long been masters of repetitive, predictable tasks, but teaching them to handle the messy, unpredictable variety of the real world has remained a stubborn challenge. For years, researchers have tried to solve this by feeding robots vast libraries of past movements, hoping the machine could learn to mimic human skill without ever needing to practice in the real world. This approach, known as offline learning, offers a safe and efficient path to training, but it hits a wall when the robot encounters a situation slightly different from its training data. The robot must then decide how to act, often choosing between many different valid ways to move, much like a driver at a complex intersection who can turn left, go straight, or weave through traffic. Traditional methods struggle to capture this rich variety of choices, often forcing the robot into a single, rigid path that fails when conditions change. To move beyond these limitations, scientists are turning to generative models, a type of artificial intelligence capable of imagining a wide spectrum of possible actions rather than just one.

The core difficulty lies in teaching these imaginative models to be not just creative, but also smart. A robot needs to know which of its many imagined actions will actually lead to success. In the past, trying to guide a robot's imagination toward a good outcome required a computationally heavy process where the computer had to trace every single step of the robot's thought process backward to see where it went wrong. This was slow, unstable, and often made the training process too complex to scale. Researchers at Cornell University have now introduced a new method called Value-Guided Flow Matching that sidesteps this bottleneck entirely. Instead of forcing the computer to retrace its steps through time, this new approach allows the robot to receive guidance at every single moment of its decision-making process, ensuring that even its intermediate thoughts are steering it toward a successful outcome.

The team, led by Prajwal Koirala and Mark Campbell, designed a system where the robot's policy, or its strategy for moving, is built like a continuous flow rather than a series of disconnected jumps. Imagine a river flowing from a source to a destination; the robot starts with a simple, random idea of movement and gradually shapes it into a specific action. The innovation here is that the system checks the quality of this action at every point along the river's path, not just at the end. By predicting the final destination of the movement at each intermediate step, the system can apply a "value" signal—a measure of how good that action is—without needing to unravel the entire generative process. This means the robot learns to refine its movements continuously, guided by a critic that evaluates the quality of the action in real-time, all while avoiding the heavy computational cost of backtracking through time.

To test this idea, the researchers put their system through a rigorous battery of challenges using a standard benchmark called OGBench, which includes a wide array of difficult tasks. These tasks ranged from navigating complex mazes with four-legged and humanoid robots to manipulating objects with robotic arms. In the maze scenarios, the robots had to find their way through winding corridors, while the manipulation tasks required them to stack cubes or interact with objects in cluttered environments. The system was trained on static datasets of past movements, meaning it never saw the environment during the learning phase, only the recorded data. The results were striking: the new method consistently outperformed or matched the best existing techniques across almost all these diverse tasks. It achieved high success rates in navigating the AntMaze and HumanoidMaze environments and excelled at the precise movements required for the Cube and Scene manipulation tasks.

A key feature of this approach is its flexibility during the actual use of the robot. Once the system is trained, engineers can adjust how much computing power is used to generate a single action without needing to retrain the model. The researchers found that by simply increasing the number of steps the computer takes to calculate the final movement, the robot could perform more precise and complex tasks. This trade-off between speed and accuracy is crucial for real-world applications, where a robot might need to move quickly in a simple situation but slow down to be precise in a delicate one. The study showed that this scalability comes with almost no extra cost in terms of time, allowing the robot to become more expressive and capable simply by using more processing power at the moment of decision.

The success of this method suggests a new path forward for robotic control, one that balances the need for safety and data efficiency with the demand for complex, adaptable behavior. By removing the heavy computational burden of tracing every step backward, the researchers have made it possible to train robots that can handle long, complicated sequences of actions with a level of expressiveness previously out of reach. The system does not rely on magic or shortcuts; it simply rethinks how guidance is applied, allowing the robot to learn from a dense stream of feedback throughout its entire decision-making journey. As the field of robotics continues to push toward machines that can operate in unstructured human environments, methods like this provide a scalable and effective way to teach them the subtle art of choosing the right action from a world of possibilities.

Technical Summary: VGFM – Expressive Robot Policies via Dense Value Guidance in Flow Matching

1. Problem Statement

Offline Reinforcement Learning (RL) aims to train control policies using pre-collected static datasets to avoid the safety risks and costs of online exploration. A central challenge in this setting is distribution shift, where a learned policy may select out-of-distribution (OOD) actions, leading to unsafe or ineffective behaviors. To mitigate this, policies are often regularized to remain close to the behavior distribution.

Recent advances in generative models (e.g., diffusion and flow matching) offer expressive, multimodal policy representations capable of capturing complex behavioral diversity in robotic datasets. However, integrating these generative policies into value-based offline RL frameworks (like DDPG+BC) presents significant difficulties:

Backpropagation Through Time (BPTT): Standard actor-critic frameworks require computing gradients of the value function with respect to the policy's output action. When the policy is a multi-step generative process (solving an ODE), this necessitates backpropagating through the entire generative trajectory. This incurs substantial computational overhead, numerical instability, and memory costs, particularly for high-dimensional or long-horizon tasks.
Trade-offs in Prior Solutions: Existing methods attempt to bypass BPTT but introduce other limitations. For instance, Flow Q-Learning (FQL) distills the generative policy into a single-step network, sacrificing inference-time scalability. Other approaches (e.g., SORL, SSCP) either retain BPTT during training or rely on auxiliary losses (distillation, shortcut, or completion losses) that complicate training and limit the benefits of multi-step generation.

2. Methodology: Value-Guided Flow Matching (VGFM)

The authors propose VGFM, a scalable offline RL framework that enables dense value-guided policy improvement within a flow-based policy without requiring BPTT or auxiliary distillation losses.

Core Concept

VGFM parameterizes the policy as a conditional flow-matching model in action space. Instead of treating the generative process as a black box where gradients must flow backward from the final action, VGFM leverages an x-prediction formulation.

Flow Formulation: The policy defines a continuous trajectory $\{a_t\}_{t \in [0,1]}$ from a latent noise sample $a_0$ to a terminal action $a_1$ . The dynamics are governed by a learned velocity field $u_\theta(s, a_t, t)$ .
X-Prediction: The neural network directly predicts the terminal action $\hat{a}_\theta(s, a_t, t)$ at any intermediate time step $t$ . The velocity is recovered analytically as $u_\theta = (\hat{a}_\theta - a_t) / (1-t)$ .

Key Mechanisms

Action-Space Validity (Lemma 1): Under the x-prediction parameterization, the network's output $\hat{a}_\theta$ at any intermediate step $t$ is a valid action in the action space $\mathcal{A}$ . Crucially, this prediction can be directly evaluated by a standard, time-independent critic $Q_\phi(s, a)$ without differentiating through the generative trajectory.
Dense Value Supervision: Because intermediate predictions are valid actions, the framework applies a standard value-based loss (maximizing $Q_\phi$ ) directly to these intermediate predictions. This allows value gradients to guide the policy at every sampled flow step, providing dense credit assignment without BPTT.
Combined Objective: The training objective combines two terms:
- Behavior Regularization ( $L_{FM}$ ): A flow-matching loss that anchors the trajectory to the empirical data distribution, ensuring conservatism.
- Value Guidance ( $L_{\pi Q}$ ): A negative Q-loss applied to the predicted terminal action $\hat{a}_\theta$ at sampled time steps.
  $L(\theta) = \alpha L_{FM}(\theta) + L_{\pi Q}(\theta)$
  This structure approximates the KL-regularized policy improvement objective, where $L_{FM}$ enforces behavior matching and $L_{\pi Q}$ implements value guidance.

Inference and Scalability

At inference time, actions are generated by numerically integrating the learned ODE from $t=0$ to $t=1$ . A key feature of VGFM is inference-time scalability: the number of integration steps ( $T$ ) can be increased to trade computation for performance (improving control precision) without retraining the model.

3. Key Contributions

Value-Guided Flow Formulation: The authors introduce a formulation where each step of the flow is interpreted as a return-optimizing decision in action space, applying a DDPG+BC-style objective at every step to enable value-guided improvement.
BPTT-Free Policy Improvement: By utilizing an end-target prediction objective with velocity-space losses, VGFM enables policy improvement without backpropagation through the generative trajectory, resolving the tension between dense value guidance and training stability.
Scalable and Expressive Performance: The approach supports inference-time scalability via adjustable ODE discretization and achieves strong empirical performance across diverse robotic manipulation and locomotion tasks.

4. Experimental Results

The method was evaluated on the OGBench benchmark, covering 40 distinct tasks across locomotion (AntMaze, HumanoidMaze, AntSoccer) and manipulation (Cube, Scene) environments.

Performance: VGFM achieved the best or near-best performance on the majority of task groups, consistently matching or surpassing established baselines including Gaussian methods (BC, IQL, ReBRAC), diffusion-based methods (IDQL, CAC), and other flow-based methods (IFQL, FQL, SORL).
Training Efficiency: The method required minimal hyperparameter tuning, relying on a single coefficient ( $\alpha$ ) to balance regularization and value guidance, unlike other methods requiring multiple auxiliary loss weights.
Inference Scalability: Experiments varying the number of Euler integration steps ( $T \in \{1, 2, 5, 10\}$ ) demonstrated that increasing test-time compute improves task success rates and control precision. Notably, the wall-clock inference time remained comparable to single-step baselines (like FQL) despite the increased number of function evaluations, enabling practical test-time scaling.

5. Significance and Claims

The paper claims that VGFM provides a simple, scalable, and effective approach for learning expressive policies in long-horizon, goal-oriented robotic control. Its significance lies in:

Decoupling Training and Inference Complexity: It avoids the computational burden of BPTT during training while retaining the flexibility of multi-step generation during inference.
Eliminating Auxiliary Overhead: Unlike prior works that require distillation losses or shortcut models, VGFM relies solely on the core flow-matching and value-guidance terms.
Robustness in Offline RL: By embedding behavior regularization directly into the flow parameterization and applying dense value guidance, it effectively addresses distribution shift while leveraging the multimodal capacity of generative models.

The authors position VGFM as a practical alternative that bridges the gap between the expressiveness of generative models and the stability required for offline reinforcement learning, particularly in settings where action diversity and multimodality are essential.