Mean Flow Policy with Instantaneous Velocity Constraint for One-step Action Generation

This paper introduces the Mean Velocity Policy (MVP), a novel one-step generative policy that employs an Instantaneous Velocity Constraint (IVC) to theoretically guarantee high expressiveness while achieving state-of-the-art performance and significantly faster training and inference speeds on challenging robotic manipulation tasks compared to existing flow-based baselines.

The Gist

🚀 The Big Idea: Teaching a Robot to Move in a Single Leap

Imagine you are teaching a robot arm to pick up a cup and pour water. In the world of Artificial Intelligence (AI), there are two main ways to teach the robot how to move:

1. The "Slow and Steady" Way (Current Standard): The robot thinks about the move in tiny, slow steps. It asks, "Where am I? Where do I want to go? Let me take a tiny step." Then it asks again, and again, and again. It's like walking up a staircase one step at a time. It's accurate, but it takes a long time to get to the top.
2. The "Super Leap" Way (What this paper proposes): The robot looks at the start and the finish, and instantly calculates the perfect jump to get there in one go. It's like a superhero leaping from the ground to the roof in a single bound.

The Problem: The "Super Leap" is usually too hard to learn. If you try to teach a robot to jump directly without practice steps, it often misses the target or learns the wrong way.

The Solution: The authors created a new method called MVP (Mean Velocity Policy). It allows the robot to make that "Super Leap" (one-step action) but teaches it in a way that is just as smart and accurate as the slow, step-by-step methods.

---

🧠 The Core Concepts (Explained with Analogies)

1. The Problem: The "Blindfolded Hiker"

Most modern AI robots use a technique called Flow Matching. Imagine a hiker trying to get from a valley (noise/randomness) to a mountain peak (the perfect action).

• Old Way: The hiker takes 10 small steps, checking a map at every step. This is slow but safe.
• The Goal: We want the hiker to take one giant leap to the peak.
• The Issue: If you just tell the hiker "Leap to the peak," they might overshoot or land in a ditch. Mathematically, the path they learn is "wobbly" because there are infinite ways to get from A to B, and the robot doesn't know which one is the right one.

2. The MVP Solution: The "Average Speed" Trick

Instead of teaching the robot the instantaneous speed at every tiny moment (which requires 10 steps), MVP teaches the robot the Mean Velocity (the average speed needed to get from start to finish).

• Analogy: Imagine you are driving from New York to Los Angeles.
  • Old Method: You check your speedometer every second and adjust the gas pedal constantly.
  • MVP Method: You calculate the average speed you need to maintain to arrive exactly on time. If you maintain that average speed, you get there in one smooth, continuous drive.
  • Result: The robot can generate the perfect move in one single calculation (one step) instead of ten. This makes it incredibly fast.

3. The Secret Sauce: The "Instantaneous Velocity Constraint" (IVC)

Here is the tricky part. If you only teach the robot the average speed, it might still be wrong.

• The Math Problem: Think of a river flowing from a waterfall to the ocean. If you only know the average flow of the river, you don't know exactly how fast the water is moving right at the edge of the waterfall. There are infinite possibilities.
• The Fix (IVC): The authors added a rule called the Instantaneous Velocity Constraint.
  • Analogy: Imagine a teacher telling a student, "The average speed of your trip must be 60mph." The student might drive 100mph for a minute and 20mph for the rest.
  • The IVC Rule: The teacher adds, "But, at the very start of the trip (the instant you leave the driveway), you must be moving at exactly 60mph."
  • Why it works: By forcing the robot to get the speed right at the very beginning, it locks the entire path into place. It stops the robot from guessing and forces it to learn the exact correct path. It acts like a "boundary condition" that makes the math solvable and the learning accurate.

---

🏆 Why This Matters (The Results)

The authors tested this on 9 difficult robot tasks (like stacking blocks, lifting cans, and moving cubes).

1. Speed: Because MVP only needs one step to decide what to do, it is 3x to 5x faster at training and running than the current best methods.
   • Real-world impact: This means robots can react in real-time. If a robot is catching a ball, it can't wait 10 milliseconds to think; it needs to think instantly. MVP makes that possible.
2. Smarts: Despite being faster, it didn't get "dumber." In fact, it was often more successful than the slow methods. It solved the hardest tasks (like moving three cubes at once) better than anyone else.
3. Efficiency: It saves computer power. Instead of running a complex simulation 10 times to get one answer, it runs it once.

📝 Summary in One Sentence

The authors invented a new AI "brain" (MVP) that lets robots learn to move in a single, perfect leap instead of taking many small steps, using a special "start-speed rule" (IVC) to ensure the leap is accurate, resulting in robots that are both super fast and super smart.

Technical Summary

Here is a detailed technical summary of the paper "Mean Flow Policy with Instantaneous Velocity Constraint for One-Step Action Generation" (ICLR 2026 Oral).

1. Problem Statement

Reinforcement Learning (RL) requires policies that are both expressive (capable of modeling complex, multi-modal action distributions) and efficient (capable of fast inference for real-time control).

• The Bottleneck: Existing generative policies, such as Diffusion Models and Flow Matching, excel at expressiveness but rely on iterative multi-step sampling (solving Ordinary Differential Equations or SDEs over many steps). This introduces significant computational overhead, hindering training speed (especially in online RL) and causing inference latency that prevents high closed-loop performance in real-time robotic systems.
• The Gap: Current attempts to create one-step policies often sacrifice expressiveness or rely on distillation from multi-step models, which can be unstable or suboptimal. There is a need for a policy function that achieves fastest one-step action generation without losing the ability to model complex distributions.

2. Methodology: Mean Velocity Policy (MVP)

The authors propose Mean Velocity Policy (MVP), a novel generative policy function that models the mean velocity field rather than the instantaneous velocity field used in standard Flow Matching.

A. Core Concept: Mean Velocity Field

Instead of learning an instantaneous velocity $v(x(t), t)$ that requires integrating from $t=0$ to $t=1$ via numerical methods, MVP learns the mean velocity $u$ over a time interval $[t, r]$:
$$u(a(t), t, r, s) \triangleq \frac{1}{r - t} \int_{t}^{r} v(a(\tau), \tau, s) d\tau$$

• One-Step Generation: If the mean velocity field is perfectly learned, the action can be generated in a single step from Gaussian noise $a(0)$ to the target action $a(1)$:
  $$a(1) = a(0) + u^*(a(0), 0, 1, s)$$
  This eliminates the need for iterative ODE solvers during inference.

B. Training Objective: Mean Flow Identity

To train the mean velocity model $u_\theta$, the authors derive a "Mean Flow Identity" by differentiating the definition of mean velocity. The training loss ($L_{MF}$) minimizes the residual of this identity:
$$L_{MF}(\theta) = \mathbb{E} \left\| u_\theta - \text{sg}\left( v - (t-r)\frac{d}{dt}u_\theta \right) \right\|^2$$
where $v$ is the target instantaneous velocity ($a^* - a(0)$) and "sg" denotes the stop-gradient operator.

C. The Critical Innovation: Instantaneous Velocity Constraint (IVC)

The authors identify a theoretical flaw in training mean velocity models using only the Mean Flow Identity: the underlying Ordinary Differential Equation (ODE) lacks explicit boundary conditions, leading to a multiplicity of solutions (an arbitrary constant bias can persist).

To solve this, they introduce the Instantaneous Velocity Constraint (IVC):

• Mechanism: At the boundary where the time interval shrinks to zero ($r \to t$), the mean velocity must equal the instantaneous velocity.
• Loss Function: An auxiliary loss term is added:
  $$L_{IVC}(\theta) = \mathbb{E} \left\| u_\theta(a(t), t, t) - v \right\|^2$$
• Theoretical Justification: The paper proves (Theorem 3) that minimizing $L_{IVC}$ forces the integration constant of the ODE solution to zero, ensuring the uniqueness of the learned mean velocity field and eliminating cumulative fitting errors. This acts as a crucial boundary condition to stabilize learning.

D. RL Framework: Generate-and-Select

MVP is integrated into an off-policy RL framework (similar to FQL or BFN):

1. Generate: Sample $N$ candidate actions from the MVP (one-step generation).
2. Select: Use a Critic ($Q$-function) to select the action with the highest value ($\text{argmax}_{a_i} Q(s, a_i)$).
3. Update: The selected action serves as the target for policy imitation and value training.
   The authors prove that this "Best-of-N" update guarantees policy improvement, provided the mean flow matching error is minimized (which IVC helps achieve).

3. Key Contributions

1. Mean Velocity Policy (MVP): A new generative policy architecture that enables fastest one-step action generation by modeling mean velocity fields, removing the iterative sampling overhead of standard flow policies.
2. Instantaneous Velocity Constraint (IVC): A theoretical and practical training enhancement that serves as an explicit boundary condition. It resolves the non-uniqueness of the mean flow ODE, significantly improving learning accuracy and policy expressiveness.
3. State-of-the-Art Performance: Empirical validation showing MVP achieves superior success rates on challenging robotic manipulation tasks while offering substantial speedups in both training and inference compared to existing baselines.

4. Experimental Results

The method was evaluated on 9 sparse-reward robotic manipulation tasks from Robomimic (Lift, Can, Square) and OGBench (Cube-double/triple tasks).

• Success Rates: MVP achieved State-of-the-Art (SOTA) results, securing the top position on 8 out of 9 tasks.
  • On the most difficult task (Cube-triple-task4), MVP achieved 0.52 ± 0.11 success rate, significantly outperforming the next best baseline (QC at 0.46) and far exceeding FQL and BFN.
  • Average success rate across all tasks: 0.88 ± 0.05.
• Efficiency (Training Speed):
  • MVP achieved an average online training speed of 153.6 iter/s, significantly faster than FQL (108.5 iter/s), QC (92.6 iter/s), and BFN (68.0 iter/s).
• Efficiency (Inference):
  • MVP inference time is ~10.9 ms (CPU-only), comparable to FQL (which uses a distilled one-step policy) and 10x faster than multi-step baselines like BFN and QC (~117 ms).
• Ablation Studies:
  • Removing IVC ($\lambda=0$) caused a significant drop in performance (e.g., success rate on Cube-triple-task4 dropped from 0.52 to 0.30), validating the theoretical necessity of the boundary condition.
  • Comparisons with "one-step" variants of other baselines showed that naive one-step flows fail on long-horizon tasks, whereas MVP succeeds due to its specific mean-flow formulation and IVC.

5. Significance

This work bridges the critical gap between expressiveness and efficiency in RL policies.

• Real-Time Applicability: By enabling one-step generation without sacrificing the ability to model multi-modal distributions, MVP makes generative policies viable for real-time, high-frequency robotic control systems where latency is a major constraint.
• Theoretical Insight: The introduction of IVC provides a rigorous mathematical solution to the boundary condition problem in mean flow modeling, offering a new perspective on training generative models for control.
• Practical Impact: The method offers a practical path forward for deploying complex generative policies in online RL settings, potentially accelerating the adoption of advanced RL techniques in autonomous robotics and industrial automation.