💻 computer science

CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL

This paper introduces CreFlow, a novel online reinforcement learning framework that employs a compositional Linear Temporal Logic-based reward model and specialized loss functions to effectively correct physical constraint violations in sparse-reward embodied video diffusion models, significantly improving downstream manipulation task success.

Original authors: Zhenyang Ni, Yijiang Li, Ruochen Jiao, Simon Sinong Zhan, Sipeng Chen, Zhenfei Yin, Minshuo Chen, Philip Torr, Zhaoran Wang, Qi Zhu

Published 2026-05-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhenyang Ni, Yijiang Li, Ruochen Jiao, Simon Sinong Zhan, Sipeng Chen, Zhenfei Yin, Minshuo Chen, Philip Torr, Zhaoran Wang, Qi Zhu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Robots That "Hallucinate"

Imagine you have a super-talented artist (a Video Generation Model) who can draw beautiful pictures of a robot arm picking up a cup and putting it in a drawer. The artist is great at making the lighting look real and the robot's hand look shiny.

However, this artist has a blind spot: Physics.
Sometimes, the artist draws the robot's hand passing through the table (like a ghost), or the cup magically teleports from the table to the drawer without moving. To the artist, the picture looks perfect. But if you tried to build a real robot based on that drawing, it would crash immediately.

Current methods try to fix this by showing the artist a "score" at the end of the video: "Good job!" or "Bad job!" The problem is that this score is too vague. It's like a teacher giving a student a failing grade on a 10-page essay because of one typo on page 7, but not telling the student where the typo is. The student then tries to rewrite the whole essay, accidentally messing up the good parts on pages 1–6.

The Solution: CreFlow

The authors propose a new system called CreFlow (Corrective Reflow). Think of it as a smart editor that doesn't just give a pass/fail grade, but acts like a detective to find exactly where and why the video failed, and then only fixes those specific parts.

CreFlow does this in two main steps:

1. The "Logic Detective" (Compositional Constraint Monitor)

Instead of just looking at the video and guessing if it looks good, CreFlow translates the robot's task into a strict set of logical rules, similar to a checklist a security guard uses.

The Analogy: Imagine the task is "Put the cup in the drawer."
- Rule 1 (Persistence): The cup must exist in every frame. (No teleporting!)
- Rule 2 (Kinematics): The robot arm must move smoothly. (No ghosting through walls!)
- Rule 3 (Causality): The drawer must be opened before the cup goes in.
- Rule 4 (Goal): The cup must end up inside the drawer.

The system checks the video against these rules. If the video fails, the system doesn't just say "Fail." It points a finger and says: "You failed Rule 2 at Frame 15 because the arm went through the table." It creates a mask (a digital highlighter) that covers only the robot's arm and the table, ignoring the background, the lighting, or the cup's color.

2. The "Smart Editor" (The Two-Part Training)

Once the system knows exactly where the mistake happened, it uses two clever techniques to fix the video without ruining the rest of it.

Part A: The "Credit-Aware" Fix (Don't Touch the Good Stuff)

The Old Way: If a video fails, the old methods would tell the AI to change every single pixel in the video to try to get a better score. This is like rewriting the whole 10-page essay because of one typo. It wastes time and often makes the good parts worse.
The CreFlow Way: Using the "highlighter" from the Logic Detective, CreFlow tells the AI: "Only change the pixels inside this highlighted box (the robot arm and the table). Leave the rest of the video exactly as it is."
The Result: The background stays beautiful and stable, while the robot learns to move correctly.

Part B: The "Group Hug" Correction (Learn from Success)

The Old Way: When the AI fails, it tries to guess what a "good" video looks like by imagining the opposite of the bad one. This is often a shaky guess.
The CreFlow Way: Imagine the AI tries to solve the puzzle 10 times. 7 times it fails, but 3 times it succeeds. Instead of guessing, CreFlow looks at those 3 successful videos, averages them together to create a "Perfect Example," and tells the failed videos: "Look at this perfect example. Copy exactly how the successful ones moved in the highlighted area."
The Result: The AI learns much faster because it is being shown a clear, real example of success rather than just being told what not to do.

The Results

The paper tested this on eight different robot tasks (like stacking bowls or putting bottles in a bin).

Better Judging: CreFlow's "Logic Detective" was much better at spotting real failures than previous methods, matching human judgment 88% of the time.
Better Robots: When they used CreFlow to train the video models, the robots actually succeeded at the physical tasks 23.8% more often than before.

Summary

CreFlow is like a teacher who stops giving vague grades. Instead, it uses a logical checklist to find the exact mistake in a robot's plan, highlights only that mistake, and shows the robot a perfect example of how to fix it, leaving everything else alone. This makes the robot learn faster and stop "hallucinating" impossible physics.

Technical Summary: CreFlow: Corrective Reflow for Sparse-Reward Embodied Video Diffusion RL

1. Problem Statement

Video generation models (VGMs) trained on heterogeneous web-scale data often produce visually plausible rollouts that violate physical constraints when applied to embodied manipulation tasks. While reinforcement learning (RL) post-training offers a pathway to align these models with specific reward targets, existing approaches face two critical limitations in the context of manipulation:

Inadequate Reward Design: Current video-RL rewards often rely on low-level visual metrics (e.g., visual quality, motion smoothness, text alignment) or single-dimension scores. These fail to capture the compositional nature of manipulation tasks, where success depends on the simultaneous satisfaction of multiple logical conditions (e.g., object persistence, kinematic validity, causal ordering). Consequently, a video may receive a high reward score despite containing subtle, decisive errors (e.g., object teleportation or kinematic violations) that lead to downstream execution failure.
Inefficient Optimization in High Dimensions: Manipulation videos are high-dimensional, yet task-critical errors often occupy only sparse spatio-temporal regions. Applying uniform RL updates across the entire video tensor wastes gradients on reward-irrelevant pixels and time steps. This can perturb regions that are already plausible or unrelated to the task, leading to off-manifold drift and policy collapse, particularly when dealing with sparse binary rewards.

2. Methodology: CreFlow

The authors propose CreFlow (Corrective Reflow), an online RL framework designed to post-train embodied video diffusion models using sparse, compositional rewards. The framework consists of two main components: a compositional constraint-based reward model and a novel training objective.

2.1 Compositional Constraint Monitor

Instead of relying on holistic visual scores, CreFlow formulates task success as a set of Linear Temporal Logic (LTL) constraints.

Specification Synthesis: For a given task context (instruction and initial observation), a coding agent synthesizes a task-specific LTL formula $\Phi_c$ . This formula is a conjunction of atomic predicates (e.g., GRASP(arm, cup), IN(cup, drawer)) covering four families: persistence, terminal placement, causal coupling, and ordering.
State Lifting & Evaluation: For each generated rollout, the system lifts the video into a state trace using vision foundation models (SAM3 for masks/trajectories, IDM for robot states, VLM for attribute checks). A finite-trace LTL monitor evaluates the rollout against $\Phi_c$ .
Output: The monitor returns a binary success label ( $r \in \{0, 1\}$ ) and, crucially, violation traces ( $V$ ) identifying exactly which constraints were violated, on which entities, and at which time steps.

2.2 Training Objectives

CreFlow converts the sparse binary feedback into localized, actionable supervision through two key loss functions:

Credit-Aware NFT Loss:
- Standard Negative-Aware Finetuning (NFT) applies updates uniformly. CreFlow aggregates the violation traces from a group of rollouts to construct a spatio-temporal mask ( $M$ ).
- This mask identifies the specific entities and frames responsible for task failure.
- The loss restricts the positive-negative velocity contrast to this mask, confining RL updates to reward-relevant regions. This prevents the model from perturbing unrelated, plausible regions of the video.
Corrective Reflow Loss:
- Existing methods often treat failed rollouts in isolation or use a reflected target based on an Exponential Moving Average (EMA) of the policy, which introduces noise.
- CreFlow aggregates within-group successful rollouts (generated under the same initial observation and instruction) to form an empirical mean ( $\bar{x}^+_0$ ).
- This mean serves as an explicit corrective target for failed rollouts on their violated regions. Instead of merely suppressing negative samples, the model is explicitly guided to move toward the distribution of successful outcomes within the same condition.

The final objective combines the credit-aware NFT loss, the corrective reflow loss, and a KL regularizer to a frozen reference policy.

3. Key Contributions

Compositional Constraint-Based Reward Model: The authors introduce a reward model that automatically formulates task requirements as LTL constraints. This provides not only a faithful binary success signal aligned with downstream execution but also localized diagnostic information (violations per constraint, entity, and frame) essential for sparse-reward learning.
CreFlow Framework: A novel online RL framework that transforms sparse compositional violations into localized corrective estimators. It uniquely combines:
- A credit-aware NFT loss that restricts updates to failure-relevant spatio-temporal regions.
- A corrective reflow loss that uses within-group positive samples as explicit correction targets, stabilizing training and reducing variance.
Empirical Validation: Extensive experiments on eight bimanual manipulation tasks (RoboTwin) demonstrate that CreFlow outperforms existing video-RL baselines and policy-level references.

4. Experimental Results

The paper evaluates CreFlow on eight RoboTwin bimanual manipulation tasks (e.g., pick-and-place, articulated manipulation, multi-object arrangement).

Downstream Success: CreFlow improves downstream execution success by 23.8 percentage points on average compared to the base Vidar model and significantly outperforms other RL post-training methods (DanceGRPO, DiffusionNFT) and reward baselines (EVA, WMReward).
Reward Fidelity: The compositional reward model achieves 88.0% agreement with human success labels and 91.0% agreement with simulator execution labels, substantially outperforming continuous reward models and VLM-as-judge baselines.
Training Efficiency: CreFlow converges faster and reaches higher accumulated rewards than baselines. Ablation studies confirm that both the credit-aware mask and the corrective reflow loss are necessary; removing either leads to performance degradation or unstable training.
Qualitative Analysis: Visual ablations show that without the credit-aware mask, standard RL updates degrade off-task regions (e.g., background objects), whereas CreFlow preserves the pretrained prior in non-critical areas while correcting task-specific failures.

5. Significance and Claims

The paper claims that CreFlow addresses a fundamental gap in embodied video generation: the mismatch between global visual plausibility and the logical, compositional requirements of physical manipulation.

Faithful Evaluation: By moving from low-level visual metrics to logic-based verification, CreFlow provides reward signals that are truly aligned with task success, avoiding the "hallucination" of success in videos that look good but fail physically.
Sparse Reward Handling: The framework demonstrates that sparse binary rewards can be effectively leveraged for high-dimensional video generation if the optimization is localized to the specific regions of failure and guided by explicit corrective targets derived from successful samples.
Practical Impact: The method enables the post-training of general-purpose video models for specific robotic tasks without requiring dense, hand-crafted rewards, improving execution success rates significantly.

Limitations: The authors acknowledge that the current monitor reasons over 2D image-plane states and cannot fully resolve fine-grained 3D spatial relations. They identify extending the constraint evaluation with depth or 3D state estimation as a direction for future work.