🤖 machine learning

RoDiF: Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedback

This paper introduces RoDiF, a robust fine-tuning method for diffusion policies that leverages a Unified MDP formulation to enable reward-free Direct Preference Optimization and employs a geometric hypothesis-cutting strategy to effectively handle corrupted human feedback in long-horizon robotic tasks.

Original authors: Amitesh Vatsa, Zhixian Xie, Wanxin Jin

Published 2026-02-03

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Amitesh Vatsa, Zhixian Xie, Wanxin Jin

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Teaching a Robot with a Noisy Teacher

Imagine you have a very talented robot chef who learned to cook by watching a master chef on TV. The robot is great at following the recipe (this is the Diffusion Policy). However, sometimes the robot makes dishes that are "okay" but not exactly what you want. Maybe you prefer your steak well-done, but the robot keeps making it medium-rare.

To fix this, you want to teach the robot your specific taste. You do this by showing it two meals and saying, "I like this one better" (this is Human Feedback).

The Problem:

The Robot's Brain is Complicated: Unlike a simple recipe, this robot chef doesn't just pick a dish; it "cooks" the dish in many tiny, invisible steps (like slowly removing noise from a blurry photo until it becomes clear). Standard teaching methods get confused by this multi-step process.
The Teacher is Tired: Humans make mistakes. Sometimes you are tired, or you change your mind, or you accidentally label a bad meal as "good." If the robot tries to learn from these mistakes, it might start cooking terrible food, thinking that's what you actually wanted.

The Solution: RoDiF
The authors created a new method called RoDiF (Robust Direct Fine-Tuning). Think of it as a super-smart teaching assistant that helps the robot learn from a teacher who might be making mistakes.

How RoDiF Works: Three Simple Steps

1. The "Unified Map" (Unified MDP)

Usually, the robot's internal cooking steps and the real-world kitchen environment are treated as two separate things. This makes it hard to apply standard teaching rules.

The Analogy: Imagine trying to navigate a city while also trying to solve a puzzle inside your car at the same time. It's confusing.
The Fix: RoDiF draws a single, giant map that combines the puzzle-solving steps and the driving steps into one continuous journey. This allows the robot to apply a powerful new teaching rule (called DPO) directly to its complex brain without getting lost.

2. The "Hypothesis Cutting" Game

The core idea of RoDiF is about how the robot processes your feedback.

The Analogy: Imagine the robot has a giant box of thousands of different "personality settings" (hypotheses). Every time you say, "I like Meal A more than Meal B," it's like taking a knife and cutting off a huge chunk of the box that contains all the personalities that would have chosen Meal B.
The Goal: You keep cutting away the "wrong" personalities until only the "right" one is left in the box.

3. The "Conservative Cutter" (Handling Mistakes)

This is the most important part. What if you accidentally cut off the right personality because you made a mistake?

The Problem: In standard methods, one bad cut can destroy the whole box. If you accidentally say "I hate the good meal," the robot might throw away its best personality forever.
The RoDiF Fix: RoDiF uses a Conservative Strategy. Instead of requiring every single cut to be perfect, it asks: "If I ignore the worst 20% of the cuts (the likely mistakes), does the right personality still survive?"
The Result: Even if 30% of your feedback is wrong (corrupted), RoDiF looks at the "votes" from all your feedback. It ignores the noisy, conflicting votes and keeps the personality that satisfies the majority of the reliable cuts. It's like a judge who ignores the screaming, angry crowd and listens only to the calm, consistent witnesses.

What Did They Prove?

The authors tested this on robots doing long, complicated tasks like stacking blocks, sorting items, and pushing objects.

Better Alignment: They showed that RoDiF could successfully steer the robot to do exactly what the human wanted (e.g., "Stack the red block first"), even when the robot naturally wanted to do something else.
Superior Noise Resistance: When they intentionally messed up 30% of the feedback labels (telling the robot the wrong thing), standard methods failed and the robot got confused. RoDiF, however, kept performing well, almost as if it could "see through" the lies.
Works on Different Brains: They tested this on three different types of robot "brains" (architectures), and it worked for all of them.

Summary

RoDiF is a new way to teach robots using human preferences. It solves two big problems:

It figures out how to teach robots that think in complex, multi-step ways.
It is tough enough to ignore human mistakes. It doesn't panic when you give it bad advice; instead, it filters out the noise and learns the right lesson anyway.

This makes it much safer and more reliable to use human feedback to train robots for real-world jobs.

Technical Summary: RoDiF – Robust Direct Fine-Tuning of Diffusion Policies with Corrupted Human Feedback

1. Problem Statement

Diffusion policies have emerged as a powerful paradigm for robotic control, offering superior capabilities in capturing multi-modal distributions and handling high-dimensional action spaces compared to traditional Gaussian or deterministic policies. However, fine-tuning these pre-trained policies to align with specific human preferences presents two fundamental challenges:

Structural Mismatch: Unlike autoregressive language models, diffusion policies operate via a multi-step denoising chain nested within the environmental Markov Decision Process (MDP). Existing Direct Preference Optimization (DPO) methods do not natively account for this dual temporal structure (internal denoising steps vs. external environmental dynamics), making the derivation of a tractable DPO objective non-trivial.
Corrupted Human Feedback: Human feedback in robotic control is often stochastic and contains inconsistencies beyond standard noise assumptions (e.g., label flips, annotator disagreement, or non-stationary criteria). Naively fine-tuning on such data causes models to overfit to conflicting comparisons, effectively optimizing for erroneous constraints rather than human intent, which degrades both alignment and task performance.

2. Methodology

The proposed framework, RoDiF (Robust Direct Fine-Tuning), addresses these challenges through a unified theoretical formulation and a geometrically grounded optimization strategy.

2.1 Unified MDP Formulation

To enable DPO for diffusion policies, the authors introduce a Unified MDP that coherently integrates the diffusion denoising chain with environmental dynamics.

State and Action Definition: The unified state $s^+$ combines the current environment state $s_t$ and the initial noise vector $a^K_t$ . The unified action $a^+$ represents the sequence of denoising steps $(a^{K-1}_t, \dots, a^0_t)$ .
Transition Dynamics: The transition function flattens the process, treating the random noise injection and the environment transition as a single step.
Policy Factorization: The unified policy $\pi_\theta(a^+|s^+)$ is factorized into the product of the conditional denoising probabilities at each step $k$ , allowing the application of standard DPO reparameterization tricks to the diffusion process without explicit reward modeling.

2.2 Geometric Hypothesis-Cutting Perspective

RoDiF reinterprets the DPO objective through a geometric lens.

Hypothesis Space: The parameter space of the diffusion policy is viewed as a hypothesis space $\Theta$ .
Preference as Cuts: Each human preference pair (winning trajectory $\tau_w$ vs. losing trajectory $\tau_l$ ) induces an inequality constraint (a "cut") on the parameter space. A valid policy parameter must satisfy $\Delta Q_\theta \geq 0$ for the preferred pair.
The Noise Problem: In the presence of corrupted labels, a single incorrect preference can create a "false cut" that excludes the true optimal parameter $\theta_H$ from the intersection of all cuts, irreversibly derailing the learning process.

2.3 Conservative Voting Strategy

To achieve robustness without assuming a specific noise distribution, RoDiF employs a conservative cutting strategy:

Voting Mechanism: Instead of requiring a parameter to satisfy all preference cuts (which fails if any label is corrupted), the method treats each cut as a vote.
Conservative Threshold: A conservativeness level $\gamma \in [0, 1]$ is introduced, representing an estimated upper bound on the corruption rate. The optimization objective is relaxed to retain parameters that satisfy at least $\lfloor(1-\gamma)N\rfloor$ cuts.
Robust Loss Function: The standard DPO loss is modified to a robust loss $L_{RoDiF}$ . The non-differentiable Heaviside step function (used to count votes) is softened using a sigmoid function to enable gradient-based optimization. This allows the policy to ignore up to $\gamma N$ inconsistent labels while still converging toward the true human-aligned policy.

3. Key Contributions

Unified MDP Formulation: A novel formulation that integrates diffusion denoising steps with environmental transitions, enabling the derivation of a tractable, reward-free DPO objective for diffusion policies.
RoDiF Algorithm: A geometrically grounded algorithm that robustly fine-tunes diffusion policies by leveraging conservative hypothesis cutting. It mitigates the impact of inconsistent preference labels without requiring prior knowledge of the noise distribution.
Empirical Validation: Extensive experiments demonstrating that RoDiF significantly outperforms state-of-the-art baselines (including standard DPO, CPO, IPO, and SimPO) on high-dimensional, long-horizon manipulation tasks, maintaining high alignment performance even under 30% corrupted preference labels.

4. Experimental Results

The method was evaluated on five long-horizon manipulation tasks (Avoid, Pushing, Sorting, Aligning, Stacking) using three different diffusion backbones (UNet, Transformer, MLP).

Alignment Effectiveness: RoDiF successfully steered pre-trained policies toward user-specified modes, even when those modes were non-dominant in the pre-trained policy (e.g., achieving >84% mode alignment in the "Avoid" task where the baseline was <21%).
Robustness to Noise:
- Under 0% corruption, RoDiF performed comparably to standard DPO.
- Under 20% and 30% corruption, standard DPO and other baselines (like SimPO) degraded significantly (e.g., SimPO success rate dropped from 43% to 10% under 20% noise).
- RoDiF maintained high performance (>84% success rate) across all noise levels, effectively isolating the steering signal from systematic label noise.
Architecture Agnosticism: The method proved effective across diverse diffusion architectures (UNet, Transformer, MLP).
Sample Efficiency: The framework demonstrated high sample efficiency, achieving strong alignment with as few as 10 winner-loser pairs in specific tasks.

5. Significance and Claims

The paper claims that RoDiF advances the safety and reliability of preference-based robot learning. By explicitly mitigating the effects of erroneous or inconsistent human feedback, the framework reduces the risk of brittle or unstable policy updates. This is presented as a critical step toward deploying robotic systems in real-world, safety-sensitive environments where human annotators may provide imperfect data. The work emphasizes robustness as a foundational requirement for responsible deployment, offering a solution that does not rely on perfect data or complex reward modeling.