🤖 machine learning

What Demonstration Curation Metrics Do to Your Policy

This paper reveals that demonstration-curation metrics optimized for defect detection often fail to improve downstream policy performance due to confounding factors like episode length, arguing that curation methods must be evaluated by the quality of the resulting policy rather than their defect-flagging accuracy.

Original authors: Aarav Bedi

Published 2026-06-10

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Aarav Bedi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to pick up a bowl and carry it to a plate. You have a huge pile of video demonstrations showing how to do this. But here's the problem: 80% of those videos are "broken." In the broken ones, the robot drops the bowl halfway through the carry, then keeps moving its empty hand to the plate.

If you just show the robot all these videos, it learns to drop the bowl every time. It fails almost 100% of the time.

To fix this, you need a curator. A curator is a tool that looks at the videos, scores them, and throws away the bad ones so the robot only learns from the good ones.

This paper asks a very simple, but surprising question: Does the curator that is best at spotting the bad videos actually produce the best robot?

The answer is a resounding no. In fact, the paper finds that the curator who is best at spotting the bad videos often creates the worst robot.

Here is the breakdown of their findings using simple analogies:

1. The "Spotter" vs. The "Teacher"

The researchers tested seven different "curator tools."

The Spotter's Job: Look at a video and say, "This is bad!" or "This is good!" They measure how good a tool is at this by seeing how often it correctly flags the bad videos (like a test score).
The Teacher's Job: Actually train the robot using only the videos the curator kept.

The Shocking Result:
One tool was the "Star Spotter." It got a near-perfect score at identifying the bad videos. But when they used its list of "good videos" to train the robot, the robot still failed miserably.

Analogy: Imagine a strict teacher who is amazing at grading essays. They can perfectly identify which essays have typos. But if they throw away all the essays with any typos, they might accidentally throw away the brilliant essays that just happened to have one small typo, while keeping the mediocre essays that were perfectly typed but had no real ideas. The robot learns from the "perfectly typed but empty" essays and fails.

2. The "Length" Trick (The Hidden Cheat Code)

The researchers discovered that five of the seven tools were "cheating."

The Cheat: The bad videos (where the robot dropped the bowl) were longer than the good videos. The bad videos ran until the very last second because the robot kept moving after dropping the bowl. The good videos stopped early because the task was finished.
The Trick: Some tools didn't actually look at how the robot moved; they just looked at how long the video was. They thought, "Short video = Good. Long video = Bad."
The Fix: When the researchers cut all videos to the exact same length before testing, the "Star Spotters" suddenly became terrible at their jobs. Their scores dropped from near-perfect to random guessing.
Analogy: It's like a judge trying to pick the best marathon runners by only looking at how long they ran. If the losers kept running until they collapsed (long time) and the winners stopped when they crossed the finish line (short time), the judge would pick the winners just by picking the shortest runners. But if you force everyone to run the same distance, the judge can't tell who is actually fast anymore.

3. The "Good Enough" Winner

The tool that actually produced the best robot wasn't the one with the highest "Spotting Score." It was a tool that simply checked if the robot's overall path looked like the average path of a successful run.

This tool had a mediocre "Spotting Score," but it kept the right videos.
The robot trained with this tool succeeded 90% of the time, which is almost as good as if the researchers had magically known exactly which videos were perfect from the start (the "Oracle" score of 93.3%).

The Big Lesson

The paper argues that we have been looking at the wrong scoreboard.

Old Way: We pick the tool that is best at flagging bad data.
New Way: We should pick the tool that actually trains the best robot.

The Takeaway:
Just because a tool is great at finding the "bad apples" in a barrel doesn't mean it knows which "good apples" to keep for a pie. Sometimes, the tool that is best at finding the bad stuff accidentally throws away the best stuff, too.

To build a good robot, don't just ask, "How good is your detector?" Ask, "How good is the robot you built with your list?"

Technical Summary: Demonstration Curation Metrics Do to Your Policy

Problem Statement

In behavior cloning (BC), policy performance is fundamentally limited by the quality of the training demonstrations. While large-scale datasets collected in the wild often contain heterogeneous quality due to operator skill variations or scripted policy failures, the standard remedy—curation (scoring and discarding low-quality episodes)—lacks a unified evaluation framework. Existing literature proposes diverse quality metrics (e.g., trajectory smoothness, outlier detection) but validates them primarily on detection accuracy (how well a metric ranks clean vs. defective episodes).

This paper identifies a critical gap: detection accuracy does not necessarily correlate with curation value (how much the downstream policy improves when trained on the curated subset). A metric might excel at flagging unusual trajectories without successfully identifying those that are most harmful to the learned policy. Furthermore, the authors hypothesize that many existing benchmarks suffer from a methodological confound where metrics exploit episode length as a trivial proxy for defects, inflating reported performance.

Methodology

The authors designed a controlled testbed using the LIBERO pick-and-place benchmark (built on robosuite) to decouple detection accuracy from downstream policy performance.

Experimental Setup

Task: A contact-rich pick-and-place task involving four phases: approach, descend, lift, and transport.
Policy Architecture: A phase-conditioned behavior cloning policy (MLP with 2 hidden layers, 256 units) that takes 26-dimensional observations (end-effector position, gripper state, initial bowl position, and phase one-hot encoding) and outputs 4-dimensional actions.
Defect Injection: A structural defect was introduced at an 80% contamination rate. The defect consists of early gripper release during the LIFT phase (between 30% and 70% of the phase), causing the object to be dropped before transport.
- Clean Data: 16 scripted demonstrations (100% success).
- Defective Data: 64 demonstrations (0% success).
- Contaminated Baseline: Training on all 80 demonstrations yields only 3.3% task success.
- Oracle Ceiling: Training on only the 16 clean demonstrations yields 93.3% success.

Evaluation Protocol

The study evaluates seven curation metrics on two axes:

Detection: Area Under the ROC Curve (AUROC) of the metric's quality score against ground-truth success labels.
Downstream Performance: Task success rate of a BC policy trained on the top 75% of demonstrations (60/80) curated by each metric.

Critical Control: Episode Length Normalization

The authors identified that defective episodes in their setup run to the time limit (500 steps) because the robot continues moving after dropping the object, whereas successful episodes terminate early (~325 steps). To prevent metrics from exploiting this length difference as a proxy for defects, all demonstrations were truncated to T=324 steps before feature extraction. This ensures metrics rely on content-based signals rather than duration.

Key Contributions and Metrics Evaluated

The paper evaluates seven metrics, ranging from simple heuristics to complex ensemble methods:

Smoothness: Spectral Arc Length (SPARC) of action speed.
Entropy: Negative standard deviation of the action sequence.
Gripper Timing: Normalized timestep of the first gripper opening (specifically targets the injected defect).
Isolation Forest: Outlier detection in action summary feature space.
Ensemble: Weighted combination of Smoothness and Gripper Timing.
kNN: Negative mean distance to nearest neighbors in trajectory feature space.
Trajectory Alignment: Cosine similarity between a demonstration's mean state trajectory and the clean set's mean.

Results

1. Sharp Decoupling of Detection and Curation

The central finding is a lack of correlation (Spearman $\rho = -0.14$ ) between detection AUROC and downstream policy success.

The "Worst" Detector: Gripper Timing achieved the highest detection AUROC (0.804) but produced the worst policy (13.3% success). It failed because it retained defective demonstrations that opened the gripper later (but still too early) while discarding clean demonstrations that happened to have slightly earlier closures.
The "Best" Detector: Trajectory Alignment had a low AUROC (0.638) but produced a policy with 90.0% success (within 3.3 percentage points of the Oracle). It succeeded because the structural defect (dropping the bowl) caused a large deviation in the mean state trajectory, allowing even an imprecise metric to separate the most harmful episodes.
The Oracle: The Ensemble method achieved 91.1% success with near-zero variance, demonstrating that combining global motion quality with phase-localized defect indicators is highly effective.

2. The Episode-Length Confound

Before truncation, five of the seven metrics (including Isolation Forest and kNN) achieved near-perfect AUROC (1.000) simply by detecting the difference in episode length (325 vs. 500 steps). After truncation:

Length (cumulative path): Dropped from 1.000 to 0.500 (chance).
Isolation Forest: Dropped from 1.000 to 0.440.
Gripper Timing: Remained high (0.804) because it is content-aware and length-normalized.
This confirms that many reported "high-performing" metrics in literature may be measuring episode duration rather than demonstration quality.

3. Variance and Reliability

Metrics like kNN and Smoothness showed high variance in downstream success (e.g., kNN $\pm 41.7\%$ ), indicating that their curation decisions are unstable across random seeds. In contrast, Ensemble and Trajectory Alignment showed near-zero variance, making them more reliable for single-run curation pipelines.

Significance and Claims

The paper argues for a paradigm shift in how demonstration curation is evaluated:

Policy Performance is the Only Metric: Detection AUROC is a poor predictor of curation value. The "right question" is not which metric best detects defects, but which metric produces the best policy.
Methodological Rigor: Any curation benchmark must control for episode length. If defective and successful episodes differ systematically in duration, detection accuracy metrics are likely measuring a confound rather than true quality.
Practical Guidance: Practitioners should prioritize metrics that yield low-variance, high-success policies (like Trajectory Alignment or Ensembles) over those with high detection AUROC but poor downstream results.

The authors conclude that the field must move away from isolated detection benchmarks toward end-to-end policy evaluation to ensure curation methods actually improve imitation learning outcomes. The testbed, metric implementations, and evaluation pipeline are released to facilitate this shift.