Phase-Localized Curation Does Not Help: A Negative Result on Per-Phase Metric Selection for Demonstration Filtering

Technical Summary: Phase-Localized Curation Does Not Help

Problem Statement
Demonstration curation for behavior cloning is critical when training on mixed-quality data, yet recent work has established a counterintuitive decoupling between detection accuracy and curation value: a metric with the highest defect-detection AUROC does not necessarily yield the best policy when used as a filter. A natural hypothesis to remedy this is phase-localized curation. Since manipulation tasks possess temporal phase structures (e.g., approach, grasp, lift, transport) and defects often localize to specific phases, it is plausible that scoring each phase with its locally most informative metric and aggregating the results would outperform global scoring. This paper tests the hypothesis that segmenting trajectories into phases and applying per-phase metric selection improves curation outcomes compared to uniform or global strategies.

Methodology and Testbed
The authors evaluate this hypothesis on three contact-rich pick-and-place tasks from the LIBERO benchmark, built on the robosuite simulator.

• Defect Model: A controlled structural "early-release" defect is injected, where the gripper opens prematurely during the LIFT phase (30–70% of the phase duration), causing the object to drop. This defect is localized to a single phase and does not average out across the dataset.
• Data Setup: Training sets consist of 80 demonstrations (80% contamination rate: 64 defective, 16 clean). Policies are trained on the top 75% of curated demonstrations (60 trajectories).
• Policy: A phase-conditioned behavior-cloning MLP (2 hidden layers, 256 units) trained for 300 epochs.
• Curation Strategies Compared:
  1. Global: A single metric is computed over the entire truncated trajectory to rank all demonstrations.
  2. Uniform: A single metric is applied identically to every phase, and per-phase scores are aggregated. This isolates the effect of aggregation machinery from metric variation.
  3. Phase-Gated (Proposed): Trajectories are segmented into four phases (PREGRASP, DESCEND, LIFT, TRANSPORT). For each phase, a sweep identifies the locally most discriminative metric. These per-phase scores are rank-normalized and averaged to produce a final curation ranking.
• Evaluation: Success rates are measured over 30 rollouts across five random seeds per condition.

Key Results
The study yields a consistent negative result: Phase-gated curation is never the best strategy and is often the worst.

• Performance: Across all three tasks, phase-gated curation failed to outperform both the uniform baseline and the global baseline.
  • Task 1: Phase-gated (86.0%) was the worst, trailing the global (92.0%) and uniform (91.3%) strategies.
  • Task 3: Phase-gated (22.7%) was the worst, significantly underperforming the uniform (48.0%) and global (30.7%) strategies.
  • Task 2: Phase-gated (78.0%) beat the global metric (58.0%) but lost to the uniform baseline (83.3%), indicating that the improvement came from using a better metric, not the phase decomposition itself.
• Transferability: The per-phase metric selection does not transfer across tasks. Table III shows that no phase shares the same "winning" metric between any two tasks, necessitating a new, noisy, and expensive sweep for every new task.

Mechanism of Failure: Signal Dilution
The paper identifies signal dilution under rank-aggregation as the concrete mechanism for failure.

• The defect signal is concentrated entirely in the LIFT phase. The metric sensitive to this defect (gripper timing) carries a clean, high-weight signal when applied globally or uniformly.
• In the phase-gated approach, the LIFT phase is scored correctly, but the other three defect-free phases are scored by metrics selected on noise (as there is no defect to detect in those phases).
• When the ranks are aggregated, the concentrated, informative signal from the LIFT phase is diluted by the uninformative, noisy ranks from the other phases. This allows defective demonstrations (which look normal in non-defect phases) to rank higher than clean demonstrations (which may look slightly anomalous in defect-free phases).
• A confidence threshold fallback was tested but failed to rescue the method because the defect-free phases were "confidently" selected as irrelevant, and the dilution mechanism remained active.

Significance and Contributions
The paper makes four primary contributions:

1. Empirical Bound: It provides a controlled, multi-task test showing that phase-gated curation does not improve upon simpler strategies and is often inferior.
2. Mechanistic Explanation: It explains why the method fails (signal dilution), predicting that phase decomposition is detrimental when defects are concentrated in a single phase.
3. Lack of Transfer: It demonstrates that per-phase metric selection is not reusable across tasks, removing the primary practical argument for adopting the method.
4. Practical Guidance: The authors argue that practitioners should prioritize identifying a single defect-informative metric and applying it uniformly, rather than decomposing curation by phase.

Limitations and Scope
The authors explicitly bound their claims to the specific setting of structural defects concentrated in a single phase, high contamination rates, and contact-rich tasks. They do not claim that phase-gating fails for defects that are genuinely distributed across multiple phases, noting that such distributed defects are less common in practice (e.g., dropped objects, missed grasps). The results rely on consistent ordering across seeds rather than mean differences alone, given the high variance observed in behavior cloning with small curated sets.