Auditing Demonstration Curation Metrics: Action-Only Scorers Fail on the Structural Defects That Degrade Imitation Policies

Technical Summary: Auditing Demonstration Curation Metrics

Problem Statement
Imitation learning policies, particularly behavior cloning, are fundamentally limited by the quality of the demonstrations they are trained on. While large datasets are often curated using automatic metrics to filter low-quality data, there is no consensus on which metrics actually improve downstream policy performance. Existing literature validates metrics on disparate datasets with different protocols, creating a gap between a metric's ability to identify "unusual" demonstrations and its ability to identify demonstrations that actively degrade a policy. Furthermore, it remains unclear whether high detection accuracy translates to improved task success.

Methodology
The authors introduce a controlled testbed to audit seven existing demonstration curation metrics. The study isolates two distinct variables: the detection capability of a metric (how well it ranks defective demonstrations lower than clean ones) and the downstream impact (whether training on the curated subset improves task success compared to training on the uncurated, contaminated set).

• Environment: A lightweight NumPy-based pick-and-place simulator modeled on the ALOHA single-arm setup. The task involves grasping an object and moving it to a goal region.
• Defect Injection: The study introduces two regimes of defects at a 40% contamination rate, keeping defect labels hidden from the curation metrics:
  1. Subtle Perturbations: Correlated action noise (AR(1)), high-frequency tremor, early truncation, and trajectory detours. These represent statistical deviations from correct behavior.
  2. Structural Errors: An "early release" defect where the gripper opens prematurely during the carry phase, causing the object to drop. This represents a systematic, state-dependent wrong action.
• Metrics Audited: Seven metrics were tested, categorized by their input data:
  • Action-Only: Smoothness (spectral arc length), Entropy (action variance), Length, Isolation Forest, and an Ensemble of these.
  • State-Trajectory-Aware: k-Nearest Neighbors (kNN) in a trajectory feature space, and Trajectory Alignment.
• Evaluation Protocol:
  • Detection: Measured via Area Under the ROC Curve (AUROC) against hidden labels.
  • Downstream: Policies were trained on the top-scoring fraction of data (fixed to $N=150$ demonstrations to control for quantity) and evaluated over 50 fresh rollouts. Results were averaged over three random seeds.

Key Results
The study reveals a sharp divergence between detection accuracy and downstream utility, heavily dependent on the type of defect:

1. Subtle Perturbations: These are highly detectable and recoverable. Multivariate outlier detection (Isolation Forest) achieved an AUROC of 0.97. When used for curation, it nearly fully recovered the performance gap between the contaminated set and the oracle clean set (71.3% vs. 72.0% success).
2. Structural Errors: These are largely invisible to action-only metrics.
   • Inversion: Several action-only metrics (Smoothness, Entropy, Ensemble) performed worse than chance (AUROC < 0.5) on structural defects. Notably, Entropy scored defective demonstrations as higher quality because the structural error increased action variance, mimicking "exploration."
   • Downstream Harm: Using these inverted metrics for curation resulted in policies that performed at or below the uncurated baseline (e.g., Smoothness dropped to 27.3% success vs. 36.0% for the contaminated set).
   • Partial Recovery: Only state-trajectory-aware metrics (kNN and Trajectory Alignment) detected structural errors. However, even the best performer (kNN) recovered only about one-third of the performance gap (48.0% vs. 62.7% oracle), demonstrating that detection does not guarantee recovery.
3. Decoupling of Detection and Recovery: Two metrics with similar detection AUROCs for structural defects (kNN: 0.86, Trajectory Alignment: 0.76) yielded vastly different downstream results. kNN recovered significant performance, while Trajectory Alignment barely exceeded the baseline. This suggests that which clean demonstrations are discarded as collateral damage is as critical as the ranking quality.

Significance and Claims
The paper's primary contribution is the audit and the testbed itself, rather than a new curation algorithm. The authors claim three main findings:

• Regime Dependency: Curation metrics are not universally applicable; multivariate outlier scoring works for subtle noise, but only state-trajectory analysis can address structural errors.
• Risk of Action-Only Metrics: Metrics that ignore state trajectories can be actively harmful, potentially degrading policy performance below the uncurated baseline by filtering out valid data while retaining defective data.
• Evaluation Standard: Detection accuracy (AUROC) is a necessary but insufficient condition for a useful curation metric. The only valid judge of a curation method is the performance of the policy trained on its curated subset.

The authors conclude that practitioners must match the curation metric to the expected defect type and emphasize that structural errors, which are the most damaging, require state-aware analysis. The paper acknowledges limitations, including the use of a simplified simulator, injected rather than real-world defects, and a limited number of random seeds, suggesting that while the qualitative findings hold, specific numerical gaps may vary in more complex, physics-rich environments.