Evaluation-Recording Contamination in Learned Nano-Quadrotor Dynamics: A Fresh-Seed Audit

Technical Summary: Evaluation-Recording Contamination in Learned Nano-Quadrotor Dynamics

Problem Statement
Data-driven dynamics models for aerial vehicles are typically trained on short windows extracted from longer flight recordings. A critical methodological flaw arises when these windows are randomly assigned to fitting and evaluation sets. Because adjacent samples within a single flight recording share vehicle state, control history, disturbances, and sensor conditions, a random split can place strongly dependent samples from the same physical flight into both the training and evaluation sets. This creates "evaluation leakage," where the model appears to generalize well simply because it has seen the specific physical trajectory during training, rather than demonstrating true generalization to a new, unseen flight. This issue is particularly consequential in flight dynamics, where low one-step prediction errors can mask significant drift or divergence during recursive rollout.

Methodology
The authors conduct a "fresh-seed audit" using a pinned snapshot of the NanoBench Crazyflie 2.1 dataset to isolate the effect of recording-level leakage. The study employs a controlled experimental design with the following specifications:

• Dataset: 106 usable recordings from the NanoBench dataset, with 26 fixed as evaluation recordings and the remainder used for fitting.
• Experimental Arms:
  • Recording-Disjoint Arm (Safe): Uses 16,000 fitting windows drawn exclusively from non-evaluation recordings.
  • Contaminated Arm: Uses the same 16,000-window capacity but replaces 25% (4,000 windows) with windows drawn from the evaluation recordings. Crucially, exact rollout start indices are excluded from the contamination pool, though 90.5% of the contaminating windows still fall within 2 seconds of an evaluated start.
• Controls: Both arms share an identical 2,000-window validation set (drawn from non-evaluation recordings) to prevent validation leakage. They use the same optimization budget, architecture, and early stopping rules.
• Models: Two Multilayer Perceptrons (MLPs) are tested:
  1. World-Coordinate Model: Inputs include standardized world position, velocity, quaternion, reference, and battery voltage.
  2. Relative-Coordinate Model: Replaces absolute position with zero and reference with tracking error to test if coordinate representation moderates the leakage effect.
• Inference Protocol: The experiment runs 20 fresh, pre-committed training seeds (100–119) across all 26 evaluation recordings. This creates a crossed design (20 seeds × 26 recordings = 520 cells) to account for both training randomness and flight-specific heterogeneity.
• Metrics: The primary endpoint is the 1-second failure-aware position Root-Mean-Square Error (RMSE). Unlike standard RMSE, this metric retains divergent trials (where error exceeds 2m) rather than discarding them, preventing the artificial inflation of performance by excluding difficult cases.
• Statistical Analysis: A two-way random-effects model decomposes variance across recordings and seeds. The primary confirmatory decision rule requires both a point estimate showing at least 10% relative optimism (lower error in the contaminated arm) and a 95% confidence interval for the difference that lies entirely below zero.

Key Contributions

1. Machine-Audited Comparison: A direct comparison differing only in the access to evaluation recordings during the fitting phase, holding all other variables constant.
2. Failure-Aware Metrics: Introduction of a scoring method that penalizes divergence, ensuring that performance gains are not driven by the silent removal of failed rollouts.
3. Crossed Inference: A rigorous statistical approach that treats both physical recordings and training seeds as random factors, avoiding pseudoreplication.
4. Interaction Testing: A direct difference-in-differences test to determine if coordinate representation (world vs. relative) moderates the leakage effect, rather than relying on separate significance tests for each model.
5. Reproducible Protocol: A transparent record of the transition from an exploratory phase (which showed strong positive results) to a locked confirmatory phase, including the negative outcome of the latter.

Results
The confirmatory experiment yields a negative result under the pre-registered decision rule:

• Point Estimates: The contaminated arm consistently shows lower error than the disjoint arm. For the world-coordinate model at 1 second, the error drops from 0.299 m to 0.262 m, a 12.1% reduction. This exceeds the 10% effect-size threshold.
• Statistical Significance: The 95% crossed confidence interval for the difference (contaminated minus disjoint) is [-0.0735, 0.0011] m. Because this interval crosses zero, the result is not statistically significant according to the study's strict criteria.
• Secondary Findings:
  • At 0.5s and 2.0s horizons, the world-model intervals exclude zero, suggesting a stronger effect at those specific times.
  • The relative-coordinate model also shows lower error in the contaminated arm, but the direct interaction test (comparing the leakage gap between world and relative models) yields intervals that cross zero. This indicates that coordinate representation does not significantly moderate the leakage effect.
  • The primary conclusion remains negative despite the practical magnitude of the point estimate.

Significance and Claims
The paper does not claim to have proven a specific "leakage penalty" magnitude, but rather demonstrates that leakage claims in flight modeling require recording-level splits and fresh-seed replication to be confirmed.

• Methodological Rigor: The study highlights that large apparent performance gaps observed in single-run or poorly controlled experiments may not hold up under strict recording-level separation and randomized seed replication.
• Benchmark Construction: The authors argue that for applications targeting future flights, benchmarks must split data at the recording level before window generation. Preprocessing, hyperparameter tuning, and early stopping must rely solely on training-side recordings.
• Evaluation Standards: The paper advocates for "failure-aware" metrics that do not discard divergent trials and for paired, multi-seed comparisons to be treated as a core design element rather than a cosmetic error bar.
• Modest Conclusion: The authors conclude that while the data suggest a practically relevant signal of optimism due to leakage, the signal remains uncertain under a controlled recording-level analysis. The study provides a reusable protocol to make this uncertainty visible, preventing the progression from exploratory "go" results to selectively reported confirmatory claims.