Technical Summary: Robust Trajectory Distillation

Problem Statement

Dataset Distillation (DD) aims to condense large-scale datasets into compact, information-rich synthetic subsets to enable efficient model training and reuse. However, existing DD frameworks implicitly assume clean supervision. In real-world scenarios characterized by noisy labels (due to crowdsourcing errors, human bias, or semantic ambiguity), standard DD methods risk condensing corrupted associations alongside useful signals, leading to degraded robustness and generalization.

Current noisy-label learning techniques (e.g., sample selection, loss weighting, label correction) often suffer from three limitations when applied to DD:

1. Tight Coupling: They tightly couple noise estimation with model optimization, creating a self-referential feedback loop that amplifies confirmation bias.
2. Dependency on Clean Anchors: Many require clean validation subsets or heavy iterative refinement, limiting scalability.
3. Capacity Constraints: Distilled datasets, typically parameterized as fixed-size image tensors, restrict representational capacity, causing premature information compression before noise and clean signals can be disentangled.

The core challenge is to improve both the quality (fidelity of synthetic data) and capacity (amount of high-quality supervision absorbed) of distilled data under noisy supervision without requiring relabeling or clean subsets.

Methodology

The authors propose Robust Trajectory Distillation, a framework that jointly suppresses noise and preserves transferable knowledge through two complementary modules integrated into a trajectory-matching DD pipeline. The framework operates via a bilevel optimization scheme where a student model trained on the synthetic dataset $S$ is optimized to match the learning dynamics of a teacher model trained on the noisy real dataset $D_{real}$.

1. Selective Guidance Reweighting (SGR)

SGR refines the teacher trajectory by assigning sample-specific weights based on reliability estimates, ensuring the teacher provides cleaner guidance signals. It employs a hybrid estimator combining:

• Dynamic Consistency ($W_{knn}$): Based on K-Nearest Neighbors (KNN) in the feature space. It measures the agreement between a sample's observed label and the predicted labels of its neighbors using Jensen-Shannon divergence.
• Static Prior ($W_{ssft}$): Derived from "Second-Split Forgetting Time" (SSFT). It quantifies sample difficulty by recording the earliest epoch a sample is correctly classified ($t_{learn}$) and the earliest epoch it is subsequently misclassified ($t_{forget}$). Samples learned late and forgotten early are assigned lower weights.

These scores are fused via a time-dependent convex interpolation ($W_t$) that transitions from static priors to dynamic consistency as training progresses. This progressive reweighting prioritizes clean supervision along the teacher trajectory.

2. Teacher-Inspired Auxiliary Targets (TIAT)

While SGR improves the teacher's signal quality, TIAT addresses the student's absorption of these signals. It injects auxiliary regularization to ensure the distilled dataset remains aligned with clean supervision in the representation space.

• Probabilistic Confidence Aggregation: The framework aggregates reliability weights across multiple teacher trajectories to estimate a sample's overall confidence.
• Uncertainty-Aware Selection: A reliable subset of samples ($D_{sub}$) is selected based on high static reliability and low variance in confidence scores across trajectories.
• Auxiliary Loss: The teacher model is fine-tuned on $D_{sub}$ to produce a "cleaner" checkpoint ($\theta^{ft}$). An auxiliary loss is computed to align the student's trajectory with this refined checkpoint, in addition to the standard trajectory matching loss.

The total objective combines the standard trajectory alignment loss ($\mathcal{L}_{match}$) and the auxiliary loss ($\mathcal{L}_{aux}$), balanced by a coefficient $\beta$.

Key Contributions

1. Progressive Hybrid Reweighting: The introduction of a mechanism that integrates dynamic (KNN-based) and static (forgetting-time-based) sample reweighting. This fuses complementary teacher-trajectory signals to robustly suppress label noise without relabeling.
2. Auxiliary Guidance Regularization: A novel module (TIAT) that enforces consistency with clean trajectories during distillation. By utilizing uncertainty-aware selection and auxiliary targets derived from intermediate teacher dynamics, it amplifies the influence of clean signals and mitigates error amplification.
3. Label-Preserving and Lightweight Design: The framework requires neither relabeling nor costly retraining schedules, making it practical for large-scale, real-world noisy datasets.
4. Empirical Validation: Extensive experiments demonstrate consistent improvements over state-of-the-art DD baselines (e.g., DATM, DANCE) across symmetric noise, asymmetric noise, and real-world noisy datasets (CIFAR-N).

Experimental Results

The method was evaluated on CIFAR-10, CIFAR-100, CIFAR-N, and Tiny-ImageNet under symmetric (20%, 40%) and asymmetric noise regimes.

• Performance Gains: The proposed method consistently outperforms strong baselines. For instance, on CIFAR-10 with 40% symmetric noise and 1000 IPC, it achieved 81.4% accuracy compared to DATM's 76.7%. Under 40% asymmetric noise, it reached 75.0% (vs. DATM's 71.4%).
• Real-World Noise: On CIFAR-N (human-annotated noise), the method showed superior resilience, particularly as the noise rate increased (e.g., "Worst" case with ~40% noise), maintaining a margin of over 3% improvement over the best baseline at higher IPCs.
• Ablation Studies:
  • Adding SGR to the baseline provided stable gains across all noise types, with significant improvements in severe noise scenarios (e.g., +4.7% at 10 IPC with 40% noise).
  • Adding TIAT further boosted performance, particularly in high-noise regimes, confirming the benefit of uncertainty-filtered auxiliary supervision.
  • Diverse Sampling: Training multiple teacher trajectories with varying static-dynamic balance coefficients ($\alpha_{max}$) proved more robust than fixed sampling strategies.
• Efficiency: The method introduces negligible additional peak GPU memory. The computational overhead is moderate (e.g., ~1.5x runtime on CIFAR-10 with 500 IPC), primarily due to reliability estimation and auxiliary target construction.

Significance

The paper positions this work as a solution to the underexplored problem of dataset distillation under noisy supervision. By decoupling noise estimation from the optimization loop through trajectory-based reweighting and auxiliary regularization, the framework offers a robust, plug-and-play approach for generating high-quality synthetic datasets from imperfect data. The authors claim this paves the way for future research in noise-resilient dataset distillation, enabling efficient model retraining and data sharing even when clean supervision is unavailable.