DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models

Technical Summary: DASH: Dual-Branch Score Distillation for Guidance-Calibrated Compact Diffusion Models

Problem Statement
Class-conditional diffusion models utilizing Classifier-Free Guidance (CFG) rely on a single network to produce both conditional ($\epsilon_y$) and unconditional ($\epsilon_\emptyset$) predictions. The guidance mechanism amplifies the gap $\Delta = \epsilon_y - \epsilon_\emptyset$ to steer generation. However, compressing these models via parameter reduction introduces a critical structural failure in existing distillation objectives.

Standard distillation supervises only the composite guided prediction ($\tilde{\epsilon}_S \approx \tilde{\epsilon}_T$) or the conditional branch alone. This creates an underdetermined optimization problem: multiple decompositions of the student's branches can satisfy the loss function, including a degenerate solution where the guidance gap collapses ($\Delta_S \to 0$). In this state, the conditional and unconditional predictions converge, rendering the guidance weight $w$ ineffective at inference, even if the composite training loss is minimized. Furthermore, existing methods fail to transfer the teacher's learned per-timestep training curriculum (importance weighting), forcing the student to relearn this schedule within limited distillation budgets, often corrupting the optimal training trajectory.

Methodology: DASH Framework
The paper proposes DASH (Dual-Branch Score Distillation), a framework designed to resolve underdetermination and preserve guidance calibration through two primary mechanisms:

1. Dual-Branch Independent Supervision:
   Instead of matching the composite output, DASH independently supervises both score branches using distinct loss terms:

   • Imitation Loss ($L_{im}$): Matches the conditional branch ($\epsilon^y_S \approx \epsilon^y_T$).
   • Unconditional Loss ($L_{un}$): Matches the unconditional branch ($\epsilon^\emptyset_S \approx \epsilon^\emptyset_T$).
   • Anchor Loss ($L_{an}$): A lightweight regularizer anchoring the conditional branch to the ground-truth noise ($\epsilon$) to prevent drift during early training.

   By enforcing constraints on both branches simultaneously, the framework uniquely specifies the target outputs for each sample, eliminating the feasible solution space that allows for guidance gap collapse. The authors define a guidance gap ratio $\rho = E[\|\Delta_S\|] / E[\|\Delta_T\|]$ to monitor calibration, where $\rho \approx 1$ indicates successful preservation.

2. TIRT Transfer (Timestep-Importance Rebalanced Training Transfer):
   The teacher model utilizes TIRT, an extension of Min-SNR weighting that learns per-timestep scaling factors ($\lambda_t$) to concentrate training on content-difficult mid-trajectory regions. DASH introduces TIRT Transfer, which copies the teacher's converged per-timestep importance curriculum ($\hat{\omega}_T$) directly into the student as a frozen prior. This prevents the student from corrupting the curriculum with distillation gradients that reward prediction-matching ease rather than content difficulty.

Key Contributions

• Resolution of Structural Underdetermination: The paper identifies and solves the degenerate collapse of the guidance gap in parameter-compressed diffusion models. It demonstrates that unconditional supervision is not merely beneficial but essential, accounting for over 60% of the total distillation gain.
• Curriculum Transfer: The introduction of TIRT Transfer, which initializes the student with the teacher's learned timestep weighting as a frozen prior, eliminating the need for the student to relearn the curriculum.
• Dual-Branch Framework: A unified distillation pipeline that independently constrains conditional and unconditional branches, validated as empirically essential for maintaining guidance fidelity under compression.

Experimental Results
Experiments were conducted on CIFAR-10 and CIFAR-100 using a teacher model (35.8M parameters) and a compressed student (6.1M parameters, 5.9× compression).

• Performance: The DASH student achieves an FID of 8.87 (CIFAR-10) and 10.47 (CIFAR-100) at 50-step DDIM sampling. This is within 4 FID points of the teacher (FID 5.47 / 6.80) and significantly outperforms training from scratch (FID 20.47 / 26.14) and single-constraint baselines.
• Guidance Calibration: DASH maintains a guidance gap ratio $\rho$ of 0.91 (CIFAR-10) and 0.89 (CIFAR-100), with directional cosine $\cos(\Delta)$ near 0.94. In contrast, composite distillation baselines suffer from partial collapse ($\rho \approx 0.68$), and conditional-only supervision leads to near-total collapse ($\rho \approx 0.09$), degrading performance below scratch training.
• Ablation Studies:
  • Removing $L_{un}$ (unconditional loss) causes a 7.77 FID point drop on CIFAR-10 and collapses $\rho$ to 0.61, confirming it as the dominant component.
  • Removing TIRT Transfer (replacing with fixed Min-SNR) results in a 2.60 FID point drop, validating the necessity of transferring the learned curriculum.
  • The cross-dataset performance ratio (1.18× for DASH vs. 1.51× for baselines) confirms robust generalization across label complexities.

Significance and Claims
The paper claims that DASH establishes a new standard for parameter-efficient diffusion by addressing a structural flaw in existing distillation objectives. The authors assert that:

1. Unconditional supervision is non-negotiable: It cannot be substituted by composite or conditional objectives; without it, CFG becomes ineffective under compression.
2. Curriculum transfer is distinct from feature distillation: Transferring the training schedule (TIRT) as a frozen prior is a novel contribution that prevents the corruption of the teacher's learned data-adaptive difficulty weighting.
3. Efficiency without Compromise: The framework enables significant parameter reduction (5.9×) while preserving the calibrated guidance mechanism, allowing for high-quality generation in resource-constrained settings without increasing inference steps.

The work positions itself at the intersection of parameter compression and branch-level calibration, offering a solution that is orthogonal to step-reduction methods and composable with them.