Technical Summary: STARS for ANN-to-SNN Data-Free Knowledge Distillation

1. Problem Statement

Spiking Neural Networks (SNNs) offer energy-efficient and low-latency inference, making them ideal for edge intelligence. However, directly trained SNNs often suffer from optimization instability and suboptimal accuracy compared to Artificial Neural Networks (ANNs). Consequently, ANN-to-SNN knowledge distillation (KD) has emerged as a primary strategy to transfer knowledge from powerful ANN teachers to SNN students.

A critical challenge arises in practical deployment scenarios (e.g., edge devices) where the original training data is unavailable due to privacy, copyright, or storage constraints. This necessitates Data-Free Knowledge Distillation (DFKD), where surrogate data is synthesized from a pretrained teacher to facilitate transfer.

Existing DFKD methods primarily rely on matching Batch Normalization (BN) statistics (mean and variance) of the teacher network to synthesize data. While effective for ANN-to-ANN distillation, the authors argue that applying these methods directly to ANN-to-SNN distillation is suboptimal. The core issue is an asymmetry in dynamics:

• ANNs process continuous activations where first- and second-order statistics (mean/variance) are sufficient to characterize the output distribution.
• SNNs operate via discrete spikes governed by Leaky Integrate-and-Fire (LIF) dynamics, where the output depends on threshold-crossing events.

The paper posits that matching BN statistics (mean and variance) leaves the tail behavior of the activation distribution under-constrained. Two distributions can share identical means and variances but possess different probability masses above the firing thresholds, leading to significantly different spike responses in the SNN student. This "BN-moment ambiguity" results in synthetic data that is valid for the ANN teacher but ambiguous or misleading for the SNN student.

2. Methodology: STARS

To address this mismatch, the authors propose STARS (Spike Tail-Aware Relational Synthesis), a plug-and-play framework that augments standard BN-guided synthesis with two complementary objectives designed to constrain the synthesis process specifically for SNN dynamics.

Core Analysis

The authors derive that the expected firing rate of an SNN neuron is determined by the exceedance probabilities of the pre-activation values over a set of time-dependent effective thresholds ($\theta_t$), rather than solely by the first and second moments.
$$R_T(P) = \frac{1}{T} \sum_{t=1}^{T} (1 - F_P(\theta_t))$$
where $F_P$ is the cumulative distribution function (CDF) of the activation distribution. This analysis proves that matching BN statistics does not uniquely determine the SNN response, as distributions with identical moments can have different tail masses above the thresholds.

The STARS Framework

STARS is applied to pre-activation features (before ReLU in the ANN and before LIF in the SNN) to avoid the non-differentiability of post-spike resets. It introduces two regularization terms:

1. Relational Consistency Alignment (RCA):

   • Goal: Preserve the global semantic structure and cross-sample relational consistency between the teacher and student.
   • Mechanism: RCA aligns the pairwise cosine similarity matrices of the global pre-activation features between the ANN teacher and the SNN student. It ensures that the relative arrangement of samples in the feature space is consistent, preventing the synthesized data from inheriting an ANN-oriented bias that distorts the student's feature organization.
   • Loss: Minimizes the squared difference between off-diagonal elements of the teacher's and student's relation matrices.

2. Tail-Aware Regularization (TAR):

   • Goal: Directly constrain the threshold-relevant tail behavior that BN matching ignores.
   • Mechanism: TAR defines a set of teacher-anchored thresholds (e.g., derived from quantiles of the teacher's feature distribution). It computes soft exceedance probabilities (using a sigmoid function) for both teacher and student features above these thresholds.
   • Loss: Minimizes the difference between the soft exceedance probabilities of the teacher and the student. This explicitly forces the synthetic data to match the probability mass in the "tails" of the distribution that are critical for triggering spikes in the SNN.

Overall Objective

The final synthesis objective combines the standard DFKD loss (classification confidence, BN matching, image regularization) with the new STARS terms:
$$\mathcal{L}_{total} = \mathcal{L}_{DFKD} + \lambda_{rca} \mathcal{L}_{RCA} + \lambda_{tar} \mathcal{L}_{TAR}$$
The synthetic batch is optimized to satisfy teacher priors while simultaneously ensuring relational consistency and tail-awareness for the SNN student.

3. Key Contributions

The paper claims the following contributions:

1. First Analysis of ANN-to-SNN DFKD: It is the first work to systematically study the mismatch in data-free distillation from ANN to SNN, analyzing how BN-moment ambiguity leads to suboptimal SNN responses.
2. STARS Framework: The proposal of a plug-and-play method (STARS) that integrates Relational Consistency Alignment (RCA) and Tail-Aware Regularization (TAR) to resolve the under-constrained nature of standard BN-guided synthesis for SNNs.
3. Empirical Validation: Demonstrating that STARS consistently improves performance across multiple DFKD baselines (DeepInversion, CMI, NAYER) and diverse datasets (CIFAR-10, CIFAR-100, Tiny-ImageNet) and model architectures (VGG, ResNet).

4. Experimental Results

The authors evaluated STARS on CIFAR-10, CIFAR-100, and Tiny-ImageNet using various ANN-SNN pairs (e.g., ResNet-19 $\to$ S-ResNet-19).

• Performance Gains: STARS consistently outperformed baseline DFKD methods.
  • On CIFAR-10, gains reached up to 4.6% (e.g., NAYER + STARS achieved 95.73% vs. 92.02% for NAYER alone).
  • On CIFAR-100, gains reached up to 6.7% (e.g., NAYER + STARS achieved 78.42% vs. 73.95% for NAYER alone).
  • On Tiny-ImageNet, improvements were consistent (e.g., +3.33% on CMI, +2.97% on NAYER), demonstrating generalization to larger, more complex datasets.
• Comparison to Data-Available KD: Notably, NAYER + STARS on CIFAR-10 (95.73%) and CIFAR-100 (78.42%) surpassed several data-available KD baselines and nearly closed the gap to the ANN teacher performance, suggesting that structured synthesis constraints can effectively substitute for real training data in this context.
• Ablation Studies:
  • Both RCA and TAR contributed positively individually, but their combination yielded the best results, confirming they address complementary aspects of the synthesis mismatch.
  • Increasing the number of tail quantiles (thresholds) in TAR improved performance monotonically, validating the importance of capturing richer tail information.
• Visualization: t-SNE visualizations showed that STARS produced SNN student features with tighter intra-class clustering and clearer inter-class separation compared to baselines.

5. Significance and Claims

The paper concludes that the mismatch between ANN-oriented synthesis priors and SNN dynamics is a fundamental challenge in data-free distillation. The significance of STARS lies in its demonstration that complementing BN matching with relational and tail-aware constraints is an effective and generalizable strategy.

The authors claim that:

• BN statistics alone are insufficient to determine SNN responses due to the threshold-crossing nature of spiking neurons.
• By explicitly regularizing the tail probabilities and relational structures, STARS generates synthetic data that is "teacher-valid" (satisfying the teacher's priors) but also "student-informative" (aligned with SNN firing dynamics).
• The method is plug-and-play, requiring no changes to the underlying DFKD framework or the teacher/student architectures, making it easily adoptable for existing pipelines.

The work is presented as a step toward enabling efficient SNN deployment in privacy-sensitive and resource-constrained environments where original training data cannot be retained or transmitted.