Improved Knowledge Distillation for Land-Use Image Classification

Technical Summary: Improved Knowledge Distillation for Land-Use Image Classification

Problem Statement
Deep learning has significantly advanced remote sensing image analysis, with recent architectures (e.g., Transformers, deep CNNs) achieving state-of-the-art accuracy. However, these high-performance models often incur substantial computational and memory costs, hindering their deployment on resource-constrained edge platforms. While Knowledge Distillation (KD) offers a solution by transferring knowledge from large "teacher" models to lightweight "student" models, existing KD approaches in remote sensing often rely on single distillation objectives. These methods may fail to fully capture the complementary probabilistic, geometric, and structural knowledge encoded in deep models, particularly given the high inter-class similarity and fine visual variations inherent in aerial imagery.

Methodology
The authors propose an improved KD framework utilizing a teacher-student paradigm.

• Architecture: A VGG16 network serves as the teacher due to its deep, uniform convolutional structure capable of learning rich hierarchical features. A MobileNetV2 network acts as the lightweight student, chosen for its reduced parameter count while maintaining discriminative feature learning capabilities.
• Loss Function Strategy: The student model is trained using a composite loss function that integrates hard supervision with a novel soft supervision strategy.
  1. Hard Loss: Standard categorical cross-entropy using ground truth labels to enforce correct classification and maintain discriminative class boundaries.
  2. Soft Loss (Probabilistic-Representational): A weighted combination of two distinct components:
     • KL Divergence Loss: Measures the divergence between the softened probability distributions of the teacher and student (using a temperature parameter $T$). This transfers probabilistic knowledge regarding relative class confidences.
     • Cosine Similarity Loss: Applied to L2-normalized logits of the teacher and student. Unlike static feature space metrics, this component aligns the structural geometry and directionality of the representations, focusing on angular relationships rather than magnitude.
  3. Total Objective: The final loss ($\mathcal{L}_{total}$) balances the hard loss and the combined soft loss ($\mathcal{L}_{soft}$) via weighting parameters $\alpha$ and $\lambda$.

Key Contributions

• Hybrid Loss Formulation: The paper introduces a framework that simultaneously optimizes for class-level semantic similarity (via KL divergence) and feature-level structural alignment (via Cosine Similarity), addressing the limitations of single-objective distillation.
• Geometric Alignment: The work uniquely exploits Cosine Similarity as a dynamic loss component to mimic the directionality of the teacher's logits, ensuring the student learns the structural geometry of the teacher's representations.
• Efficient Compression: The method successfully compresses a heavy VGG16 teacher into a highly efficient MobileNetV2 student without significant accuracy degradation.

Experimental Results
Experiments were conducted on the UC Merced land-use dataset (21 classes, 2100 images).

• Performance: The proposed method achieved a peak classification accuracy of 99.04%, outperforming the baseline MobileNetV2 training (96.47%) and various single-loss distillation approaches (e.g., Hard + KL: 97.17%, Hard + Cosine: 97.08%).
• Efficiency: The distilled student model reduced the parameter count from 14.85 million (Teacher) to 2.42 million and computational cost from 20.06 GFLOPs to 0.40 GFLOPs.
• Comparison: The method outperformed five of eight non-KD state-of-the-art baselines and achieved competitive results against recent KD-based approaches, offering a superior accuracy-parameter trade-off compared to methods using deeper teachers (e.g., ResNet50/101) and heavier students.
• Generalization: Preliminary experiments on the AID and NWPU-RESISC45 datasets demonstrated promising generalization capabilities (95.77% and 91.59% accuracy, respectively).
• Statistical Significance: An ablation study over 10 independent runs confirmed that the full framework (Hard + KL + Cosine) yields a statistically significant improvement over the baseline (p-value < 0.05).

Significance and Claims
The paper claims that by integrating hard supervision with probabilistic and representational soft losses, the proposed framework effectively preserves semantic relationships and feature geometry. This allows the lightweight student network to achieve high inter-class discrimination in aerial imagery while maintaining computational efficiency. The authors position the method as well-suited for real-time and resource-constrained remote sensing applications. They modestly acknowledge minor variance across simulations and note that while the approach is validated on UC Merced, future work should focus on extending the analysis to larger, more diverse datasets and further reducing computational overhead.