AFD-SLU: Adaptive Feature Distillation for Spoken Language Understanding

Technical Summary: AFD-SLU: Adaptive Feature Distillation for Spoken Language Understanding

1. Problem Statement

Spoken Language Understanding (SLU)—comprising Intent Detection and Slot Filling—is essential for conversational AI. However, the field faces two critical bottlenecks:

• Data Scarcity: High-quality labeled training data for specific domains or languages (like Chinese) is often limited.
• Computational Constraints: While Large Language Models (LLMs) offer superior semantic understanding, their massive memory and computational requirements make them impractical for real-world, resource-constrained edge devices or real-time applications.

Existing knowledge distillation methods often focus on task-level distillation, which lacks cross-domain adaptability, or they fail to balance the trade-off between transferring teacher knowledge and learning task-specific labels.

2. Methodology

The authors propose AFD-SLU (Adaptive Feature Distillation for SLU), a lightweight framework designed to transfer rich semantic representations from a powerful teacher to an efficient student. The architecture consists of three core components:

• Teacher Model (GTE-based): The framework utilizes a General Text Embeddings (GTE) model (derived from an LLM) as the teacher. Unlike traditional methods that use the LLM directly, the GTE model provides high-quality token-level semantic embeddings. The teacher is frozen during training to provide supervision without increasing inference cost.
• Student Model: A lightweight, joint architecture (e.g., BiLSTM with self-attention) optimized for efficient inference, performing both intent detection and slot filling simultaneously.
• Dynamic Adapter: This is the core innovation, consisting of two sub-modules:
  • Residual Projection Neural Network (RPNN): Since the student and teacher models have different embedding dimensions ($d_{es} \neq d_{et}$), the RPNN aligns these heterogeneous feature spaces. It uses a nonlinear transformation involving a linear layer, GELU activation, Layer Normalization, and a Residual-Block (a two-layer feed-forward network with skip connections) to project student features into the teacher's higher-dimensional space.
  • Dynamic Distillation Coefficient (DDC): To prevent the student from being "overwhelmed" by the teacher or ignoring task labels, the DDC uses a cosine annealing schedule. It adaptively modulates the weight ($\lambda$) of the distillation loss relative to the task loss, gradually shifting the focus from teacher-guided semantic alignment to task-specific optimization as training progresses.

Loss Function:
$$L_{total} = L_{task} + \lambda L_{distill}$$
(Where $L_{task}$ is a combination of intent and slot losses, and $L_{distill}$ is the MSE between teacher and student token embeddings).

3. Key Contributions

• Novel Teacher Selection: It is the first work to employ General Text Embeddings (GTE) as a teacher model for SLU, bridging the gap between heavy LLMs and lightweight SLU models.
• Feature Space Alignment: The introduction of the RPNN allows for effective nonlinear mapping between disparate embedding dimensions while preserving essential information via residual connections.
• Adaptive Training Strategy: The DDC provides a dynamic mechanism to balance knowledge transfer and task learning, addressing the limitations of fixed-weight distillation.

4. Experimental Results

The framework was evaluated on the ProSLU (Chinese profile-based) benchmark:

• State-of-the-Art (SOTA) Performance: Using the PRO-HAN model as a student, AFD-SLU achieved 95.67% intent accuracy and 92.02% slot F1 score, significantly outperforming baseline models.
• Efficiency: The performance gains were achieved without increasing the model's parameter count or inference latency, as the teacher model is only used during training.
• Ablation Insights:
  • The RPNN requires a balanced design; both overly simple (linear) and overly complex (deep) projections resulted in lower performance.
  • The DDC is crucial; removing it led to a significant drop in all metrics.
• Teacher Model Analysis: The study found that "bigger is not always better." Smaller, well-aligned models (like GTE-Qwen2-1.5B) outperformed much larger models (like Qwen2-7B), which tended to overfit the small ProSLU dataset.

5. Significance

AFD-SLU provides a highly practical blueprint for deploying sophisticated semantic capabilities in low-resource and resource-constrained environments. By successfully distilling "intelligence" from massive embedding models into lightweight architectures, it enables high-performance, real-time spoken language understanding on consumer-grade hardware without the need for massive labeled datasets.