Technical Summary: ZEBRA for Base-to-Novel Generalization in Audio-Language Models

1. Problem Statement

Audio-Language Models (ALMs) have demonstrated strong zero-shot capabilities by aligning audio features with textual class descriptions, eliminating the need for large labeled datasets. However, a critical limitation exists in prompt learning, a technique used to adapt these models to downstream tasks using few-shot labeled data from "base" classes.

While prompt learning significantly improves accuracy on seen (base) classes, the authors observe a base-to-novel generalization gap: optimizing prompts solely on base classes often degrades performance on unseen (novel) classes. In many cases, the adapted model performs worse on novel categories than the original zero-shot baseline. This phenomenon indicates that few-shot adaptation encourages overfitting to seen categories, distorting the semantic alignment learned during large-scale pre-training and weakening transferability to unseen concepts.

2. Methodology: ZEBRA

To address this trade-off, the authors propose ZEBRA (Zero-shot Entropy-Regularized Prompt Learning for Base-to-Novel Generalization). ZEBRA is a plug-and-play framework designed to operate on top of existing prompt-learning methods (e.g., COOP, COCOOP) without introducing additional learnable parameters or modifying the underlying model architecture.

The framework employs two complementary mechanisms:

A. Zero-Shot Logit Fusion

Instead of relying exclusively on the logits generated by the prompt-learning module ($f_{pr}$), ZEBRA fuses them with the original zero-shot logits ($f_{zs}$) during both few-shot training and inference. The fused logit vector is computed as:
$$f_{zebra}(x; \phi) = \lambda_{zs} \cdot f_{zs}(x) + \lambda_{pr} \cdot f_{pr}(x; \phi)$$
where $\lambda_{zs}$ and $\lambda_{pr}$ are weighting coefficients. This fusion anchors the adaptation process to the pre-trained decision space, preventing the model from drifting too far from the original semantic alignment.

B. Self-Entropy Regularization

To mitigate overconfidence in base classes during few-shot training, ZEBRA introduces a self-entropy regularization term ($L_{ent}$) into the loss function. The objective function becomes:
$$L_{zebra}(x, y; \phi) = L_{ce}(f_{zebra}, y) - L_{ent}(\text{softmax}(f_{zebra}))$$
where $L_{ce}$ is the standard cross-entropy loss. By maximizing the entropy term (effectively minimizing negative entropy), the method discourages over-confident predictions on base classes. This promotes smoother decision boundaries and reduces excessive specialization to seen categories, thereby enhancing generalization to novel classes.

Computational Efficiency:
ZEBRA is lightweight. The zero-shot logits are computed once and reused throughout training and inference, eliminating the need for additional forward passes through the text encoder. Consequently, the framework incurs negligible computational overhead.

3. Key Contributions

The paper outlines four primary contributions:

1. Identification of a Generalization Gap: The authors analyze and demonstrate that prompt learning in ALMs creates a trade-off where improvements on base classes come at the cost of degraded novel-class performance, often falling below zero-shot baselines.
2. Proposal of ZEBRA: A plug-and-play framework that integrates zero-shot logit fusion and self-entropy regularization to preserve zero-shot transferability while benefiting from supervised adaptation.
3. Parameter and Efficiency Efficiency: The method introduces no additional learnable parameters to existing baselines and maintains negligible computational overhead.
4. Empirical Validation: The approach demonstrates consistent improvements in novel-class generalization across multiple datasets, significantly reducing the base-to-novel performance gap compared to vanilla prompt learning.

4. Experimental Results

The authors evaluated ZEBRA on a diverse set of audio classification tasks, including instrument classification, sound event detection, emotion recognition, and music analysis, using datasets such as ESC-50, UrbanSound8K, CREMA-D, and Beijing Opera. The backbone model used was Pengi, adapted in a CLIP-like contrastive framework.

Key Findings:

• Baseline Performance: Standard prompt learning methods (COOP, COCOOP) improved base-class accuracy significantly (e.g., +26.2% and +28.5% over zero-shot) but degraded novel-class accuracy (e.g., -7.13% and -4.74% relative to zero-shot).
• ZEBRA Performance: Incorporating ZEBRA reversed this trend. It maintained strong base-class accuracy while significantly boosting novel-class performance. On average, ZEBRA-enhanced models improved novel-class accuracy by +4.19% and +4.31% over standard prompt learning, bringing performance back above the zero-shot baseline average.
• Ablation Studies: Experiments confirmed that the fusion of zero-shot logits provided the majority of the performance gain, while self-entropy regularization offered a marginal but consistent additional improvement.
• Calibration and Efficiency: ZEBRA reduced the Expected Calibration Error (ECE) on both base and novel classes, indicating better-calibrated predictions. Runtime analysis showed negligible increases in training and testing time (e.g., <10 seconds difference on average).

5. Significance and Claims

The paper claims that ZEBRA offers a simple yet effective solution to a fundamental limitation in adapting Audio-Language Models. By leveraging the stability of zero-shot knowledge and regularizing against overconfidence, ZEBRA successfully bridges the gap between few-shot adaptation and generalization to unseen categories.

The authors emphasize that their approach does not require complex architectural changes or additional training parameters. Instead, it serves as a lightweight enhancement that allows existing prompt-learning methods to retain their adaptability without sacrificing the robust generalization capabilities inherent to large-scale pre-trained models. The code is made available to facilitate further research in this domain.