ALICE: A Multifaceted Evaluation Framework of Large Audio-Language Models' In-Context Learning Ability

1. Problem Statement

While Large Audio-Language Models (LALMs) have advanced significantly, recent studies indicate they suffer from degraded instruction-following capabilities compared to text-only Large Language Models (LLMs), particularly under explicit output constraints. A critical gap exists in understanding how LALMs perform In-Context Learning (ICL) when conditioned on audio. Specifically, it is unknown whether LALMs can infer task objectives and output formats solely from audio-conditioned demonstrations (few-shot examples) without explicit textual instructions. Current research lacks a systematic framework to disentangle the influence of acoustic inputs from in-context demonstrations to determine if LALMs can truly generalize task inference across modalities.

2. Methodology: The ALICE Framework

The authors introduce ALICE (Audio-Language In-Context learning Evaluation), a systematic, three-stage evaluation framework designed to progressively reduce textual guidance while keeping audio inputs and demonstration outputs constant. This isolates the model's ability to infer tasks from audio-conditioned examples.

• Evaluation Testbed: The framework utilizes a subset of Speech-IFEval, covering four audio understanding tasks:
  • Automatic Speech Recognition (ASR)
  • Speech Emotion Recognition (SER)
  • Gender Recognition (GR)
  • Massive Multi-Task Audio Understanding (MMAU)
• Constraint Categories:
  • Closed-Ended Questions (CEQ): Enforce strict deterministic formats (e.g., JSON, uppercase, specific prefixes).
  • Chain-of-Thought (CoT): Requires explicit intermediate reasoning steps before the final answer.
• The Three Stages:
  1. Stage 1 (Explicit Constraint): Demonstrations and test samples include both task descriptions and explicit format constraints. This serves as an upper-bound baseline.
  2. Stage 2 (Implicit Constraint): Explicit format instructions are removed from both demonstrations and test samples. The model must infer formatting rules solely from the example outputs.
  3. Stage 3 (Audio Only): Task descriptions are also removed. The model receives only audio inputs paired with correctly formatted outputs. The model must jointly infer the task objective and formatting patterns from the audio-text relationship. (Note: MMAU is excluded here as it requires text instructions to be defined).
• Experimental Setup:
  • Models: Evaluated 6 LALMs (5 open-source: Qwen2-Audio, DeSTA2.5-Audio, BLSP-Emo, Qwen2.5-Omni, Phi-4-Multimodal; 1 proprietary: Gemini 2.5 Flash).
  • Data: 988 evaluation samples and 912 demonstration samples, ensuring disjoint data sources and label balancing.
  • Metrics:
    • Format Compliance Rate (FCR): Measures adherence to output constraints.
    • Task Performance: Word Error Rate (WER) for ASR; Accuracy for SER, GR, and MMAU.

3. Key Contributions

1. First Systematic Analysis: The paper provides the first comprehensive study of LALMs' ICL deduction abilities under audio conditioning.
2. Decoupling Instruction-Following from ICL: It offers empirical evidence that strong instruction-following capabilities do not guarantee robust ICL deduction abilities from demonstrations alone.
3. Identification of Cross-Modal Limitations: The study reveals that providing reasoning traces (CoT) fails to improve core task performance, suggesting that LALM limitations stem from the integration of auditory information rather than a deficiency in text-based reasoning.

4. Key Results

The evaluation across six models and four tasks revealed a consistent asymmetry between format compliance and core task performance:

• Format Compliance vs. Task Performance:
  • Format: In-context demonstrations reliably improve format compliance (FCR). Even in Stage 3 (Audio Only), many models maintained or improved FCR compared to Stage 2, suggesting they can learn surface-level formatting patterns from audio-label pairs.
  • Task Performance: Demonstrations failed to improve (and often degraded) core task performance (e.g., ASR accuracy, emotion recognition). This held true even when reasoning traces were provided.
• The "Instruction-Following" Paradox: Models with the strongest instruction-following abilities (e.g., DeSTA2.5-Audio, Gemini 2.5 Flash) showed the largest drops in format compliance when explicit constraints were removed (Stage 1 to Stage 2). This indicates that reliance on explicit instructions hinders the ability to deduce rules from examples alone.
• The Role of CoT:
  • In Stage 3, CoT performance remained closer to Stage 2 than CEQ performance did. However, the authors argue this is not due to genuine audio-text integration. Instead, models rely on textual cues within the reasoning traces to infer the task, as evidenced by the persistent failure in CEQ (where no such textual guidance exists).
• Stage 3 Counter-Intuition: Removing task descriptions (Stage 3) sometimes improved format compliance compared to Stage 2. The authors suggest that textual task descriptions in Stage 2 introduce distracting variability, whereas Stage 3 forces the model to rely on stable surface-level audio-label patterns.

5. Significance and Conclusion

The findings highlight a fundamental limitation in current LALMs: they can mimic surface-level formatting patterns but struggle with cross-modal semantic grounding.

• Mechanism Distinction: Format inference relies on pattern matching (which LALMs handle well), whereas task inference requires integrating auditory perception with semantic context to map audio to a correct response (which LALMs currently fail to do reliably via ICL).
• Implications: The "ICL" observed in LALMs is largely a form of pattern matching rather than true reasoning or task generalization. The inability to leverage reasoning traces to improve task performance suggests that the bottleneck lies in the multimodal integration architecture, not the reasoning capabilities of the underlying LLM.
• Future Directions: The paper calls for new training paradigms focused on better cross-modal integration. Future work should extend this framework to open-ended generation, diverse tasks (e.g., speech synthesis), and investigate optimal demonstration selection and ordering.

In summary, ALICE demonstrates that while LALMs are becoming better at following explicit instructions, their ability to "learn" tasks from audio examples without explicit guidance remains critically underdeveloped, revealing a significant gap in true cross-modal understanding.