Unlocking In-Context Learning in Audio-Language Models from Decentralized Medical Audio

Technical Summary: Federated Self-Contextualization (FSC) for Clinical Audio Diagnosis

Problem Statement
Clinical audio diagnosis is inherently a grounded reasoning task where clinicians interpret acoustic evidence (e.g., wheezes, murmurs) against a backdrop of medical knowledge. However, dominant machine learning approaches treat this as a closed-set classification problem requiring large, centrally aggregated, expert-annotated datasets. This paradigm fails in low-resource settings where annotated recordings are scarce, distributed across institutional silos, and governed by strict privacy constraints. Furthermore, standard models cannot adapt to diagnostic concepts absent from their training data or leverage the open-vocabulary nature of real-world diagnosis. While medical language models (LLMs) possess internalized clinical knowledge and native in-context learning (ICL) capabilities, grounding them in clinical audio is paradoxical: training audio-conditioned ICL requires structured audio-label episodes, yet expert-annotated pairs are precisely what is unavailable across privacy-constrained institutions.

Methodology: Federated Self-Contextualization (FSC)
The authors propose Federated Self-Contextualization (FSC), a multimodal framework that decouples the acquisition of diagnostic reasoning skills from the availability of real medical annotations. The framework operates under a federated protocol where raw audio never leaves its source institution.

1. Core Insight: The reasoning skill of contextual diagnosis and the clinical knowledge required to apply it can be acquired from separate sources. FSC trains a model using semantically void pseudo-labels (e.g., "Mountain Breeze," "Sun Ray") derived from unsupervised clustering of audio embeddings. Because these labels carry no medical semantics, the model cannot memorize disease-sound associations; instead, it learns the abstract skill of mapping acoustic patterns to contextual label descriptions.
2. Architecture:
   • Audio Encoder: Uses CaReAQA, a medical audio encoder pretrained on diverse tasks, to generate embeddings.
   • Language Model Backbone: Uses MedGemma-4B-IT, a 4-billion-parameter instruction-tuned medical LLM. Its pretraining provides the semantic grounding for real clinical terms at inference.
   • Projection: A linear layer maps audio embeddings to prefix tokens inserted between visual boundary tokens in the LLM, enabling audio-conditioned generation via standard self-attention without architectural modification.
3. Progressive Three-Stage Training Pipeline:
   • Stage I (Alignment): Non-episodic training aligns audio embeddings with the language model using individual audio-pseudo-label pairs. Only the encoder and projection are trained; the LLM is frozen.
   • Stage II (Episodic Refinement): Training shifts to episodic formats (support-query pairs) using pseudo-labels. The encoder and projection are further trained to produce embeddings discriminative within a relative support set context.
   • Stage III (LM Adaptation): The encoder and projection are frozen. LoRA adapters are applied to the LLM to specialize its attention mechanisms for audio-conditioned in-context reasoning while preserving its pretrained medical knowledge.
4. Federated Protocol: Implemented using Flower and FedProx. Clients perform local clustering to generate pseudo-labels independently. Only model parameters (encoder/projection in Stages I-II; LoRA weights in Stage III) are aggregated. No raw audio or shared label taxonomies are exchanged.
5. Inference: At test time, the model receives a few-shot support set with real clinical descriptions (e.g., "Wheeze," "Atrial Septal Defect"). The LLM's pretrained knowledge grounds these terms semantically, while the learned ICL skill matches the query audio to the appropriate description.

Key Contributions

• Formalization of Federated Self-Contextualization: A paradigm where semantically void pseudo-labels teach an audio-language model the transferable skill of in-context diagnostic reasoning, while the LLM's pretrained knowledge provides the necessary semantic grounding. This decouples reasoning skill acquisition from annotation availability.
• Progressive Episodic Training: A three-stage pipeline that addresses cross-modal alignment, in-context learning, and data sovereignty jointly, operating without centralized data access or predefined label spaces.
• Empirical Validation: Extensive evaluation across seven respiratory and cardiac datasets (over 22,000 recordings) demonstrating that FSC consistently outperforms recent audio-language baselines.

Results
Evaluated on held-out respiratory and cardiac conditions using an N-way K-shot episodic protocol (where diagnostic labels are unseen during training):

• Performance: In the 2-way 2-shot setting, FSC achieves 71.6% accuracy, outperforming the strongest audio-language baseline (Qwen2.5-Omni-7B) by over 9 percentage points (71.6% vs. 62.1%).
• Generalization: The model demonstrates strong performance across diverse conditions, including URTI (89.3%), bronchiolitis (74.8%), wheeze (74.3%), and atrial septal defect (74.9%).
• Ablation Studies:
  • Removing Stage I alignment reduces accuracy by 4.3 points.
  • Jointly training projection and LoRA from scratch degrades performance by 5.7 points, validating the progressive pipeline.
  • Replacing the medical LLM with general-purpose LLMs (LLaMA3.2, Qwen2.5) causes substantial drops (to ~49-50%), confirming the necessity of medically pretrained priors.
  • Federated vs. Centralized: Federated training under the same pseudo-label strategy outperforms centralized training by over 6 points (71.6% vs. 65.4%), suggesting that cross-institutional diversity acts as a natural regularizer.

Significance
The paper claims that FSC establishes the practical viability of privacy-preserving, few-shot clinical audio diagnosis. By resolving the paradox of needing structured episodes without access to real labels, the framework enables models to interpret acoustic evidence against any clinical concept expressible in natural language. The results indicate that decentralized training can be more effective than centralized training under identical labeling constraints, leveraging institutional diversity to prevent overfitting to a single clustering structure. This approach offers a pathway for deploying diagnostic tools in low-resource settings where annotated data is scarce and privacy is paramount.