⚡ electrical engineering

Resp-Agent: An Agent-Based System for Multimodal Respiratory Sound Generation and Disease Diagnosis

Resp-Agent is an autonomous multimodal system featuring an Active Adversarial Curriculum Agent that orchestrates a Diagnoser and a flow-matching Generator to overcome information loss and data scarcity in respiratory sound analysis, leveraging a new 229k-record benchmark to achieve superior diagnostic robustness under class imbalance.

Original authors: Pengfei Zhang, Tianxin Xie, Minghao Yang, Li Liu

Published 2026-03-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pengfei Zhang, Tianxin Xie, Minghao Yang, Li Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery inside a patient's lungs. The clues are the sounds they make when they breathe: a wheeze, a crackle, or a rattle. For a long time, computers have tried to be these detectives, but they've been struggling with two big problems:

They miss the fine details: When computers try to "see" sound (by turning it into a picture called a spectrogram), they often blur out the tiny, split-second sounds that are actually the most important clues. It's like trying to identify a specific bird by looking at a blurry photo of a forest; you miss the bird's unique song.
They don't have enough cases: To learn how to spot rare diseases, a computer needs to hear thousands of examples. But rare diseases are, well, rare. The computer ends up only studying the common stuff and gets confused when it sees something unusual. It's like a student who only studies for the final exam by reading the same three chapters over and over, then panics when the test asks about Chapter 10.

Enter Resp-Agent, a new AI system designed to fix these problems. Think of it not as a single robot, but as a high-tech medical detective agency with three specialized team members working together in a perfect loop.

The Three Team Members

1. The "Thinker" (The Case Manager)

Imagine a brilliant, overworked detective chief named Thinker.

What it does: Instead of just waiting for cases to come in, Thinker actively looks at the team's past mistakes. "Hey, we keep failing to diagnose Bronchiolitis," it says. "And we keep getting confused when the sound comes from a specific type of microphone."
The Magic: Thinker doesn't just complain; it creates a to-do list. It tells the other team members exactly what kind of fake cases they need to generate to practice on. It's like a coach saying, "We're bad at free throws, so let's practice 500 free throws before we play the next game."

2. The "Generator" (The Master Forger)

This team member is a creative sound artist with a very specific skill: they can make up fake lung sounds that sound exactly real.

What it does: When Thinker says, "We need more examples of Bronchiolitis recorded on an old microphone," the Generator listens to the instructions. It takes the "medical story" (the disease) and the "sound style" (the microphone type) and weaves them together to create a brand new, high-quality audio clip.
The Analogy: Imagine a chef who can cook a perfect steak (the disease) but also change the seasoning and plating style (the recording device) to match exactly what the restaurant needs. The Generator creates these "fake" but perfect examples so the detective team can practice on rare diseases without needing real patients.

3. The "Diagnoser" (The Expert Detective)

This is the final judge, the one who actually listens to the sounds and makes the diagnosis.

What it does: In the past, computers looked at the sound or the patient's notes separately. The Diagnoser is special because it weaves them together. It reads the patient's medical history (text) while simultaneously listening to the breathing sound (audio).
The Analogy: Imagine a detective who reads the suspect's alibi while listening to the recording of the crime. The Diagnoser uses a special "spotlight" (called Audio Anchors) that can zoom in on tiny, split-second sounds (like a crackle that lasts 20 milliseconds) and connect them directly to the text description. It ensures the computer doesn't miss the tiny, fleeting clues that human doctors catch by ear.

How They Work Together (The Loop)

The genius of Resp-Agent is that these three don't work in a straight line; they work in a circle:

The Diagnoser tries to solve a case but gets stuck on a rare disease.
The Thinker notices this weakness and says, "We need more practice on this!"
The Generator creates 50 new, perfect examples of that rare disease.
The Diagnoser studies these new examples, gets smarter, and tries again.
The Thinker checks the results, finds the next weakness, and the cycle repeats.

The Result: A New Library of Clues

To make this work, the team built a massive new library called Resp-229k. It contains 229,000 lung sounds, but with a twist: every single sound is paired with a clear, written summary of the patient's condition, written by an AI that read the medical records. This gives the system a huge amount of "text + sound" data to learn from.

Why This Matters

For Doctors: It acts as a super-powered assistant that can spot rare diseases that human ears might miss, especially when the data is messy or comes from different types of microphones.
For Patients: It means better early detection. If a computer can learn to recognize a rare, dangerous cough after hearing just a few examples (thanks to the Generator), it can help diagnose patients faster.
For Science: It proves that instead of just waiting for more data to appear, we can use AI to create the missing data we need to learn, as long as we do it carefully and intelligently.

In short, Resp-Agent is a self-improving detective agency that learns from its mistakes, creates its own practice tests, and uses a super-ear to listen to the tiny, fleeting sounds that save lives.

1. Problem Statement

The paper identifies three fundamental challenges hindering the advancement of deep learning-based respiratory auscultation:

Unimodal Representational Bottleneck: Existing audio models often convert signals into spectrograms, discarding phase information and blurring fine temporal structures (e.g., transient crackles). Conversely, text-only models lack objective acoustic evidence. Current large-scale pretraining remains unimodal, failing to capture the synergy between clinical context and acoustic transients.
Data Scarcity and Imbalance: Public respiratory sound datasets are small, cover few conditions, and suffer from severe class imbalance (long-tailed distribution). Existing augmentation methods (e.g., SpecAugment, noise injection) are untargeted and often degrade cross-domain generalization.
Disconnect Between Analysis and Generation: Research is heavily skewed toward diagnostic classification, leaving generative modeling underutilized. No existing framework unifies diagnostic analysis and generative synthesis in a single coherent system to create a closed-loop learning process.

2. Methodology: Resp-Agent Framework

The authors propose Resp-Agent, a centrally orchestrated, closed-loop multi-agent system. It integrates a Thinker (planner), a Generator (synthesizer), and a Diagnoser (classifier) to create an active adversarial curriculum.

A. The Thinker (Active Adversarial Curriculum Agent - Thinker-A2CA)

Role: A compute-aware planner (based on DeepSeek-V3.2-Exp) that acts as the central controller.
Function: It parses semantic intents, routes tasks, and analyzes diagnostic weaknesses (error profiles and calibrated confidence).
Mechanism: Instead of static data augmentation, the Thinker actively identifies failure modes (e.g., rare diseases or specific domain shifts) and schedules targeted synthesis requests to the Generator. This creates a self-correcting training loop where data generation is driven by model failure.

B. The Generator (Controllable Synthesis)

Goal: Synthesize high-fidelity respiratory audio that disentangles pathological content (what to generate) from acoustic style (how it sounds).
Architecture (Resp-MLLM):
- Base: A lightweight text-only LLM (Qwen3-0.6B-Base) retooled via Modality Injection.
- Style Conditioning: Acoustic style is captured via BEATs features (from a 10s reference audio), compressed into style descriptors, and injected into the LLM's hidden space as "style prefix tokens."
- Content Conditioning: Disease semantics are provided as text prompts.
- Training: The model autoregressively predicts discrete acoustic units from a BEATs codebook. A "leak-free" masking strategy prevents the model from peeking at oracle tokens during training.
Waveform Reconstruction:
- Predicted units are fed into a Conditional Flow Matching (CFM) decoder (Diffusion Transformer).
- CFM reconstructs mel-spectrograms, which are then converted to waveforms using a neural vocoder (Vocos).
- This two-stage design ensures stable, phase-aware reconstruction of transient events.

C. The Diagnoser (Robust Diagnosis)

Goal: Fuse clinical text (EHR summaries) with audio to improve diagnostic robustness.
Architecture:
- Modality Weaving: Unlike late-fusion approaches, the Diagnoser weaves audio and text tokens into a single sequence at the input layer. EHR text tokens are interleaved with a block of audio placeholders.
- Strategic Global Attention: Built on a Longformer backbone. It employs a sparse set of global attention tokens to handle long-range dependencies efficiently ( $O(N)$ complexity).
- Audio Anchors: A key innovation where specific audio tokens (anchors) are selected at a stride (e.g., every 4 frames, ~80ms resolution) to act as "cross-modal hubs." This allows distant text symptoms (e.g., "wheeze") to directly query transient acoustic events without quadratic computational costs, capturing millisecond-level events.

3. Key Contributions

Resp-229k Benchmark:
- A large-scale corpus of 229,101 respiratory recordings (408 hours) spanning 16 diagnostic categories.
- Multimodal Pairing: Each audio clip is paired with a standardized clinical narrative synthesized from EHR metadata using an LLM (DeepSeek-R1-Distill-Qwen-7B), rigorously audited for accuracy.
- Strict Evaluation: Features a source-disjoint split (training on ICBHI, SPRSound, UK COVID-19; testing exclusively on unseen KAUH and COUGHVID datasets) to stress-test cross-domain generalization.
Controllable Synthesis (Resp-MLLM):
- The first multimodal LLM trained with aligned text-audio supervision for controllable respiratory sound synthesis.
- Successfully decouples disease semantics from recording style (device, timbre), enabling the generation of "hard-to-diagnose" samples for data augmentation.
Modality Weaving & Strategic Global Attention:
- Introduces a novel architecture that interleaves modalities at the input level and uses anchor-based global attention to capture transient acoustic events (~80ms resolution) efficiently.
Closed-Loop Agent System:
- Demonstrates that coupling a diagnostic agent with a generative agent via an active planner (Thinker) outperforms static augmentation and passive training, effectively turning data scarcity into a solvable curriculum problem.

4. Experimental Results

Diagnostic Performance (ICBHI):
- Resp-Agent achieved an ICBHI Score of 72.7% (Specificity: 79.3%, Sensitivity: 66.1%), surpassing the previous state-of-the-art (67.55%) by over 5 absolute points.
Cross-Domain Generalization (Resp-229k):
- On the strict cross-domain test set (Test-CD), the full multimodal Diagnoser achieved 84.9% Accuracy and 0.212 Macro-F1 (on imbalanced data).
- Under the balanced regime (using Thinker-guided synthetic data), Macro-F1 jumped to 0.598, a massive improvement over the audio-only baseline (0.1935) and non-generative balancing methods.
Ablation Studies:
- Thinker Efficacy: The Thinker-A2CA planner significantly outperformed static planners (Class-Prior, Random) and non-generative loss re-weighting (Focal Loss), proving that targeted synthesis is superior to random augmentation.
- Architecture: Removing "Modality Weaving" or "Audio Anchors" caused significant performance drops, confirming their necessity for capturing transient events and cross-modal alignment.
- Generator Quality: The Resp-Agent Generator achieved superior style similarity (0.92) and lower Fréchet Audio Distance (1.13) compared to fine-tuned StableAudio Open and AudioLDM 2.

5. Significance

Paradigm Shift: Moves respiratory sound analysis from passive, static pipelines to an active, closed-loop agent system where generation and diagnosis co-design each other.
Solving Data Scarcity: Provides a robust solution for the long-tail problem in medical AI by generating high-fidelity, pathology-specific synthetic data that improves model fairness and robustness on rare diseases.
Clinical Relevance: The ability to synthesize specific pathological sounds with controlled style supports medical education, data augmentation for rare conditions, and interpretability research.
Generalizability: The "Modality Weaving" and "Strategic Global Attention" mechanisms offer a blueprint for other multimodal medical tasks involving transient signals and long-range context.

The code and data (Resp-229k) are open-sourced, facilitating reproducibility and further research in medical audio intelligence.