Transformer Architectures for Respiratory Sound Analysis and Multimodal Diagnosis
This paper proposes a multimodal diagnostic framework utilizing Audio Spectrogram Transformers and vision-language models to analyze respiratory sounds and patient metadata, demonstrating superior accuracy and interpretability compared to traditional physician auscultation and previous CNN-based approaches for asthma diagnosis and remote monitoring.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Listening to a patient's breath is one of the oldest tools in a doctor's kit. For centuries, physicians have pressed their ears or stethoscopes against a chest to detect the subtle rattle of fluid, the high-pitched whistle of narrowed airways, or the smooth silence of healthy lungs. This practice, known as auscultation, relies entirely on the human ear and the experience of the listener. While invaluable, it is also subjective; two doctors might hear different things in the same breath, and a diagnosis can depend heavily on how much practice a physician has had. As respiratory diseases like asthma affect millions of people worldwide, the medical community has long sought a way to make this listening process more objective, consistent, and accessible, especially for young children or those in remote areas where specialists are scarce.
In recent years, scientists have turned to computers to help listen. By converting the sound of breathing into a visual map called a spectrogram—where time runs along the bottom and pitch runs up the side—researchers can treat a breath like an image. This allows them to use powerful computer programs, originally designed to recognize cats or cars in photographs, to identify patterns of disease. However, these early attempts often relied on older computer models that looked only at the sound, ignoring the rest of the patient's story, such as their age or gender. Furthermore, comparing different computer programs has been difficult because they were often tested on different sets of data, making it hard to know which one truly worked best.
A team of researchers from Ben-Gurion University in Israel and Perm State Medical University in Russia set out to solve these problems by testing the newest generation of computer models on a single, shared collection of patient data. They wanted to see if modern artificial intelligence could not only listen to breath sounds but also understand the context of the patient, mimicking the way a human doctor combines what they hear with what they know about the person. Their work focuses on distinguishing between patients with asthma and those without, a task that is critical for managing a condition that affects up to 29 percent of the population in various countries.
The researchers gathered a large set of recordings from over 1,300 participants, ranging from infants to adults. These recordings captured the sound of quiet breathing from four different spots on the body: the mouth, the windpipe, the chest, and the back. Each recording was accompanied by structured information about the patient, including their age, sex, and where the sound was taken. The team then fed this data into three different types of computer models to see which could diagnose asthma most accurately.
The first model they tested was a well-established type of computer program known as a convolutional neural network, specifically a version called DenseNet201. This model had been used in their previous work and served as a reliable baseline. It looked only at the visual map of the sound, ignoring the patient's personal details. The second model was a cutting-edge system called the Audio Spectrogram Transformer. This model was designed specifically to understand sound patterns and was trained on vast libraries of audio before being adapted for this medical task. The third model was a multimodal vision-language system named Moondream2. This was the most ambitious attempt; it was designed to look at the sound map while simultaneously reading a text description of the patient's age, sex, and recording location, hoping that combining these two streams of information would lead to a better diagnosis.
The results revealed a clear winner. The specialized sound model, the Audio Spectrogram Transformer, outperformed all others with remarkable precision. It achieved a success rate of 98.7% across accuracy, sensitivity, specificity, and F1-score, with a Youden Index of 0.974. This performance was significantly higher than the older DenseNet model, which was correct about 87 percent of the time. The multimodal model, Moondream2, performed similarly to the older model, achieving about 86.5 percent accuracy. Interestingly, the researchers found that the multimodal model relied heavily on the text information it was given. When they removed the patient's age and sex from the input, the model's ability to diagnose asthma collapsed, producing errors on every single asthma case. This suggested that while the model could use the text to make a decision, it struggled to learn the sound patterns on its own as effectively as the model built specifically for audio.
The study also explored how these computer models make their decisions, a crucial step for gaining the trust of doctors. For the older model, researchers could use a technique that highlights the specific parts of the sound map that triggered the diagnosis, much like a heat map showing where a camera focused. For the new sound model, they analyzed the internal attention mechanisms to see which parts of the sound the computer was focusing on. They found that both models zeroed in on the same frequency range, roughly between 800 and 1800 hertz, particularly during the third cycle of a breath. This consistency suggests that the computers are not guessing randomly but are actually detecting the same physical acoustic features that doctors listen for.
The researchers concluded that while combining patient details with sound analysis is a logical step, the current generation of general-purpose models that try to do both may not be as effective as a model designed specifically for sound. The specialized audio model proved that it could learn the nuances of respiratory disease from the sound alone, surpassing the performance of human auscultation reported in medical literature. The findings suggest that the future of computer-assisted diagnosis may lie in using highly specialized tools for specific tasks, rather than trying to build one single system that does everything. These models offer a promising path toward continuous, remote monitoring of patients, potentially allowing for earlier detection of asthma attacks and more consistent care for people everywhere, regardless of their access to a specialist.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.