Speech-to-SOAP: End-to-End Summarization of Medical Dialogues: KIT@BeTraC 2026

In the daily rhythm of a doctor's visit, a significant portion of time is often spent not on the patient, but on the computer. As a physician listens to symptoms and asks questions, they must simultaneously type notes into a digital record, a task that pulls their attention away from the person in front of them. This documentation burden is a well-known source of fatigue for healthcare workers. To solve this, researchers have long looked toward speech recognition, hoping to turn spoken words directly into medical records. However, a simple transcript of what was said is rarely enough; the goal is to create a structured summary known as a SOAP note. This format organizes a visit into four distinct sections: the patient's subjective complaints, the objective findings from the exam, the doctor's assessment, and the plan for treatment. The challenge lies in teaching a computer to listen to a conversation, understand the medical context, and write this specific type of summary without human intervention, all while preserving subtle cues like a cough or a pause that might be lost if the computer first had to write down every word before summarizing it.

A team of researchers from the Karlsruhe Institute of Technology and Carnegie Mellon University set out to tackle this problem in the 2026 BeTraC challenge, a competition designed to test how well machines can automate medical documentation. Their approach, titled "Speech-to-SOAP," focused on building a system that could listen to a doctor-patient dialogue and generate the final medical note in one continuous step, skipping the intermediate step of writing a full transcript first. They utilized a powerful type of artificial intelligence model known as a foundation model, which had already been trained on vast amounts of general language and speech data. The researchers' main innovation was not just using this existing model, but teaching it how to handle the messy, varied nature of real medical conversations. They created a pipeline that combined several different datasets, including some that were entirely synthetic and others that contained real recordings, to train the system. For conversations that lacked audio recordings, they used computer-generated speech to fill the gaps, ensuring the model could learn from a massive volume of examples totaling over 1.6 million hours of audio.

To make the training data useful, the team had to teach the computer what a SOAP note actually looks like. Since many of their source conversations did not come with pre-written summaries, they used another AI tool to generate these summaries automatically. They crafted specific instructions to ensure these generated notes followed the correct structure and used consistent medical terminology, effectively creating a massive library of examples for the main system to study. The researchers then tested various ways to teach the model. They experimented with different ways of giving instructions to the computer, finding that detailed instructions worked best when placed as a direct command to the model rather than as a background rule. They also investigated whether it helped to show the model the text transcript of the conversation alongside the audio. Their findings suggested that seeing both the sound and the text together helped the model identify medical concepts more accurately, even if the final written summary looked similar to one generated from audio alone.

The team also explored whether breaking the task into smaller steps would help, such as having the computer first write down the transcript and then summarize it, or asking it to explain its reasoning before writing the note. The results showed that while these intermediate steps offered some benefits, the most effective approach for overall summary quality was to train the model to go directly from speech to the final summary. However, they discovered that Chain-of-Thought (CoT) supervision specifically achieved the highest score for extracting clinically relevant concepts, suggesting that explicitly modeling intermediate reasoning steps can improve the identification of medical details, even if it did not surpass direct generation in other metrics. Regarding data cleaning, the team found that removing segments where computer-generated speech did not match the text did not improve performance; in fact, the best results were obtained using the uncleaned dataset filtered to a maximum duration of 21 minutes. They also found that limiting the length of the conversations used for training to twenty-one minutes was the sweet spot; longer conversations introduced too much noise to be helpful.

In the final stage of their work, the researchers combined the best versions of their trained models to create a single, robust system. This merged system was tested against a specific contrastive submission from the same authors using three different sets of test data: conversations that were very similar to the training material, simulated role-play scenarios, and realistic recordings of actual medical visits. The combined system consistently outperformed this contrastive submission on all official test sets, particularly in the realistic scenarios where the differences between the training data and the test data were the greatest. This success suggests that averaging the knowledge from several differently trained models makes the system more adaptable and less likely to be confused by new situations. The study concludes that by unifying diverse data sources and using a direct speech-to-summary approach, it is possible to create a tool that significantly reduces the documentation burden on healthcare workers, allowing them to focus more on the human element of care.