VoxSumm: A Multilingual Corpus of Long-Form Spoken News for Joint Summarization and Translation
The paper introduces VoxSumm, a multilingual corpus of over 10,000 BBC article-summary pairs spanning 24 languages and 703 hours of speech, to establish the first benchmark for joint speech summarization and translation (JSumT) and evaluate the performance of current speech-language models on this task.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you are standing in a bustling global newsroom where reporters are shouting stories into microphones in dozens of different languages. You want to know what's happening, but you only speak English, and the stories are hours long. You need a super-smart assistant who can listen to a three-hour speech in Swahili, figure out the most important parts, and instantly whisper a short, clear summary to you in English. This is the dream of multilingual speech summarization.
For a long time, computers have been great at two separate tricks: reading text and summarizing it, or listening to speech and translating it word-for-word. But combining these skills—taking a long, messy spoken story in one language and turning it into a short, punchy summary in another—has been a huge blind spot. It's like having a translator who can only copy-paste entire books, or a summarizer who only works with written notes. The real world, however, is full of long, spoken audio (like podcasts, news broadcasts, and lectures) that needs to be both compressed and translated to be useful for everyone.
This paper, titled VoxSumm, steps into that gap. The researchers built a massive new playground called VOXSUMM to test how well AI handles this specific, difficult task. They gathered over 10,000 pairs of news articles and their summaries in 24 different languages, turning the text into about 703 hours of synthesized speech. They then put three different AI models to the test to see if they could listen to a long audio clip in one language and spit out a one-sentence summary in another.
Here is what they found:
The "Do It All at Once" vs. "Do It in Steps" Debate
One of the biggest questions was: Should the AI translate the whole long audio first and then summarize it? Or should it summarize the audio first (in the original language) and then translate just the short summary?
The paper suggests that the "translate first" approach is a trap. When the AI tries to translate a three-hour speech before summarizing it, it often gets tired, confused, or forgets the instructions, leading to hallucinations or missing the point entirely. It's like trying to carry a heavy backpack full of rocks (the whole translation) while trying to write a haiku (the summary); you're more likely to drop the rocks or forget the poem. The researchers found that summarizing first, then translating the short summary, is a much more reliable pipeline. It keeps the AI focused on the core message before worrying about the language switch.
The Language Direction Matters
The direction of the translation also changes the game. The AI models generally did a better job when they were summarizing a non-English speech into English, rather than summarizing English speech into a non-English language.
Think of it like a chef who is a master at cooking a complex dish but only has a perfect recipe book in English. If you ask them to cook a French dish and then describe it in English, they might struggle with the French ingredients. But if you ask them to cook a French dish and describe it in French, they might be more confident. In this case, the models seemed more comfortable "thinking" in English first, condensing the information, and then translating that small, clean summary, rather than trying to juggle complex foreign grammar while summarizing.
The Models and the "Magic" of Few-Shot Learning
The researchers tested three different AI "brains": a massive proprietary model (Gemini 3.1-Pro), a mid-sized open model (Qwen 3-Omni), and a smaller, edge-friendly model (Gemma 4-12B).
The results showed that Gemini 3.1-Pro was the clear winner, consistently producing the most accurate and faithful summaries. However, the paper highlights a fascinating trick called Few-Shot prompting. This is like giving the AI a few example problems and solutions before asking it to solve a new one. When the researchers gave the models just five examples of "listen to this audio, here is the summary," the performance jumped significantly, especially for the stronger models. It's as if showing the AI a few sample recipes made it suddenly understand the cooking style much better.
The "Speech" Factor
Finally, the paper confirms that listening to actual audio is harder than just reading a transcript. When the AI had to process the raw sound waves (with all the pauses, breathing, and tone), it performed slightly worse than when it just read the text. This suggests that the "noise" of real speech—like hesitation or background sounds—still trips up even the smartest models, causing them to lose a tiny bit of information compared to reading a clean text file.
In short, the paper doesn't claim to have solved the problem of perfect AI summarization. Instead, it provides a new, rigorous test track (VOXSUMM) and shows us that the current best strategy is to summarize first, translate second, and to show the AI a few examples before asking it to work. It's a solid step toward building assistants that can truly help us navigate the world's spoken information, even if they still need a little help with the heavy lifting of long, foreign speeches.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.