Do What I Say: A Spoken Prompt Dataset for Instruction-Following

This paper introduces DoWhatISay (DOWIS), a multilingual dataset of human-recorded spoken and written prompts designed to evaluate Speech Large Language Models under realistic spoken instruction conditions, revealing that text prompts generally outperform spoken ones except in tasks requiring speech output.

The Gist

Imagine you've built a brilliant, all-knowing robot assistant. You've tested it a thousand times by typing commands into a keyboard, and it passes every test with flying colors. You think, "Perfect! This robot is ready for the world."

But then, you try talking to it out loud. Suddenly, the robot stammers, misunderstands your casual requests, or gives you the wrong answer. Why? Because you only tested it on how it reads, not how it listens.

This is exactly the problem the paper "Do What I Say" (or DOWIS) addresses.

The Problem: The "Text-Only" Blind Spot

Most researchers test "Speech Large Language Models" (AI that can hear and speak) using text prompts. It's like testing a chef's ability to cook a meal by only reading the recipe to them, never letting them hear you say, "Hey, could you chop these onions?"

The authors argue this is unfair. In the real world, we talk to our devices. We say, "Summarize this meeting," or "Translate what that person just said." If we only test AI with typed text, we get an overly optimistic view of how smart it really is.

The Solution: DOWIS (The "Voice-First" Toolkit)

The researchers created DOWIS, a massive new dataset. Think of it as a universal remote control for testing AI voices.

• It's Multilingual: They didn't just do English. They covered 11 languages (like German, Spanish, Russian, etc.), making sure the AI works across different cultures.
• It's Diverse: They didn't just say "Translate this." They recorded 10 different ways to ask for the same thing:
  • Formal: "Please provide a translation."
  • Casual: "Hey, can you translate this?"
  • Short: "Translate."
  • Detailed: "Translate this, but keep the tone friendly."
• It's Human: Unlike other datasets where computers read text to simulate voices, DOWIS used real humans recording these prompts on their phones and laptops. This captures the natural "noise" and rhythm of real human speech.

The Experiment: Putting AI to the Test

The authors took two of the smartest AI models available today (named Phi-4 and Qwen) and put them through the DOWIS gauntlet. They asked the models to do nine different tasks, from translating languages to summarizing meetings and answering questions.

The Big Reveal: The "Text vs. Voice" Gap

Here is what they found, using a simple analogy:

1. The "Text" Advantage (The Optimistic Lie)
When the AI was given instructions via text, it performed brilliantly. It was like a student acing a written exam.

• The Catch: When the exact same instructions were given via spoken voice, the AI's performance dropped significantly, especially for tasks where the answer is written text (like translation or summarization).
• The Metaphor: It's like a musician who can play perfectly when reading sheet music but freezes up when someone hums the tune to them. The text prompts were hiding the AI's weaknesses.

2. The "Voice" Exception
Interestingly, when the task required the AI to speak back (like Text-to-Speech or translating spoken words to spoken words), the gap disappeared. The AI performed just as well with voice prompts as it did with text.

• Why? It seems the AI is comfortable "thinking" in text but struggles to "listen" to text-like instructions when it needs to output a written answer.

3. The "Style" Factor
The way you ask matters.

• Formal and Detailed instructions worked best. The AI loves clear, structured orders.
• Casual and Short instructions (like "Hey, do this thing") were the hardest for the AI to follow. It's like the AI gets confused by slang or brevity.

4. The Gender Bias
They also noticed that the AI sometimes performed slightly differently depending on whether a man or a woman was speaking the prompt. This suggests the AI might have hidden biases based on the speaker's voice, a problem that needs fixing.

The Takeaway

The paper concludes that we cannot trust current AI evaluations that only use text. If we want to build AI that truly understands us in the real world, we need to test it the way we actually use it: by talking to it.

DOWIS is the tool that finally lets researchers say, "Okay, let's stop testing the robot's reading skills and start testing its listening skills." It ensures that when we finally talk to our AI assistants, they won't just be smart on paper—they'll be smart in the conversation.

Technical Summary

Here is a detailed technical summary of the paper "Do What I Say: A Spoken Prompt Dataset for Instruction-Following" (DOWIS).

1. Problem Statement

Speech Large Language Models (SLLMs) have advanced rapidly, yet their evaluation remains flawed. Current benchmarks primarily rely on text prompts to evaluate instruction-following (IF) capabilities, which fails to reflect real-world scenarios where users interact via voice.

• The Gap: Existing spoken instruction benchmarks (e.g., SpeechInstructBench, Uro-Bench) are limited by:
  • Synthetic Data: Instructions are often generated via Text-to-Speech (TTS) rather than human recording.
  • Lack of Reusability: Instructions are pre-concatenated with task inputs, preventing pairing with other datasets.
  • Narrow Scope: They focus mostly on monolingual English/Chinese QA and lack support for cross-lingual tasks or diverse speech-to-speech modalities.
  • Limited Diversity: They rarely cover variations in prompt style (formal vs. informal) or speaker demographics.

2. Methodology: The DOWIS Dataset

The authors introduce DoWhatISay (DOWIS), the first multilingual dataset of human-recorded parallel spoken and textual prompts designed to be decoupled from task inputs, allowing it to be paired with any existing benchmark.

• Scope:
  • Tasks: 9 distinct tasks covering the full SLLM modality spectrum:
    • Speech-to-Text: Automatic Speech Recognition (ASR), Speech QA (SQA), Audio Chaptering (ACHAP), Speech Translation (ST), Speech Summarization (SSUM).
    • Text-to-Text: Machine Translation (MT), Text Summarization (TSUM).
    • Text-to-Speech: Text Synthesis (TTS).
    • Speech-to-Speech: Speech-to-Speech Translation (S2ST).
  • Languages: 11 languages (German, English, Italian, Czech, Spanish, French, Hungarian, Dutch, Portuguese, Russian, Swedish).
  • Prompt Variants: 10 prompts per task-language pair, categorized into 5 styles: Basic, Formal, Informal, Detailed, and Short.
• Data Collection:
  • Prompt Generation: Researchers specialized in specific tasks created 2 English prompts per style (10 total per task).
  • Translation: Native speakers translated these prompts into the 10 target languages, ensuring natural phrasing.
  • Recording: 19 native/proficient speakers (9 male, 10 female) recorded the prompts using personal devices (phones/laptops) to simulate realistic meeting scenarios.
  • Processing: Audio was converted to .wav, trimmed of silence using loudness-based VAD (Voice Activity Detection), and padded to preserve natural onset/offset.
  • Total Volume: 3 hours and 17 minutes of audio.

3. Experimental Setup

• Models Evaluated: Two state-of-the-art SLLMs:
  • Phi-4 Multimodal (supports text and speech input; text output only).
  • Qwen2.5-Omni-7B (supports text and speech input/output).
• Benchmarks: DOWIS prompts were paired with standard datasets (e.g., FLEURS for ASR/MT/ST, MCIF for Summarization/QA, YTSeg for Chaptering).
• Metrics: Standard metrics were used based on task type:
  • Text Output: WER (ASR), CometKiwi (MT/ST), BERTScore (Summarization/QA).
  • Speech Output: UTMOS (quality), WER/CometKiwi (content accuracy after transcribing generated audio).
• Variables: The study analyzed the impact of Prompt Modality (Text vs. Speech), Prompt Style (5 categories), Language, and Speaker Gender.

4. Key Results

The evaluation revealed significant discrepancies between text-based and speech-based evaluations:

• Text vs. Speech Prompts:
  • Text Output Tasks: Text prompts consistently outperformed spoken prompts. For tasks like ASR, MT, and ST, text prompts significantly overestimated model capabilities.
    • Example: Phi-4 failed catastrophically on ASR with spoken prompts (WER > 100), whereas it performed well with text prompts.
  • Speech Output Tasks: For TTS and S2ST, spoken prompts performed on par or slightly better than text prompts, suggesting the gap closes when the output modality matches the input.
• Prompt Style Impact:
  • Informal and Short prompts were consistently the most challenging, leading to lower performance across all tasks.
  • Formal and Detailed prompts yielded the best results, indicating models respond better to structured, explicit instructions.
  • Nuance: For TTS, formal/detailed prompts worked better with speech, while informal/short prompts worked better with text.
• Language and Modality Interplay:
  • Models struggled significantly more with spoken instructions in low-resource or cross-lingual settings (e.g., Czech, Dutch, Portuguese, Swedish) compared to high-resource languages.
  • Performance drops were attributed to the model's inability to follow spoken instructions rather than audio quality (transcription WER was low).
• Speaker Gender Bias:
  • Models showed inconsistent preferences for male vs. female speakers depending on the task (e.g., Qwen performed better with male prompts for summarization, but female prompts for translation).
  • This suggests the presence of speaker-related biases in the models, unrelated to audio intelligibility.

5. Key Contributions

1. DOWIS Dataset: The first multilingual, human-recorded, parallel spoken-text prompt dataset that is decoupled from task inputs, enabling flexible evaluation across any benchmark.
2. Comprehensive Coverage: Covers 9 tasks, 11 languages, and 5 distinct prompt styles, addressing the lack of diversity in current spoken IF benchmarks.
3. Empirical Evidence of Evaluation Bias: Demonstrates that relying solely on text prompts paints an overly optimistic picture of SLLM capabilities, particularly for text-output tasks and cross-lingual scenarios.
4. Style and Demographic Analysis: Highlights that prompt style (formality) and speaker gender significantly influence model performance, factors often ignored in standard evaluations.

6. Significance and Conclusion

The paper argues that the current reliance on text prompts for evaluating SLLMs is insufficient for real-world deployment. The findings indicate that:

• Realism is Critical: Evaluations must use human-recorded spoken instructions to accurately gauge model robustness.
• Modality Matters: The gap between text and speech performance is task-dependent; models are not yet robust enough to handle spoken instructions for text-generation tasks as effectively as they handle text prompts.
• Future Directions: The community needs diverse, human-recorded datasets like DOWIS to detect biases (gender, style) and improve the instruction-following capabilities of SLLMs in natural, spoken interaction scenarios.

The authors conclude that DOWIS serves as a vital resource for the community to move beyond synthetic or text-only evaluations toward a more realistic and comprehensive assessment of Speech Large Language Models.