StarDrinks: An English and Korean Test Set for SLU Evaluation in a Drink Ordering Scenario

1. Problem Statement

Despite rapid advancements in Large Language Models (LLMs) and speech assistants, their evaluation often relies on controlled, simplified scenarios that fail to capture the complexity of real-world user interactions. Specifically:

• Lack of Realism: Standard benchmarks use clean, structured inputs with limited vocabularies, ignoring spontaneous speech phenomena like hesitations, self-corrections, and disfluencies.
• Domain Complexity: Drink ordering involves diverse named entities, complex attribute combinations (size, temperature, milk type, toppings), and brand-specific terminology that vary in order and expression.
• Data Scarcity: There is a notable shortage of publicly available datasets for spoken drink ordering, particularly in multilingual settings. Existing resources (e.g., SNIPS, ATIS, SLURP) cover other domains but lack the specific linguistic variability and Korean language support required for realistic beverage ordering systems.

2. Methodology

The authors introduced StarDrinks, a test set designed to evaluate Speech-to-Slots (SLU), Transcription-to-Slots (NLU), and Speech-to-Transcription (ASR) capabilities. The dataset creation followed a four-stage pipeline:

A. Structured Data Generation

• Source: Real drink order receipts from a popular coffee chain in South Korea.
• Synthesis: Starting with 2,500 real receipts, the authors used a semantic schema to generate synthetic variations. They prioritized low-frequency structures to ensure diversity, creating up to 83,974 potential structures.
• Selection: A subset of 326 structures was selected to ensure coverage of all attribute-value pairs (e.g., drink types, sizes, customizations) for the annotation phase.

B. Data Collection

• Platform: Participants were recruited via Prolific.
• Languages & Demographics:
  • English: 32 native speakers (UK-based, ages 21–50).
  • Korean: 29 native speakers (recruited from UK/US, ages 21–50).
• Task: Participants viewed a digital receipt and recorded a single natural utterance ordering the items, including all necessary options (quantity, size, customizations, temperature).
• Naturalism: Participants were instructed to speak naturally; no specific disfluencies were encouraged, though they emerged naturally during recording.

C. Validation and Annotation

• ASR Transcription: Initial transcripts were generated using whisper-large-v3.
• Human Correction: Authors manually corrected transcripts and slot annotations to ensure alignment with the structured data. Utterances where participants failed the task were discarded.
• Final Dataset:
  • English: 255 validated samples (53 mins total).
  • Korean: 295 validated samples (45.7 mins total).
  • Total Slots: 15 distinct slot types covering 45 unique drink types and various customizations.

3. Key Contributions

1. StarDrinks Dataset: The first publicly available dataset for spoken drink ordering in both English and Korean.
2. Realistic Benchmark: Unlike synthetic datasets, StarDrinks captures spontaneous speech phenomena (hesitations, repetitions) and complex, brand-specific terminology.
3. Multi-Task Evaluation: The dataset supports three distinct evaluation modes:
   • ASR: Speech-to-Transcription.
   • NLU: Transcription-to-Slots (Text-based).
   • SLU: Speech-to-Slots (End-to-End).
4. Generalization Focus: The dataset is designed to test model robustness against previously unseen named entities (new menu items) and vocabulary variations.

4. Experimental Results

The authors evaluated a baseline agent using whisper-large-v3 for ASR and GPT-4o for NLU/SLU slot filling.

ASR Performance (whisper-large-v3)

• Word Error Rate (WER): 9.2% (English) and 22.9% (Korean).
• Observation: The model struggled significantly with domain-specific named entities (e.g., "youthberry" transcribed as "yuleberry," "cafe americano" as "caffi americanas"). This highlights the difficulty of adapting ASR to unseen vocabulary without fine-tuning.

NLU/SLU Performance (GPT-4o)

• Metrics: Unordered Exact Match (UEM) and Slot F1.
• Impact of ASR Noise:
  • In 0-shot settings, replacing gold transcripts with ASR output caused a massive drop in UEM: -11.76% for English and -17.96% for Korean.
  • In 3-shot settings, the gap narrowed significantly (only -2.75% for English and -5.08% for Korean), suggesting few-shot prompting helps the model correct ASR errors.
• Language Comparison: Korean results generally outperformed English (e.g., 89.83% vs. 87.06% UEM for 3-shot gold), likely because English allows for more flexible syntactic ordering of order attributes, making parsing harder.
• Best Performance:
  • English: 87.06% UEM (Gold Transcripts + 3-shot).
  • Korean: 89.83% UEM (Gold Transcripts + 3-shot).
• Limitation: Even the best scores are below the "near-perfect" threshold required for real-world deployment, indicating a need for significant in-domain adaptation data.

5. Significance and Future Directions

• Bridging the Gap: StarDrinks provides a critical resource for evaluating the robustness of LLMs and speech assistants in a linguistically rich, real-world task.
• Research Directions: The results emphasize the need for:
  • Context-biasing and Test-time ASR adaptation: To handle menu changes and new items without full retraining.
  • Robust NLU: Developing models that can infer correct slots despite noisy ASR input.
• Availability: The dataset is open for download to facilitate the development of more context-aware and robust spoken language systems.

In conclusion, StarDrinks exposes the limitations of current state-of-the-art models when faced with spontaneous, domain-specific speech, serving as a rigorous benchmark for the next generation of task-oriented dialogue systems.