KVoiceBench, KOpenAudioBench, and KMMAU: Agent-Driven Korean Speech Benchmarks for Evaluating SpeechLMs

Technical Summary: KVoiceBench, KOpenAudioBench, and KMMAU

Problem Statement
While Speech Language Models (SpeechLMs) have advanced significantly by extending Large Language Models (LLMs) to speech modalities, their evaluation remains heavily skewed toward English. Existing multilingual resources (e.g., SD-QA, FLEURS) improve coverage for dialectal QA and ASR but lack scalable frameworks for transferring modern SpokenQA benchmarks or constructing target-language audio understanding benchmarks.

Direct transfer strategies—such as translating transcripts, normalizing text, and synthesizing speech via TTS—introduce critical artifacts:

1. Instruction Corruption: Language-specific constraints (e.g., uppercase/lowercase distinctions in English) often fail to translate meaningfully to languages like Korean, which lack such orthographic features.
2. Normalization Errors: Converting written text to speech-friendly forms (handling numbers, dates, abbreviations) is error-prone. In SpokenQA, a minor normalization error can alter the answerability of a question.
3. Paralinguistic Loss: Transferring source-language audio fails to preserve target-language speaker attributes, accents, and paralinguistic properties, which are essential for audio understanding tasks.

Methodology
The authors propose two human-agent collaborative frameworks to construct high-quality Korean speech benchmarks, addressing the limitations of naive transfer.

1. SpokenQA Benchmark Transfer Framework (KVoiceBench & KOpenAudioBench)
This framework transfers English SpokenQA benchmarks (VoiceBench and OpenAudioBench) into Korean through four stages:

• Ground-Truth Correction: Two frontier LLM agents (a reviewer and a meta-reviewer) audit deterministic source samples (multiple-choice and short-answer) to identify and correct erroneous annotations (e.g., stale factual answers or overly restrictive answer sets) before translation.
• Hypertranslation: A core component where an LLM agent, guided by a human-constructed rulebook, converts source samples into Korean. The rulebook explicitly encodes grammatical, orthographic, and writing-system-specific properties. It distinguishes between:
  • Direct Adaptation: Localizing surface forms (e.g., option labels, proper nouns).
  • Language-Specific Redesign: Replacing or removing tasks dependent on English-specific features (e.g., converting English adjective-ordering tasks into Korean particle/conjugation judgments).
• Speech-Friendly Normalization: A separate rulebook guides the conversion of hypertranslated text into speech-friendly forms suitable for TTS. This handles context-dependent readings for numbers, units, symbols, and mixed-language expressions to ensure the synthesized audio matches the intended question semantics.
• TTS Synthesis: Normalized text is synthesized using Qwen3-TTS conditioned on Korean reference voices. Samples exceeding a Word Error Rate (WER) threshold against the normalized input are re-synthesized.

2. Audio Understanding Benchmark Construction Framework (KMMAU)
Rather than transferring source audio, this framework constructs benchmarks from naturally occurring Korean ASR corpora (KSS, KMSAV, Seoul Corpus) using audio, transcriptions, and speaker metadata. The construction method is selected based on the capability being tested:

• Rule-Based from Metadata: Generates questions about speaker attributes (age, gender, number of speakers) using provided metadata.
• Rule-Based from Transcriptions: Generates lexical questions (word order, word frequency) based on transcript analysis.
• LLM-Generated with Human Review: Creates questions for fact extraction and topic summarization, verified by humans to ensure naturalness and answerability.
• Fully Manual: Used for holistic capabilities (general counting, role/profession) that require listening to the audio for dialogue structure or non-lexical evidence.

Key Contributions

1. Frameworks: The proposal of reproducible human-agent collaborative frameworks for transferring SpokenQA benchmarks and constructing target-language audio understanding benchmarks.
2. Datasets: The construction and public release of three Korean benchmarks comprising 12,345 samples:
   • KVoiceBench: 7,306 samples derived from VoiceBench.
   • KOpenAudioBench: 2,835 samples derived from OpenAudioBench.
   • KMMAU: 2,204 samples constructed from Korean ASR corpora.
   • Note: The construction process includes "rulebooks" that make the methodology auditable and reusable for other target languages.
3. Evaluation: A comprehensive evaluation of eight recent SpeechLMs across English and Korean SpokenQA and audio understanding tasks.

Results
Evaluation of eight SpeechLMs (including Raon-Speech, Qwen2.5-Omni, MiniCPM-o 4.5, etc.) reveals:

• Performance Gaps: There is a substantial performance drop from English to Korean SpokenQA, but the degradation is not uniform. For instance, Raon-Speech maintains the highest performance on Korean benchmarks, while other models show significant drops.
• Task Sensitivity: Reasoning tasks (e.g., BBH) show relative robustness, whereas knowledge-heavy and structured tasks suffer larger drops. Safety behaviors also vary; some models drastically reduce refusal rates in Korean compared to English.
• Divergent Rankings: Audio understanding rankings diverge from SpokenQA rankings. For example, a model ranking second in Korean SpokenQA may rank third in audio understanding, while another model with weaker SpokenQA performance ranks second in audio understanding.
• Capability Specifics: In KMMAU, models show varying strengths: some excel at word order detection, others at contextual fact extraction, while speaker count detection remains difficult for all models (≤36% accuracy).

Significance
The paper claims that English-only evaluations fail to reveal complementary weaknesses in SpeechLMs. By introducing these Korean benchmarks, the authors demonstrate that:

1. Target-language question answering and naturally grounded audio understanding probe different capabilities.
2. The performance gap between English and Korean varies significantly across models and task families, suggesting that model robustness is task-dependent rather than monolithic.
3. The proposed human-agent frameworks provide a replicable methodology for extending speech benchmarks to additional languages, moving beyond simple translation to address language-specific validity issues.

The authors note limitations, including the use of synthesized speech for SpokenQA (which lacks natural human variability) and the fact that KMMAU is not a direct translation pair of English benchmarks, making cross-benchmark comparisons references rather than controlled language-pair gaps.