Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Code-Switching Beyond Standard UD Assumptions

Technical Summary: Lost in Speech

Problem Statement
Spoken code-switching (CSW) presents a fundamental challenge for syntactic parsing that is distinct from the challenges observed in written text. While parsers and Large Language Models (LLMs) perform well on written data, they frequently fail on spoken CSW due to intrinsic conversational features such as disfluencies, repetitions, ellipsis, fragmentary clauses, and discourse-driven structures. These phenomena routinely violate the assumptions underlying canonical annotation frameworks like Universal Dependencies (UD), specifically regarding clause completeness and canonical head selection.

Furthermore, existing evaluation practices exacerbate the problem. Standard metrics, such as Labeled Attachment Score (LAS) and Unlabeled Attachment Score (UAS), enforce strict matching against a single gold tree. This approach conflates genuine structural errors with linguistically plausible variations acceptable in spoken discourse. Consequently, parsers are penalized for producing analyses that are defensible within the context of spoken language, leading to an underestimation of their performance and a misinterpretation of failure modes. The authors argue that scaling model capacity or data alone is insufficient, as many challenges stem from the mismatch between written-language annotation conventions and the incremental, ambiguous nature of speech.

Methodology
The paper adopts a systems-oriented approach, separating the handling of spoken-language phenomena from core syntactic analysis. The methodology is built upon four key components:

1. Taxonomy of Spoken CSW Phenomena: Through a data-driven analysis of the Miami Corpus (English-Spanish), the authors developed a linguistically grounded taxonomy of nine categories that undermine canonical parsing assumptions. These include:

   • Repetition: Recurring words/phrases used for floor-holding.
   • Discourse Elements: Pragmatic tokens (e.g., interjections) distinct from structural material.
   • Ellipsis: Incomplete structures not subsequently repaired.
   • Contractions: Tokens encoding multiple syntactic units (e.g., "wasn't", "del").
   • Compound/MWEs: Multi-word expressions functioning as single units.
   • Break of Thought: Abandoned phrases replaced by revised continuations.
   • Filler Words: Specific tokens used to hold the floor (e.g., "umm").
   • Slang/Curses: Context-dependent items with unclear syntactic roles.
   • Enclisis: Verbs with attached clitic pronouns (common in Spanish).

2. SpokeBench: A gold-standard benchmark designed to evaluate parsing under spoken-language conditions. Unlike existing resources, SpokeBench contains expert-annotated, dispute-resolved, and category-balanced UD annotations for 126 sentences from the Miami Corpus. It specifically targets structural complexity and spoken phenomena, utilizing a graded acceptability protocol to distinguish between substantive errors and acceptable variation.

3. FLEX-UD Metric: A new evaluation metric designed to be ambiguity-aware. Unlike standard metrics that enforce a single gold tree, FLEX-UD employs graded tolerances to compare system outputs against gold standards. It distinguishes between:

   • Catastrophic Errors: Violations undermining global structural validity (e.g., missing MWEs, invalid head references), which incur high penalties.
   • Minor Errors: Linguistically plausible deviations (e.g., tolerant POS substitutions, near-miss dependency relations), which incur low penalties.
     This allows FLEX-UD to reward linguistically defensible analyses that standard metrics would reject.

4. DECAP (DECoupled Agentic Parser): A modular parsing framework that isolates spoken-language handling from core syntactic analysis. DECAP utilizes a fixed sequence of specialized agents:

   • Spoken-Phenomena Handler (SPH): Identifies and localizes disfluencies (repetitions, fillers, etc.) and proposes tokenization edits without assigning syntactic relations.
   • Language-Specific Resolver (LSR): Applies conservative, language-aware normalization (e.g., expanding contractions, validating MWEs) and provides lemmas.
   • Core UD Structure Assigner: Constructs the dependency parse over the normalized representation, enforcing constraints like a single root and preserving disfluencies as structural elements rather than forcing ill-formed reconstructions.
   • Verifier and Ranker (V/R): Ensures global structural well-formedness (acyclicity, valid heads) and outputs confidence scores and penalty logs.
     The framework is instruction-grounded, leveraging LLMs to follow linguistic guidelines without requiring fine-tuning on spoken data.

Key Contributions

• Taxonomy: A structured account of spoken CSW phenomena that complicate parsing, derived from empirical analysis of the Miami Corpus.
• SpokeBench: The first expert-annotated gold benchmark for spoken CSW parsing that balances coverage of spoken phenomena with structural complexity.
• FLEX-UD: An ambiguity-aware evaluation metric that penalizes catastrophic structural errors more heavily than acceptable variation, revealing qualitative improvements masked by standard metrics.
• DECAP: A decoupled agentic parsing framework that improves robustness and interpretability by isolating spoken-phenomena handling from core syntactic analysis, achieving significant performance gains without retraining.

Results
Experiments conducted on SpokeBench using English-Spanish code-switched data compared DECAP against traditional bilingual parsers and the LLM-based BiLingua pipeline.

• Standard Metrics: Under traditional LAS and UAS metrics, all systems showed substantial degradation on spoken phenomena (repetition, ellipsis, discourse) compared to canonical sentences. DECAP outperformed baselines, improving overall LAS from 0.31 (BiLingua) to 0.48, but standard metrics compressed these gains.
• FLEX-UD Evaluation: FLEX-UD revealed clearer separation between systems. DECAP achieved the highest overall score (76.2), significantly outperforming BiLingua (70.7) and traditional parsers (30.4). The metric highlighted that DECAP's success was driven not just by higher head/label accuracy, but by a substantial reduction in catastrophic structural errors.
• Linguistic Analysis: The study found that while UPOS-LAS performance correlates with surface lexical availability (repetition performing best), LAS performance is driven by structural ambiguity and recoverability (ellipsis and repetition performing worst). Bilingualism acts as a complexity multiplier, intensifying these ambiguities.

Significance and Claims
The paper claims that progress in spoken CSW parsing requires rethinking both parsing architectures and evaluation practices, rather than simply scaling model capacity. The authors argue that the "modality gap" between written and spoken language necessitates speech-aware language abstractions. By introducing SpokeBench, FLEX-UD, and DECAP, the work provides a foundation for evaluating and parsing spoken CSW that acknowledges structural uncertainty as an inherent property of the data. The authors emphasize that their approach allows for extensibility to new languages and phenomena without retraining, offering a scalable path forward for handling the non-canonical structures of conversational speech.

Limitations
The authors note that SpokeBench is intentionally small and curated to prioritize depth and linguistic complexity over broad coverage across speakers or dialects. The experiments are limited to English-Spanish code-switching, though the identified phenomena are common in other bilingual settings. Finally, DECAP relies on instruction-grounded LLMs rather than learned representations, making it dependent on prompt quality and the underlying model's capabilities.