Measuring User's Mental Models of Speech Translation in Human-AI Collaboration

Technical Summary: Measuring Users' Mental Models of Speech Translation in Human-AI Collaboration

Problem Statement
While millions of users rely on machine translation (MT) daily, particularly in speech-to-text scenarios, there is limited understanding of how users perceive a system's capabilities and limitations. Existing Human-AI collaboration research often focuses on classification or regression tasks with clear error boundaries, which do not translate well to MT where outputs are natural language and can be imperfect in diverse ways. Users need "MT literacy"—an accurate mental model (MM) of when a system succeeds or fails—to make informed decisions about trusting outputs or seeking human intervention. This is especially critical in speech translation, where audio variability amplifies output quality issues. The paper addresses the gap in measuring and understanding how users form these mental models in the context of speech translation.

Methodology
The authors propose a novel framework based on cross-lingual question answering (QA) to measure and update users' mental models.

• Task Design: Participants are presented with French speech passages translated into English by an MT system (Whisper). They must answer multiple-choice reading comprehension questions based on the translated text.
• Intervention Mechanism: For each sentence in the passage, users decide whether to accept the MT output "as-is" or request a "professional" re-translation (gold reference).
• Reward Structure: The user's score is determined by the correctness of the QA system's answer to the final passage, minus a penalty for every re-translation request. This incentivizes users to identify only the segments containing critical errors that would cause the QA system to fail, rather than re-translating everything.
• Experimental Variables:
  • Proficiency: Participants were categorized into Basic, Intermediate, and Fluent groups based on self-reported French proficiency and performance on quality estimation (QE) tasks.
  • Input/Output Features: The study analyzed errors across four categories: rare words/named entities, phonetic ambiguities/noise, domain-specific errors (Sports vs. Science), and surface-level cues (incompleteness/unnaturalness).
  • Explanations: Three conditions were tested: Default (no explanation), Transcription (displaying the ASR text), and Error Span (highlighting specific error segments in the translation).

Key Contributions

1. A New Framework for MM Measurement: The paper introduces a downstream task (cross-lingual QA) that moves beyond intrinsic quality estimation. It measures "fitness for purpose," forcing users to weigh the cost of re-translation against the risk of translation errors affecting their final decision.
2. Empirical Analysis of MM Evolution: The study tracks how mental models evolve over time through interaction, analyzing reward trends, QA accuracy, and "simple regret" (the gap between optimal and actual performance).
3. Feature Salience Analysis: It identifies which error characteristics users rely on to update their mental models, distinguishing between surface-level cues and deeper semantic or domain-specific errors.
4. Evaluation of Explanations: The paper compares the efficacy of different explanation types (transcriptions vs. error spans) in fostering robust mental models versus merely improving immediate task accuracy.

Key Results

• Learning Over Time: Users generally refine their mental models as they progress through the task, evidenced by increasing rewards and QA accuracy, and decreasing simple regret.
• Impact of Proficiency:
  • Fluent and Intermediate users showed consistent improvements in building mental models.
  • Basic users struggled to update their models, often resorting to over-reliance (selecting too many sentences for re-translation) to avoid penalties, indicating a failure to develop an accurate MM.
• Feature Detection:
  • Surface-level cues (incompleteness, unnaturalness) and phonetic noise were the most salient features; users quickly learned to detect these errors.
  • Topic-related errors (e.g., domain-specific terminology) were the hardest to detect and leverage for model updates.
  • Rare words/named entities were detected early but did not show significant learning curves, suggesting users relied on these cues from the start.
• Explanation Effectiveness:
  • Transcription: Providing ASR transcriptions was the most effective method for developing robust mental models, particularly for intermediate users. It offered clues about error sources (audio representation) without encouraging blind over-reliance.
  • Error Spans: While highlighting error spans increased QA accuracy (users selected more correct segments), it led to lower final scores due to excessive re-translation requests. This suggests error spans encourage over-reliance on the system's detection rather than fostering an independent mental model of the MT system's behavior.

Significance and Claims
The paper claims that cross-lingual QA is a promising interactive framework for studying and improving MT literacy. It demonstrates that users can learn to predict system failures, but this learning is heavily dependent on language proficiency and the type of feedback provided.

• For Design: The findings suggest that interfaces should provide transcriptions to help users understand error sources (especially for intermediate users) rather than simply highlighting errors, which may lead to inefficient behavior.
• For Research: The study provides a foundation for extending mental model research beyond simple classification tasks to complex generation tasks like summarization and dialogue.
• Modesty: The authors acknowledge limitations, noting that the study was restricted to French as the source language and had a limited sample size. They caution that their findings regarding language proficiency and specific error features may not fully generalize to languages with different phonetic or morphological structures without further validation. They frame the work as a step toward understanding human-AI collaboration in high-stakes translation scenarios rather than a complete solution for all MT literacy challenges.