Technical Summary: Interactive In-Meeting Speaker Correction with Human Feedback

Problem Statement

Automatic Speech Recognition (ASR) and speaker diarization systems typically operate in an "open loop" mode, lacking mechanisms for real-time user intervention. While state-of-the-art systems have advanced, they still struggle with noisy backgrounds, overlapping speech, and long-tail vocabulary. Crucially, speaker attribution errors—who said what—can severely undermine transcript reliability. Existing correction workflows are predominantly post-hoc, requiring costly manual annotation after the meeting concludes. There is a need for a practical, lightweight approach that allows speakers to correct attribution errors during the conversation without disrupting the natural flow or requiring them to review overwhelming amounts of text.

Methodology

The authors propose an LLM-assisted, in-meeting interactive correction system designed to operate under three practical constraints: (1) users must not be overwhelmed by full transcripts; (2) feedback must be lightweight (e.g., simple verbal corrections); and (3) the underlying ASR and diarization must be streaming (making immediate decisions without future context).

The workflow consists of the following components:

1. Streaming Transcription & Segmentation: The system processes live audio using a streaming ASR (Google Speech-to-Text) and a speaker embedding model (ECAPA-TDNN) to generate a speaker-attributed transcript.
2. Split-When-Merged (SWM) Refinement: To address the issue where streaming ASR segments often merge speech from multiple speakers, the system employs a post-processing step. SWM recursively partitions segments into smaller, speaker-consistent sub-segments using word-level speaker evidence and a sliding window voting strategy. This increases the "purity" of speaker assignment within units.
3. LLM Summarization: Instead of displaying the full transcript, an LLM generates concise summaries of the most recent $L$ utterances. These summaries highlight each speaker's main contributions, allowing users to quickly identify attribution errors without reading the entire log.
4. User Feedback & Correction: Users provide brief corrective feedback (e.g., "Hey Cobi, it wasn't Sara who mentioned overfitting; it was Rosa").
5. LLM Corrector & Remediation: An LLM processes the feedback to:
   • Narrow the Correction: Identify the specific text span within a segment that requires re-assignment, rather than correcting the entire segment.
   • Re-assign Speakers: Update the transcript with the correct speaker label for the identified span.
   • Online Enrollment: Extract a new speaker embedding from the corrected span (if it meets duration criteria, e.g., 5–20 seconds) and add it to the speaker enrollment set. This dynamically improves future speaker assignment by adapting to the speaker's voice in the current acoustic environment.

Evaluation Strategy

Due to the difficulty of conducting large-scale live user studies, the authors developed an LLM-driven user feedback simulation. In this pipeline:

• An LLM compares the system-generated summary against the ground-truth transcript to identify attribution errors.
• If an error is found, the LLM generates a natural-language correction message mimicking a user (starting with the wake word "Hey Cobi").
• A verification step ensures the proposed correction is reasonable before it is fed into the system.
• This simulation was validated on 8 meetings from the AMI training set, achieving 78.9% correction accuracy.

Key Results

Experiments were conducted on the AMI Headset-mix test set (16 meetings). The proposed workflow was compared against a streaming baseline (Google ASR + ECAPA) and an oracle "up-front enrollment" strategy.

• Performance Gains: The full proposed workflow reduced the Diarization Error Rate (DER) by 31.99% and the Speaker Error (SErr) by 52.68% relative to the uncorrected baseline.
• Component Analysis:
  • SWM alone reduced DER from 36.32% to 31.60%.
  • Online Enrollment alone reduced DER to 26.82%.
  • Combined Approach: The combination of SWM and Online Enrollment yielded the best results (24.70% DER), demonstrating that the components are complementary.
• Comparison to Up-Front Enrollment: The interactive online enrollment strategy outperformed a strategy where speakers provided multiple enrollment samples at the start of the meeting. This suggests that correcting errors dynamically based on actual attribution mistakes is more effective than relying solely on initial samples, likely because it captures acoustic conditions (e.g., noise changes) encountered later in the meeting.
• Summary vs. Full Transcript: Using LLM-generated summaries for correction was more effective and efficient than displaying the full conversation, providing sufficient context while minimizing user cognitive load.

Significance and Claims

The paper claims to demonstrate that precise, adaptive feedback can substantially improve speaker-attributed transcription in real-time settings. The significance of the work lies in:

1. Workflow Innovation: Proposing a practical, in-meeting correction loop that balances user convenience (lightweight feedback) with system accuracy (fine-grained updates).
2. Dynamic Adaptation: Showing that online enrollment, triggered by user corrections, is a powerful mechanism for overcoming domain shifts and improving future diarization accuracy without burdensome initial setup.
3. Scalable Evaluation: Introducing an LLM-driven simulation method that allows for reproducible, large-scale evaluation of interactive correction workflows on standard datasets like AMI.

The authors acknowledge limitations, noting that the evaluation relies on simulated rather than live user feedback and that the current system focuses specifically on speaker attribution rather than lexical ASR errors. They suggest future work could extend this interactive framework to handle word-level corrections and boundary errors.