Beyond Transcripts: Iterative Peer-Editing with Audio Unlocks High-Quality Human Summaries of Conversational Speech

Technical Summary: Beyond Transcripts: Iterative Peer-Editing with Audio Unlocks High-Quality Human Summaries of Conversational Speech

Problem Statement
The field of conversational speech summarization lacks established benchmarks for unstructured, casual settings, relying heavily on formal meeting datasets like AMI and ICSI. While Large Language Models (LLMs) offer powerful generation capabilities, embedding their outputs into benchmarks risks propagating systemic errors, hallucinations, and prompt-dependent variability. Consequently, human annotation remains the gold standard for creating robust ground truth. However, using human annotators introduces specific tradeoffs:

1. Input Modality: Annotators working directly with audio (rather than transcripts) face higher cognitive loads due to the temporal nature of speech, potentially resulting in summaries that are less informative or overly compressed.
2. Quality Comparison: Human-written summaries are often perceived as less fluent and coherent than LLM-generated outputs, raising concerns that human benchmarks may be of lower overall quality.

This study investigates whether specific human annotation workflows can mitigate these tradeoffs, specifically testing if iterative peer-editing can enable audio-based summaries to match the informativeness of transcript-based summaries and LLM outputs.

Methodology
The researchers designed a systematic study involving 13 telephone conversations from the Switchboard Dialogue Act (SWBDA) and CallHome datasets. To ensure data integrity, conversations were standardized to five minutes, and manual transcripts were used to avoid ASR contamination.

• Annotators: 18 native English-speaking U.S. citizens (students and professionals) were employed.
• Annotation Workflows: Ten distinct workflows were tested, varying input modality (audio, transcript, or both) and editing strategies (self-editing, peer-editing, and iterative peer-editing).
  • Baseline: Audio-only (A) and Transcript-only (T) summaries.
  • Single-Edit: Self-editing with alternate modality (ATself, TAself) and peer-editing with single or dual modalities (A2peer, T2peer, ATApeer, TTApeer).
  • Iterative Peer-Editing: Four rounds of peer-editing where a new annotator edits the summary using only the original modality (A2–5peer, T2–5peer).
• LLM Benchmarks: Four models generated comparison summaries: GPT-4o, Gemini-2.5-flash, Llama-3.2-1B-Instruct (text-based), and GPT-4o-Audio-preview (audio-based).
• Evaluation Metrics:
  • Quality: G-Eval (fluency, coherence, consistency, relevance).
  • Informativeness: CREAM (Comparison-Based Reference-Free ELO-Ranked Automatic Evaluation), which extracts key facts from concatenated summaries and measures the ratio of facts supported by individual summaries.
  • Lexical/Semantic: ROUGE, BertScore, BartScore, lexical overlap, and compression ratios.

Key Contributions

1. Validation of Modality Tradeoffs: The study confirms that un-edited, audio-based summaries are significantly shorter, less informative, and less abstractive than transcript-based summaries. They exhibit higher lexical overlap with the source but lower CREAM scores.
2. Iterative Peer-Editing as a Solution: The authors identify iterative peer-editing as the most effective methodology to bridge the gap between modalities. Specifically, iterative peer-editing of audio-based summaries (using only audio as the reference) yields summaries that are as informative as their transcript-based counterparts.
3. Human vs. LLM Parity: The study demonstrates that human-written summaries, when subjected to iterative peer-editing, achieve informativeness levels comparable to the best-performing LLMs (specifically Gemini and GPT-4o). In many cases, iteratively edited human summaries are statistically indistinguishable in informativeness from top-tier automated outputs.

Results

• Audio vs. Transcript: Initial audio summaries (Row 2 in Table 1) had a CREAM score of 0.19 compared to 0.38 for transcript summaries (Row 1). They were also more compressed (0.27 vs. 0.55 ratio).
• Editing Impact: Self-editing (ATself) did not significantly improve informativeness over the baseline. However, a single round of peer-editing on audio (A2peer) increased the CREAM score to 0.41, effectively closing the gap with transcript-based peer-edited summaries (T2peer at 0.44).
• Stabilization: After the first round of iterative peer-editing, further rounds (A3–5peer) did not yield significant gains in informativeness, suggesting the informational content stabilizes quickly.
• LLM Comparison: While LLMs generally outperformed un-edited human summaries, the iteratively peer-edited human summaries (A3–5peer and T2–5peer) were not significantly less informative than any of the four LLM benchmarks. Notably, the Llama model produced the least informative summaries, while Gemini produced the most.
• Abstractiveness: Peer-editing with audio significantly increased the abstractiveness of summaries (lower lexical overlap with the transcript) compared to self-editing or dual-modality editing.

Significance and Claims
The paper claims that its findings validate iterative peer-editing among human annotators as a viable workflow for creating high-quality, prosody-informed speech benchmarks. This is significant because it enables the collection of robust datasets in domains where high-quality transcripts are unavailable or too expensive to produce manually (e.g., where ASR systems perform poorly).

The authors assert that this workflow allows human annotators to produce summaries that are:

1. As informative as transcript-based summaries.
2. As informative as state-of-the-art LLM summaries.
3. Capable of incorporating prosodic cues without sacrificing informational density.

The study concludes that while human annotators initially struggle with the cognitive load of audio-only summarization, a structured, iterative peer-review process effectively mitigates these limitations, offering a rigorous alternative to automated benchmark generation. The authors note limitations regarding the small corpus size (13 conversations) and the lack of a specific metric to quantify the inclusion of prosodic information, identifying these as areas for future work.