SEAM: Shortcut-Aware Real-Time Detection of Scripted vs. Spontaneous Speech for Interview Guardrails

Technical Summary: SEAM – Shortcut-Aware Real-Time Detection of Scripted vs. Spontaneous Speech

1. Problem Statement

The detection of scripted versus spontaneous speech is critical for applications such as conversational system guardrails and corpus curation. However, existing benchmarks often suffer from inflated performance metrics driven by "shortcuts" rather than genuine speaking-style recognition. Models frequently rely on spurious correlations tied to corpus identity, channel conditions (e.g., microphone quality, room acoustics), and recording artifacts (e.g., codec effects) instead of prosodic or temporal cues.

In real-world deployment scenarios, such as AI-assisted job interviews, a model that learns these shortcuts fails when faced with modest shifts in recording conditions or corpus sources. The challenge is twofold: developing a system compact enough for low-latency, real-time inference alongside other speech components, and ensuring the training and evaluation pipeline actively discourages shortcut learning to guarantee robustness in external domains.

2. Methodology: The SEAM Framework

The authors propose SEAM (Shortcut-aware Evaluation, Augmentation, and Modeling), a framework designed to coordinate task design, data curation, preprocessing, and evaluation to mitigate shortcut learning.

• Data Curation and Preprocessing:

  • Corpora: The training data aggregates four English corpora: People's Speech and PodcastFillers (spontaneous) and LibriSpeech and Spoken Wikipedia (scripted).
  • Uniform Waveform Preprocessing: To suppress channel-level variation, all audio is converted to mono 16 kHz and processed through a pipeline including DC removal, a 70 Hz high-pass filter, integrated loudness normalization (−23 LUFS), and peak limiting.
  • Seam-Aware Sampling: Training windows are sampled strictly from provenance-consistent segments within a single source recording. This prevents the model from learning artificial boundaries or "seams" created by joining different recordings, which could serve as unintended style cues.
  • Non-Speech Augmentation: To break the heuristic that "clean audio implies scripted speech," a dedicated noise bank (containing room tone, breathing, and microphone artifacts) is injected into training windows with a random overlap ratio of 40–70%.

• Model Architecture:

  • Backbone: The system utilizes DistilHuBERT, a compact self-supervised speech encoder chosen for its favorable quality-efficiency tradeoff.
  • Classifier: A lightweight 2-layer MLP with ReLU activations and dropout follows temporal mean pooling of the encoder outputs.
  • Training Strategy: The authors employ shallow adaptation regimes rather than full end-to-end fine-tuning. The final configuration unfreezes only the top transformer layer of the encoder while keeping the rest frozen, optimizing for real-time deployment constraints.

• Evaluation Protocol:

  • Internal: Grouped 80/10/10 splits (train/eval/test) based on speaker, user, or recording-family keys to minimize data leakage.
  • External: A proprietary interview-domain dataset is used exclusively for evaluation. This set is deliberately adversarial, crossing speaking styles with varied recording conditions (clean vs. mixed room/microphone) to test robustness against corpus and channel shifts.

3. Key Results

• Performance: Under the full-training regime (240 hours per class), the final SEAM model achieves 0.971 ± 0.004 ROC-AUC on the external interview-domain evaluation set and 0.9766 ± 0.0045 on the internal test set.
• Ablation of Shortcut Prevention: Removing shortcut-prevention components (noise augmentation and seam-aware sampling) improves internal held-out metrics (e.g., internal test AUC rises from 0.9287 to 0.9557) but sharply degrades external performance (external AUC drops from 0.8991 to 0.7324). This confirms that naive benchmark gains often reflect shortcut learning rather than improved style modeling.
• Hyperparameters:
  • Window Length: 8-second windows provide the optimal balance between accuracy and latency; 12-second windows offer marginal gains at the cost of higher latency.
  • Adaptation Depth: Unfreezing only the top transformer layer yields superior results compared to training only the head or unfreezing deeper layers.
• Deployment Efficiency:
  • Quantization: Post-training quantization reduces the model footprint from ~90 MB to 41.8 MB (INT4) with negligible loss in external accuracy (95.35% vs. 95.17%) and AUC (0.9743 vs. 0.9713).
  • Latency: The system operates with a real-time factor (RTF) of approximately $5.60 \times 10^{-4}$ on an NVIDIA L4 GPU.

4. Significance and Claims

The paper argues that robust scripted-versus-spontaneous detection depends as much on data construction and evaluation design as on the choice of the backbone model. The primary contribution is demonstrating that "shortcut-aware" components—specifically uniform preprocessing, seam-aware sampling, and non-speech augmentation—are essential for transferring performance from internal benchmarks to real-world, external domains.

The authors position SEAM not as a standalone decision-maker but as a narrow integrity signal for human review in interview guardrails. They acknowledge limitations, noting that the system is English-first, that scriptedness remains entangled with genre and recording practices, and that zero-shot transfer to non-English languages is only partial. The work emphasizes that standard held-out accuracy can be misleading if it rewards corpus-specific heuristics, advocating for transfer-oriented evaluation as a necessary standard for deployment-ready audio models.