Development and multi-center evaluation of domain-adapted speech recognition for human-AI teaming in real-world gastrointestinal endoscopy

1. Problem Statement

Automatic Speech Recognition (ASR) is a critical interface for Human-AI teaming in gastrointestinal endoscopy, enabling hands-free communication between physicians and AI agents. However, deploying general-purpose ASR models in this specific clinical setting faces two major challenges:

• Domain-Specific Terminology: Physicians use highly specialized medical vocabulary (e.g., Clopidogrel, Paris classification IIa, BBPS scores) that is sparsely represented in general-domain training corpora, leading to high transcription errors.
• Complex Acoustic Conditions: Endoscopy rooms are characterized by persistent operational noise (towers, suction devices, alarms) and multi-speaker interactions, which degrade the performance of standard models.
• Data Scarcity: Ethical constraints and privacy concerns limit the availability of large-scale, annotated intra-procedural speech data, making it difficult to train robust models.
• Real-World Generalization: Existing studies often rely on single-center or simulated data, failing to validate performance across heterogeneous hospitals, acoustic environments, and clinician speaking styles.

2. Methodology

The authors propose EndoASR, a domain-adapted Chinese ASR system designed for real-time, edge-deployed clinical workflows. The methodology involves a systematic framework of data construction, a two-stage adaptation strategy, and rigorous multi-center evaluation.

A. Data Construction

• Synthetic Training Corpus: To overcome data scarcity, the authors converted structured endoscopic reports (containing dense medical terminology) into synthetic speech using a commercial Text-to-Speech (TTS) system.
• Noise Augmentation: Real-world endoscopy room noise was recorded and mixed with the synthetic speech at controlled Signal-to-Noise Ratios (SNR) of 20–28 dB (more challenging than the real-world average of ~30 dB) to simulate adverse conditions.
• Evaluation Datasets:
  • Retrospective: 600 real intraoperative recordings from 6 endoscopists at a single center.
  • Prospective: 300 real intraoperative recordings from 5 independent endoscopy centers, covering 6 clinical content categories (PreOp, IntraOp, Lesion, Cancer, Inflammation, PostOp).

B. Model Architecture and Training Strategy

• Base Model: The system uses Seaco-Paraformer, a non-autoregressive end-to-end ASR model (220M parameters), chosen for its balance of accuracy and inference efficiency.
• Two-Stage Fine-Tuning:
  1. Stage 1 (Linguistic Adaptation): The base model is fine-tuned exclusively on clean synthetic speech. This adapts the model to domain-specific vocabulary, sentence structures, and reporting conventions without acoustic interference.
  2. Stage 2 (Acoustic Robustness): The Stage 1 model is further fine-tuned on noise-augmented synthetic speech. This enhances robustness to the specific acoustic interference found in endoscopy rooms while preserving the learned linguistic knowledge.
• Constraints: No real-world clinical data was used during training; all real data was reserved strictly for evaluation to ensure unbiased testing of generalization.

C. Evaluation Protocol

• Metrics: Character Error Rate (CER), BLEU-1, BERTScore, and a domain-specific Medical Term Accuracy (Med ACC).
• Workflow Integration: The system was tested in an end-to-end loop with a Large Language Model (LLM) to assess its impact on downstream structured information extraction and clinician-AI interaction.

3. Key Contributions

1. EndoASR System: A fully open-source, domain-adapted ASR system specifically optimized for Chinese gastrointestinal endoscopy, addressing a gap in medical AI infrastructure.
2. Synthetic Data Strategy: A novel approach to generate large-scale, high-quality domain-specific training data from structured reports, bypassing the need for massive annotated speech datasets.
3. Two-Stage Adaptation: A decoupled training strategy that separately addresses linguistic specialization and acoustic robustness, achieving superior performance without architectural changes.
4. Rigorous Multi-Center Validation: The first prospective evaluation of an endoscopic ASR system across five independent centers, demonstrating robustness to institutional and acoustic heterogeneity.
5. Efficiency: A compact model (220M parameters) with a Real-Time Factor (RTF) of 0.005, significantly faster than Whisper-large-v3 (RTF 0.055), enabling efficient edge deployment.

4. Key Results

Performance Improvements

• Retrospective Evaluation (Single Center):
  • CER: Reduced from 20.52% (Baseline Paraformer) to 14.14% (EndoASR-noise).
  • Medical Term Accuracy (Med ACC): Increased dramatically from 54.30% to 87.59%.
• Prospective Multi-Center Evaluation:
  • CER: Reduced from 16.20% to 14.97% compared to the baseline.
  • Med ACC: Improved from 61.63% to 84.16%.
  • Generalization: The model maintained consistent performance across all 5 centers and diverse clinical content categories, despite significant acoustic variability (visualized via t-SNE of MFCCs).

Downstream Impact

• LLM Integration: Improved ASR quality directly enhanced the accuracy of downstream structured information extraction (lesion location, morphology, procedural actions) by the LLM.
• Latency: The system provides near real-time feedback (~200ms delay), enabling fluid clinician-AI interaction.
• Error Propagation: Qualitative analysis showed that replacing the default ASR with EndoASR significantly reduced downstream extraction errors caused by misrecognized medical terms.

5. Significance

• Clinical Reliability: The study demonstrates that domain-adapted ASR can serve as a reliable interface for Human-AI teaming in high-stakes procedural environments, moving AI from offline analysis to real-time, interactive clinical support.
• Scalability: The synthetic data approach offers a scalable solution for other medical domains where annotated speech data is scarce.
• Safety and Trust: By significantly improving the recognition of critical medical terminology, the system reduces the risk of documentation errors, thereby enhancing patient safety and clinician trust in AI-assisted workflows.
• Edge Deployment: The model's small size and high efficiency make it feasible for deployment on local hardware (edge devices) within endoscopy suites, ensuring low latency and data privacy.

In conclusion, EndoASR bridges the gap between general-purpose speech models and the specific, noisy, terminology-heavy reality of gastrointestinal endoscopy, providing a validated foundation for the next generation of agent-based clinical AI.