Technical Summary: VSRo-200

Problem Statement

Visual Speech Recognition (VSR), or lip reading, faces significant challenges in low-resource languages due to the inherent ambiguity of visual speech (e.g., multiple phonemes mapping to the same viseme) and a scarcity of large-scale, high-quality annotated datasets. While English VSR has benefited from large sentence-level corpora (e.g., LRS2, LRS3) and multilingual approaches utilizing weak supervision, Romanian VSR remains under-resourced. Existing Romanian resources, such as LRRo, are limited to small-scale, isolated word recognition tasks. Furthermore, a fundamental trade-off exists in current VSR datasets: they are either small but carefully manually annotated or large-scale but rely predominantly on pseudo-labels generated by Automatic Speech Recognition (ASR) systems. This gap prevents a controlled analysis of how supervision quality impacts model performance and robustness in low-resource settings.

Methodology

The authors introduce VSRo-200, the first large-scale dataset for Romanian VSR, comprising 200 hours of real-world podcast videos sourced from YouTube.

Data Construction and Preprocessing

• Source Material: The dataset consists of podcast episodes featuring a host and a single guest in a fixed multi-camera setup. To ensure identity diversity, episodes with single guests were selected, ensuring each individual appears in only one episode.
• Segmentation: Long-form recordings were partitioned into shots based on camera angle changes. Visual filtering using InsightFace retained only segments with a single visible speaker, removing host appearances via face embedding comparison.
• Audio Filtering: Speaker diarization (Pyannote) was applied to identify the guest as the primary speaker within a shot. Audio was segmented into 3–30 second clips containing only guest speech.
• Synchronization: Clips underwent face tracking (S3FD) and audio-visual synchronization verification (SyncNet). Clips with synchronization offsets >±10 frames or face resolutions <96×96 pixels were discarded. Final outputs are 224×224 face crops at 25 FPS with aligned 16 kHz audio.
• Annotation Strategy:
  • Pseudo-labels: The entire 200-hour dataset was annotated using a Whisper-large model fine-tuned for Romanian.
  • Human Annotations: A subset of 100 hours, selected for visual clarity and lack of overlapping speakers, was transcribed by humans following strict orthographic guidelines. This allows for a controlled comparison between weak (pseudo) and full (human) supervision.

Experimental Setup

The authors established a benchmark using a transformer-based VSR architecture (following MultiVSR), consisting of a Visual Transformer Pooling (VTP) encoder and a Transformer encoder-decoder.

• Supervision Analysis: Models were trained on varying data scales (10h to 200h) using either human annotations or pseudo-labels.
• Robustness Evaluation:
  • Noise: Audio-Visual Speech Recognition (AVSR) was tested under Gaussian and babble noise at various Signal-to-Noise Ratios (SNR). A confidence-aware dynamic fusion strategy was employed, weighting modalities based on prediction entropy.
  • Domain Shift: Out-of-Distribution (OOD) test sets were curated from YouTube covering vlogs, specific domains (medical, religious, etc.), noisy environments, and black-and-white archival footage.
• Transfer Learning: Representations learned on VSRo-200 were evaluated on the LRRo benchmark for isolated word recognition.

Key Contributions

1. VSRo-200 Dataset: The first large-scale (200h) Romanian sentence-level VSR dataset, featuring both pseudo-labels and a human-annotated subset (100h).
2. Controlled Supervision Analysis: A unified framework enabling the study of supervision quality, demonstrating the trade-off between annotation fidelity and data scale.
3. Robustness Benchmark: Systematic evaluation of VSR under domain shift (OOD) and acoustic noise, highlighting the complementary nature of multimodal fusion.
4. Transferability Validation: Demonstration that representations learned on VSRo-200 effectively transfer to isolated word recognition tasks on the LRRo benchmark.

Results

Impact of Supervision Quality

• Scaling vs. Quality: At fixed data scales, models trained with human annotations achieved slightly lower Word Error Rates (WER) than those trained with pseudo-labels (e.g., 53.3% vs. 53.6% on unseen speakers at 100h).
• Scalability: However, as data volume increased, the performance gap narrowed. Crucially, pseudo-labels enabled training beyond the human-annotated subset. Scaling from 100h to 200h of pseudo-labeled data further reduced WER (from 53.6% to 48.8% on unseen speakers), indicating that increased data volume can compensate for label noise.

Robustness to Acoustic Noise

• Audio-Only Degradation: Audio-only models (Whisper) suffered severe performance degradation under noise (e.g., WER >90% at -5dB SNR).
• Visual Stability: Visual-only models remained stable across all noise levels (constant WER ~47.8%).
• Multimodal Fusion: Confidence-aware fusion significantly improved robustness in noisy conditions. For instance, under -5dB Gaussian noise, fusion reduced WER from 73.49% (fine-tuned audio-only) to 38.87%. The benefits of fusion diminished as SNR increased, confirming the visual modality's role as a stable complement to unreliable audio.

Robustness to Domain Shift

Performance varied significantly across OOD categories:

• Vlogs: Best performance (58.61% WER) with low Out-of-Vocabulary (OOV) rates.
• Specific Domains: Degraded performance (63.01% WER) correlated with high OOV rates, suggesting vocabulary mismatch is a primary failure mode.
• Black-and-White: Worst performance (87.97% WER), indicating high sensitivity to visual appearance shifts (resolution, contrast, technology) despite moderate OOV rates.

External Validation (LRRo)

Representations learned on VSRo-200 transferred effectively to the LRRo isolated word benchmark:

• Lab LRRo: Achieved 95.0% Top-1 accuracy, substantially outperforming the previous Inception-V4 baseline (71.0%).
• Wild LRRo: Achieved 72.7% Top-1 accuracy, improving upon previous ResNet-based cross-lingual approaches (51.6%).
• Preprocessing: Positioning mouth crops in the lower center of the input canvas yielded better results than direct resizing, highlighting the importance of spatial layout preservation.

Significance and Claims

The paper positions VSRo-200 as a critical testbed for studying supervision, domain generalization, and multimodal fusion in low-resource VSR. The authors claim that:

1. Pseudo-labels are viable for scaling: While human annotations offer superior performance at small scales, pseudo-labels enable continued improvements through data scalability, making them a practical solution for low-resource languages where manual annotation is prohibitive.
2. Multimodal fusion is essential for robustness: In noisy environments, visual cues provide a stable signal that significantly mitigates acoustic degradation, validating the necessity of AVSR for real-world deployment.
3. Domain shift is multifaceted: Robustness is compromised not just by acoustic noise, but also by vocabulary mismatches and visual appearance changes (e.g., archival footage), necessitating holistic approaches to generalization.
4. Transferability: High-quality sentence-level representations learned on VSRo-200 generalize effectively to word-level tasks, suggesting the dataset's utility extends beyond its primary sentence-level design.

The work concludes that VSRo-200 addresses the gap between small, high-quality datasets and large, weakly supervised corpora, providing a foundation for future research into robust, low-resource visual speech recognition.