HumanOmni-Speaker: Identifying Who said What and When

1. Problem Statement

Current Omni-modal Large Language Models (LLMs) struggle with a fundamental aspect of human interaction: accurately deciphering complex, multi-person conversational dynamics to answer "Who said what and when."

• The "Illusion of Competence": Existing models often achieve high scores on benchmarks by exploiting visual biases (e.g., identifying the speaker because they are centered in the frame or holding a microphone) rather than performing genuine cross-modal alignment between audio and subtle visual cues (like lip movements).
• Architectural Perception Gap: Most Omni-models rely on sparse visual sampling (typically 1–2 fps). This discards high-frequency temporal dynamics essential for speaker tracking, such as visemes (visual phonemes) and micro-expressions.
• Token Explosion: Simply increasing the frame rate in standard Vision Transformers (ViTs) leads to a quadratic explosion in token count, drowning out subtle motion signals in redundant background noise.
• Evaluation Flaws: Current benchmarks often isolate tasks (Speech Recognition, Speaker Diarization, etc.) or allow "shortcut learning" via static visual heuristics, failing to test true end-to-end spatio-temporal identity binding.

2. Methodology

The authors propose HumanOmni-Speaker, a unified Omni-modal model designed specifically for speaker-centric tasks, alongside a new rigorous evaluation framework.

A. The Benchmark: HumanOmni-Speaker & VR-SDR

To eliminate shortcuts and rigorously test capabilities, the authors introduce:

• Visual-Registered Speaker Diarization and Recognition (VR-SDR): A holistic task where the model must output structured records (Identity, Timestamp, Transcribed Text) based only on natural language visual descriptions (e.g., "The woman in the red shirt") and the audio-visual stream. No pre-extracted speaker embeddings are allowed.
• Hard vs. Easy Sets: The benchmark includes a "Hard" set where obvious visual biases (close-ups, single subjects) are manually removed, forcing models to rely on genuine audio-visual alignment.
• Atomic Subtasks: To diagnose bottlenecks, the benchmark includes four subtasks: Speech Recognition (SR), Speaker Verification (SV), Speaker Localization (SL), and Speaker Identification (SI).
• Metrics: Introduction of Identity-Fixed SA-WER (measuring if the correct text is attributed to the correct visual ID) and Identity-Fixed IER (measuring precise temporal alignment of identities).

B. Architecture: Visual Delta Encoder

The core innovation is the Visual Delta Encoder, designed to bridge the gap between high-frequency audio and sparse visual semantics without token explosion.

• Dual-Stream Visual Processing:
  1. Visual Base Encoder: Operates at low frequency (1–2 fps) to capture stable spatial identity and environmental context (shared with Qwen2.5-Omni).
  2. Visual Delta Encoder: Operates at 25 fps to capture high-frequency temporal dynamics.
• Token Compression (Structured Visual Tokenizer - SVT):
  • Uses a lightweight ResNet-18 backbone to extract local motion features.
  • Applies hierarchical spatial and large-receptive-field temporal convolutions to compress dense CNN features into just 6 structured tokens per frame.
  • This allows 25 fps sampling without catastrophic token overhead.
• Function: The encoder explicitly models inter-frame motion residuals, enabling the model to track speaker trajectories and extract fine-grained visemes directly from raw video without intrusive face cropping.

C. Training Strategy

A progressive three-stage training paradigm is employed:

1. Pre-training: The Visual Delta Encoder is pre-trained on large-scale speaker datasets (AVSpeech, LRS2/3, VoxCeleb2) using multi-task objectives (face center-point regression, lip-reading).
2. Modality Alignment: The LLM is aligned exclusively with the Visual Delta Encoder features (bypassing the Base Encoder) to force the model to learn high-fidelity viseme-to-semantics translation without relying on static shortcuts.
3. End-to-End Fine-Tuning: Joint optimization on ~1.5 million audio-visual instructions covering ASR, VSR, localization, and VR-SDR, using hybrid parameter-efficient tuning (full fine-tuning for early layers, LoRA for deeper layers).

3. Key Contributions

1. Redefining Evaluation: Introduced the VR-SDR task and HumanOmni-Speaker Benchmark, which strictly eliminates visual shortcuts and evaluates true cross-modal spatio-temporal identity binding.
2. Architectural Innovation: Proposed the Visual Delta Encoder, which enables 25 fps video processing and fine-grained viseme modeling via a highly compressed 6-token-per-frame representation, solving the token explosion problem.
3. System Performance: Developed HumanOmni-Speaker, the first Omni-model capable of end-to-end lip-reading and high-precision speaker localization directly from raw video, outperforming both open-source and closed-source baselines.

4. Experimental Results

The model was evaluated against state-of-the-art open-source models (Qwen-Omni, OLA, VITA) and closed-source models (Gemini3-Pro).

• Speaker Localization: HumanOmni-Speaker achieved a 0.8% error rate, drastically outperforming Qwen3-Omni (2.8%) and Gemini3-Pro (12.8%). This confirms the critical role of high-frequency sampling (25 fps) for spatial tracking.
• Speaker Identification (Hard Set): On the bias-free "Hard" set, the model reduced error rates from ~33% (baseline) to 21.1%, demonstrating robustness against visual shortcuts.
• VR-SDR (Holistic Task):
  • Who said what (SA-WER): Improved from 52.1% to 47.1%.
  • Who said when (IER): Improved from 31.5% to 28.5%.
  • The model significantly outperformed open-source baselines and approached the performance of closed-source Gemini3-Pro.
• Visual Speech Recognition (VSR) & AVSR:
  • Achieved 29.8% WER on LRS2 and 33.4% WER on LRS3 for VSR without any preprocessing (no lip cropping/alignment), matching specialized VSR models.
  • Achieved 1.36% WER (LRS2) and 0.76% WER (LRS3) for Audio-Visual Speech Recognition (AVSR), outperforming other LLM-based AVSR models.
• Ablation Studies: Confirmed that the Visual Delta Encoder alone reduces Speaker Localization error by 73% compared to the base model. Token counts below 6 or frame rates below 25 FPS significantly degrade performance.

5. Significance

This work addresses a critical gap in Omni-modal AI: the inability to handle dynamic, multi-person interactions.

• Scientific Impact: It proves that "illusion of competence" in current benchmarks masks a severe inability to bind visual identity with acoustic dynamics. The new benchmark sets a higher standard for future research.
• Technical Breakthrough: The Visual Delta Encoder demonstrates that high-frequency temporal dynamics can be captured efficiently without token explosion, enabling native lip-reading and precise speaker tracking.
• Practical Application: The system enables the deployment of autonomous embodied agents, wearable AI assistants, and meeting diarization systems that can accurately track "Who said what and when" in real-world, noisy, multi-person environments.