POLY-SIM: Polyglot Speaker Identification with Missing Modality Grand Challenge 2026 Evaluation Plan

Based on the provided paper, here is a detailed technical summary of the POLY-SIM: Polyglot Speaker Identification with Missing Modality Grand Challenge 2026.

1. Problem Statement

The paper addresses two critical limitations in current multimodal speaker identification (SI) systems:

• Missing Modality: Real-world scenarios often lack complete audio-visual data due to occlusions, camera failures, or privacy constraints. Existing models typically degrade significantly when one modality (usually visual) is missing.
• Cross-Lingual Generalization: Most SI models are trained on a single language (e.g., English) and fail to generalize when tested on speakers using a different language (e.g., Urdu) due to acoustic and phonetic variations.

The core challenge is to develop robust models that can identify speakers using only audio inputs (missing visual modality) in a different language than the one used during training, while maintaining high accuracy.

2. Methodology & Challenge Design

The Grand Challenge is structured around a specific experimental setup designed to simulate these real-world constraints.

A. Task Formulation

• Training Phase: Models are trained on paired face images and speech audio in English ($\ell_{en}$).
• Inference/Testing Phase:
  • Missing Modality: The visual (face) modality is completely removed; the model receives only audio.
  • Cross-Lingual: The audio input is in Urdu ($\ell_{ur}$), a language different from the training set.
• Goal: Predict the speaker identity $y_j$ based solely on the audio embedding of the Urdu speech, using a model trained on English audio-visual pairs.

B. Dataset: MAV-Celeb

The challenge utilizes a subset of the MAV-Celeb dataset, specifically curated for bilingual speakers.

• Speakers: Each speaker appears in videos speaking both English and Urdu.
• Statistics: The dataset includes 262 English videos and 415 Urdu videos for training, with specific splits for validation and testing (see Table I in the paper).
• Features: Pre-extracted features (embeddings) for both face and audio are provided, generated by state-of-the-art pre-trained architectures.

C. Evaluation Protocols

The challenge evaluates performance across four distinct configurations to measure robustness:

1. P3 (In-language, Multimodal): Train and test on English with both Face and Audio. (Baseline performance).
2. P4 (Missing-Modality): Train on English (Face+Audio), test on English with Audio only.
3. P5 (Cross-lingual, Multimodal): Train on English (Face+Audio), test on Urdu with Face+Audio.
4. P6 (Cross-lingual, Missing-Modality): Train on English (Face+Audio), test on Urdu with Audio only. (The primary challenge).

D. Baseline Model

A starter kit is provided based on the FOP (Face-Oral-Phonetic) framework:

• Architecture: A two-branch network (Face encoder + Audio encoder).
• Encoders:
  • Face: Pre-trained CNN on large-scale facial recognition data.
  • Audio: Speaker recognition network trained on the training language.
• Optimization: Uses a loss function imposing orthogonality constraints on multimodal embeddings of different speakers to ensure distinctiveness.

3. Key Contributions

• New Benchmark: Introduces the first standardized benchmark specifically targeting the intersection of missing modality and cross-lingual speaker identification.
• Realistic Simulation: Moves beyond idealized lab conditions to simulate scenarios where cameras fail and speakers switch languages, which is common in global multimedia applications.
• Comprehensive Evaluation: Defines a multi-faceted evaluation protocol (P3–P6) that isolates the impact of language shift and missing data, allowing researchers to pinpoint specific failure modes.
• Open Resources: Provides a public dataset split, pre-extracted features, a baseline code repository, and a submission platform (CodaBench) to foster reproducibility.

4. Results (Baseline Performance)

The paper reports baseline accuracy results (Table II) highlighting the severity of the challenges:

• In-Language (P3): High accuracy (~98%) when both modalities are present in the same language.
• Missing Modality (P4): Accuracy drops drastically to 37.75% (Progress) / 52.53% (Eval) when visual data is removed, even in the same language.
• Cross-Lingual (P5): High accuracy (~98%) is maintained if both modalities are present, suggesting the visual modality can compensate for language shifts.
• The Critical Gap (P6): When both missing modality and cross-lingual conditions are applied, accuracy plummets to 31.70% (Progress) / 43.87% (Eval).
• Overall Score: The average baseline score across all four configurations is roughly 66.34% (Progress) and 73.37% (Eval), indicating significant room for improvement.

5. Significance

• Robustness for Industry: The challenge directly addresses the reliability of biometric systems in security, media analytics, and human-computer interaction, where perfect data conditions are rare.
• Advancement in Representation Learning: It pushes the research community to develop better domain adaptation techniques, cross-modal alignment strategies, and language-invariant speaker representations.
• Fairness and Inclusivity: By testing on non-English languages (Urdu), the challenge promotes the development of speaker identification systems that work globally, rather than being biased toward English-centric datasets.

The POLY-SIM 2026 challenge aims to bridge the gap between theoretical multimodal learning and practical, deployable systems that function reliably under imperfect and diverse real-world conditions.