Multimodal Speaker Identification in Classroom Environments

Technical Summary: Multimodal Speaker Identification in Classroom Environments

Problem Statement
Automated analysis of K-12 classroom discourse faces significant hurdles due to the "perfect storm" of acoustic conditions: high levels of non-stationary background noise, significant reverberation, and the highly variable spectral characteristics of children's speech. Traditional speaker identification (SID) systems relying solely on acoustic biometrics frequently fail in these environments, struggling to distinguish target speakers from the pervasive "babble noise" of peer discussions. Specifically, purely acoustic models achieve low accuracy when identifying individual students, often confounding them with one another or misattributing speech to the wrong role (teacher vs. student). This limitation hinders the development of scalable, equitable feedback systems capable of tracking individual student participation and instructional quality.

Methodology
The study proposes a multimodal framework that anchors acoustic embeddings with semantic context derived from Large Language Models (LLMs) and transcripts. The approach utilizes data from the Center for Educational Data Science and Innovation (EDSI) dataset, specifically a subset of eight mathematics classrooms containing 2,801 labeled utterances.

1. Data Preparation and Annotation:

   • Ground Truth: Two trained annotators identified speakers using mixed-audio tracks, video recordings, seating charts, and class rosters. Discrepancies were resolved through consensus, resulting in a final dataset of 2,801 utterances with a Cohen's kappa of 0.90 (near-perfect agreement).
   • Voice Enrollment: Reference embeddings were created from 30–60 second enrollment recordings of consenting teachers and students, segmented into 3-second overlapping windows.

2. Feature Engineering:
   The system extracts features from two primary modalities:

   • Acoustic Features: Speaker embeddings are generated using the SpeechBrain ECAPA-TDNN model (pre-trained on VoxCeleb), producing 192-dimensional vectors. Features include cosine distances between utterance embeddings and enrollment embeddings, as well as intra-group similarity statistics.
   • Semantic/Contextual Features: An LLM (GPT-5-mini) processes the transcript to perform "contextual anchoring." It infers speaker identity based on discourse cues (e.g., a teacher nominating a student by name). This generates a binary feature indicating LLM confidence and distance statistics to other utterances assigned to the same speaker by the LLM.
   • Additional Features: Utterance duration and transcript-based diarization labels (e.g., "S1", "S2") are included. Missing values are handled via sentinel values (-999).

3. Model Architecture:

   • The task is formulated as a binary classification problem: predicting whether a candidate speaker is the true speaker for a given utterance.
   • An XGBoost gradient boosting classifier is employed to integrate heterogeneous features (acoustic distances, semantic inferences, duration).
   • Training Strategy: A nested cross-validation approach is used. An outer loop performs "Leave-One-Session-Out" validation (training on 7 sessions, testing on 1), while an inner loop handles hyperparameter tuning via Bayesian optimization (Optuna).
   • Calibration: Sigmoid calibration (Platt scaling) is applied to ensure predicted probabilities reflect true likelihoods.

Key Results
The multimodal approach demonstrated significant performance gains over an acoustic-only baseline (ECAPA-TDNN using only VoxCeleb embeddings):

• Role Differentiation: The model achieved 99.3% accuracy in distinguishing teachers from students, a substantial improvement over the 88.0% baseline. This performance was robust even for utterances shorter than 1 second.
• Individual Student Identification:
  • Overall: The multimodal model improved exact student identification accuracy from 39.0% (acoustic baseline) to 50.3%.
  • Longer Utterances: For utterances exceeding 5 seconds, accuracy rose to 76.9% (vs. 64.9% baseline).
  • Top-k Performance: The system proved highly effective as a candidate filter. For utterances over 10 seconds, the Top-3 accuracy reached 95.8%, and the Top-10 accuracy reached 100%.
• Duration Dependency: Performance scales strictly with utterance length. Short utterances (0–1s) showed a Top-1 accuracy of 41.0%, while substantive turns (>10s) reached 79.2%.

Significance and Claims
The paper claims that integrating LLM-derived semantic context with acoustic embeddings creates a robust method for attributing classroom discourse in adversarial environments. The primary significance lies in:

1. Feasibility of Automated Feedback: The near-perfect teacher-student differentiation (99.3%) enables the reliable automation of metrics focused on teacher behavior (e.g., "Uptake" or "Questioning"), which previously required manual analysis.
2. Granular Student Tracking: While identifying specific students in short, context-poor utterances remains challenging, the model successfully isolates "high-value" contributions (longer, substantive turns) with high reliability (95.8% Top-3 accuracy). This allows for tracking individual student mastery and engagement patterns at scale.
3. Paradigm Shift: The study validates a shift from purely acoustic speaker identification to "LLM-adaptive diarization," where linguistic context serves as a necessary anchor to resolve ambiguities that acoustic signals alone cannot, particularly given the acoustic similarity among peer speakers and the out-of-domain nature of child speech for adult-trained models.

The authors conclude that this approach is a crucial step toward equitable, scalable classroom analytics, though they acknowledge the need for future work to improve resolution for rapid, sub-second classroom discourse.