Cross-modal Affinity-aligned Multimodal Learning Analytics for Predicting Student Collaboration Satisfaction in Game-Based Learning

Technical Summary: Cross-modal Affinity-aligned Multimodal Learning Analytics (AAMLA)

Problem Statement
Collaborative game-based learning environments offer rich opportunities for knowledge construction, yet automatically predicting student collaboration satisfaction remains challenging. A primary barrier is modality degradation: in educational deployments, individual modalities (such as eye gaze) often exhibit inconsistent informativeness across different student cohorts due to sensor noise, occlusion, and behavioral variation. Existing multimodal approaches typically rely on implicit cross-attention fusion mechanisms that assume all modalities contribute equally informative signals. When a weak or noisy modality dominates the attention mechanism, it causes cascading errors, degrading prediction robustness. Simply discarding affected modalities is not a viable solution, as they may still carry useful signals in other contexts.

Methodology: The AAMLA Framework
The authors propose Affinity-Aligned Multimodal Learning Analytics (AAMLA), a framework designed to predict collaboration satisfaction by explicitly modeling inter-modal relationships and enforcing cross-modal consistency. The framework processes four complementary modalities extracted from student interactions in the EcoJourneys environment:

1. Facial Action Units (AU): 17 intensity values per frame.
2. Head Pose: 3D translation and rotation coordinates.
3. Eye Gaze: 3D coordinates per eye.
4. Interaction Trace Logs: Encoded via a BERT-based sentence encoder.

The methodology consists of three core stages:

1. Modality-Specific Encoding and Projection:
   Each modality stream is processed by a specific encoder (GRU for AU, pose, and gaze; feedforward network for trace logs). To handle heterogeneous feature dimensionalities, modality-specific learnable linear projection layers map all features into a unified semantic space ($d=128$) prior to alignment.

2. Cross-modal Affinity-guided Modality Alignment (CAMA):
   This is the core contribution of the paper. Unlike implicit fusion, CAMA explicitly models relationships between modalities using an affinity matrix.

   • Stabilization: To mitigate feature discrepancies caused by behavioral variation, the model computes batch-level averages of original and degraded feature pairs.
   • Affinity Modeling: A symmetric affinity matrix is computed to capture explicit semantic relationships among the modalities.
   • Contrastive Learning: Guided by the affinity matrix, a contrastive loss ($L_{aff}$) is applied. This loss pulls together embeddings of different modalities belonging to the same student activity (positive pairs) and pushes apart embeddings from different activities (negative pairs).
   • Adaptive Suppression: This mechanism allows the model to learn to adaptively suppress uninformative or degraded modalities (e.g., inconsistent gaze signals) without discarding them entirely, ensuring stable representations even under severe degradation.

3. Training Objectives:
   The total loss function combines the standard cross-entropy classification loss ($L_{CE}$) with the affinity alignment loss ($L_{aff}$), weighted by a hyperparameter $\alpha$ (set to 0.25). This joint optimization ensures both classification accuracy and cross-modal feature consistency.

Key Contributions

• CAMA Module: A novel module that explicitly models inter-modal relationships via affinity matrices and enforces consistency through contrastive learning, addressing the limitations of implicit fusion in the presence of modality degradation.
• Adaptive Handling of Degradation: The framework demonstrates the ability to suppress uninformative modalities dynamically rather than discarding them, maintaining robust performance when specific signals (like gaze) are noisy or missing.
• Unified Semantic Space: The use of modality-specific projection layers allows heterogeneous features to be mapped into a coherent space for direct cross-modal comparison.

Experimental Results
Experiments were conducted on data from 50 middle school students in the EcoJourneys environment, utilizing a student-level 10-fold cross-validation protocol.

• Performance: AAMLA achieved a macro F1-score of 0.79 and accuracy of 0.77, outperforming unimodal baselines (F1 $\approx$ 0.65–0.66) and the prior cross-attention approach (F1 = 0.72).
• Robustness under Degradation: AAMLA demonstrated consistent superiority under three degradation conditions:
  • Gaze Dropout: Even with 70% of gaze features zeroed, AAMLA maintained an F1 of 0.72, compared to 0.52 for the cross-attention baseline.
  • Feature Perturbation: Under Gaussian noise added to AU and pose, AAMLA significantly outperformed the baseline.
  • Full Modality Absence: When entire modalities were removed at inference, AAMLA maintained robust performance (e.g., F1 = 0.73 without AU), whereas the baseline dropped significantly.
• Ablation Studies: Removing the CAMA module or the contrastive loss ($L_{aff}$) resulted in fragmented feature spaces and reduced performance, confirming that both explicit alignment and contrastive supervision are necessary.
• Interpretability:
  • t-SNE Visualizations: Showed that AAMLA produces compact, semantically coherent clusters with clear inter-class separation, whereas removing CAMA led to cross-modality drift.
  • SHAP Analysis: Confirmed that trace log features were the most predictive, while gaze features were absent from the top-20 contributors, validating the model's adaptive suppression of uninformative modalities.

Significance and Claims
The paper claims that AAMLA addresses a critical gap in multimodal learning analytics by providing a mechanism to handle the inherent variability and unreliability of behavioral data in educational settings. The authors assert that their explicit affinity-based alignment strategy offers a robust alternative to implicit fusion, particularly when dealing with "modality degradation" where certain signals are inconsistent across student cohorts. The work demonstrates that such explicit alignment techniques can transfer effectively to educational domains, offering interpretable insights into the behavioral patterns that predict collaboration satisfaction without requiring the complete removal of potentially useful, albeit noisy, data streams.