A Robust and Explainable Multimodal Fusion Framework for Emotion Recognition Using Visual–Textual Feature Learning and Ensemble Optimization

Technical Summary: A Robust and Explainable Multimodal Fusion Framework for Emotion Recognition

Problem Statement
Current emotion recognition systems often rely on unimodal data (either visual or textual), which limits their ability to capture the multidimensional nature of human emotions. Unimodal approaches frequently suffer from poor contextual awareness, ambiguity, and noise, leading to suboptimal classification performance. Furthermore, existing multimodal models often function as "black boxes," lacking interpretability, which hinders their deployment in sensitive domains like healthcare and education. Additionally, many studies lack rigorous robustness validation, such as cross-validation, statistical significance testing, and ablation studies, making it difficult to verify if performance gains are generalizable or merely artifacts of specific data splits.

Methodology
The authors propose an explainable multimodal fusion framework that integrates visual and textual features using ensemble learning and rigorous validation techniques. The methodology follows a structured pipeline:

1. Dataset and Preprocessing: The study utilizes a multimodal music emotion dataset containing 5,866 aligned samples with 14 emotion classes. The data undergoes preprocessing to handle duplicates and missing values, followed by Z-score normalization.
2. Feature Engineering:
   • Visual and Textual Features: The framework utilizes a total of 28 engineered features derived from both visual and textual modalities. Visual attributes include saturation, texture, brightness, hue, and motion, along with interaction terms, while unstructured text is converted into structured numerical representations.
   • Fusion: The framework employs early feature-level fusion, concatenating visual and textual vectors to create a unified representation that captures cross-modal relationships.
3. Model Architecture:
   • Base Classifiers: Eight supervised machine learning algorithms are benchmarked: Logistic Regression, SVM-RBF, Gradient Boosting, Random Forest, Extra Trees, XGBoost, LightGBM, and CatBoost.
   • Ensemble Learning: A stacked ensemble approach is implemented. Heterogeneous base classifiers generate predictions, which are then fed into a meta-classifier to learn the optimal combination of these predictions, aiming to reduce bias and variance.
4. Explainability (XAI): To address the "black box" issue, the framework integrates SHAP (SHapley Additive exPlanations) for global feature importance analysis and LIME (Local Interpretable Model-agnostic Explanations) for instance-level interpretation.
5. Validation and Robustness: The study employs stratified 5-fold cross-validation, ablation analysis, confidence interval estimation, and statistical significance testing (paired t-tests, Friedman test, and Nemenyi post-hoc test) to ensure the reliability and generalizability of the results.

Key Contributions

• Multimodal Integration: The paper presents a framework that fuses complementary visual and textual information at the feature level, demonstrating that this approach yields richer emotional representations than unimodal methods.
• Systematic Benchmarking: A comprehensive evaluation of eight diverse machine learning algorithms under identical experimental conditions identifies the optimal base learners for this specific task.
• Stacked Ensemble Optimization: The study demonstrates that combining heterogeneous base classifiers via a stacked ensemble significantly enhances predictive accuracy, robustness, and generalization compared to individual models.
• Explainability Implementation: The integration of SHAP and LIME provides both global and local explanations, identifying key drivers of emotion predictions and increasing model transparency.
• Rigorous Validation: Unlike many prior studies, this work includes extensive robustness checks, including stratified cross-validation, ablation studies, and statistical significance testing, to validate that performance improvements are not due to random fluctuations.

Results

• Multimodal Superiority: Multimodal fusion consistently outperformed unimodal (visual-only and text-only) models across all evaluation metrics.
• Classifier Performance: Among individual classifiers, Extra Trees achieved the highest accuracy of 95.81%, followed closely by Random Forest and LightGBM. Traditional linear models (Logistic Regression, SVM-RBF) performed significantly worse.
• Ensemble Performance: The proposed stacked ensemble achieved the best overall performance with an accuracy of 98.41%, a Macro F1-score of 98.40%, an MCC of 0.9829, and a ROC-AUC of 0.9986. This represents an improvement of approximately 2.6 percentage points over the best individual classifier (Extra Trees).
• Feature Importance: SHAP analysis revealed that textual features contributed 52.11% to the predictions, while visual features contributed 47.89%. The top influential features identified included saturation, texture, brightness–contrast interaction, hue, and brightness, along with other interaction terms.
• Robustness: The stacked ensemble demonstrated the lowest standard deviation in 5-fold cross-validation (0.31% for accuracy), indicating high stability. Statistical tests confirmed that the multimodal and ensemble results were significantly superior (p < 0.05) to unimodal and single-classifier baselines.
• Error Analysis: The primary confusion occurred between semantically or visually similar emotions (e.g., "Sentimental" vs. "Sad," "Mysterious" vs. "Dark"), suggesting that errors are often due to inherent emotional ambiguity rather than model failure.

Significance and Claims
The paper claims that the proposed framework offers a robust, accurate, and interpretable solution for multimodal emotion recognition. By combining feature-level fusion, stacked ensemble learning, and XAI techniques, the study addresses critical gaps in current literature regarding model transparency, robustness validation, and the underutilization of ensemble methods in multimodal contexts. The authors assert that their approach provides a trustworthy foundation for deploying emotion recognition systems in real-world applications where explainability and reliability are paramount. The study concludes that while the current results are promising, future work should explore the integration of transformer-based foundation models and additional modalities (such as audio and physiological signals) to further enhance scalability and generalization.