Learning Human-Aligned Explanations: AMeta-Model Approach to Combining XAIMethods for Arabic Sentiment Analysis

Problem Statement

Explainable Artificial Intelligence (XAI) aims to make complex "black box" models, such as Transformers, transparent. However, existing post-hoc XAI methods (e.g., LIME, SHAP, Integrated Gradients, DeepLIFT) often yield inconsistent results because they capture different aspects of model behavior. Previous ensemble approaches, including the authors' earlier work, attempted to combine these methods using fixed aggregation strategies (e.g., simple averaging or weighted averages). These fixed strategies suffer from two main limitations:

1. They rely on manually defined weights that do not reflect human preferences.
2. They fail to capture non-linear interactions between different XAI methods.

Furthermore, there is a scarcity of research regarding XAI aggregation specifically for the Arabic language, which presents unique challenges due to morphological complexity and dialectal variation. The core problem addressed is the lack of a dynamic, data-driven mechanism to learn how to combine XAI outputs to maximize alignment with human judgment (plausibility) while acknowledging the inherent trade-off with model consistency (faithfulness).

Methodology

The authors propose a supervised meta-model framework designed to learn human-aligned token-level explanations for Arabic sentiment analysis. The pipeline consists of four stages:

1. Data Preparation and Annotation:

   • Dataset: A subset of 303 Arabic TripAdvisor hotel reviews (160 positive, 102 negative, 41 neutral) was manually annotated at the token level.
   • Ground Truth: Human annotators used the XS2A platform to assign binary labels (1 = important, 0 = non-important) to tokens they deemed influential for sentiment. This resulted in 16,925 token-level instances.

2. Feature Construction:
   The meta-model takes a rich feature vector for each token, integrating:

   • XAI Scores: Raw importance scores and binary rationales (top 25% tokens) from LIME, SHAP, Integrated Gradients (IG), DeepLIFT, and a previous Ensemble XAI baseline.
   • Transformer Internal States: Attention weights (averaged across all layers/heads and specifically from the final layer's [CLS] token) and contextual embeddings from the base model (CAMeLBERT-MSA). Embeddings were reduced from 768 to 10 dimensions using PCA and UMAP.
   • Linguistic and Structural Features: Part-of-speech (POS) tags, token frequency, token position, token length, subword status, punctuation, and comment-level features (sentiment polarity, model confidence).

3. Meta-Model Training:
   Three supervised machine learning models were trained to predict the binary human importance label based on the features above:

   • Logistic Regression: A linear baseline.
   • XGBoost: A gradient-boosted decision tree capable of capturing non-linear interactions.
   • CatBoost: An algorithm optimized for categorical features and robust against overfitting.
   • The dataset was split 90/10 at the review level to prevent data leakage, with 5-fold cross-validation used for hyperparameter tuning.

4. Evaluation:
   Performance was measured along two dimensions:

   • Plausibility (Human Alignment): Precision, Recall, F1-score, and Jaccard Index against human annotations.
   • Faithfulness (Model Consistency): Comprehensiveness, Sufficiency, Correlation with Leave-One-Out (LOO), Insertion/Deletion AUC, and a previously proposed Combined Faithfulness metric.

Key Contributions

• First Human-Supervised XAI Aggregation in Arabic: This study presents the first investigation of a supervised meta-model framework for learning human-aligned token-level explanations by combining multiple post-hoc XAI methods specifically for Arabic NLP.
• Novel Meta-Model Architecture: Unlike prior ensemble methods that use fixed heuristics, this approach trains a supervised model to dynamically learn the optimal combination of XAI scores, attention weights, embeddings, and linguistic features.
• Empirical Characterization of the Trade-off: The work provides a principled empirical analysis of the tension between plausibility and faithfulness, quantifying the gains in human alignment against the costs in model fidelity.

Experimental Results

Plausibility (Human Alignment):
The meta-models significantly outperformed individual XAI methods and the previous weighted ensemble approach.

• CatBoost and XGBoost achieved the highest performance, with an F1-score of 0.84 and a Jaccard index of 0.74.
• This represents a substantial improvement over individual methods (LIME, SHAP, IG, DeepLIFT), which ranged between 0.64 and 0.70 F1-score.
• Feature Analysis: Ablation studies revealed that contextual embeddings (reduced via UMAP) provided the strongest individual signal for predicting human rationale. However, the best results were achieved by fusing embeddings with XAI scores, attention weights, and linguistic features.

Faithfulness (Model Consistency):
While plausibility improved, faithfulness decreased, confirming a fundamental trade-off.

• The Combined Faithfulness metric for the meta-models ranged from 0.54 to 0.61 (CatBoost: 0.60, XGBoost: 0.61).
• In contrast, the Ensemble XAI (weighted) and LIME achieved higher faithfulness scores of 0.76.
• Qualitative analysis showed that while meta-models successfully identified sentiment-bearing words (e.g., "beautiful," "bad"), they sometimes diverged from the specific tokens the underlying CAMeLBERT classifier relied upon most heavily (e.g., negation particles), leading to lower faithfulness scores.

Significance and Claims

The paper claims to offer the first investigation of human-supervised XAI aggregation in the Arabic domain. Its primary significance lies in demonstrating that learning to combine XAI methods under human supervision yields explanations that are significantly more plausible to humans than static ensembles or individual methods.

However, the authors modestly conclude that this gain in plausibility comes with a measurable cost to faithfulness. They argue that this trade-off is theoretically coherent: human annotators prioritize semantically salient and pragmatically interpretable terms, whereas Transformer classifiers rely on complex, often non-intuitive statistical patterns. The study does not claim to have solved the tension between human alignment and model fidelity but rather provides a new methodology to navigate it and offers a principled understanding of the fundamental challenges in explainable NLP.

Limitations Acknowledged:

• Scalability: The meta-model is specific to the CAMeLBERT model, the TripAdvisor dataset, and the sentiment analysis task; it requires retraining for different models or domains.
• Token Independence: The current design treats tokens as independent instances, failing to capture inter-token dependencies (e.g., negation scope or discourse connectives like "but").
• Computational Overhead: The pipeline is slower as it requires running multiple XAI methods and then a meta-model inference for every token.
• Data Constraints: The annotation effort was limited to 303 reviews by two annotators, limiting statistical power and generalizability to rare dialects.