💻 computer science

Benchmarking Individual and Ensemble XAI Methods for Arabic Transformer-Based Text Classification

This paper presents a comprehensive benchmark of individual and ensemble Explainable AI methods for Arabic transformer-based text classification, revealing that while selective ensembling offers modest improvements in model-centered faithfulness, it does not significantly enhance human-centered plausibility, alongside the introduction of a cost-effective automated rationale-selection method.

Original authors: Youssef Chafiqui, Houda Anoun

Published 2026-10-28

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Youssef Chafiqui, Houda Anoun

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the modern world of artificial intelligence, computers have become remarkably good at reading and understanding human language. They can sort through thousands of tweets to find angry voices, or scan reviews to determine if a customer was happy or disappointed. These systems, built on complex structures called transformers, act like powerful engines that process text by looking at how words relate to one another in a sentence. However, there is a catch: while these engines are excellent at making decisions, they are often opaque black boxes. When a computer decides that a specific sentence is hateful or positive, it does not naturally explain which words led it to that conclusion. This lack of transparency is a problem, especially in critical situations where understanding the "why" behind a decision is just as important as the decision itself. Scientists have developed tools to shine a light into these black boxes, a field known as explainable artificial intelligence. These tools attempt to highlight the specific words or phrases that a computer model considered most important when it made a prediction. Yet, for languages like Arabic, which is rich in dialects and complex word structures, it remains unclear which of these explanation tools actually work best, or if combining them together creates a clearer picture than using them alone.

A team of researchers at Hassan II University in Casablanca set out to solve this uncertainty by putting the most popular explanation tools to the test. They focused on two very different tasks: determining the sentiment of text, such as whether a review is positive or negative, and detecting hate speech. To do this, they trained three different computer models specifically designed for the Arabic language and then asked four distinct explanation methods to justify the models' decisions. These methods, which include techniques that test how predictions change when words are removed or altered, were evaluated not just on whether they looked logical to humans, but on whether they accurately reflected the internal logic of the computer model itself. The researchers also tested a new, automated way to decide exactly how many words should be highlighted in an explanation, aiming to find a balance between being too vague and being too specific.

The study revealed that while no single tool was perfect, combining two specific methods—one that learns from small changes in the text and another that calculates the mathematical contribution of each word—produced the most reliable results. When the researchers paired these two approaches, the resulting explanation was slightly more accurate at reflecting the model's true reasoning than using either tool on its own. This improvement was consistent across both the sentiment analysis and the hate speech detection tasks, suggesting that the benefit is real and not just a fluke of one specific dataset. However, the gain was modest; the combined method did not revolutionize the quality of the explanation, but rather provided a small, statistically significant boost in reliability. The researchers found that simply adding more tools to the mix did not help; in fact, including a third or fourth method sometimes made the explanation worse. This suggests that the quality of the explanation depends more on choosing the right partners than on having a large crowd of them.

Perhaps the most surprising discovery was that being faithful to the computer's logic does not always mean being convincing to a human. The researchers compared the computer-generated explanations against a set of text where humans had manually highlighted the words they believed were important. They found that the method which performed best at mimicking the computer's internal logic was not necessarily the one that matched the human annotations the best. In some cases, the tool that was most accurate to the machine's math was only marginally better at aligning with human intuition than a tool that was less accurate to the machine. This distinction is crucial: it means that a system can be technically correct about how it works without necessarily making sense to the person reading the explanation. The study concludes that to truly understand these systems, we must evaluate them from both angles—the machine's perspective and the human's—rather than assuming one guarantees the other.

To make these explanations practical, the team also introduced a new, automated method for deciding how many words to highlight. Traditionally, researchers had to guess a number or set a fixed rule, which often led to explanations that were either too long or too short. The new method looks at the pattern of importance scores and automatically finds the point where the importance of words drops off sharply, similar to how a hiker might spot a sudden change in the terrain to decide where a trail ends. This approach requires no manual settings and runs much faster than existing automated techniques, taking only a fraction of a second to process a sentence. While it did not outperform every other method in every single category, it offered a highly efficient way to generate concise, relevant explanations without needing a human to tweak the settings. The work provides a clear roadmap for developers working with Arabic text, showing that while combining tools can help, the most effective strategy involves careful selection and a clear understanding that what is mathematically true for the model is not always what feels right to a human reader.

Technical Summary: Benchmarking Individual and Ensemble XAI Methods for Arabic Transformer-Based Text Classification

Problem Statement

While transformer-based models have achieved state-of-the-art performance in Natural Language Processing (NLP), their complex architectures often render predictions difficult to interpret. Explainable Artificial Intelligence (XAI) aims to address this by providing human-understandable rationales. However, the application of XAI to Arabic NLP remains underexplored compared to English. Existing studies on Arabic XAI have largely focused on individual explanation methods, leaving a gap in systematic evidence regarding ensemble explanation methods. Furthermore, explanation quality is multidimensional; improvements in faithfulness (how well an explanation reflects the model's internal behavior) do not necessarily correlate with plausibility (agreement with human-annotated rationales). There is a need for controlled benchmarks that compare individual and ensemble attribution methods across multiple Arabic transformer models and tasks, evaluating both faithfulness and plausibility.

Methodology

The study establishes a comprehensive benchmark framework to evaluate individual and ensemble XAI methods for Arabic text classification.

1. Experimental Setup

Datasets: Three Arabic datasets were utilized:
- D1 (Tweet Sentiment Multilingual): Used for primary faithfulness evaluation (3,033 instances).
- D2 (HateDay): Used for cross-task evaluation on hate speech detection (2,000 instances).
- D3 (Tripadvisor Reviews): Used for plausibility evaluation against manually annotated human rationales (303 instances).
Models: Three pre-trained and fine-tuned Arabic transformer models were evaluated:
- CAMeLBERT-MSA: Fine-tuned on sentiment datasets.
- AraBERT-Arabic-SA: Fine-tuned on Arabic tweets (including Saudi dialect).
- MarBERT-HateSpeech: Fine-tuned for binary hate speech detection.
XAI Methods: Four individual attribution methods were tested: LIME, SHAP, Integrated Gradients (IG), and DeepLIFT.
Ensemble Strategy: An ensemble framework was implemented where token-level attribution vectors from selected methods were independently min-max normalized to $[-1, 1]$ and aggregated using either arithmetic mean or median.

2. Evaluation Framework

The study employs a multidimensional evaluation approach:

Faithfulness Metrics: Five complementary metrics were used:
- Comprehensiveness: Model confidence drop after removing rationale tokens.
- Sufficiency: Model confidence using only rationale tokens.
- Correlation with Leave-One-Out (LOO): Agreement between attribution scores and empirical token removal impact.
- Insertion & Deletion AUC: Performance of adding/removing tokens progressively.
- Combined Score: A unified metric aggregating the five metrics (inverting Sufficiency and Deletion AUC where lower is better).
- Borda Count: A rank-based aggregation to assess relative ordering independent of metric scales.
Plausibility Metrics: Agreement with human-annotated rationales was measured using Precision, Recall, F1-score, and Jaccard similarity.
Rationale Selection: To convert continuous attribution scores into discrete "hard" rationales, the study compared fixed strategies (Top-n, fixed thresholds) against automated methods. The authors proposed a parameter-free, residual-based elbow detection method that fits a single global linear trend (Ordinary Least Squares) to sorted scores and identifies the elbow as the point of maximum residual deviation.

Key Contributions

Comprehensive Benchmark: A rigorous, reproducible benchmark comparing individual and ensemble XAI methods across three Arabic transformer models and two distinct classification tasks (sentiment analysis and hate speech detection).
Multidimensional Evaluation: A framework integrating five faithfulness metrics, rank-based aggregation (Borda count), and plausibility evaluation against human rationales, highlighting the divergence between model-centered and human-centered evaluation.
Automated Rationale Selection: The introduction of a parameter-free, residual-based elbow detection method. This approach achieves competitive faithfulness with significantly lower computational cost compared to existing automated techniques (e.g., Kneedle, DFDT, L-Method).

Results

1. Ensemble Performance

Modest Faithfulness Gains: Selective ensembling yields statistically significant but modest improvements in faithfulness. The LIME+SHAP ensemble consistently achieved the highest Combined scores.
- In sentiment analysis, LIME+SHAP improved the primary Combined score by an average paired difference of approximately 0.012 over LIME alone.
- This trend persisted in the cross-task hate speech evaluation.
Diminishing Returns: Increasing ensemble size did not guarantee better performance. Adding DeepLIFT to the LIME+SHAP+IG combination actually decreased the Combined score, suggesting that ensemble effectiveness depends on the quality and complementarity of constituent explainers rather than the sheer number of methods.
Inter-Method Agreement: Pearson correlation analysis revealed that LIME and SHAP share the highest agreement (0.43), while gradient-based methods (IG, DeepLIFT) showed lower correlation with perturbation-based methods.

2. Faithfulness vs. Plausibility

Divergent Trends: Improvements in model-centered faithfulness did not translate to stronger human-rationale alignment.
- While LIME+SHAP achieved the best faithfulness scores, its advantage in plausibility (F1-score and Jaccard similarity) over the best individual method (SHAP) was negligible.
- SHAP achieved the highest Precision (0.299), while DeepLIFT achieved the highest Recall (0.903), indicating distinct behaviors in human-rationale alignment.
Conclusion: Faithfulness and plausibility must be evaluated as complementary, not interchangeable, dimensions.

3. Rationale Selection Efficiency

Trade-offs: Fixed Top-n strategies showed that larger rationales (e.g., Top-20) artificially inflate faithfulness scores (Comprehensiveness/Sufficiency) but reduce conciseness.
Proposed Method: The proposed residual-based elbow method selected an average of 51.9% of tokens, achieving a Combined score of 0.673.
- It was ~4.6x faster than Kneedle, ~10x faster than DFDT, and ~30x faster than the classical L-Method.
- While DFDT achieved slightly higher faithfulness, the proposed method offered a superior trade-off between performance and computational efficiency.

Significance and Claims

The paper claims to provide rigorous, evidence-based guidance for the selection of XAI methods in Arabic NLP. Its significance lies in:

Demonstrating that selective ensembling (specifically LIME+SHAP) offers a reliable, albeit modest, boost in explanation robustness across tasks.
Establishing that ensemble size is not a proxy for quality; careful selection of complementary explainers is more critical.
Highlighting the gap between faithfulness and plausibility, cautioning researchers that optimizing for model behavior does not guarantee alignment with human reasoning.
Offering a computationally efficient, parameter-free alternative for generating hard rationales, addressing a practical bottleneck in automated XAI evaluation.

The authors conclude that no single XAI method or rationale selection strategy universally dominates across all metrics, emphasizing the necessity of multi-metric, cross-task, and human-centered evaluation frameworks for reliable explainability in Arabic NLP.