Reliability of Urdu–Arabic Abusive Language Detection with Cross- Lingual Transformers and Explainable AI
This study evaluates a bilingual XLM-RoBERTa model for detecting abusive language in Urdu and Arabic, revealing that while joint training yields high accuracy, the system lacks robust cross-lingual transfer and context sensitivity, underscoring the necessity of targeted reliability tests over aggregate metrics alone.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
On social media, the volume of harmful speech has grown faster than human moderators can read it. To keep pace, researchers are building computer programs that can automatically spot abusive language. For years, these tools have worked well for English, but they often stumble when faced with languages like Urdu and Arabic, which are spoken by hundreds of millions of people. A major hurdle is that these languages are written right-to-left and have complex scripts, making them difficult for standard software to parse. Furthermore, even when a computer program correctly identifies a bad post, it usually offers no explanation for why it made that choice. This lack of transparency makes it hard for human moderators to trust the tool or to understand if the computer is biased against certain groups. The goal of modern research in this field is not just to build a detector that gets the right answer most of the time, but to build one that can explain its reasoning and remain reliable even when the text is messy or written in different dialects.
A team of researchers from universities in Pakistan and Saudi Arabia set out to solve these specific problems by creating a single computer model capable of understanding both Urdu and Arabic. They gathered a massive collection of over 65,000 social media posts, carefully labeled as either abusive or harmless, to teach their system. Unlike previous attempts that treated each language separately, they trained their model to learn from both languages at the same time. The results showed that this joint approach worked well: the model correctly identified abusive content in about 82 percent of cases for both languages, performing with nearly equal skill in Urdu and Arabic. However, when they tested whether the model could understand one language after only being trained on the other, it failed completely, guessing correctly only about half the time. This finding revealed a crucial truth: the model's success did not come from a magical ability to translate between languages, but from the specific work of learning both languages together.
To ensure the model was not just guessing, the researchers asked it to point out the specific words that triggered its decision. They used two different mathematical methods to check which words mattered most and found that these methods agreed with each other only about a third of the time. This suggests that no single method is perfect for finding the "reason" behind a decision. The team then tried to make the computer's reasoning more human-readable by feeding its findings into a large language model, which wrote short explanations in plain English. While these explanations sounded good and quoted many of the important words, a strict test showed that they were not always faithful to the computer's actual logic. When the researchers removed the words highlighted by the computer, the prediction changed significantly, but when they removed the words the computer's written explanation had quoted, the prediction often stayed the same. This means the written explanations were sometimes just echoing the right words without truly capturing the computer's internal reasoning.
The study also tested how robust the system was when the text was imperfect, which is common in real-world social media. The model handled missing accents and random typos reasonably well, but it struggled significantly when Urdu was written using Latin letters, a common practice in informal online chats. This drop in performance highlighted a gap in the training data. Perhaps most importantly, the researchers discovered that the model was easily fooled by context. It would flag a sentence as abusive simply because it contained a sensitive word, even if the sentence was actually giving moral advice or using the word in a harmless way. For example, it might mistake a warning about indecent behavior for an insult. By adding a small amount of extra training that focused specifically on these tricky contexts, the researchers fixed this error, allowing the model to distinguish between an insult and a warning without losing its overall accuracy.
The final result of this work is a system that does more than just flag bad posts; it provides a "reliability card" for every decision. This card lists the confidence level of the prediction, the specific word that triggered the alert, the target of the attack, and the reason for the classification. The researchers emphasize that this tool is designed to assist human moderators, not replace them. By showing exactly why a post was flagged and admitting when it is unsure, the system helps humans make better final decisions. The study concludes that for multilingual systems to be truly reliable, they must be tested against these specific challenges—like context and script variations—rather than just looking at a single overall score. The path forward involves refining these tools to handle the messy reality of human speech while keeping the human in the loop to ensure fairness and accuracy.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.