Arabic Author Gender Profiling with Large Language Models: An Empirical Investigation of Capability, Calibration, Bias, and Prompt Sensitivity
This study presents the first comprehensive evaluation of large language models for Arabic author gender profiling, revealing that they underperform classical systems, exhibit well-calibrated abstention behavior, and display prompt-dependent biases and specific morphological error modes unique to the Arabic language.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Every time we read a message, we instinctively try to guess who is holding the pen. Is it a friend, a stranger, a man, or a woman? For decades, computer scientists have taught machines to do the same thing, a field known as author profiling. By analyzing word choices, sentence structures, and topics, algorithms attempt to infer the demographics of a writer. This skill is vital for security, marketing, and understanding human behavior online. In recent years, a new kind of artificial intelligence called a large language model has emerged. These are systems trained on vast amounts of text from the internet, capable of understanding and generating human language with remarkable fluency. In English, these systems have become so skilled at guessing an author's gender that they have surpassed older, more traditional computer programs. They can look at a block of text and make a guess with high accuracy, often without needing any special training on that specific task.
But does this brilliance translate to other languages? Arabic is spoken by over 400 million people and is famous for its complex grammar, where words change shape to indicate gender, tense, and who is being spoken to. This richness makes it a difficult test for computers. Until now, it was unclear whether these powerful new AI systems could handle the nuances of Arabic writing as well as they handle English. A researcher set out to find the answer, not by building a new tool, but by putting the most advanced existing systems to the test. They wanted to see if the AI could correctly identify the gender of Arabic authors, how often it would admit when it was unsure, and whether it carried hidden biases that might lead it to make unfair or incorrect guesses.
The researcher gathered a large collection of real-world data: one thousand Twitter profiles from the year 2018, each containing one hundred tweets. Half of these accounts belonged to men and half to women. They fed this data into three different state-of-the-art language models, including two massive global systems and one specifically designed for Arabic. They asked the computers to look at the full profiles and guess the gender of the author. The results were surprising. Unlike in English, where the new AI systems easily beat the old ones, here they fell short. None of the three models managed to outperform the best traditional computer program from 2018. In fact, the two global models scored lower than the old benchmark, and the Arabic-specific model scored significantly lower still. This suggests that the incredible success these systems have in English does not automatically carry over to Arabic, at least not without extra help.
The study also looked at how the models behaved when they were unsure. The researcher gave the computers two types of instructions: one that allowed them to say "I don't know" if the text was too vague, and another that forced them to guess anyway. When allowed to abstain, the models were surprisingly honest. They refused to guess on about 84 percent of single tweets, recognizing that a single short message often lacked enough clues to make a reliable call. When the researcher forced the models to guess on these difficult, single tweets anyway, their accuracy collapsed to near-random levels, barely better than flipping a coin. This showed that the models were not being overly cautious; they were correctly identifying when the information was insufficient.
Perhaps the most revealing discovery concerned bias. When the models were asked to guess using English instructions, they consistently leaned toward guessing "female," even when the author was male. This happened across all three systems, suggesting a deep-seated tendency to associate certain writing styles with women. However, the researcher found a way to neutralize this. When they forced the models to make a binary choice without allowing them to express uncertainty, the bias largely disappeared for two of the models. This indicates that the bias was not a fixed rule in the computer's mind, but rather a result of how the model weighed its confidence. When it felt unsure, it defaulted to a stereotype; when forced to commit, it became more balanced.
The study also uncovered a unique challenge specific to Arabic. The researcher found that the models often got confused by the grammar itself. In Arabic, a man writing to a woman might use feminine verb forms to address her. The computers mistakenly interpreted these grammatical markers as evidence that the author was a woman, rather than realizing the author was simply speaking to a female friend. This is a type of error that does not happen in English, highlighting a specific blind spot in how these systems understand Arabic grammar.
Finally, the team tested whether the language of the question mattered. They asked the Arabic-specific model to perform the task using instructions written in Arabic instead of English. While this slightly improved the accuracy of the guesses the model did make, it came at a steep cost: the model refused to guess on far more tweets, and when it did guess, it flipped its bias. Instead of leaning toward "female," it began leaning heavily toward "male." This reversal suggests that the model's internal understanding of gender is deeply tied to the language of the prompt, and that neither English nor Arabic instructions are free from bias.
The researcher concluded that while these large language models are powerful tools, they are not yet ready to replace human judgment or older methods for analyzing Arabic text without careful oversight. They found that the models are good at knowing when they don't know, but they struggle with the complex grammar of Arabic and carry significant biases that change depending on how they are asked. The study serves as a reminder that as we deploy these advanced systems, we must understand their specific limitations and the unique ways they interpret the rich, complex languages of the world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.