Evaluating the Generation and Detection of Arabic-Based Phishing Attacks Using LLMs
This study empirically demonstrates that large language models can generate highly convincing Arabic phishing emails that outsmart human detection, while LLM-based classifiers significantly outperform humans in identifying these culturally nuanced attacks, highlighting the urgent need for AI-driven cybersecurity defenses in the Gulf region.
Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the digital world, a phishing attack is a digital trap disguised as a trusted message. It looks like an email from a bank, a boss, or a government office, but its true purpose is to trick the recipient into handing over passwords or money. For decades, creating these traps required a human criminal with enough skill to write a convincing lie in the right language. That barrier has now vanished. Large language models, which are powerful computer programs trained to understand and generate human language, can now write these deceptive messages instantly and in any language, removing the need for a skilled human author. This shift has made the threat of phishing more widespread and harder to stop, especially in regions where the language is complex and culturally specific, such as the Arabic-speaking world.
A team of researchers at Zayed University in the United Arab Emirates set out to understand how this new technology is changing the game for Arabic speakers. They wanted to know two things: could these computer programs actually write convincing phishing emails in Arabic, and could other computer programs or regular people spot them? To find out, they built a controlled experiment. They asked three different large language models to write hundreds of simulated emails. Some of these emails were designed to look like legitimate business messages, while others were crafted to look like scams, asking for urgent action or sensitive information. The researchers then asked three other computer models and a group of forty-five human volunteers to read these emails and decide which ones were real and which were fake.
The results revealed a surprising twist in the story of digital deception. The computer programs were remarkably good at writing the scams. All three models they tested successfully generated Arabic phishing emails that bypassed their own safety filters when given a specific academic request. More importantly, the content they produced was not just grammatically correct; it was culturally convincing. One model, which was smaller and developed specifically with a focus on the Gulf region, created emails that felt the most authentic to local readers. These emails used the specific language and tone of local financial institutions so well that they were the hardest for humans to identify as fakes. This finding challenges the common assumption that the biggest, most powerful computer models are the only ones capable of creating dangerous threats; in this case, a smaller, regionally tuned model proved to be the most deceptive.
When it came to spotting the traps, the computer models outperformed the humans by a significant margin. The human volunteers, who represented a mix of professions and backgrounds, correctly identified the phishing emails only about sixty-eight percent of the time. This means that more than one in five of the computer-generated scams successfully fooled the people reading them. The humans often missed the scams because the messages looked so professional and lacked obvious warning signs. They also made mistakes by flagging legitimate emails as scams, usually because the sender's email address looked slightly different from what they expected.
In contrast, the computer models used to detect the scams were far more accurate. One of the detector models achieved a perfect score, correctly identifying every single phishing email and every legitimate message. Another performed with very high accuracy, while a third struggled by flagging too many safe emails as dangerous. The researchers noted a clear difference in how the two groups thought about the problem. The humans relied heavily on checking the sender's email address, looking for small errors in the domain name. The computer models, however, analyzed the entire text for patterns of urgency, requests for personal information, and the overall tone of the message. Because the humans missed so many scams while the computers rarely did, the researchers suggest that the best defense is a partnership between the two. A system where a computer model acts as a first line of defense to catch the obvious threats, leaving only the difficult edge cases for a human to review, would likely be far more effective than relying on either one alone.
The study also highlighted a critical gap in current security training. The ability to spot a phishing email did not depend on how much education a person had, but rather on their profession. People working in finance, technology, or security were better at detecting the scams than those in other fields. This suggests that generic safety training might not be enough; instead, organizations need to tailor their warnings to the specific risks faced by different job roles. Furthermore, the researchers found that the most effective defense against these sophisticated, computer-generated lies is a simple technical check: verifying the sender's address. Since the human participants were most often fooled by emails that looked real but came from slightly wrong addresses, enforcing strict rules on email authentication could stop many of these attacks before they are even seen.
Ultimately, this research provides the first clear evidence of how artificial intelligence is reshaping the landscape of cybercrime in the Arabic language. It shows that the threat is no longer limited to skilled human hackers but is now an automated, accessible capability that can produce highly convincing local content. The study confirms that while computers are currently better at both creating and detecting these scams than humans, the most dangerous aspect is not the size of the computer program, but its ability to mimic the cultural and linguistic nuances of a specific region. As these tools continue to evolve, the defense against them will require a blend of advanced technology and human judgment, with a sharp focus on the specific details that make a message feel real.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.