💻 computer science

Adversarial Robustness in Smishing Detection: A Comparative Analysis of Adversarial Fragility in Classical vs. Transformer-Based Detection Systems

This study demonstrates that while transformer-based models exhibit significantly greater adversarial robustness against obfuscation and structural perturbations in smishing detection compared to classical lexical models, clean-text performance is an unreliable predictor of resilience, highlighting the need for architecture-specific defenses against evolving attack vectors.

Original authors: Denzel Chiuseni, Athanase Bahizire, Silva Hama, Jema David Ndibwile

Published 2026-08-14

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Denzel Chiuseni, Athanase Bahizire, Silva Hama, Jema David Ndibwile

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to spot a fake text message, like a scammer pretending to be your bank. This is the world of "smishing" detection, a branch of cybersecurity where algorithms act as digital bouncers, checking SMS messages to see if they are friendly or dangerous. For a long time, scientists trained these bouncers using clean, perfect text, assuming that scammers would always write in plain English or Swahili without trying to trick the system. But in the real world, scammers are clever. They use "adversarial attacks," which are like wearing a disguise to walk past a security guard. They might swap a letter for a look-alike symbol (like an 'a' for a 'α'), add weird invisible spaces to break up words, or mix two languages together in one sentence to confuse the guard. The big question researchers have been asking is: If we build a super-smart guard, will they actually be tougher against these disguises, or will they just be better at spotting the obvious stuff?

This paper dives into that exact question by pitting two different types of "guards" against a swarm of tricky, disguised text messages. On one side, you have the "Classical" guards (like Random Forest and XGBoost), which are like old-school detectives who memorize specific keywords and patterns. On the other side, you have the "Transformer" guards (like mBERT and XLM-RoBERTa), which are like modern, context-aware detectives that understand how words fit together in a sentence. The researchers took a dataset of over 27,000 text messages and subjected both types of guards to three different kinds of "disguises": Character Obfuscation (swapping letters for look-alikes), Structural Perturbation (messing with spaces and punctuation), and Code-Switching (mixing languages). They measured how much each guard's performance dropped using a score called the "Robustness Degradation Ratio" (RDR), where 0 means the guard stayed perfect and 1 means they completely failed.

The results revealed a dramatic split in the ranks. The Classical guards, despite being very good at spotting clean messages, fell apart almost instantly when faced with disguises. When scammers used character swaps or messed with spacing, the Classical models suffered near-total collapse, with their performance dropping by up to 98.8% (an RDR of 0.988). It was as if the detective forgot how to read the moment the suspect changed their font. In contrast, the Transformer guards were much tougher. They held their ground, with their performance dropping by a maximum of only 35.1% (an RDR of 0.351). However, they weren't invincible; their biggest weakness turned out to be structural perturbations, where messing with spaces confused their internal word-breakers.

Perhaps the most surprising twist in the story is that being "smarter" in a normal test doesn't mean you are safer in a fight. The researchers found that the Transformer model with the highest score on clean text (XLM-RoBERTa) actually crumbled more under attack than its slightly less accurate sibling (mBERT). This proves that a high score on a standard test is a false sense of security; it doesn't predict how well a system will handle a real-world trick. The study concludes that while modern AI is a huge step up from older methods, we can't just rely on accuracy numbers. We need to build systems that are specifically trained to handle these "disguises," because in the world of smishing, the bad guys are always trying to change their appearance to slip right past the guard.

Technical Summary: Adversarial Robustness in Smishing Detection

Problem Statement

Smishing (SMS phishing) detection systems are predominantly trained and evaluated on clean, monolingual text. However, in low-resource settings characterized by linguistic diversity and limited labeled data, attackers frequently bypass these systems using sophisticated evasion tactics: character obfuscation (homoglyphs, misspellings), structural perturbation (spacing manipulation), and cross-lingual code-switching. Current literature often treats these evasion strategies as theoretical edge cases rather than persistent environmental variables. Consequently, there is a lack of empirical evidence regarding how different model architectures—from classical lexical models to modern transformers—degrade under adversarial conditions, and whether high clean-text accuracy reliably predicts adversarial robustness.

Methodology

The study evaluates the adversarial robustness of five model architectures using a dataset of 27,037 SMS messages (356 malicious, 1.3% prevalence), comprising English and Swahili content. The models are divided into two tiers:

Classical Lexical Models: Random Forest, XGBoost (both TF-IDF based), and a hybrid CNN+BiLSTM.
Multilingual Transformers: mBERT and XLM-RoBERTa.

Attack Generation and Strategy:

Classical Models: Evaluated under a black-box threat model. Adversarial examples were generated using the Gemini API to apply three attack types across three intensity levels (low, medium, high):
- Character Obfuscation: Substitution with homoglyphs and non-printable characters.
- Structural Perturbation: Insertion/removal of spaces to disrupt tokenization.
- Code-Switching: Replacing trigger phrases with translations in the alternate language.
Transformer Models: Evaluated using attention-guided targeting. Preliminary tests showed generic attacks yielded negligible degradation; thus, perturbations were applied exclusively to the top-K tokens identified by final-layer attention weights ( $K=3, 6, 10$ for low, medium, high intensity) to simulate worst-case white-box scenarios.

Evaluation Metric:
The primary metric is the Robustness Degradation Ratio (RDR), defined as $(F1_{clean} - F1_{adversarial}) / F1_{clean}$ . This normalizes degradation relative to each model's baseline, allowing for direct comparison of architectural fragility independent of absolute performance.

Statistical Validation:
Non-parametric tests (Mann-Whitney U, Friedman, and Cliff's $\delta$ ) were employed to confirm that observed differences are attributable to architecture rather than sampling artifacts.

Key Results

1. Architectural Divide

A distinct boundary exists between classical and transformer architectures:

Classical Models: Exhibit near-catastrophic failure under character obfuscation and structural perturbation. Random Forest and XGBoost reached RDR values of 0.9533 and 0.9877 respectively under high-intensity character obfuscation, indicating total signal collapse. CNN+BiLSTM showed better resilience (RDR ~0.48) due to contextual recovery but still suffered severe degradation.
Transformers: Demonstrated significantly greater resilience, with maximum RDR values capped at 0.3511 (XLM-RoBERTa under structural perturbation). They remained robust against character obfuscation (RDR < 0.24) and code-switching (RDR < 0.05).

2. Attack Specificity

Classical Models: Most vulnerable to character obfuscation (TF-IDF signal collapse) and structural perturbation (n-gram boundary shattering). Code-switching caused the widest divergence within this tier, with Random Forest (RDR 0.70) failing significantly more than CNN+BiLSTM (RDR 0.27) due to monolingual lexical bias.
Transformers: Most vulnerable to structural perturbation, which disrupts subword tokenization. Interestingly, code-switching was the least effective attack (RDR < 0.05), as multilingual pre-training maps cross-lingual equivalents to proximate embedding regions.

3. Performance-Robustness Decoupling

The study found no correlation between clean-text performance and adversarial robustness:

XLM-RoBERTa achieved the highest clean-text F1 (0.9565) but suffered greater degradation (RDR 0.2326) under high-intensity character obfuscation than mBERT (F1 0.9429, RDR 0.1515).
Within the classical tier, CNN+BiLSTM had the lowest clean-text F1 (0.897) but the lowest RDR (0.4868) under character obfuscation compared to tree-based models.

4. Statistical Significance

Mann-Whitney U Test: Confirmed that classical models exhibit significantly higher RDR than transformers ( $U=50.0, p=0.0024$ ), with a large effect size (Cliff's $\delta = 0.85$ ).
Friedman Test: Validated that degradation scales non-linearly with intensity, confirming a critical threshold where performance drops sharply.

Key Contributions

Systematic Empirical Comparison: The study provides the first direct comparison of adversarial fragility across the spectrum from classical lexical models to transformers in a low-resource, multilingual setting.
RDR Metric Application: Introduces and applies the Robustness Degradation Ratio to isolate architectural failure points from baseline capability, demonstrating that clean-text accuracy is a poor predictor of security.
Attack Strategy Refinement: Demonstrates that generic black-box attacks underestimate transformer fragility, necessitating attention-guided targeting to reveal worst-case vulnerabilities.
Identification of Failure Modes: Explicitly links specific failure modes to architectural properties:
- TF-IDF: Collapse due to exact surface-form dependence.
- Transformers: Vulnerability to subword tokenization disruption rather than surface character changes.

Significance and Claims

The paper posits that smishing detection must be framed as an adversarial cybersecurity challenge rather than a static classification task. The findings underscore that:

Clean-text performance is not a reliable proxy for security. Optimizing for accuracy may yield models that are more fragile under targeted attack.
Architecture-specific defenses are required. There is no universal solution; TF-IDF models require Unicode normalization and n-gram augmentation, while transformers require attention-aware adversarial training.
Contextual Embeddings offer partial insulation. In resource-constrained environments where transformers are computationally prohibitive, CNN+BiLSTM architectures serve as the most viable classical alternative, offering better resilience than bag-of-words models.

The authors conclude that security professionals must evaluate models based on adversarial resilience metrics rather than baseline accuracy, as models with comparable clean-text performance can exhibit starkly different real-world defensive capabilities.