💻 computer science

Adversarially Robust and Semantically Aware Phishing Detection using Calibrated Multimodal Learning

This paper presents a robust, semantically aware multimodal phishing detection framework that fuses URL, text, and visual features with calibrated confidence and adversarial training to significantly outperform unimodal baselines while maintaining high accuracy and reliability under deceptive attacks.

Original authors: Dinesh Patil, Madhuri Gedam, Chhaya dhavale

Published 2026-09-15

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Dinesh Patil, Madhuri Gedam, Chhaya dhavale

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital landscape, a phishing attack is a deception that relies on trust. Criminals create websites that look and sound exactly like legitimate services—banks, email providers, or popular retailers—to trick people into handing over passwords or credit card numbers. For years, security software has tried to stop these traps by looking at the web address, or URL, of a site. These addresses are the strings of characters that appear in a browser's bar. If a URL looks strange, contains too many numbers, or uses a suspicious web address structure, the software flags it as dangerous. However, this approach has a weakness. A clever attacker can craft a web address that looks perfectly normal on the surface while the actual page it leads to is a convincing forgery. The address might be clean, but the content is a lie. To catch these sophisticated tricks, security experts realized they needed to look at more than just the address; they needed to understand the story the page tells and what it looks like to the human eye.

A team of researchers set out to build a system that does exactly this, combining three different ways of looking at a website to decide if it is a trap. Instead of relying on a single clue, their system examines the web address, reads the text on the page, and even looks at a screenshot of what the page actually displays. They treated this as a three-part investigation. First, they analyzed the structure of the web address, looking for hidden patterns that humans might miss. Second, they used a computer program trained to understand the meaning of words, allowing it to detect if the text on the page is using urgent language or asking for sensitive information in a way that feels unnatural. Third, they fed a picture of the webpage into a visual analyzer to spot if the design is copying a trusted brand or if the layout looks suspiciously wrong. By weaving these three perspectives together, the researchers created a detector that is much harder to fool than those looking at just one piece of evidence.

The study began with a massive collection of over ten thousand websites, half of which were known phishing sites and half were legitimate. The researchers were careful to ensure that the data they used for training and testing did not overlap in a way that would give the system an unfair advantage. They split the data so that the system learned from some domains and was tested on completely different ones, mimicking how it would face new, unseen threats in the real world. When they tested their new multimodal system, the results were striking. The system that combined the web address and the page text achieved a high level of accuracy, correctly identifying phishing sites in nearly every case. It performed significantly better than systems that relied only on the web address or only on the text. The visual component, while useful, required a smaller amount of data in this specific study, but it still added a layer of insight that the other methods missed. The most effective way to combine these three streams of information was through a mechanism that allowed the system to weigh the importance of each clue, paying the most attention to the text when it was the strongest signal, but still listening to the address and the image.

A critical part of this research was testing how well the system could withstand an attacker trying to trick it. In the real world, criminals constantly try to find small changes they can make to a phishing site to bypass security filters. The researchers simulated these attacks by making tiny, almost invisible alterations to the web addresses and the text on the pages. They found that systems looking at only one type of information were easily fooled; a small change to the address could make a bad site look good to a simple detector. However, the combined system was remarkably resilient. When an attacker tried to disguise the web address, the system could still recognize the danger because the text on the page remained suspicious. Conversely, if the text was altered, the web address often gave the game away. This redundancy acted as a safety net, ensuring that if one clue was tampered with, the others could still sound the alarm. The researchers also trained the system specifically to expect these tricks, which made it even tougher to fool, with only a tiny drop in its ability to spot normal, safe websites.

Beyond just catching the bad sites, the researchers were concerned with how much the system trusted its own decisions. In a real-world security setting, a computer needs to know not just if a site is bad, but how sure it is. If a system is overconfident, it might block a safe site by mistake, causing frustration for users. If it is too unsure, it might let a dangerous site through. The team developed a method to "calibrate" the system, adjusting its confidence scores so they matched reality. They found that this calibration made the system's predictions much more reliable without hurting its ability to catch phishing sites. The system became a more trustworthy partner for security teams, able to say "I am very sure this is a trap" or "I am not sure, let a human check" with greater accuracy.

The study also highlighted the limits of what can be achieved with current resources. While the visual part of the system was promising, the researchers noted that it required a large amount of image data to reach its full potential, and their current setup was limited by the computing power available. They found that the text analysis was the strongest single clue, often carrying more weight than the visual appearance or the web address alone. This suggests that understanding the language used on a page is currently the most powerful tool for spotting these deceptions. The research concludes that while no system is perfect, combining multiple ways of seeing a website, protecting against deliberate tricks, and ensuring the system knows its own confidence levels creates a far more robust defense. This approach offers a practical path forward for building security tools that are not just accurate, but also resilient and trustworthy in the face of evolving threats.

Technical Summary: Adversarially Robust and Semantically Aware Phishing Detection using Calibrated Multimodal Learning

1. Problem Statement

Current automated phishing detection systems often rely on single information sources, particularly lexical URL features. While fast, these unimodal detectors are brittle; they fail when deceptive webpages appear lexically benign but remain semantically or visually misleading. Furthermore, existing literature often neglects two critical properties for deployable security controls: adversarial robustness (vulnerability to targeted perturbations designed to evade detection) and probability calibration (the reliability of confidence scores, which determines operational actions like auto-quarantine or analyst paging). This study addresses the need for a framework that jointly integrates multimodal fusion, adversarial defense, and calibrated confidence estimation.

2. Methodology

2.1 Dataset Construction

The study utilizes the "Phishing Website Dataset" (Putra, 2023), a multimodal collection of 10,395 labeled websites (5,151 phishing, 5,244 legitimate) impersonating 86 brands. Uniquely, this dataset provides co-located data for the same sites: full URLs, raw HTML/visible text, rendered screenshots, and domain metadata (WHOIS, SSL, IP).

Preprocessing: The dataset was cleaned to 10,391 records. Splitting was performed at the registered-domain level (70/15/15 train/validation/test) to prevent data leakage from near-duplicate pages sharing a domain.
Subsampling: Due to CPU-only constraints, the visual modality and tri-modal ablation studies were conducted on a bounded subsample of 395 sites with verified screenshots.

2.2 Multimodal Architecture

The framework employs three modality-specific encoders mapping inputs to a common 128-dimensional space:

URL Encoder: Combines 25 engineered lexical/structural features (e.g., entropy, subdomain depth) with a trainable character-level 1D CNN (using 3x5x7 kernels) to capture structural patterns.
Text Encoder: Uses a frozen all-MiniLM-L6-v2 sentence-transformer to generate semantic embeddings of visible page text, capturing urgency and credential requests without manual feature engineering.
Visual Encoder: Uses a frozen ResNet-18 (ImageNet-pretrained) to extract features from page screenshots.

2.3 Fusion Mechanisms

Three fusion strategies were evaluated:

Concatenation (Early Fusion): Direct concatenation of embeddings.
Gated Fusion: A learned scalar gate weights the contribution of each modality.
Cross-Modal Attention: Stacks embeddings as a token sequence processed by a multi-head self-attention layer, followed by mean pooling.

2.4 Adversarial Framework

Attacks: Offline, semantics-preserving perturbations were generated.
- URL: Homoglyph substitution, benign subdomain prepending, path appending, percent-encoding, and hyphen insertion.
- Text: WordNet synonym replacement and character transposition (typo noise).
- Visual: White-box $L_\infty$ -norm attacks (FGSM and PGD) on pixel tensors.
Defense: An adversarial training protocol was implemented for the URL+text branch, augmenting the training set with one perturbed copy of every example.

2.5 Calibration

Post-hoc calibration was applied on the validation set using three methods: Temperature Scaling, Platt Scaling, and Isotonic Regression. These were evaluated against the uncalibrated baseline using Expected Calibration Error (ECE), Brier score, and reliability diagrams.

3. Key Contributions

The paper claims five primary contributions:

Dataset Pipeline: A domain-aware, leakage-controlled pipeline using a permissively licensed dataset that jointly provides URL, text, screenshot, and metadata for the same sites.
Fusion Architecture: A cross-modal attention fusion model that empirically outperforms unimodal baselines and simpler fusion mechanisms (concatenation, gating) on the full URL+text dataset.
Adversarial Evaluation: A systematic, offline evaluation protocol covering single and combined modality attacks, quantified via clean/robust accuracy and Attack Success Rate (ASR).
Calibration Study: A comprehensive comparison of calibrators for both the multimodal model and unimodal ablations, demonstrating improved reliability without accuracy loss.
Comprehensive Ablation & Explainability: A full ablation study over all modality subsets and fusion types, supported by SHAP (for engineered features) and attention-weight analysis, alongside external validation on an independent benchmark.

4. Key Results

4.1 Detection Performance

Multimodal Superiority: The URL+Text Cross-Modal Attention model achieved the best performance with an F1-score of 0.9506 and MCC of 0.8847.
Modality Comparison: Semantic text embeddings were significantly stronger than engineered URL features alone (Best Text MCC: 0.8928 vs. Best URL MCC: 0.6418).
Visual Modality: On the limited subsample, visual-only detection was weaker (F1 0.7123) but provided discriminative signal. In tri-modal settings, simple concatenation outperformed gated/attention fusion, suggesting overfitting of complex fusion mechanisms on small sample sizes.

4.2 Adversarial Robustness

Vulnerability: Unimodal models were highly vulnerable. The URL-only model suffered a 16.07% Attack Success Rate (ASR) under URL attacks, and the text-only model suffered 10.34% under text attacks.
Fusion Benefit: The fused model demonstrated redundancy. Under URL-only attacks, the fused model's ASR dropped to 0.79% (approx. 20x lower than unimodal) because it could rely on the un-perturbed text signal.
Visual Vulnerability: The visual-only classifier collapsed under FGSM/PGD attacks (Accuracy dropped to 0.0000), highlighting a severe unmitigated vulnerability.
Adversarial Training: Retraining with adversarial examples reduced relative F1 degradation under text and combined attacks by roughly an order of magnitude (e.g., 6.09% $\to$ 0.66%) with a negligible clean-accuracy cost (<1%).

4.3 Calibration

Post-hoc calibration significantly improved reliability. For the multimodal model, Temperature Scaling reduced ECE from 0.0472 to 0.0338 (28.4% reduction) without degrading F1.
Isotonic Regression was most effective for the text-only SVM model (ECE 0.0343 $\to$ 0.0233).

4.4 External Validation

When the URL-only deep model was tested on an independent benchmark (Hannousse & Yahiouche, 2021), performance degraded (F1 0.7854 vs. 0.8771 in-distribution), confirming the generalization gap under distribution shift, though the model remained better than random chance (MCC 0.5944).

5. Significance and Claims

The paper positions its significance not merely as achieving state-of-the-art clean accuracy, but as the first study to jointly report calibration metrics (ECE, Brier score) and adversarial robustness metrics for the same multimodal phishing detection system.

Practical Value: The authors argue that a model that is both robust and well-calibrated is more practically valuable than one that is merely accurate on clean data. Calibrated confidence allows for actionable policies in security operations (e.g., auto-quarantine vs. analyst review).
Defense-in-Depth: The results demonstrate that multimodal fusion acts as a natural defense against single-modality attacks, as the model can fall back on uncorrupted signals.
Honest Scoping: The authors explicitly acknowledge limitations, particularly the small sample size for visual experiments (n=395) and the use of frozen encoders due to CPU constraints. They caution that the severe vulnerability of the visual branch requires specific adversarial training before deployment.

In conclusion, the study validates that combining URL structural features with semantic text embeddings via cross-modal attention, while applying adversarial training and probability calibration, creates a more trustworthy and robust phishing detection pipeline than unimodal or uncalibrated approaches.