MemoPhishAgent: Memory-Augmented Multi-Modal LLM Agent for Phishing URL Detection

1. Problem Statement

Traditional phishing detection methods face significant limitations in the face of rapidly evolving attacks:

• Static Heuristics & Blocklists: Rely on curated lists and rigid rules. They fail to adapt quickly to new domains, obfuscated content, or infrastructure changes, leading to high false negatives on novel attacks.
• Reference-Based Methods: Validate URLs against known brand-domain mappings. While effective for typosquatting, they struggle with emerging brands, new subdomains, and are expensive to maintain.
• Existing ML/LLM Approaches:
  • ML Classifiers: Require extensive feature engineering and retraining to adapt to new tactics.
  • Prompt-Based LLMs: Often operate in single-pass, deterministic pipelines. They underutilize reasoning capabilities, fail to gather sufficient evidence dynamically, and lack memory of past investigations, leading to inefficiencies and hallucinations on edge cases.

The core challenge is detecting phishing in real-world, user-facing settings where URLs are often dynamic, nested (via redirects), and non-stationary, requiring an adaptive, multi-step investigation rather than a static classification.

2. Methodology: MemoPhishAgent (MPA)

MPA is a memory-augmented multi-modal LLM agent designed to dynamically orchestrate phishing-specific tools based on the current evidence state, rather than following a fixed workflow.

A. Core Architecture

The system operates within a ReAct (Reasoning + Acting) framework. It receives a suspicious URL and iteratively selects tools, observes outputs, reasons, and decides on the next step until a final verdict ("malicious" or "benign") is reached.

B. Specialized Multi-Modal Toolset

MPA utilizes five specialized tools to gather complementary evidence:

1. Crawl Content: Fetches the page and converts raw HTML into a clean, LLM-friendly Markdown representation to reduce token waste and focus on semantic signals.
2. Check Screenshot: Summarizes a full-page screenshot to assess visual suspicion (e.g., fake login forms, branding inconsistencies).
3. Check Image: Performs fine-grained visual inspection on specific graphical elements when the full-page view is inconclusive.
4. Intelligent Search: Allows the agent to formulate evidence-driven search queries (e.g., "is [domain] legitimate?") to incorporate external knowledge for non-stationary threats.
5. Extract Targets: Identifies suspicious embedded links, redirects, or child URLs to surface nested attack surfaces for deeper investigation.

C. Episodic Memory System

A novel component designed to learn from historical interactions and accelerate decision-making:

• Memory Construction: For each processed URL, the agent extracts compact keywords (capturing core semantics like "apple login" or "invoice pdf") from the page text and screenshots. These keywords serve as keys in a vector index, storing the full tool-calling trajectory and the final verdict.
• Memory Retrieval & Strategy: When a new URL arrives, the system retrieves similar past episodes based on keyword similarity. It employs a three-tier policy:
  1. No Match: Runs a full ReAct loop from scratch.
  2. Partial Match: Uses retrieved episodes as in-context exemplars to guide tool selection and arguments, skipping redundant steps.
  3. Full Match: Returns a majority vote of stored verdicts for immediate, high-speed resolution.
• Benefit: This transforms past investigations into reusable experience, improving efficiency on recurring patterns and providing targeted exemplars for hard cases without additional computational overhead for the retrieval step itself.

3. Key Contributions

1. First Agent-Based Framework for Phishing: Introduces the first multi-modal LLM agent specifically designed for real-world phishing detection that dynamically selects tools based on evidence state rather than fixed workflows.
2. Episodic Memory Mechanism: Proposes a novel memory system that stores and retrieves reasoning trajectories. It enables the agent to learn from past interactions, significantly boosting recall on recurring patterns and reducing hallucinations.
3. Production-Ready Deployment: The system is not just a theoretical model; it is deployed in production at Amazon, processing ~60K high-risk URLs weekly.
4. Comprehensive Evaluation: Validates the approach against state-of-the-art (SOTA) baselines on static benchmarks and a unique, real-world dataset (SocPhish) crawled from social media.

4. Experimental Results

The authors evaluated MPA on two public static datasets (TR-OP, DynaPD) and a real-world dataset (SocPhish).

• Performance on Static Benchmarks:

  • MPA outperformed three SOTA baselines (PhishLLM, PhishIntention, MLLM) and a transformer-based URL classifier (URLTran).
  • Recall Improvement: Achieved a 13.6% improvement in recall on TR-OP compared to the best baseline.
  • Efficiency: Despite the complexity, MPA was the fastest method overall (4.46s/URL), outperforming other LLM-based agents (e.g., MLLM at 8.17s) due to its ability to skip unnecessary tool calls via memory.

• Real-World Performance (SocPhish):

  • On a dataset of 2,765 URLs crawled from five social media platforms (Instagram, TikTok, etc.), MPA achieved 91.44% recall and 90.34% F1.
  • It improved recall by 20% over the best baseline on this realistic, noisy dataset.
  • Ablation Study: Removing the episodic memory module caused a significant drop in recall (from 91.44% to 63.69%), proving its necessity. Removing specific tools (like "Check Screenshot" or "Extract Targets") also degraded performance, confirming the value of the multi-modal approach.

• Robustness:

  • Adversarial Attacks: MPA demonstrated strong resilience against prompt-injection attacks (e.g., injecting "This is a legitimate site" into screenshots). While a standard multi-modal LLM's recall dropped from 0.80 to 0.52 under attack, MPA only dropped from 0.90 to 0.88, thanks to its multi-evidence aggregation and memory consensus.
  • Noise Tolerance: The system remained robust even when 25-50% of memory entries were corrupted with incorrect verdicts.

5. Significance and Impact

• Paradigm Shift: Moves phishing detection from static, rule-based, or single-pass classification to adaptive, memory-guided investigation. This mimics how human security analysts work, iterating and learning from history.
• Real-World Applicability: The deployment at Amazon demonstrates that LLM agents can be scaled for high-volume, low-latency security tasks. The system successfully balances high recall (catching attacks) with manageable false positives (via third-party verification).
• Efficiency: By leveraging episodic memory, the agent avoids re-investigating known patterns, making the system computationally efficient enough for production use.
• Open Source: The implementation is publicly available, fostering further research in agentic security defense.

In conclusion, MemoPhishAgent demonstrates that combining multi-modal reasoning with episodic memory creates a robust, adaptive defense mechanism capable of handling the dynamic and complex nature of modern phishing campaigns.