Eliciting Least-to-Most Reasoning for Phishing URL Detection

Technical Summary: Eliciting Least-to-Most Reasoning for Phishing URL Detection

Problem Statement
Phishing remains a primary vector for cyberattacks and data breaches. While traditional detection methods (machine learning, deep learning, and URL-based approaches) offer high accuracy, they often function as "black boxes," lacking interpretability regarding why a specific URL is classified as benign or malicious. Furthermore, these supervised models typically require large, labeled datasets and struggle with generalization across different datasets. Although Large Language Models (LLMs) have shown promise in phishing detection via few-shot prompting, existing approaches often treat the model's reasoning as a single-step process or a black box, failing to analyze how the reasoning process itself influences the final prediction.

Methodology
The authors propose a Least-to-Most prompting framework specifically tailored for phishing URL detection. Unlike standard prompting strategies (such as One-shot or Chain-of-Thought) that generate predictions in a single prompt or a single chain, this framework instructs the LLM to decompose the classification problem into smaller, iterative sub-problems.

A novel component of this work is the "answer sensitivity" mechanism. In this iterative process:

1. The LLM generates a sub-question regarding a specific feature of the URL (e.g., domain structure, directory names).
2. The model answers the sub-question and provides a percentage estimate of the URL's likelihood of being phishing (0% = benign, 100% = phishing).
3. This percentage acts as a guide for the iteration. The process continues until the sensitivity score crosses user-defined thresholds (e.g., ≥90% for phishing or ≤10% for benign).
4. To prevent infinite loops, the framework programmatically limits iterations to 10. If the threshold is not crossed by the 10th iteration, the URL is conservatively classified based on the final score.

Key Contributions

1. Framework Proposal: The introduction of a Least-to-Most prompting framework for phishing URL detection that explicitly decomposes the problem into iterative sub-questions.
2. Answer Sensitivity Mechanism: The development of a mechanism that guides the iterative reasoning process via percentage-based confidence estimates, allowing the model to refine its prediction over multiple steps rather than committing to a single-shot answer.
3. Empirical Evaluation: A comprehensive evaluation using three public URL datasets (HP, EBBU, ISCX) and four state-of-the-art LLMs (Gemma 3, Llama 3.1, GPT-4.1, Gemini 2.5-Flash). The framework was compared against a One-shot prompting baseline and a supervised BERT-based model (URLTran).

Results
The experimental results demonstrate that the Least-to-Most framework consistently outperforms the One-shot baseline while approaching the performance of the supervised URLTran model, despite requiring no labeled training data.

• Performance Gains: The Least-to-Most approach achieved an average F1 score of 0.9040 across all datasets and models, compared to 0.8726 for the One-shot baseline (an average improvement of 0.03 F1).
• Comparison to Supervised Models: In its best-performing setting (Gemini 2.5-Flash), the framework achieved an average F1 of 0.9621, which is within 0.03 F1 of the supervised URLTran model's score of 0.99.
• Iterative Effectiveness: Analysis revealed that while many correct predictions required few iterations, a significant number of correct predictions were only possible after "outlier" numbers of iterations. The answer sensitivity mechanism allowed the model to correct initial misclassifications (e.g., a URL initially scored at 25% phishing rising to 90% after six iterations) that the One-shot method would have missed.
• Consistency: In almost all cases, Least-to-Most produced more unique correct predictions and fewer incorrect predictions than the One-shot method, even for lower-performing models like Llama.

Significance
The paper claims that this work demonstrates the value of structured, multi-step reasoning in LLMs for security tasks. By moving beyond single-step prompts, the framework elicits stronger reasoning capabilities that improve prediction accuracy without the need for extensive training data or few-shot examples. The "answer sensitivity" mechanism is highlighted as a critical factor in driving these gains, as it enables the model to iteratively refine its assessment of a URL's maliciousness. The authors conclude that this simple yet powerful prompting strategy offers a robust, interpretable alternative to both traditional supervised models and standard prompting techniques for phishing detection.