Technical Summary: Detecting Fake Information Through Source Verification with NLP

Problem Statement

The proliferation of fake information, particularly email-based phishing, poses a critical threat to digital security and trust. Existing countermeasures generally fall into two categories: educational awareness (which often fails against evolving tactics or non-technical users) and technical detection systems. Current technical solutions primarily rely on static analysis of email content, URL structures, or machine learning models trained on fixed datasets. These approaches suffer from three fundamental limitations:

1. Lack of Real-Time Source Verification: They treat detection as a pattern-matching problem over static features without verifying if the claimed source (e.g., an organization mentioned in the email) actually corresponds to the provided URL.
2. Dependency on Static Datasets: They rely on large, pre-labeled corpora, making them vulnerable to "concept drift" when attackers evolve their strategies beyond the distribution of the training data.
3. Absence of Incremental Learning: They lack mechanisms to accumulate verified source evidence over time to improve future detection.

The authors argue that a robust solution must prioritize source verification, assuming that while attackers can craft deceptive URLs and text, they cannot compromise official organizational domains or manipulate authoritative search engine rankings to replace legitimate sources.

Methodology

The proposed system is a five-stage pipeline that integrates Natural Language Processing (NLP) with web crawling and scraping techniques to perform dynamic source verification. Unlike traditional classifiers, this approach operates without relying on pre-built static training datasets for the core detection logic.

1. Input Collection and Knowledge Base Check

The system accepts user input in the form of an email (text + URL) or a URL alone. Before processing, it queries a Knowledge Base (KB) to check if the entity-URL pair has been previously verified. If a match exists, the system retrieves the historical recommendation, reducing latency.

2. Entity Extraction (NLP)

Using NLP techniques (specifically spaCy), the system extracts key entities from the email body, focusing on organization names and domain-level identifiers. Simultaneously, the input URL is decomposed into its structural components (protocol, host, path, parameters) to isolate the domain name.

3. Web Mining and Source Retrieval

The extracted organization name is used to query search engines (e.g., Google, Bing) via web mining. The system employs crawling and scraping algorithms to locate the official website associated with the identified entity. This step dynamically retrieves the "ground truth" URL for the organization.

4. Consistency Verification and Matching

The system performs a comparative analysis:

• Text-URL Correlation: It checks if the entity extracted from the text matches the domain structure of the embedded URL.
• Source Verification: It compares the URL provided in the email against the official website retrieved via web mining.
• Similarity Assessment: If the URLs do not match, the system flags the discrepancy as a potential phishing attempt. If they match, the source is deemed legitimate.

5. Knowledge Base Update and Recommendation

The results are stored in the Knowledge Base, which records:

• Fake URLs: Deceptive links detected.
• Benign URLs: Verified official links.
• Discovery Entities: The specific linguistic cues that led to the verification.
  This incremental learning mechanism allows the system to reuse verified mappings, improving efficiency and accuracy over time. Finally, a recommendation is generated for the user (e.g., "Legitimate" or "Phishing Alert").

Key Contributions

The paper outlines three primary contributions:

1. A Source-Centric Framework: A novel detection framework that combines NLP for entity extraction with dynamic web mining to validate the authenticity of cited sources, rather than relying solely on textual or structural features.
2. Dynamic Knowledge Base: A mechanism that accumulates verified source evidence over time, enabling the system to reduce redundant processing and adapt to new patterns without retraining on large static corpora.
3. Empirical Evaluation: A comprehensive evaluation demonstrating that the approach outperforms established detection services and classical machine learning baselines.

Experimental Results

The system was evaluated on two datasets:

• Dataset 1 (URLs): 1,664 URLs (832 phishing, 832 legitimate) from PhishTank and Alexa Top Sites.
• Dataset 2 (Emails): 101 manually curated emails (48 phishing, 53 legitimate) in English and French.

Performance Metrics

The proposed model achieved the following results, outperforming both public services (PhishTank, URLVOID) and machine learning baselines (Logistic Regression, Random Forest, SVM, Naive Bayes):

• Dataset 1 (URLs):

  • AUC-ROC: 0.96
  • Recall: 1.00 (Perfect detection of phishing URLs)
  • F1-Score: 0.93
  • Comparison: Outperformed URLVOID (F1 0.76) and Random Forest (F1 0.89). PhishTank performed poorly (Recall 0.11) due to its reliance on database lookups.

• Dataset 2 (Emails):

  • AUC-ROC: 0.99
  • Recall: 0.92
  • F1-Score: 0.95
  • Comparison: Significantly outperformed the best ML baseline (SVM, F1 0.78).

Ablation Study

An ablation study confirmed the value of the Knowledge Base:

• Accuracy: Enabled a modest increase in F1-score (0.91 to 0.93).
• Efficiency: Reduced average processing time per URL by 65% (from 4.8s to 1.7s) due to a 38.2% cache hit rate, bypassing the need for real-time web mining for known entities.

Significance and Claims

The authors claim that their work demonstrates that dynamic source verification can effectively reduce dependence on static training datasets and improve resilience against evolving phishing strategies. By shifting the focus from "is this text suspicious?" to "does this source match the official entity?", the system addresses the root cause of impersonation attacks.

The paper acknowledges that the approach is not a silver bullet. It relies on the assumption that attackers cannot compromise official domains or permanently manipulate search engine rankings. Limitations include potential failures with typosquatting (visual similarity), SEO poisoning, compromised legitimate sites, and URL shorteners. However, the authors assert that the method provides a scalable, adaptable framework for real-time fact-checking that complements traditional content-based analysis. Future work is proposed to address these limitations through visual similarity analysis, multilingual expansion, and integration of WHOIS/SSL data.