Technical Summary: Can LLMs Reason About Brand Ownership?

Problem Statement

Defenders in cybersecurity face a critical ambiguity when encountering new domains resembling popular brands: is the domain an attacker-created squatting site for phishing, or is it a legitimate domain registered by the brand itself for defensive purposes (e.g., blocking typosquats) or new product launches?

Current security systems that flag all brand-adjacent domains as malicious without verifying ownership generate significant false positives. Blocking a brand-owned domain (e.g., google-web.com owned by Google) harms end users, erodes trust in security tools, and exposes defenders to legal and reputational risks. The core challenge is Brand Intelligence: the ability to determine, at scale and near-real-time, whether a specific domain is owned by or affiliated with a known brand. This task is difficult because brands continuously register new domains, acquire others, and retire old ones, while WHOIS data is often privacy-proxied.

Methodology

The authors present the first systematic empirical evaluation of Large Language Models (LLMs) as brand intelligence engines. The study evaluates four state-of-the-art models (Gemini 2.5 Flash, Gemini 3.5 Flash, Claude Sonnet 4.5, and Claude Sonnet 4.6) across 36 of the most-phished brands (spanning finance, technology, social media, and infrastructure).

The evaluation is structured around three distinct tasks:

1. Q1: Domain Enumeration: Can the LLM enumerate 50 legitimately owned domains for a brand from memory?
2. Q2: Brand Attribution: Given an unseen domain (owned or squatting), can the LLM identify the associated brand?
3. Q3: Ownership Verification: Can the LLM determine if a specific domain belongs to a specific brand (binary classification)?

Experimental Settings:
Models were tested under four retrieval configurations:

• ICL (In-Context Learning): No external tools; relies solely on training data.
• Search: Access to a DuckDuckGo web search tool.
• WHOIS: Access to a WHOIS lookup tool (providing registrant org, registrar, and creation date).
• Search+WHOIS: Combined access to both tools.

Dataset Construction:
Ground truth was established via a six-step pipeline involving Tranco WHOIS scans, Crunchbase matching, DNStwist variants, search queries, TLD sweeps, and Certificate Transparency (CT) logs. The dataset includes up to 50 owned domains and 50 squatting/non-brand domains per brand.

Key Results

1. Domain Enumeration (Q1)

• Memory is Strong: LLMs possess significant memorized knowledge of brand portfolios. Claude Sonnet 4.5 achieved the highest mean Precision@50 of 0.821 in ICL mode.
• Search Limitations: Web search did not consistently improve performance. For Gemini 2.5 Flash, search caused a catastrophic drop in precision (to 0.047) due to tool timeouts and subsequent hallucinations. Gemini 3.5 Flash was the only model to reliably benefit from search, slightly improving its ICL score.

2. Brand Attribution (Q2)

• High Accuracy: Attribution is generally accurate, with Claude Sonnet 4.6 achieving up to 0.950 accuracy when using WHOIS.
• Tool Impact: WHOIS helps confirm identity for owned domains but can confuse models on squatting domains (where the registrant is the attacker, not the brand). Claude Sonnet 4.5 showed the largest improvement from retrieval augmentation (+0.307 points), suggesting it compensates for weak memorized knowledge with tool access.

3. Ownership Verification (Q3) – The Critical Task

• ICL Failure: Without external tools, models fail to distinguish defensive registrations from attacks. In ICL mode, macro-F1 scores were low (max 0.37), and recall was extremely poor (8–26%). Models exhibited a strong bias toward predicting "no" (not owned), leading to a 74–92% false-negative rate for brand-owned domains (meaning 74–92% of legitimate domains would be incorrectly passed to downstream malicious classifiers).
• WHOIS is Decisive: Augmenting with WHOIS lookup transformed performance. Macro-F1 scores rose by 0.47 to 0.65 points across all models.
  • Claude Sonnet 4.5 with Search+WHOIS achieved a macro-F1 of 0.870 with near-perfect precision (0.999) and 80.3% recall.
  • Claude Sonnet 4.6 with Search+WHOIS achieved 1.000 precision (zero false positives on squatting domains) but lower recall (0.570), leaving 43% of brand-owned domains still misclassified.
• Search Alone: Web search alone did not help Q3 and often degraded performance by providing less context than ICL when queries failed.

4. Cost Efficiency

• Gemini vs. Claude: While Claude models generally achieved higher raw accuracy, they cost 40–200× more per token than Gemini models.
• Best Value: Gemini 3.5 Flash with Search+WHOIS offered the best performance-per-dollar, delivering a macro-F1 of 0.847 and Q2 accuracy of 0.933 at a fraction of the cost of Claude.

Significance and Claims

The paper claims to provide the first rigorous empirical study of LLMs for the full brand intelligence pipeline. Its primary contributions are:

1. Demonstrating the "Stark Dichotomy": LLMs have strong memorized knowledge for enumeration and attribution but are fundamentally unreliable for ownership verification without external tools.
2. Identifying the False-Positive Risk: The study quantifies that relying on ICL alone for ownership verification results in unacceptably high false-positive rates (74–92% of brand-owned domains would be flagged as threats).
3. Validating WHOIS as the Critical Tool: The authors conclude that WHOIS augmentation is transformative and necessary for reducing false positives. It is the only setting that corrects the model's bias toward "not owned."
4. Practical Pipeline Recommendations: The authors propose a three-stage pipeline for practitioners:
   • Use Claude Sonnet 4.5 (ICL) to build an initial domain inventory.
   • Use Claude Sonnet 4.6 or Gemini 3.5 Flash (with WHOIS) for attribution.
   • Use WHOIS-augmented models as a mandatory gate before any malicious classification to verify ownership.

The paper concludes that while LLMs are promising zero-configuration engines, they must be paired with live WHOIS lookups to be reliable for brand protection. Web search is deemed unreliable for this specific task. The authors suggest future work should explore fine-tuning on brand intelligence tasks and integrating live certificate-transparency feeds to handle domains registered after the model's training cutoff.