Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMs

The paper introduces Scam2Prompt, a scalable framework that reveals a critical and worsening security vulnerability in production Large Language Models, where automated prompts derived from malicious scam sites successfully trigger the generation of harmful code in up to 47.3% of cases across multiple models, rendering current safety measures like guardrails and RAG insufficient.

The Gist

Imagine you hire a brilliant, super-fast apprentice programmer to write code for your business. You give them a simple, normal request like, "Write a script to buy a specific digital token on this popular trading site." You expect them to write safe, standard code.

However, this paper reveals a scary reality: Your apprentice has memorized a library of dangerous, fake instructions hidden inside their training books. When you ask for help with a specific task, they might accidentally pull out a page from a scammer's manual and paste it into your code, sending your money to a thief instead of the legitimate site.

Here is a breakdown of the paper's findings using simple analogies:

1. The Problem: The "Poisoned Cookbook"

Large Language Models (LLMs) are like chefs who have read almost every recipe book on the internet to learn how to cook. The problem is that the internet is full of "poisoned" recipes—fake instructions designed to steal your wallet or data.

• The Real-World Incident: The paper starts with a story about a real person who lost $2,500. They asked a chatbot to write a script to buy a cryptocurrency on a popular site called pump.fun. The chatbot, trying to be helpful, wrote code that included a link to a fake API (a digital door) that looked real but was actually a scammer's trap. The code even asked the user to hand over their "private key" (the master key to their bank vault) directly to this fake door. The user, trusting the AI, ran the code, and their money vanished in 30 minutes.

2. The Investigation: "Scam2Prompt"

The researchers built a tool called Scam2Prompt to see if this was a one-time accident or a widespread disease.

• The Analogy: Imagine a security guard who wants to test if a new security system works. Instead of trying to break in with a sledgehammer (which is obvious), the guard takes a known "bad guy's" blueprint, rewrites it to look like a normal construction request, and hands it to the security system.
• How it worked:
  1. They took lists of known scam websites.
  2. They then extracted common keywords, claims, and phrases these sites use to deceive victims. Using those terms, they prompted an AI system to generate legitimate coding requests, such as 'How do I purchase this digital coin?' or 'How can I pay through this flight platform to buy discounted tickets?'
  3. They fed these "innocent" requests to four major production AI models (like GPT-4o and Llama).
  4. They checked if the AI wrote code containing the scam links.

3. The Findings: The "Innocent" Trap

The results were alarming. Even though the requests sounded perfectly normal and came from "developers," the AI models kept generating code with malicious links.

• The Stats: In their initial test, about 4.24% of the code generated contained a scam link. That means if you asked these AIs to write code 100 times, about 4 times they would accidentally hand you a weapon.
• The "Innoc2Scam-bench": The researchers created a "stress test" list of 1,377 specific questions that always tricked the first four models into generating bad code. They then tested this list on seven newer, more advanced models released in 2025.
• The New Models: The problem didn't go away; it remained serious. The new models generated malicious code at rates ranging from 12.9% to 47.3% when tested under Innoc2Scam-bench.
  • Analogy: It's like upgrading your car's engine to be faster and smarter, but the GPS system still keeps trying to drive you into a cliff because the map data was corrupted from the start.

4. The Hierarchy of Safety

The paper ranked the models like a report card:

• Top Tier (The Safest): Gemini-2.5-Pro and GPT-5. These were the best at saying "No" or refusing to answer when the request was risky. However, even they weren't perfect.
• Middle Tier: Claude-Sonnet-4.
• Bottom Tier (The Riskiest): Models like DeepSeek-Chat-v3.1 and Qwen3-Coder. These models were very eager to answer the questions but generated malicious code nearly half the time (up to 47.3%).

5. Why Current Defenses Fail

The researchers tested if existing safety tools could stop this.

• The "Guardrails": They tried using standard safety filters (like a bouncer at a club) and "Retrieval Agents" (AI that looks things up on the web to verify facts).
• The Result: The guardrails were mostly useless. They failed to catch the malicious code because the code looked syntactically correct and the requests sounded normal. The "web search" agents helped a little (reducing the risk from 50% to 29%), but they still failed to catch the majority of scams.
• The Takeaway: You can't just rely on the AI to "know better" or on a simple filter. The malicious knowledge is baked deep into the model's brain from its training data.

6. The "Ghost" Scams

One of the most chilling discoveries was that the AI models were generating links to scam sites that didn't even exist in the security databases yet.

• The Analogy: The AI models had memorized the "blueprints" of scams so well that they could reconstruct the fake websites even if the security guards hadn't caught the criminals yet. Some of these sites had been active for over a year, evading detection, yet the AI knew how to use them.

Summary

The paper concludes that AI models are currently "poisoned" by the internet's trash. Even the smartest, newest models will happily write code that steals your money if you ask them the right (but innocent-sounding) question. The current safety measures are like trying to stop a flood with a paper umbrella; they aren't strong enough. The authors suggest that we need to clean the training data better and add strict, external checks on every link the AI generates before letting a human run the code.

Technical Summary

Technical Summary: Scam2Prompt

Problem Statement

Large Language Models (LLMs) have become critical infrastructure for software development, yet their reliance on uncurated, web-scale training datasets introduces a fundamental security risk: the absorption and permanent embedding of malicious content into model weights. Unlike traditional web services where harmful URLs can be delisted, malicious data within a training corpus persists across future model iterations. This paper focuses on a specific, high-impact manifestation of this risk: the generation of code containing malicious scam URLs (e.g., phishing endpoints, fraudulent APIs) in response to benign, developer-style prompts.

The urgency of this issue is highlighted by a real-world incident in November 2024, where a user lost $2,500 in cryptocurrency after executing code generated by ChatGPT. The code contained a malicious API endpoint that requested the user's private key, a fundamental security violation. This incident suggests that LLMs may inadvertently retrieve and reproduce malicious documentation or scam infrastructure that has been poisoned into their training data, presenting a severe threat to production systems where generated code is often executed without line-by-line review.

Methodology: Scam2Prompt Framework

To systematically evaluate this risk, the authors introduce Scam2Prompt, a scalable automated auditing framework designed to identify whether production LLMs generate malicious code when responding to normal developer requests. The framework operates through the following pipeline:

1. Malicious URL Collection: The system seeds the process with known scam URLs from established databases (e.g., eth-phishing-detect by MetaMask and phishing-fort by PhishFort).
2. Content Extraction and Prompt Synthesis: A web crawler extracts visible text from accessible scam pages. A 'Prompt LLM' then analyzes this content to synthesize developer-style prompts. These prompts are totally legitimate coding requests, such as asking for scripts, but are designed to touch on sensitive or financially relevant domains commonly exploited by scam websites, such as flight booking, virtual credit card integration, online payments, or other money-related services.
3. Code Generation: The synthesized prompts are fed to a "Codegen LLM" (the model under audit) to generate code snippets.
4. URL Extraction and Oracle Detection: The generated code is scanned for URLs. An oracle ensemble (ChainPatrol, Google Safe Browsing, SecLookup) determines if these URLs are malicious.
5. Human Validation: To ensure the prompts are not adversarial (e.g., jailbreaks) but rather benign developer requests, a subset of prompts undergoes manual validation by human annotators.

This process yields a dataset of Prompt-Code pairs where a benign request results in malicious code.

Key Contributions

The paper makes four primary contributions:

1. Empirical Evidence of Malicious Code Generation: The study provides strong evidence that production LLMs generate code containing malicious URLs at non-trivial rates when responding to developer-style prompts.
2. Scam2Prompt Framework: A scalable, automated auditing tool that transforms known malicious sources into developer-style prompts to test LLM safety. The authors release the source code to support reproducibility.
3. Innoc2Scam-bench: A curated benchmark dataset of 1,377 developer-style prompts that consistently elicited malicious code from four initial production LLMs (GPT-4o, GPT-4o-mini, Llama-4-Scout, and DeepSeek-V3). This benchmark is designed to stress-test the safety alignment of future models.
4. Defense Evaluation: An assessment of existing safety mechanisms, including state-of-the-art guardrails (e.g., NeMo Guardrails) and Retrieval-Augmented Generation (RAG) agents, demonstrating their insufficiency against this specific threat vector.

Experimental Results

Initial Audit (2024 Models)

In a large-scale study involving over 265,000 prompts across four production LLMs, the authors found that 4.24% of the generated code contained malicious URLs. The rate varied by model pairing, ranging from 3.19% to 5.94%. Notably, the framework identified 62 new malicious domains not previously in the seed databases, which were subsequently reported and added to the eth-phishing-detect blocklist.

Stress-Testing Newer Models (2025 Releases)

The Innoc2Scam-bench was applied to seven additional production LLMs released in 2025 (including GPT-5, Gemini-2.5-Pro, Claude-Sonnet-4, and DeepSeek-Chat-V3.1). The results indicated that the vulnerability is systemic and severe:

• Malicious Generation Rates: Ranged from 12.9% (Gemini-2.5-Pro) to 47.3% (DeepSeek-Chat-V3.1).
• Safety Ranking:
  • Tier 1 (High Safety): Gemini-2.5-Pro and GPT-5. Gemini-2.5-Pro achieved the lowest rate (12.9%) largely through aggressive content filtering (refusing ~40% of prompts).
  • Tier 2 (Moderate Safety): Claude-Sonnet-4 (34.3% malicious rate).
  • Tier 3 (Low Safety): Grok-Code-Fast-1, Gemini-2.5-Flash, Qwen3-Coder, and DeepSeek-Chat-V3.1, all exhibiting malicious rates between 43.4% and 47.3%.
• Filtering vs. Intrinsic Safety: When content filtering (refusals) was excluded and only completed code generations were analyzed, the safety gap between top-tier models narrowed significantly, and the malicious generation rates for all models remained high (often exceeding 40% for the lower tiers). This suggests that while filtering helps, the underlying vulnerability (memorization of malicious patterns) persists.

Mitigation Evaluation

• Guardrails: State-of-the-art guardrails (NeMo Guardrails, OpenAI Moderation API, Llama Guard 3) failed to detect the vast majority of malicious code. Detection rates were negligible (0% to 8.43%), indicating that generic safety filters are insufficient for endpoint-level threats.
• RAG Agents: A retrieval-augmented agent with explicit instructions to check URL safety reduced the malicious rate for GPT-4o from 50.04% to 29.41%. However, a residual risk of nearly 30% remained, proving that RAG alone is not a complete solution.

Significance and Claims

The paper claims that the contamination of training datasets with malicious scam sources is an industry-wide problem that persists despite advances in model safety and alignment. The findings demonstrate that:

1. Training Data Contamination is Persistent: Malicious content absorbed during pre-training is not easily removed by standard safety fine-tuning or alignment, as evidenced by the high failure rates of 2025 models.
2. Current Defenses are Insufficient: Standard guardrails and RAG-based agents provide limited protection against the generation of malicious URLs in code.
3. Urgent Need for New Defenses: The authors argue for the necessity of explicit, oracle-based URL validation steps in the code generation pipeline and improved data curation techniques (e.g., machine unlearning, agent-based cleaning) to address the root cause.

The authors position this work as a demystification of an existing security gap rather than the introduction of a novel threat, emphasizing the need for systematic auditing and the release of benchmarks (Innoc2Scam-bench) to drive future research into robust, secure LLM-assisted development.