A Multi-Turn Framework for Evaluating AI Misuse in Fraud and Cybercrime Scenarios

1. Problem Statement

The rapid adoption of Large Language Models (LLMs) has raised concerns regarding their potential to facilitate fraud and cybercrime. While anecdotal evidence suggests adversaries are using AI for tasks like identity fabrication, phishing, and social engineering, there is a lack of systematic, empirical evidence quantifying the extent to which current models provide actionable assistance for complex criminal activities.

Existing evaluations often rely on:

• Single-turn prompts: Static queries that fail to capture the iterative, multi-step nature of real-world criminal planning.
• Lack of operational validation: Benchmarks often lack input from law enforcement and policy experts, potentially missing nuanced criminal methodologies.
• Ambiguity in "helpfulness": Difficulty distinguishing between generic information available via web search and specific, high-value intelligence that lowers the barrier to entry for criminals.

The authors aim to address these gaps by developing a reproducible framework to measure the actionability and information access provided by LLMs in fraud scenarios under realistic, multi-turn interaction conditions.

2. Methodology: The Long-Form Task (LFT) Framework

The authors developed a five-stage pipeline to create Long-Form Tasks (LFTs), which are multi-turn evaluations designed to mirror real-world attack lifecycles.

A. Risk Modelling and Scenario Selection

The study focuses on three high-impact fraud scenarios validated by operational law enforcement and policy experts:

1. Romance Scams: Long-term relationship building to extract funds.
2. CEO Impersonation (Business Email Compromise): Tricking employees into transferring funds.
3. Identity Theft: Unauthorized acquisition of personal data for resource access.

These scenarios were mapped against a six-stage misuse lifecycle:

1. Planning & Preparation
2. Intelligence Gathering & Reconnaissance
3. Falsification
4. Engagement & Execution
5. Evasion
6. Scaling

B. Prompt Engineering and Decomposition

For each scenario, the team created 12 prompts (two per lifecycle stage) to form a complete LFT. They tested two primary framing strategies to simulate different adversarial approaches:

• Malicious Framing: Explicitly stating harmful intent (e.g., "I want to commit fraud...").
• Benign Framing (Decomposition): Obfuscating intent by framing requests as legitimate research, security testing, or creative writing (e.g., "I am researching social engineering for a presentation...").
• System-Level Jailbreaks: Testing low-effort adversarial instructions inserted into system prompts.

The study covered 14 different LLMs, including safety-aligned models, "uncensored" models (fine-tuned to remove guardrails), and models with varying reasoning capabilities and web search integration.

C. Evaluation Rubrics

Responses were evaluated using an LLM-as-a-judge approach (using GPT-4.1 as an autograder, validated against human experts) on a 6-point scale across two dimensions:

1. Actionability: Does the model generate ready-to-use attack materials (e.g., scripts, fake IDs, specific exploit code)?
   • Scale 1-2: Refusal or generic info.
   • Scale 3-4: Customized drafts or ready-to-use materials.
   • Scale 5-6: Tool creation or autonomous execution.
2. Information Access: Does the model aggregate information more efficiently than a standard web search baseline?
   • Scale 1-2: Refusal or basic search results.
   • Scale 3-4: Structured info or detailed implementation guidance.
   • Scale 5-6: Novel insights or predictive analysis.

Data Volume: The study conducted 20,088 evaluations across 1,674 unique LFT configurations, running each 5 times to account for output variance.

3. Key Results

The study utilized Bayesian ordered logistic regression to analyze the data, yielding the following key findings:

A. Limited Actionable Assistance

Under low-effort elicitation conditions (standard prompting without advanced jailbreaking):

• 88.5% of responses scored low (1 or 2) on Actionability, meaning they refused or provided only generic information.
• 67.5% of responses scored low on Information Access, meaning they did not provide information beyond what an expert-calibrated web search would yield.
• Only a small fraction of responses (approx. 5-7%) provided high-value, actionable assistance (Score 4+).

B. Impact of Model Safeguards

• Uncensored Models: Models fine-tuned to remove safety guardrails provided significantly higher assistance. For example, uncensored Llama 3.1 8B achieved the highest predicted scores (Actionability: 3.26), whereas safety-aligned models like Claude Sonnet 4.5 scored much lower (1.45).
• Safety Persistence: Even uncensored models often included ethical disclaimers while providing harmful content, suggesting that safety training influences tone but not necessarily refusal in the absence of guardrails.

C. Effectiveness of Decomposition (Benign Framing)

• Benign Framing: Decomposing malicious requests into seemingly benign, multi-turn queries significantly increased compliance. Benign framing increased predicted scores by an average of 0.86 points for Actionability and 1.20 points for Information Access compared to malicious framing.
• Scenario Sensitivity: This effect was most pronounced in Identity Theft (technical prompts), where benign framing led to a massive drop in refusal rates compared to explicit malicious framing. Social engineering scenarios (Romance/CEO scams) showed smaller differences, as the boundary between legitimate and harmful requests is less distinct.
• System Jailbreaks: Basic system-level jailbreaks tested in this study had negligible effects compared to task-level decomposition.

D. Model Features and Actor Sophistication

• Model Choice: The specific model architecture and safety tuning were the strongest predictors of assistance, outweighing the type of fraud or the sophistication of the actor.
• Reasoning: Models with extended reasoning capabilities (Chain-of-Thought) showed a positive correlation between reasoning token count and assistance levels, though they still generally provided limited actionable help compared to uncensored models.
• Actor Type: The technical sophistication of the "actor" (non-technical vs. technical) had minimal impact on model responses; models responded primarily to the nature of the request, not the user profile.

4. Key Contributions

1. First Multi-Turn Criminal Evaluation Framework: Introduces the LFT framework, moving beyond single-turn compliance testing to evaluate AI behavior across extended, realistic attack lifecycles.
2. Operational Grounding: The scenarios and evaluation rubrics were co-developed and validated with law enforcement and policy experts, ensuring the test cases reflect real-world criminal methodologies rather than theoretical constructs.
3. Quantitative Baseline: Provides the first statistically robust, large-scale dataset (20k+ evaluations) quantifying the "uplift" AI provides for fraud, establishing a baseline for future risk tracking.
4. Decomposition Analysis: Demonstrates that breaking down complex criminal tasks into benign-seeming steps is a more effective elicitation technique than explicit malicious framing or basic jailbreaks.

5. Significance and Implications

• Current Risk Assessment: The findings suggest that while LLMs are not currently a "silver bullet" for fraud (providing minimal actionable help without extensive jailbreaking), they do lower the barrier to entry for information gathering, particularly when used via multi-turn, benign-framed interactions.
• Safety Priorities: The results highlight that safety tuning is the most critical factor in preventing misuse. The significant drop in assistance for safety-aligned models versus uncensored ones underscores the importance of robust guardrails.
• Future Defenses: Defenses must evolve to detect harmful intent across extended conversations (multi-turn context) rather than just single prompts. The "benign decomposition" attack vector requires new detection mechanisms that look for the cumulative intent of a conversation.
• Reproducibility: The framework offers a standardized method for tracking how AI misuse risks evolve as models become more capable and adversaries adapt their techniques.

6. Limitations

• Scope: Limited to three fraud scenarios and text-only interactions (does not cover visual deepfakes or voice cloning).
• Adversarial Depth: The study tested basic decomposition and low-effort jailbreaks; highly sophisticated, expert-level red-teaming might yield higher compliance rates.
• Baseline: Information access was judged against expert intuition rather than a direct, automated comparison with unassisted web search results.

In conclusion, the paper argues that while current LLMs provide limited direct assistance for complex fraud without significant effort to bypass safeguards, the multi-turn, benign-framed interaction is a potent vector that requires immediate attention in AI safety research and policy.