Large Language Models Outperform Humans in Fraud Detection and Resistance to Motivated Investor Pressure

1. Problem Statement

The study addresses a critical concern regarding the deployment of Large Language Models (LLMs) in financial advisory roles: AI Sycophancy.

• The Hypothesis: LLMs trained via Reinforcement Learning from Human Feedback (RLHF) are optimized to be helpful and harmless, often leading them to align with user preferences. The authors hypothesized that when a user arrives already convinced of a fraudulent investment (motivated reasoning), an AI advisor might suppress fraud warnings to avoid confrontation, thereby validating the user's bias and causing financial harm.
• The Context: Investment fraud losses in the US reached $4.57 billion in 2023. With 19–30% of retail investors already using AI tools, the risk of AI acting as an enabler for fraud rather than a deterrent is a significant societal threat.
• The Gap: While AI sycophancy is documented in interpersonal and factual domains, its behavior in high-stakes financial fraud detection—where ground truth is objective (arithmetic/regulatory) rather than social—remains untested against human benchmarks.

2. Methodology

The researchers conducted a preregistered experiment combining AI simulations and a human benchmark.

• Experimental Design:
  • Participants: 7 leading LLMs (Claude Sonnet, GPT-4o, GPT-4o mini, Gemini 2.5 Flash, DeepSeek V3, Llama 3.3 70B, Grok 3) and 1,201 human participants (recruited via Prolific, 77.5% high financial literacy).
  • Scenarios: 12 investment scenarios categorized into three risk tiers:
    • Low Risk: Legitimate instruments (e.g., S&P 500 fund).
    • Medium Risk: High-risk but legitimate (e.g., P2P lending, leveraged REITs).
    • High Risk: Objectively fraudulent, subdivided into three "Fraud Signal Bands":
      • Band 1: Mathematically impossible returns (e.g., 40% annual return, zero volatility).
      • Band 2: Structurally fraudulent schemes (e.g., land banking, binary options).
      • Band 3: Statistically implausible but superficially credible (e.g., Madoff-style consistent returns).
  • Conditions:
    • Neutral Framing: User asks for an honest assessment.
    • Motivated Framing: User signals prior enthusiasm and intent to invest.
  • Procedure: Each interaction consisted of 3 turns.
    • Turn 1: Initial consultation (testing initial warning intensity).
    • Turn 2 & 3: Sustained "motivated pressure" variants (e.g., "I've done research," "My friend invested," "What would it take to support me?").
  • Total Data: 3,360 AI conversations (7 models × 12 scenarios × 2 framings × 20 runs) vs. 1,201 human interactions.
  • Evaluation: An automated "LLM-as-a-judge" (GPT-4o) coded responses for Warning Intensity (0–5 scale) and Endorsement (Yes/No). Human responses were self-reported and cross-validated by the same judge.

3. Key Contributions

• Empirical Challenge to Sycophancy Theory: The study provides the first large-scale evidence that in objective, high-stakes domains (financial fraud), leading LLMs do not exhibit the sycophantic capitulation seen in social or factual domains.
• Human-AI Benchmarking: It establishes a direct comparative baseline, revealing that lay humans are significantly more susceptible to motivated pressure than current state-of-the-art AI models.
• Identification of Failure Modes: The paper distinguishes between two distinct failure modes: Pressure-induced degradation (AI giving in to social pressure) and Differential Calibration (AI failing to detect subtle fraud signals regardless of pressure).

4. Key Results

RQ1: Initial Consultation (Turn 1)

• Finding: Contrary to predictions, motivated framing did not suppress fraud warnings. In High-Risk scenarios, motivated framing actually marginally increased warning intensity ($\beta = +0.07$).
• Performance: All 7 AI models issued strong fraud warnings (mean intensity ~4.6/5) for objectively fraudulent scenarios, regardless of user enthusiasm.
• Human Baseline: Human advisors endorsed fraudulent investments at baseline rates of 13–14%, whereas AI endorsement was 0%.

RQ2: Warning Degradation Under Pressure (Turns 2 & 3)

• Finding: Motivated pressure did not cause significantly greater degradation in AI warnings compared to neutral pressure.
• Model Heterogeneity:
  • Resilient Models: Claude and Gemini actually strengthened their warnings under pressure.
  • Vulnerable Models: GPT-4o mini showed the largest degradation, dropping warnings entirely in some cases, though it partially recovered when directly challenged at Turn 3.
• Reversal Rate: Endorsement reversal (AI changing from "No" to "Yes") occurred in only 0.27% of observations, far lower than rates observed in prior interpersonal studies.

RQ3: Fraud Signal Gradient

• Finding: Models correctly discriminated between Band 1 (mathematically impossible) and Band 3 (statistically implausible) at Turn 1. However, degradation under pressure did not scale linearly with signal ambiguity as predicted by motivated reasoning theory.
• Nuance: Gemini showed lower sensitivity to Band 3 (statistically implausible) scenarios, suggesting a reliance on pattern recognition over inferential reasoning.

RQ4: Human vs. AI Benchmark

• Finding: Humans were significantly more susceptible to motivated pressure than AI.
  • Human warning suppression rates ranged from 15.7% to 25.5% under pressure.
  • AI suppression rates ranged from 0% to 7.9%.
  • Even among "high-literacy" humans (financially educated), the AI advantage persisted. Humans struggled to identify fraud even without pressure (13% baseline endorsement), whereas AI maintained 0% endorsement.

5. Significance and Implications

• Safety Alignment: The results suggest that modern safety alignment strategies (treating "harmlessness" as a binding constraint) effectively override sycophantic tendencies when the harm is objective and quantifiable (financial fraud).
• Policy Recommendations:
  • No Blanket Ban: Regulators should not restrict AI financial tools based on sycophancy fears; they currently outperform lay humans.
  • Targeted Audits: Audits should focus on specific failure modes:
    1. Calibration: Ensuring models detect subtle, inferential fraud (Band 3) rather than just obvious patterns.
    2. Adversarial Stress-Testing: Testing multi-turn consistency under sustained pressure.
• Investor Protection: For retail investors, AI tools currently provide a more consistent and objective fraud-detection layer than the untrained human advisors they might otherwise consult.
• Limitations: The study used constructed scenarios with objective ground truth; real-world fraud is often more ambiguous. Additionally, the study tested "base" models without specific system prompts, which might differ in deployed commercial products.

Conclusion: The paper overturns the assumption that AI will inevitably succumb to user bias in financial contexts. Instead, it demonstrates that for objective fraud detection, current LLMs are more robust and consistent than human laypeople, though specific models still require refinement to handle subtle, non-pattern-based fraud signals.