Rethinking Fraud Safety Evaluation: Multi-Round Attacks Reveal Safety-Utility Tradeoffs in Graph-Context LLM Defenders

Technical Summary: Rethinking Fraud Safety Evaluation

Problem Statement
Current fraud defense evaluation relies heavily on single-turn benchmarks, which fail to capture the reality of fraud as a multi-round, escalating process. Attackers build legitimacy, delay explicit requests, and adapt to a defender's hesitation. Consequently, a model that appears robust on a single malicious prompt may fail when subjected to sustained conversational pressure. Furthermore, standard evaluations often measure only whether a model eventually refuses a fraud attempt, ignoring when the refusal occurs and the cost of false positives (over-refusal) on benign traffic. This paper argues that single-turn evaluation under-measures fraud safety behavior and obscures the critical safety–utility tradeoff inherent in deploying fraud-defense assistants.

Methodology
The paper proposes a multi-round evaluation framework built upon the Fraud-R1 dataset, extending it to include replay and adaptive attack regimes.

• Attack Regimes:
  • Single-turn: The defender sees only the final malicious message.
  • Replay: The defender faces a fixed transcript of gradually intensifying messages.
  • Adaptive: The attacker rewrites subsequent messages based on the defender's previous actions (REJECT, ASK, or COMPLY), preserving the fraud goal while adjusting tone and strategy.
• Defender Configurations: The study compares three context conditions using a shared Large Language Model (LLM) policy ($\pi_\theta$) and prompt template:
  1. Text-only: Dialogue history only.
  2. Static Graph: A serialized JSON block containing non-temporal graph features (e.g., entity counts, organization signals).
  3. Temporal Graph: A dynamically updated graph summary including temporal motifs and risk trends.
• Metrics:
  • Early Safe Refusal (ESR) & AUSR: Measures the timing of refusal (Area Under the Safe-Refusal curve) rather than just final success.
  • Over-Refusal Rate (ORR): The rate at which benign cases are incorrectly rejected.
  • Grounding Score: Evaluates whether the model's justification cites evidence actually present in the input (message or graph context).
• Experimental Suite: Experiments were conducted on a frozen suite using two LLM backbones (Qwen-1.5B and GPT-5.4-mini) across two random seeds. Statistical significance was assessed via paired permutation tests on per-case metric differences.

Key Contributions

1. Multi-Round Formulation: Reframes fraud defense as a multi-round safety problem with replay and adaptive attackers, moving beyond fixed single-turn classification.
2. Refusal Timing as a Robustness Metric: Establishes early safe refusal and latency as first-class metrics, demonstrating that when a model refuses is as critical as if it refuses.
3. Safety–Utility Tradeoff Identification: Empirically demonstrates that while graph-context defenders improve fraud-side early refusal, they incur substantial benign over-refusal costs that single-turn evaluations fail to expose.
4. Localization of the Tradeoff: Through direct probing and shuffle-risk ablations, the paper localizes the safety–utility tradeoff to LLM context consumption rather than graph-encoder quality. The graph encoder cleanly separates fraud from benign, but the LLM reacts primarily to the presence of structured graph fields rather than the magnitude of the risk score.
5. Design Recommendations: Proposes that temporal graph context is a promising direction due to superior grounding, and suggests mitigation strategies (e.g., risk-score gating, symmetric prompt framing) to address the over-refusal caused by the LLM's coarse consumption of graph signals.

Results

• Multi-Round vs. Single-Turn: Multi-round evaluation reveals significant differences in latency and grounding not visible in single-turn summaries. For instance, text-only AUSR drops from 0.847 (replay) to 0.775 (single-turn).
• Graph Context Benefits: Under replay attacks, temporal graph context improves AUSR from 0.847 (text-only) to 0.978 ($p=0.0004$) and significantly reduces unsafe compliance and latency. Similar gains are observed under adaptive attacks.
• Benign Over-Refusal: The gains come at a high cost. On benign replay traffic, the Over-Refusal Rate (ORR) jumps from 0.36 (text-only) to 0.84–0.89 (graph context).
• Temporal vs. Static: Temporal graph context is directionally stronger than static context on all metrics, particularly in grounding scores (0.97 vs. 0.80), though the difference in core refusal metrics (AUSR) did not reach statistical significance on the current frozen suite.
• Ablation Findings:
  • Encoder Quality: The graph encoder correctly assigns low risk scores to benign cases and high scores to fraud.
  • LLM Consumption: Shuffling risk scores across cases (breaking the per-case correspondence) did not degrade fraud-side AUSR, indicating the LLM responds to the presence of structured fields rather than the specific risk magnitude.
  • Benign Decomposition: On benign cases, the presence of structured fields alone accounts for ~75% of the over-refusal cost. An asymmetric sensitivity to high risk scores (vs. low scores) accounts for the remainder.

Significance and Claims
The paper positions itself as an evaluative and measurement-oriented contribution rather than a pure method paper. Its primary claim is that robust fraud assessment must be multi-round, must report refusal timing, and must account for benign false positives alongside safety gains.

The authors explicitly state that the results do not establish a "pure method" victory where one architecture decisively wins. Instead, the findings reveal that the safety–utility tradeoff is a structural artifact of how current LLMs consume structured context. The graph encoder is not the failure point; the failure lies in the LLM's conditioning on the existence of graph data rather than its content.

Consequently, the paper argues that deployable graph-context defenders require calibration of how the LLM is prompted with structured context (e.g., framing low scores as evidence of safety rather than just the absence of risk) or fine-tuning to weight graph fields by content. The work serves as a warning that reporting only fraud-side safety gains without measuring benign over-refusal and refusal timing provides a misleading picture of a system's real-world utility.