Technical Summary: FraudSMSWalker

Problem Statement

SMS fraud has evolved into cross-channel attacks where a text message (smishing) directs a user to a webpage, and the ultimate risk depends on the alignment between the SMS claim, the page content, and the requested user action. Existing evaluation methods suffer from two primary limitations:

1. Message-only focus: Traditional smishing benchmarks classify the SMS in isolation, ignoring the webpage evidence that often determines the actual risk.
2. Reputation shortcuts: Phishing and web-agent benchmarks often expose URLs, domains, or reputation metadata. This allows models to rely on "location cues" (e.g., known malicious domains) rather than analyzing the semantic consistency of the service flow.

Consequently, current evaluations fail to isolate whether agents can make fraud judgments based solely on visible evidence when direct reputation and location shortcuts are suppressed.

Methodology: The FraudSMSWalker Benchmark

The authors introduce FraudSMSWalker, a controlled benchmark designed to evaluate "URL-masked SMS-to-webpage fraud judgment."

Dataset Construction

• Scale and Composition: The dataset contains 699 bilingual (Chinese/English) chains, comprising 332 fraudulent and 367 benign cases across ten service scenarios (e.g., logistics, banking, telecom).
• Input Sanitization: To prevent reliance on shortcuts, the model-visible input consists of:
  • The SMS context (with real-world service claims preserved but sensitive fields redacted).
  • Sanitized webpage evidence (title, visible text, and form signals like login/payment widgets).
• Withheld Information: Raw URLs, hosts, domains, IP addresses, redirect chains, and reputation metadata are strictly withheld from the model.
• Hard Benign Cases: A significant portion of the benign cases includes legitimate service flows containing login forms, payment widgets, or verification steps. These are designed to be plausible within their context but share surface patterns with scam flows, testing the model's ability to avoid false positives.

Task Definition

The task is a binary classification of the entire service chain ($y_i \in \{0, 1\}$):

• Input: SMS context ($s_i$) + Sanitized webpage evidence ($w_i$).
• Output: ANSWER: YES (Fraudulent) or ANSWER: NO (Benign).
• Evaluation Criteria:
  1. Binary Correctness: Standard accuracy, precision, and recall.
  2. Benign Recall: A primary metric, as the benchmark emphasizes distinguishing legitimate flows from scams.
  3. Evidence Support: An "LLM-as-Judge" audit verifies whether the agent's final conclusion is grounded in the observed browser trajectory. The judge checks for hallucinations, unsupported claims, and forbidden reasoning (e.g., inferring reputation from hidden URL structures).

Experimental Setup

Nine web agents (including Qwen3.6-Plus, GPT-5.5, Kimi-K2.6, etc.) were evaluated under a masked browser-agent protocol. The evaluation included ablation studies varying:

• Evidence Access: Browser interaction vs. text-only input.
• URL Visibility: Masked URLs vs. visible URLs.

Key Results

1. Benign Recall is the Primary Bottleneck

Under the main URL-masked setting, all nine agents demonstrated a significant disparity between fraud detection and benign recognition:

• Fraud Recall: Ranged from 64.16% to 90.13%.
• Benign Recall: Ranged from 12.81% to 30.25%.
• Accuracy: Most models hovered near the label-prior baseline (approx. 47–52%), with the best (Qwen3.6-Plus) achieving 50.93%.

The authors interpret this as a failure of benign discrimination. When reputation cues are hidden, agents over-weight generic risk indicators (login forms, payment fields) and under-weight service-flow consistency, leading to excessive false positives on legitimate pages.

2. Weak Evidence Grounding

The "Evidence Support" audit revealed that correct predictions are often not supported by the observed trajectory:

• Low Support Rates: Even the best-performing agent (OpenAI GPT-5.5) had a support rate of only 32.19%.
• Unsupported Positives: Eight of nine agents exhibited "unsupported-YES" rates exceeding 63%, meaning they flagged benign chains as fraud without sufficient evidence in the trace.
• Residual Shortcuts: Even with URLs masked, some agents engaged in "reputation-like reasoning," making unsupported claims about domain trust or official status.

3. Ablation Findings

• URL Visibility Impact: Revealing URLs significantly shifted the operating point. In the snapshot setting, visible URLs increased fraud recall (29.82% $\to$ 75.90%) but collapsed benign recall (83.11% $\to$ 26.16%), confirming that models rely heavily on domain reputation when available.
• Browser vs. Text: Once URLs were masked, the difference between browser-agent and text-only protocols was minimal regarding binary accuracy, suggesting that the primary challenge is evidence interpretation rather than navigation capability.

Significance and Claims

The paper positions FraudSMSWalker not as a measure of general web navigation, but as a diagnostic tool for safety-critical judgment.

• Shift in Evaluation Focus: The authors argue that high fraud recall is insufficient for robust security. A robust agent must preserve benign recall and ensure decisions are evidence-grounded.
• Current Limitations: The results indicate that current web agents struggle to calibrate subtle chain-level cues when direct reputation shortcuts are removed. They tend to treat routine interface elements (like login forms) as definitive proof of fraud.
• Benchmark Utility: The benchmark provides a controlled environment to measure whether agents can make restrained, evidence-supported judgments, separating "lucky" guesses from reasoned analysis.

The authors conclude that the main bottleneck in SMS-to-webpage fraud detection is not the sensitivity to suspicious signals, but the ability to distinguish deceptive flows from legitimate service flows that share similar interface patterns, particularly when reputation metadata is unavailable.