Poster: Exploring the Limits of Audio-Based Detection of Turkish Phone Call Scams

Technical Summary: Exploring the Limits of Audio-Based Detection of Turkish Phone Call Scams

Problem Statement
Phone scams represent a rapidly growing global threat, yet technical detection research has remained heavily skewed toward English and other high-resource languages. This leaves low-resource language communities, such as Turkish speakers, vulnerable. Detection in these contexts is complicated by two factors: the scarcity of annotated data and the limitations of current defenses. Furthermore, existing scam detection research predominantly relies on Natural Language Processing (NLP) using text transcripts. This approach overlooks critical affective and prosodic cues—such as intonation, stress, and vowel lengthening—that scammers exploit to manipulate trust and urgency. These cues are often lost when speech is reduced to text, creating a gap in detection capabilities, particularly for agglutinative languages like Turkish which present unique challenges for Automatic Speech Recognition (ASR) due to morphological complexity and colloquial variations.

Methodology
To address these gaps, the authors introduced the first public multi-modal dataset specifically for Turkish scam detection. The dataset comprises 100 aligned audio-transcript pairs, balanced between 50 scam calls and 50 benign calls. The audio was sourced from publicly available YouTube videos explicitly identified as scams, subsequently verified by a native Turkish speaker to confirm fraudulent content. The calls cover diverse tactics including financial fraud, identity theft, and employment scams.

The study evaluated seven Large Language Models (LLMs) across three families: Gemini 2.5 (Flash, Flash-Lite, Pro), GPT-4o, and Qwen (Max, Plus, Turbo). The models were tested under three distinct input conditions without fine-tuning or prompt optimization between samples:

1. Method 1 (Raw Audio): Direct input of raw audio files to the multimodal LLM.
2. Method 2 (Unchecked Transcript): Audio converted to text via the Scribe V1 ASR system, then passed to the LLM without human correction.
3. Method 3 (Checked Transcript): ASR output reviewed and corrected by a fluent Turkish speaker before LLM input.

Key Results
The evaluation revealed distinct performance patterns across modalities:

• Transcript Superiority: Transcript-based inputs consistently outperformed raw audio processing. The mean F1 scores for checked and uncorrected transcripts were 0.995 and 0.992, respectively, compared to 0.969 for raw audio.
• ASR Quality Impact: The difference between human-corrected and uncorrected transcripts was negligible (an average difference of 0.008 F1 points), suggesting that once a transcript exists, the specific quality of the ASR output has minimal impact on detection performance.
• Audio Failure Modes: The performance gap in audio processing was primarily driven by content-filter refusals. Models frequently refused to process calls containing profanity, aggressive tones, or themes like police impersonation and extortion. Since these are common tactics in real-world scams, these refusals were counted as false negatives. In contrast, the same content presented as text was less likely to trigger safety filters. Additionally, some models struggled with overlapping speech and background noise, and specific hallucination artifacts were observed in Qwen Audio pipelines, likely stemming from training data containing subtitled video.

Key Contributions

• Dataset Creation: The release of the first public, multi-modal dataset of 100 Turkish scam and benign call pairs, addressing a critical data scarcity in low-resource language safety research.
• Empirical Evaluation: A systematic comparison of seven state-of-the-art LLMs across raw audio and varying transcript qualities, providing baseline performance metrics for Turkish scam detection.
• Insight into Safety Mechanisms: The identification of a fundamental tension between model safety mechanisms (content filters) and practical utility in adversarial detection, where safety refusals inadvertently degrade detection performance on high-risk scam calls.

Significance and Claims
The paper claims that for Turkish scam detection, transcript-based inputs are currently more robust and effective than direct audio processing across evaluated LLMs. It highlights that the cost of native speaker review for transcripts may not be justified given the comparable performance of uncorrected ASR outputs.

The authors emphasize the urgent need for culturally and linguistically inclusive AI safety research. They argue that relying solely on high-resource languages leaves vast populations unprotected. However, the paper maintains a modest stance regarding its findings, acknowledging that the high F1 scores may partly reflect the limited scope (100 calls) and potential artifacts in the dataset, such as inconsistent audio quality. The work serves as a foundational step rather than a final solution, pointing toward the necessity of expanding datasets and evaluating robustness under broader real-world conditions.