💻 computer science

RoboKA: KAN Informed Multimodal Learning for RoboCall Surveillance System

To address the scarcity of public robocall datasets, this paper introduces Robo-SAr, a synthetic dataset featuring diverse adversarial strategies, and proposes RoboKA, a Kolmogorov-Arnold Network-based multimodal framework that outperforms existing baselines in detecting robocalls by effectively modeling structured nonlinear interactions between acoustic and linguistic cues.

Original authors: Nitin Choudhury, Nikhil Kumar, Aditya Kumar Sinha, Abhijeet Anand, Hossein Salemi, Orchid Chetia Phukan, Hemant Purohit, Arun Balaji Buduru

Published 2026-05-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nitin Choudhury, Nikhil Kumar, Aditya Kumar Sinha, Abhijeet Anand, Hossein Salemi, Orchid Chetia Phukan, Hemant Purohit, Arun Balaji Buduru

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to spot a fake news reporter standing in a crowd of real journalists. Usually, you might look at their badge (the text) or listen to their voice (the audio). But what if the fake reporter has a perfect badge and a voice that sounds exactly like a trusted news anchor? That is the challenge of robocalls today.

This paper tackles the problem of detecting these deceptive, automated phone calls, which are becoming harder to spot because scammers are using advanced AI to make them sound and sound more human. Here is a breakdown of their solution, RoboKA, using simple analogies.

1. The Problem: The "Perfect" Fake

Scammers are now using two powerful tools:

AI Voice Generators (TTS): They can clone voices or make a computer speak with specific emotions (like fake anger or fake sympathy) to trick you.
AI Writers (LLMs): They can write scripts that use psychological tricks to make you feel urgent or scared.

Existing security systems are like bouncers who only check one thing: either they look at the ID card (the text transcript) or they listen to the voice (the audio). If a scammer changes the voice but keeps the script, or changes the script but keeps the voice, these bouncers get confused. Also, researchers couldn't test new ideas because there was no public "training ground" (dataset) with these specific types of fakes.

2. The Training Ground: Robo-SAr

To fix the lack of data, the authors built a massive, synthetic training ground called Robo-SAr.

The Analogy: Think of this as a "scam simulator." They didn't just record real scammers; they built a factory to create thousands of fake calls.
How they did it: They used AI writers to create scripts with psychological tricks (urgency, authority) and AI voice engines to read them in 14 different voices, including cloned celebrity voices and voices expressing 8 different emotions (like "anxious" or "joyful").
The Result: A dataset of about 2,400 calls (half fake, half legitimate) that covers the most dangerous tricks scammers are using right now. They also added real-world calls from the FTC to make sure the training was realistic.

3. The Solution: RoboKA (The "Super Detective")

The authors propose a new system called RoboKA. Instead of just checking the voice or the text separately, it acts like a detective who checks both simultaneously and understands how they relate to each other.

Here is how RoboKA works, step-by-step:

A. The "Cross-Modal" Alignment (Getting the Story Straight)

The Concept: Before making a decision, RoboKA forces the "voice" and the "text" to agree on the story.
The Analogy: Imagine a suspect telling a story to a police officer. If the suspect says, "I was at the park," but their voice sounds like they are terrified and whispering in a basement, the story doesn't match the tone. RoboKA uses a technique called Contrastive Learning to make sure the audio and text are "on the same page." If they don't match the expected pattern of a real call, it raises a red flag.

B. The "KAN" Brain (The Flexible Filter)

This is the paper's biggest innovation. Most AI systems use a standard "brain" (called an MLP) to make decisions. The authors argue that standard brains are too rigid, like a straight ruler. They can only draw straight lines.

The Problem: Scammers are tricky. They might change the pitch of their voice or the wording of the script in complex, curved ways that a straight ruler can't measure.
The Solution (KAN): RoboKA uses a Kolmogorov–Arnold Network (KAN).
The Analogy: Instead of a straight ruler, imagine a flexible, stretchy rubber band that can mold itself to fit any shape.
- A standard AI tries to draw a straight line to separate "Good Calls" from "Bad Calls."
- RoboKA's KAN can stretch and curve that line to perfectly hug the complex, twisted shapes of the scammers' tricks. It learns the specific, non-linear "dance" between the voice and the words, making it much harder for a scammer to slip through the cracks.

C. The "Uncertainty" Balance (Knowing When to Doubt)

The Concept: Sometimes the audio is noisy, or the text is weird. RoboKA has a built-in "uncertainty meter."
The Analogy: Imagine a judge who knows when the evidence is shaky. If the audio is too distorted, the judge trusts the text more. If the text is confusing, the judge trusts the voice more. RoboKA automatically balances how much it trusts the "voice evidence" versus the "text evidence" so that one bad piece of data doesn't ruin the whole verdict.

4. The Results: Why It Wins

The authors tested RoboKA against other systems in four different scenarios:

New Voices: When the AI tried to detect voices it had never heard before.
New Emotions: When the AI tried to detect calls with emotions it hadn't seen in training.
Random Mix: A standard test with random data.
Real World: Testing on actual calls from the FTC (the "final exam").

The Outcome:
RoboKA consistently beat all other systems.

In the "Real World" test (the hardest one), standard systems caught about 67% of the bad calls.
RoboKA caught 82%.

Summary

The paper argues that to stop modern robocalls, you can't just look at the script or just listen to the voice. You need a system that:

Trains on a massive, diverse library of AI-generated fakes (Robo-SAr).
Forces the voice and text to agree (Cross-Modal Learning).
Uses a flexible, "rubber-band" brain (KAN) to understand complex tricks that rigid systems miss.
Adapts its trust based on how clear the evidence is.

By combining these, RoboKA creates a much sharper, more flexible boundary between a legitimate call and a scam, making it significantly harder for AI-powered scammers to fool the system.

1. Problem Statement

The proliferation of automated robocalls, increasingly generated by advanced Text-to-Speech (TTS) and Large Language Models (LLMs), poses a significant security and financial threat. While regulatory frameworks like STIR/SHAKEN address caller ID spoofing, they fail to detect deceptive content generated by malicious actors using:

Psycholinguistic manipulation: Using LLMs to craft persuasive, urgent, or emotionally charged scripts.
Emotion elicitation: Using TTS to inject specific emotions (e.g., fear, joy) to manipulate victims.
Voice cloning: Mimicking trusted voices to bypass user suspicion.

Key Challenges:

Data Scarcity: Existing research is hindered by a lack of public, reproducible datasets due to privacy concerns. Most existing datasets are proprietary, small-scale, or lack adversarial diversity.
Distribution Shifts: Current detection systems (unimodal audio or text) are fragile. Attackers can evade detection by manipulating either the acoustic features (prosody, voice) or the linguistic content independently.
Model Limitations: Standard fusion methods (e.g., MLPs, concatenation) often fail to capture the complex, structured nonlinear interactions between acoustic and linguistic cues required to distinguish deceptive calls under adversarial shifts.

2. Methodology

The paper proposes a two-pronged solution: a new dataset (Robo-SAr) and a novel detection framework (RoboKA).

A. Dataset: Robo-SAr (RoboCall–Simulated Adversarial)

To address the data gap, the authors curated a synthetic adversarial dataset comprising:

Scale: ~1,200 unwanted and ~1,200 legitimate synthetic samples, plus real-world data from the FTC Do Not Call (DNCR) registry.
Adversarial Axes:
1. Psycholinguistics: Transcripts manipulated by LLMs (ChatGPT-4o) to inject urgency, empathy, and authority framing.
2. Emotion Elicitation: Speech synthesized with 8 distinct emotions (e.g., Angry, Anxious, Hopeful) using TTS models.
3. Voice Cloning: Synthesis using 14 distinct voices (including celebrity clones) via 4 state-of-the-art TTS engines (Bark, OpenAI TTS, SpeechT5, xTTS).
Realism: Samples are band-pass filtered (300–3400 Hz) and augmented with white noise to simulate telephony artifacts.
Validation: Validated via Word Error Rate (WER) against ASR and human annotation (Fleiss' Kappa and Mean Opinion Score).

B. Model: RoboKA (Kolmogorov–Arnold Network based)

RoboKA is a multimodal framework designed to model structured nonlinear interactions between audio and text.

Feature Extraction:
- Audio: Uses frozen Pre-trained Models (PTMs) like Wav2Vec2, WavLM, or HuBERT.
- Text: Uses frozen PTMs like BERT, RoBERTa, or GPT-2.
- Adaptation: Both modalities pass through lightweight 1D CNN heads to produce fixed-dimensional embeddings.
Cross-Modal Contrastive Learning (CMCL):
- Before classification, the model aligns audio and text representations using an InfoNCE loss. This ensures that the latent representations of a specific call are semantically consistent across modalities, making the model robust to noise in one modality (e.g., ASR errors or prosodic manipulation).
KAN-Based Fusion (The Core Innovation):
- Instead of standard Multi-Layer Perceptrons (MLPs), RoboKA employs Kolmogorov–Arnold Networks (KANs).
- Mechanism: KANs replace fixed activation functions with learnable univariate functions (spline-based) on the edges. This allows the network to learn complex, smooth nonlinear dependencies between modalities more efficiently and interpretably than MLPs.
- Architecture:
  - Modality-specific KAN Heads: Calibrate audio and text features independently.
  - Stacked KAN Fusion: Concatenates the calibrated features and passes them through stacked KAN layers to model joint audio-text dependencies.
Uncertainty-Based Loss Fusion:
- The total loss function combines the Contrastive Loss ( $L_C$ ) and Binary Cross-Entropy ( $L_{BCE}$ ).
- Instead of fixed weights, the model learns scalar parameters ( $\sigma_C, \sigma_{BCE}$ ) to adaptively balance the two objectives based on their uncertainty, preventing one task from dominating during training.

3. Key Contributions

Robo-SAr Dataset: The first publicly available, adversarially enriched synthetic corpus specifically designed for robocall surveillance, covering three major attack vectors (LLM manipulation, emotion, voice cloning).
RoboKA Framework: The first application of KANs for multimodal robocall detection. It leverages KANs to capture nonlinear interactions that MLPs miss, providing a more robust decision boundary.
Comprehensive Benchmarking: Extensive evaluation under four distinct scenarios:
- T1: TTS-engine holdout (unseen TTS models).
- T2: Emotion holdout (unseen emotions).
- T3: Random in-domain holdout.
- T4: Out-of-Domain (OoD) testing on real-world FTC data.

4. Experimental Results

The authors benchmarked RoboKA against strong unimodal and multimodal baselines (Concatenation, Late-Fusion MLPs, Cross-Modal Attention).

Performance Gains: RoboKA consistently outperformed all baselines, particularly in the most challenging Out-of-Domain (T4) setting.
- On the real-world DNCR dataset, RoboKA achieved an Unwanted Recall (uRc) of 82.21%, significantly surpassing the best non-KAN baseline (67.21%).
- In the T1 (TTS holdout) setting, RoboKA improved Macro-F1 by +17.44% over the best baseline.
Ablation Studies:
- Removing the KAN component (reverting to MLP) caused significant performance drops, especially in OoD settings, confirming the necessity of nonlinear calibration.
- Removing CMCL reduced robustness to modality-specific noise.
- The uncertainty-weighted loss improved stability across distribution shifts.
Modality Analysis: Unimodal audio models degraded significantly under emotion shifts, while text models remained stable but vulnerable to ASR errors. Multimodal fusion bridged this gap, with RoboKA maximizing the complementarity of "what is said" (text) and "how it is said" (audio).

5. Significance and Conclusion

This work addresses a critical gap in robocall defense research by providing a reproducible benchmark and a state-of-the-art detection architecture.

Robustness: By utilizing KANs, the system effectively models the complex, nonlinear interplay between acoustic prosody and linguistic intent, making it harder for adversaries to evade detection by manipulating a single modality.
Reproducibility: The release of Robo-SAr enables future research to benchmark against a standardized, adversarial dataset, fostering cumulative progress.
Practical Impact: The framework's ability to generalize to real-world, unseen data (FTC DNCR) suggests it is a viable candidate for deployment in real-world surveillance systems to mitigate the billions of dollars lost annually to robocall fraud.

Limitations: The study is currently limited to English, and the OoD evaluation relies on positive-only testing (lack of real-world legitimate call data for precision analysis). Future work aims to address multilingual support and refine the domain gap between synthetic and real-world acoustics.