RF-Analyzer: Can Vision-Language Models Learn RF Understanding from Synthetic Data?

Technical Summary: RF-Analyzer: Can Vision-Language Models Learn RF Understanding from Synthetic Data?

Problem Statement
Understanding the wireless spectrum is a fundamental requirement for intelligent communication systems. However, interpreting spectrograms requires extracting multiple physical attributes and reasoning about signal structure, a capability traditional machine learning (ML) approaches struggle to achieve due to their reliance on discrete, predefined label spaces and lack of natural language interfaces. While Large Language Models (LLMs) and Vision-Language Models (VLMs) have shown promise in reasoning tasks, they lack direct grounding in physical environments. A critical gap exists in determining whether VLMs trained exclusively on synthetic Radio Frequency (RF) data can generalize to real over-the-air (OTA) environments. Real RF data is expensive, difficult to label, and constrained by regulations, making synthetic data an attractive alternative for training. However, it remains unclear if models trained solely on synthetic spectrograms can reliably interpret live signals where noise characteristics, interference, and propagation effects differ significantly from simulations.

Methodology
To address this gap, the authors introduce RF-Analyzer, a GPU-accelerated SDR-to-AI analysis platform. The system integrates a live spectrum capture device (Ettus USRP B210) with multiple VLM backends, including a domain-adapted model (RF-GPT, based on Qwen2.5-VL and trained exclusively on synthetic spectrograms) and general-purpose baselines (Llama 3.2 Vision and untuned Qwen2.5-VL).

The platform operates as follows:

1. Signal Acquisition: Complex baseband IQ samples are streamed from the USRP B210, processed through a DSP pipeline (Hann window, FFT, dB conversion), and rendered as a 200-row time-frequency waterfall image.
2. Inference: The rendered waterfall image and a text prompt containing only hardware settings (center frequency and sample rate) are sent to the VLM backends. Crucially, the models receive no raw IQ data, ground-truth labels, or scenario identifiers.
3. Evaluation Framework: The authors propose a benchmark comprising three specific metrics to assess signal understanding and grounding:
   • Physical Attribute Extraction Score (PAES): A 0–5 scale evaluating the model's ability to correctly identify five binary physical attributes: temporal class, spectral occupancy, SNR class, signal isolation, and technology family.
   • Prompt Leakage Rate (PLR): Measures the extent to which bandwidth estimates are derived from prompt metadata (sample rate) rather than inferred from the visual spectrogram.
   • Hallucination Count: Tracks factual claims contradicting measurable signal properties, such as false negatives (missing visible signals) or incorrect technology labels.

Experiments were conducted in Abu Dhabi across six scenarios: five category-aligned scenarios (covering LTE, FM, ISM, Wi-Fi, and GSM) and one adversarial "Killer Test" (KT) involving a Bluetooth mouse in an RF-shielding bag to simulate low-SNR, out-of-distribution conditions.

Key Results
The evaluation demonstrates that VLMs trained on synthetic spectrogram data can generalize to real RF environments, but with distinct limitations:

• Physical Attribute Generalization: RF-GPT consistently outperformed general-purpose baselines in extracting physical signal geometry. It reliably identified spectral occupancy, temporal behavior, and signal isolation across in-distribution scenarios, achieving high PAES scores (e.g., 5/5 in S2, S3, S4, S5). This indicates that synthetic data is sufficient for learning transferable representations of RF signal structure.
• Grounding vs. Leakage: RF-GPT exhibited a low Prompt Leakage Rate (15%), indicating its bandwidth estimates were grounded in the visual data rather than copied from prompt metadata. In contrast, general-purpose models showed high leakage rates (55–70%), often relying on metadata when visual cues were ambiguous.
• Failure Modes:
  • Semantic Grounding: While RF-GPT could perceive signal shapes, it frequently misclassified technology labels (e.g., labeling an LTE signal as 5G NR) due to a lack of frequency-allocation priors in the synthetic training data.
  • Low-SNR Robustness: In the adversarial Killer Test (low-SNR, attenuated Bluetooth signal), RF-GPT's performance degraded significantly (PAES 2/5), often over-segmenting the scene or misclassifying noise as overlapping signals. General-purpose baselines failed completely, reporting no signal.
  • Baseline Limitations: Untuned VLMs suffered from false-negative hallucinations, failing to detect visible signals because their pretraining did not align with the semantics of RF power maps (e.g., interpreting bright spectral regions as background texture).

Significance and Claims
The paper claims that synthetic data is sufficient for training VLMs to learn transferable representations of RF signal structure, enabling the extraction of physical attributes like spectral occupancy and SNR from real-world captures without real-data fine-tuning. However, the authors modestly conclude that this generalization is limited. The models lack reliable semantic grounding without contextual priors (such as frequency band regulations), leading to protocol misclassification. Furthermore, generalization breaks under conditions not covered in the synthetic distribution, specifically low-SNR regimes and attenuated signals.

The significance of this work lies in demonstrating that RF language models can be developed and tuned more efficiently using synthetic data for physical interpretation tasks, provided that future work addresses the need for explicit band-allocation priors and low-SNR data augmentation to improve robustness and semantic accuracy. The paper does not claim that synthetic training alone solves all RF analysis challenges, but rather establishes a baseline for what is achievable and where the current limitations lie.