Technical Summary: Can Language Models Understand mmWave Data?

Problem Statement

The integration of Large Language Models (LLMs) into human perception systems has advanced significantly in visual (RGB) and audio domains, yet remains largely unexplored for millimeter-wave (mmWave) radar. While mmWave sensing offers distinct advantages—such as robustness in low-light, occlusion, and privacy preservation due to its reliance on electromagnetic reflections rather than appearance—current approaches face three primary bottlenecks:

1. Data Scarcity: There is a lack of public radar-language pairings; existing datasets are small, lab-specific, and lack natural language annotations.
2. Heterogeneity: mmWave observations vary significantly across hardware (carrier frequency, antenna count), environmental conditions (clutter, multipath), and experimental setups, causing severe distributional shifts that hinder generalization.
3. Lack of Foundational Encoders: The field lacks standardized tokenizers or foundational encoders to bridge raw mmWave tensors with language backbones, forcing current systems to be retrained for every new dataset or scenario.

Consequently, existing mmWave human perception models are "scenario-tuned," requiring per-dataset retraining and failing to leverage the zero-shot reasoning capabilities of modern LLMs.

Methodology

1. mmWave-QA Benchmark Construction

To address these gaps, the authors introduce mmWave-QA, the first benchmark designed to evaluate language-conditioned mmWave human perception. The construction pipeline involves:

• Data Integration: Aggregating three heterogeneous public datasets (mmBody, MM-Fi, and mRI) covering diverse devices (TI IWR series, Phoenix custom boards), subjects, and environments.
• Textualization: A minimal textual interface converts 5D mmWave point cloud frames ($[x, y, z, \text{Doppler}, \text{intensity}]$) into coordinate-formatted text strings. This allows off-the-shelf LLMs to process radar data without specialized radar encoders.
• Taxonomy and QA Generation: The benchmark defines five core question types to probe different aspects of motion understanding:
  1. ActRec: Action Recognition (identifying the performed action).
  2. TrajCheck: Spatial Movement Verification (determining if the centroid changed position).
  3. ActOrder: Temporal Action Sequencing (identifying the order of actions).
  4. ActNum: Action Cardinality Estimation (counting distinct actions).
  5. LimbFocus: Primary Limb Motion Detection (identifying the moving body part).
• Quality Control: A human-in-the-loop process refines annotations, balances action distributions, and ensures semantic clarity across six scenarios: Normal, Furnished, Rain, Smoke, Dark, and Occlusion.

2. Evaluation Framework

The study evaluates commercial LLMs (Gemini 2.5, GPT-4o, GPT-5) on the mmWave-QA benchmark using three prompting strategies:

• Zero-shot: Direct inference from point cloud text and a question.
• Few-shot: In-context learning using example question-answer pairs.
• Chain-of-Thought (CoT): Generating intermediate reasoning steps before the final answer.
• Hybrid: Combining few-shot examples with CoT reasoning.

The evaluation compares performance across different action categories (upper-body, lower-body, torso, full-body) and hardware types, and contrasts mmWave performance against RGB-based Vision-Language Models (VLMs) under varying environmental conditions.

Key Results

1. Zero-Shot Reasoning Capability

LLMs demonstrate the ability to interpret textualized mmWave point clouds without task-specific fine-tuning.

• Performance Gains: Accuracy consistently improves from zero-shot to few-shot, CoT, and finally Few-shot CoT strategies across all models. For instance, GPT-5 achieved the highest accuracy (38.37%) in the Full-body action category using the Few-shot CoT setting.
• Generalization: Models show robust reasoning across heterogeneous hardware and environmental conditions, suggesting strong generalization capabilities even without radar-specific training.

2. Task-Specific Performance

• ActRec & TrajCheck: These tasks yield the highest performance. Models effectively distinguish moving vs. static cases when provided with few-shot context (e.g., static accuracy for GPT-4o improved to 32.0% with few-shot, while Gemini 2.5-flash improved to 27.2%).
• LimbFocus: Models perform well on single-limb motions but struggle with multi-limb scenarios, likely due to multipath interference and ghost reflections in radar data.
• Temporal Reasoning (ActOrder/ActNum): Performance is highest for "Present" action identification but drops for "Previous/Next" actions and high action counts, indicating challenges in capturing long-range temporal dependencies in dense point cloud sequences.

3. Robustness vs. RGB Modality

A critical finding is the comparative robustness of mmWave data under visual degradation:

• Clear Conditions: In Normal and Furnished environments, RGB-based VLMs outperform mmWave models, likely due to richer visual cues and larger pre-training datasets for vision.
• Degraded Conditions: In Dark and Occlusion scenarios, mmWave-based reasoning significantly surpasses RGB. VLMs often hallucinate or fail when visual cues are corrupted, whereas LLMs leveraging Doppler and coordinate shifts maintain stable, physically grounded reasoning.
• Weather: In Rain and Smoke, both modalities perform comparably, demonstrating radar's resilience to visibility issues.

4. Frame Count Sensitivity

Performance peaks when using approximately 16 frames. Fewer frames limit temporal context, while excessive frames (e.g., 64) introduce redundant or noisy information that hinders reasoning, suggesting an optimal temporal balance is required for radar-based motion analysis.

Significance and Claims

The paper claims three primary contributions:

1. mmWave-QA Benchmark: It establishes the first unified benchmark that harmonizes heterogeneous mmWave datasets into natural language QA tasks, enabling standardized evaluation across devices and scenarios.
2. Feasibility of Zero-Shot Radar-LLM Integration: It demonstrates that off-the-shelf LLMs, when provided with textualized point cloud data, can perform strong zero-shot reasoning on human motion, challenging the notion that radar data requires specialized encoders or heavy fine-tuning.
3. Robustness in Vision-Limited Environments: It provides empirical evidence that the mmWave modality, when paired with LLMs, offers superior robustness compared to RGB vision in low-light and occluded environments, highlighting its potential for reliable human perception in real-world, privacy-sensitive, or visually degraded settings.

The authors conclude that while current off-the-shelf models show promise, future work should focus on fine-tuning open-source models and integrating retrieval-augmented generation to further bridge non-visual sensing with language-based intelligence.