RadarFuseNet: Complex-Valued Cross-Attention Fusion of Time-Frequency IQ Radar Features for Robust Classification

1. Problem Statement

Millimeter-wave (mmWave) radar is a critical sensing modality for perception tasks, particularly in adverse conditions (fog, rain, occlusion) where cameras and LiDAR fail. While deep learning has advanced radar processing, current state-of-the-art methods often rely on transforming raw complex-valued In-phase and Quadrature (IQ) signals into image-like representations (e.g., range-Doppler maps) or point clouds.

• The Limitation: These transformations discard crucial phase information and reduce the signal's native complexity.
• The Challenge: Existing methods for classifying occluded objects and surface materials using direct IQ signal processing still struggle with generalization, particularly when testing on sensor-target distances not seen during training. There is a need to leverage the full information content of the complex signal without losing phase data while improving feature extraction robustness.

2. Methodology: RadarFuseNet

The authors propose RadarFuseNet, a bidirectional cross-attention fusion network designed to process raw mmWave radar data directly in the complex domain.

A. Signal Preprocessing

• Input Data: The system utilizes a 20×20 L-shaped antenna array from a Vayyar FMCW MIMO radar, generating a 400-channel complex signal sampled at 100 time points ($s \in \mathbb{C}^{20 \times 20 \times 100}$).
• Dual Representations: To fuse time and frequency domains, the raw signal is processed in two ways:
  1. Time-Domain (IQ): The raw complex signal $s$.
  2. Frequency-Domain (FFT): A 3D Fast Fourier Transform (FFT) is applied to generate range, azimuth, and elevation spectra ($S$).
• Reshaping: Both representations are flattened along the channel dimension to create 2D complex inputs ($x$ and $X$) for the neural networks.

B. Model Architecture

The architecture consists of three main components:

1. Dual Complex-Valued CNN Branches:

   • Two separate Convolutional Neural Networks (CNNs) extract features from the IQ signal ($x$) and the FFT signal ($X$).
   • Complex Operations: Unlike standard networks, all operations (convolution, pooling, batch normalization, and ReLU activation) are performed directly in the complex domain. This preserves the integrity of the signal phasor rather than splitting real and imaginary parts.
   • Structure: The CNNs are adapted from the SMCNet architecture, utilizing three complex convolutional layers and complex average pooling.

2. Domain Adaptation for Attention:

   • The complex-valued feature vectors ($f$ from IQ, $F$ from FFT) are converted to real-valued vectors ($\tilde{f}, \tilde{F}$) by concatenating their real and imaginary components. This is necessary because the cross-attention mechanism requires real-valued inputs.

3. Bidirectional Cross-Attention Fusion:

   • A multi-head cross-attention mechanism fuses the two feature spaces.
   • Mechanism: The model computes attention in both directions:
     • $A_1 = \text{Attn}(\tilde{f}, \tilde{F}, \tilde{F})$ (IQ attending to FFT)
     • $A_2 = \text{Attn}(\tilde{F}, \tilde{f}, \tilde{f})$ (FFT attending to IQ)
   • The outputs are concatenated to form a unified representation, allowing the model to leverage complementary features from both the time and frequency domains.

4. Classification Head:

   • The fused features pass through a fully connected layer followed by a cross-entropy loss function for classification.

3. Key Contributions

• Novel Architecture: Design of a network integrating two complex-valued CNNs with a bidirectional cross-attention mechanism specifically for raw mmWave radar signals.
• Complex-Valued Processing: Adaptation of feature representations to maintain complex-domain processing up to the fusion stage, avoiding information loss associated with separating real/imaginary components early in the pipeline.
• Enhanced Generalization: Demonstration of improved robustness in classifying objects and materials under varying measurement conditions (specifically unseen sensor distances).

4. Experimental Results

The model was evaluated on two datasets: Occluded Object Classification (10 classes) and Surface Material Classification (5 classes).

A. Occluded Object Classification

• Performance: RadarFuseNet achieved 94.20% accuracy.
• Comparison: It outperformed:
  • Single-branch complex CNN (90.70%)
  • Real-valued CNN streams (91.79%)
  • MLP-based feature fusion (89.90%)
  • ResNet-18 (93.36%)
• Significance: The fusion strategy significantly improved accuracy over single-domain approaches, proving that combining time and frequency features aids in distinguishing objects inside packaging.

B. Surface Material Classification (Generalization Test)

• Known Distances ($d_0$): The model achieved 99.92% accuracy, comparable to other top models.
• Unseen Distances ($d_1$): When tested on distances not present in the training set (60cm and 80cm), RadarFuseNet achieved 65.56% accuracy.
  • This outperformed the single-branch complex CNN (59.50%) and ResNet-18 (64.62%).
  • This demonstrates that the cross-attention fusion provides better generalization capabilities for varying physical conditions.

5. Significance and Conclusion

• Robustness: RadarFuseNet proves that directly processing complex-valued IQ signals, rather than converting them to images, yields superior feature extraction for radar tasks.
• Fusion Strategy: The bidirectional cross-attention mechanism effectively bridges the gap between time-domain (raw) and frequency-domain (spectral) representations, creating a richer feature space that is more robust to environmental variations (e.g., distance changes).
• Future Work: The authors note that current datasets are relatively small. Future research aims to scale the model to handle larger, more diverse datasets to further validate its generalization capabilities and uncover remaining limitations.

In summary, RadarFuseNet represents a significant step forward in mmWave radar perception by leveraging the full complexity of the signal through specialized complex-valued deep learning and attention-based fusion, resulting in state-of-the-art classification accuracy and improved robustness.