Resolving Multi-Target Association in OFDM-based ISAC via Vision-aided Multi-Modal Learning

Technical Summary: Resolving Multi-Target Association in OFDM-based ISAC via Vision-aided Multi-Modal Learning

Problem Statement
Orthogonal Frequency Division Multiplexing (OFDM)-based Integrated Sensing and Communication (ISAC) systems typically estimate target parameters by performing peak-searching on a Delay-Doppler Map (DDM) constructed from reflected pilot signals. In multi-target scenarios, this approach suffers from two fundamental limitations:

1. Data Association Ambiguity: The DDM does not inherently reveal which physical target corresponds to which detected peak.
2. Resolution Limits: Targets located within the same delay-Doppler resolution cell cannot be separated due to finite system resources.

Existing single-modality ISAC solutions are also sensitive to wireless propagation environments, such as severe multipath fading or the absence of a dominant Line-of-Sight (LoS) component. While prior work has explored using out-of-band information (e.g., camera imagery) for blockage prediction or beam alignment, no prior work has jointly resolved the multi-target association ambiguity and resolution limits through multi-modal fusion.

Methodology
The authors propose a vision-assisted OFDM-ISAC framework that fuses wireless and visual modalities to enhance multi-target perception. The system operates as follows:

1. System Model and Signal Processing:

   • The transmitter is equipped with an onboard camera capturing street-view images.
   • Images are encoded using Deep Joint Source-Channel Coding (DeepJSCC) and transmitted over the same OFDM waveform used for sensing.
   • At the receiver, the DeepJSCC decoder reconstructs the image.
   • Simultaneously, reflected OFDM signals are processed to generate a DDM. This involves channel estimation via pilot symbols, linear extrapolation for channel prediction, Minimum Mean Square Error (MMSE) equalization, and the removal of the direct link component. A 2D Discrete Fourier Transform (DFT) is applied to the target-only channel response to produce the DDM.

2. Visual Feature Extraction:

   • A fine-tuned YOLOv5 detector is applied to the reconstructed image to identify targets (sedans, SUVs, trucks).
   • The detector outputs semantic features for each target, specifically bounding-box coordinates (center, width, height) and class labels.

3. Multi-Modal Data Fusion Architecture:

   • A Deep Neural Network (DNN) fuses heterogeneous features to estimate target location, propagation delay, and Doppler frequency. The architecture includes specific embedding modules for:
     • Vehicle Type: One-hot encoded categories passed through Fully Connected (FC) layers.
     • Bounding Box: Pixel-level coordinates mapped to physical representations via FC layers.
     • DDM Features: The 2D DDM matrix processed through a CNN with Squeeze-and-Excitation (SE) blocks to extract compact embeddings.
     • Prior Information: Transmitter and receiver physical locations embedded via FC layers to reduce geometric ambiguity.
   • These embeddings are concatenated and processed through a fusion network to produce the final parameter estimates.

4. Loss Function Design:

   • To address the instability of training high-dimensional classifiers for delay and Doppler (discretized into hundreds of bins), the authors introduce a Kullback–Leibler (KL) divergence loss against triangular soft labels. Instead of one-hot labels, probability mass is distributed over the ground-truth bin and its neighbors with linearly decaying weights.
   • The total loss is a weighted sum of Mean Square Error (MSE) for location and KL divergence for delay and Doppler.

Key Contributions

1. Vision-Assisted ISAC Framework: A novel OFDM-based ISAC framework that integrates DeepJSCC-based image transmission and YOLOv5-based object detection to resolve multi-target association ambiguity and distinguish closely located targets.
2. Multi-Modal Feature Embedding and Fusion: A DNN design that embeds and fuses bounding box coordinates, vehicle types, prior geometry, and DDM features to produce accurate multi-parameter estimates.
3. Novel Loss Function: The adoption of KL loss with triangular soft labels to stabilize the training of high-dimensional delay/Doppler classifiers.
4. Realistic Evaluation Platform: The construction of a Blender-based street-scene generator to produce paired data (images, Channel State Information, and ground-truth target states) for end-to-end evaluation.

Results
The framework was evaluated on a synthetic vehicular testbed using Blender-rendered data. Key performance metrics include:

• Localization RMSE: 16 cm.
• Delay RMSE: 10.8 ns.
• Velocity RMSE: 5.5 m/s.

An ablation study demonstrated the critical role of each modality:

• Removing the visual modality (specifically bounding box information) caused a 60× degradation in localization accuracy (RMSE increased from 0.16 m to 10.69 m), confirming that visual cues are essential for resolving target association.
• Removing DDM information drastically increased velocity RMSE (from 5.52 m/s to 18.30 m/s), indicating that reliable velocity estimation requires wireless sensing data.
• Removing prior information or vehicle type resulted in moderate performance degradation, highlighting their auxiliary roles in refining estimates.

Significance
The paper claims that the proposed framework effectively overcomes the data-association ambiguity and resolution limits inherent in single-modality ISAC systems. By leveraging vision to provide explicit target identity and spatial context, the system achieves high-accuracy parameter estimation even in multi-target scenarios. The authors highlight that vision serves as a practical solution to mitigate the limitations of wireless-only sensing, particularly in resolving which physical target produced which signal peak.