A Camera-Cooperative ISAC Framework for Multimodal Non-Cooperative UAVs Sensing

Technical Summary: A Camera-Cooperative ISAC Framework for Multimodal Non-Cooperative UAVs Sensing

Problem Statement
The detection and tracking of non-cooperative unmanned aerial vehicles (UAVs) present significant challenges for Integrated Sensing and Communication (ISAC) systems. These challenges arise from the inherent limitations of single-modal perception and the intense competition for shared physical resources between sensing and communication (S&C) functions. Non-cooperative UAVs typically exhibit low-altitude, low-speed, and small radar cross-section (RCS) characteristics, making them difficult to detect via ISAC echoes alone. Furthermore, traditional ISAC approaches often rely on exhaustive or hierarchical beam sweeping to locate targets, which incurs substantial time and resource overhead, thereby reducing the capacity available for communication tasks. While vision-based sensing offers high-resolution spatial information, it is susceptible to environmental factors like illumination changes and occlusion. Conversely, ISAC sensing provides all-weather robustness but lacks fine-grained visual details and requires significant beam management.

Methodology
To address these issues, the authors propose a Camera-Cooperative ISAC (CC-ISAC) framework. This system integrates an external camera with an ISAC base station (BS) to create a complementary perception loop. The framework decomposes the sensing process into three synergistic tasks:

1. Visual Real-Time Airspace Detection: An external camera performs coarse-grained airspace monitoring using a YOLO-based detector to identify UAVs and extract bounding box coordinates and cropped image patches.
2. Vision-Assisted Beam Selection: The visual data is aligned with the ISAC sensing domain to guide beam selection, reducing the search space for the BS.
3. Multimodal Fusion-Based Beam Tracking: Historical sensing data and current observations are fused to enable robust, continuous tracking.

The framework relies on two core deep learning modules:

• Vision-to-Echo Data Alignment (V2EDA): This model addresses the heterogeneous representation mismatch between visual observations (image space) and beamspace angles. It employs a dual-stream feature extraction architecture (using ResMLP for bounding box vectors and CNN for cropped images) followed by a Cross-Attention (CA) mechanism. This mechanism allows the model to dynamically retrieve relevant visual cues to map image-space observations to beamspace azimuth and elevation angles, effectively learning the nonlinear mapping without explicit depth measurements.
• Multimodal Fusion-Based Estimation (MMFE): Designed for the tracking phase, this module integrates historical echo-domain sensing information (ESI) with current visual sensing information (VSI). It utilizes a Transformer encoder to capture long-range temporal dependencies in fused historical data and combines this with current visual inputs via a Multi-Layer Perceptron (MLP). The model incorporates a fallback mechanism: if visual data is unreliable (e.g., severe occlusion), it defaults to echo-only prediction; if echo data is missing, it relies on visual data or recent measurements.

Key Contributions

1. Framework-Level Innovation: The paper introduces a novel system-level design that redefines the functional roles of heterogeneous sensors. Unlike prior works that use vision solely to refine beam alignment within a fixed pipeline, CC-ISAC employs explicit hierarchical task partitioning. Vision is assigned the role of initial airspace awareness and coarse localization, while ISAC resources are reserved for collaborative fine-grained sensing. This approach realizes perception offloading, transferring coarse-grained environmental awareness from the ISAC module to the camera to reduce overall sensing overhead.
2. Enabling Multimodal Learning Modules: The authors develop the V2EDA and MMFE models specifically to support this framework. V2EDA enables practical cooperation between spatially separated sensors by aligning visual and echo features. MMFE enhances state estimation robustness through cross-modal decision fusion, reducing beam tracking overhead.
3. System-Level Optimization: The work claims to be the first to leverage multimodal cooperation to directly optimize ISAC resource efficiency (reducing beam steering time) rather than solely improving perception accuracy.

Experimental Results
The framework was evaluated using the DeepSense6G dataset, which provides synchronized RGB imagery and UAV ground-truth GPS data.

• Beam Steering Efficiency: The V2EDA-assisted beam selection strategy significantly reduced the average number of beam scans required for target acquisition. Compared to conventional hierarchical scanning, the proposed method achieved an average reduction of 71% in beam steering overhead.
• Beam Tracking Efficiency: The MMFE model demonstrated robust performance in beam tracking. Compared to an Echo-Only baseline, the MMFE approach reduced beam tracking overhead by 1.69% to 11.15% across various signal-to-noise ratio (SNR) conditions and data loss scenarios.
• Robustness: The system maintained high angular estimation accuracy even under channel degradation (SNR down to -2 dB) and partial data loss (missing echo or visual data). The fusion model adaptively leveraged the strengths of each modality, showing superior fault tolerance compared to single-modal or Kalman filter-based approaches.
• Resource Recovery: By reducing sensing overhead, the framework effectively frees up time-domain resources (OFDM symbols) within the communication frame, thereby increasing the available capacity for communication tasks.

Significance and Claims
The paper posits that the CC-ISAC framework represents a practical advancement in ISAC system design. Its primary significance lies in mitigating resource contention between sensing and communication. By offloading coarse detection to vision and utilizing ISAC only for fine-grained refinement, the system achieves reliable UAV surveillance while freeing substantial system resources for additional communication tasks. The authors emphasize that this approach transforms the traditional trade-off between sensing accuracy and ISAC resource consumption into a trade-off between sensing accuracy and multimodal sensing capabilities. The framework is particularly suited for low-altitude scenarios with dominant line-of-sight (LoS) propagation, offering a scalable solution for efficient non-cooperative UAV detection without requiring exhaustive beam searching.